Jul 02 21:18:45.925545 ip-10-0-134-124 systemd[1]: Starting Kubernetes Kubelet...
Jul 02 21:18:46.236249 ip-10-0-134-124 kubenswrapper[2586]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 21:18:46.236249 ip-10-0-134-124 kubenswrapper[2586]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 02 21:18:46.236249 ip-10-0-134-124 kubenswrapper[2586]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 21:18:46.236249 ip-10-0-134-124 kubenswrapper[2586]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 02 21:18:46.236249 ip-10-0-134-124 kubenswrapper[2586]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 21:18:46.238038 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.237951    2586 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 02 21:18:46.243229 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243212    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 21:18:46.243229 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243229    2586 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 21:18:46.243308 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243232    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 21:18:46.243308 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243235    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 21:18:46.243308 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243238    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 21:18:46.243308 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243243    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 21:18:46.243308 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243246    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 21:18:46.243308 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243249    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 21:18:46.243308 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243251    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 21:18:46.243308 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243254    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 21:18:46.243308 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243257    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 21:18:46.243308 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243260    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 21:18:46.243308 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243262    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 21:18:46.243308 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243277    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 21:18:46.243308 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243281    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 21:18:46.243308 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243284    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 21:18:46.243308 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243286    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 21:18:46.243308 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243290    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 21:18:46.243308 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243298    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 21:18:46.243308 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243301    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 21:18:46.243308 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243304    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 21:18:46.243308 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243307    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 21:18:46.243761 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243310    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 21:18:46.243761 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243312    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 21:18:46.243761 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243315    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 21:18:46.243761 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243318    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 21:18:46.243761 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243321    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 21:18:46.243761 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243323    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 21:18:46.243761 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243326    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 21:18:46.243761 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243329    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 21:18:46.243761 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243331    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 21:18:46.243761 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243334    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 21:18:46.243761 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243336    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 21:18:46.243761 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243339    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 21:18:46.243761 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243341    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 21:18:46.243761 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243344    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 21:18:46.243761 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243346    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 21:18:46.243761 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243349    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 21:18:46.243761 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243351    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 21:18:46.243761 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243353    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 21:18:46.243761 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243356    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 21:18:46.243761 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243358    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 21:18:46.244242 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243361    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 21:18:46.244242 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243364    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 21:18:46.244242 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243366    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 21:18:46.244242 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243369    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 21:18:46.244242 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243373    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 21:18:46.244242 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243377    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 21:18:46.244242 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243380    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 21:18:46.244242 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243383    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 21:18:46.244242 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243386    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 21:18:46.244242 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243394    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 21:18:46.244242 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243396    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 21:18:46.244242 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243399    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 21:18:46.244242 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243401    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 21:18:46.244242 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243404    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 21:18:46.244242 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243409    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 21:18:46.244242 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243413    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 21:18:46.244242 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243416    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 21:18:46.244242 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243419    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 21:18:46.244242 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243422    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 21:18:46.244704 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243424    2586 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 21:18:46.244704 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243427    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 21:18:46.244704 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243430    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 21:18:46.244704 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243433    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 21:18:46.244704 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243435    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 21:18:46.244704 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243438    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 21:18:46.244704 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243441    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 21:18:46.244704 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243443    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 21:18:46.244704 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243446    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 21:18:46.244704 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243452    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 21:18:46.244704 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243455    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 21:18:46.244704 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243457    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 21:18:46.244704 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243460    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 21:18:46.244704 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243462    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 21:18:46.244704 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243466    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 21:18:46.244704 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243469    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 21:18:46.244704 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243472    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 21:18:46.244704 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243475    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 21:18:46.244704 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243477    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 21:18:46.244704 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243480    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 21:18:46.245177 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243482    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 21:18:46.245177 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243485    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 21:18:46.245177 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243488    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 21:18:46.245177 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243496    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 21:18:46.245177 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243499    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 21:18:46.245177 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243502    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 21:18:46.245177 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243505    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 21:18:46.245177 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243509    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 21:18:46.245177 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243512    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 21:18:46.245177 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243515    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 21:18:46.245177 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243518    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 21:18:46.245177 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243520    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 21:18:46.245177 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243523    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 21:18:46.245177 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243526    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 21:18:46.245177 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243529    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 21:18:46.245177 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243531    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 21:18:46.245177 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243534    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 21:18:46.245177 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.243536    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 21:18:46.245177 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244232    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 21:18:46.245650 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244238    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 21:18:46.245650 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244241    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 21:18:46.245650 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244244    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 21:18:46.245650 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244247    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 21:18:46.245650 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244249    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 21:18:46.245650 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244252    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 21:18:46.245650 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244255    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 21:18:46.245650 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244258    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 21:18:46.245650 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244261    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 21:18:46.245650 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244263    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 21:18:46.245650 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244280    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 21:18:46.245650 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244283    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 21:18:46.245650 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244286    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 21:18:46.245650 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244289    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 21:18:46.245650 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244292    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 21:18:46.245650 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244294    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 21:18:46.245650 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244297    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 21:18:46.245650 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244300    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 21:18:46.245650 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244303    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 21:18:46.246144 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244306    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 21:18:46.246144 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244309    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 21:18:46.246144 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244311    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 21:18:46.246144 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244314    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 21:18:46.246144 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244317    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 21:18:46.246144 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244319    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 21:18:46.246144 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244322    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 21:18:46.246144 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244324    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 21:18:46.246144 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244327    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 21:18:46.246144 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244330    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 21:18:46.246144 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244332    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 21:18:46.246144 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244335    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 21:18:46.246144 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244338    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 21:18:46.246144 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244340    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 21:18:46.246144 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244343    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 21:18:46.246144 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244346    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 21:18:46.246144 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244348    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 21:18:46.246144 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244351    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 21:18:46.246144 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244353    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 21:18:46.246144 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244356    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 21:18:46.246648 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244359    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 21:18:46.246648 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244361    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 21:18:46.246648 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244364    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 21:18:46.246648 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244366    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 21:18:46.246648 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244369    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 21:18:46.246648 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244372    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 21:18:46.246648 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244374    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 21:18:46.246648 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244377    2586 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 21:18:46.246648 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244379    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 21:18:46.246648 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244382    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 21:18:46.246648 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244385    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 21:18:46.246648 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244388    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 21:18:46.246648 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244391    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 21:18:46.246648 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244396    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 21:18:46.246648 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244400    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 21:18:46.246648 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244403    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 21:18:46.246648 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244406    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 21:18:46.246648 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244409    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 21:18:46.246648 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244411    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 21:18:46.246648 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244414    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 21:18:46.247130 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244416    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 21:18:46.247130 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244419    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 21:18:46.247130 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244421    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 21:18:46.247130 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244424    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 21:18:46.247130 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244426    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 21:18:46.247130 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244429    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 21:18:46.247130 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244431    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 21:18:46.247130 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244434    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 21:18:46.247130 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244436    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 21:18:46.247130 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244439    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 21:18:46.247130 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244441    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 21:18:46.247130 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244445    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 21:18:46.247130 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244448    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 21:18:46.247130 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244450    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 21:18:46.247130 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244453    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 21:18:46.247130 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244455    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 21:18:46.247130 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244458    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 21:18:46.247130 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244460    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 21:18:46.247130 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244463    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 21:18:46.247592 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244466    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 21:18:46.247592 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244468    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 21:18:46.247592 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244471    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 21:18:46.247592 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244473    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 21:18:46.247592 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244476    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 21:18:46.247592 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244484    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 21:18:46.247592 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244487    2586 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 21:18:46.247592 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244491    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 21:18:46.247592 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244495    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 21:18:46.247592 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244498    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 21:18:46.247592 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244502    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 21:18:46.247592 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244505    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 21:18:46.247592 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244508    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 21:18:46.247592 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244511    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 21:18:46.247592 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244513    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 21:18:46.247592 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244516    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 21:18:46.247592 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244520    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 21:18:46.247592 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244522    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 21:18:46.247592 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244525    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 21:18:46.248056 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244528    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 21:18:46.248056 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244877    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 21:18:46.248056 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244881    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 21:18:46.248056 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244884    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 21:18:46.248056 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244887    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 21:18:46.248056 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244890    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 21:18:46.248056 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244893    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 21:18:46.248056 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244895    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 21:18:46.248056 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244898    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 21:18:46.248056 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244900    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 21:18:46.248056 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244903    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 21:18:46.248056 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244906    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 21:18:46.248056 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244908    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 21:18:46.248056 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244911    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 21:18:46.248056 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244914    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 21:18:46.248056 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244916    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 21:18:46.248056 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244919    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 21:18:46.248056 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244922    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 21:18:46.248056 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244928    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 21:18:46.248525 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244931    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 21:18:46.248525 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244933    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 21:18:46.248525 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244936    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 21:18:46.248525 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244938    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 21:18:46.248525 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244941    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 21:18:46.248525 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244943    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 21:18:46.248525 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244946    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 21:18:46.248525 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244948    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 21:18:46.248525 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244951    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 21:18:46.248525 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244954    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 21:18:46.248525 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244956    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 21:18:46.248525 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244959    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 21:18:46.248525 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244961    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 21:18:46.248525 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244964    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 21:18:46.248525 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244967    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 21:18:46.248525 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244969    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 21:18:46.248525 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244972    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 21:18:46.248525 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244974    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 21:18:46.248525 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244977    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 21:18:46.248525 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244979    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 21:18:46.249009 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244982    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 21:18:46.249009 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244984    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 21:18:46.249009 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244987    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 21:18:46.249009 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244989    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 21:18:46.249009 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244991    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 21:18:46.249009 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244994    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 21:18:46.249009 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.244997    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 21:18:46.249009 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245001    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 21:18:46.249009 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245004    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 21:18:46.249009 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245006    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 21:18:46.249009 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245009    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 21:18:46.249009 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245011    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 21:18:46.249009 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245015    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 21:18:46.249009 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245017    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 21:18:46.249009 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245020    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 21:18:46.249009 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245023    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 21:18:46.249009 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245025    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 21:18:46.249009 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245028    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 21:18:46.249009 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245031    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 21:18:46.249009 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245033    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 21:18:46.249515 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245036    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 21:18:46.249515 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245038    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 21:18:46.249515 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245040    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 21:18:46.249515 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245043    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 21:18:46.249515 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245045    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 21:18:46.249515 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245048    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 21:18:46.249515 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245050    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 21:18:46.249515 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245052    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 21:18:46.249515 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245055    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 21:18:46.249515 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245057    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 21:18:46.249515 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245060    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 21:18:46.249515 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245062    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 21:18:46.249515 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245065    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 21:18:46.249515 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245067    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 21:18:46.249515 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245069    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 21:18:46.249515 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245072    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 21:18:46.249515 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245074    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 21:18:46.249515 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245077    2586 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 21:18:46.249515 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245079    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 21:18:46.249515 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245082    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 21:18:46.249991 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245084    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 21:18:46.249991 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245087    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 21:18:46.249991 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245089    2586 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 21:18:46.249991 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245092    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 21:18:46.249991 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245098    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 21:18:46.249991 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245102    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 21:18:46.249991 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245104    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 21:18:46.249991 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245107    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 21:18:46.249991 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245110    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 21:18:46.249991 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245114    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 21:18:46.249991 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245117    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 21:18:46.249991 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245120    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 21:18:46.249991 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245123    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 21:18:46.249991 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245125    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 21:18:46.249991 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245128    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 21:18:46.249991 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245130    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 21:18:46.249991 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245133    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 21:18:46.249991 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245135    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 21:18:46.249991 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245138    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 21:18:46.250452 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245140    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 21:18:46.250452 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245143    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 21:18:46.250452 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245214    2586 flags.go:64] FLAG: --address="0.0.0.0"
Jul 02 21:18:46.250452 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245221    2586 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 02 21:18:46.250452 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245228    2586 flags.go:64] FLAG: --anonymous-auth="true"
Jul 02 21:18:46.250452 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245232    2586 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 02 21:18:46.250452 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245236    2586 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 02 21:18:46.250452 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245239    2586 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 02 21:18:46.250452 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245243    2586 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 02 21:18:46.250452 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245247    2586 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 02 21:18:46.250452 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245251    2586 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 02 21:18:46.250452 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245254    2586 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 02 21:18:46.250452 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245258    2586 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 02 21:18:46.250452 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245261    2586 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 02 21:18:46.250452 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245277    2586 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 02 21:18:46.250452 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245281    2586 flags.go:64] FLAG: --cgroup-root=""
Jul 02 21:18:46.250452 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245284    2586 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 02 21:18:46.250452 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245287    2586 flags.go:64] FLAG: --client-ca-file=""
Jul 02 21:18:46.250452 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245290    2586 flags.go:64] FLAG: --cloud-provider="external"
Jul 02 21:18:46.250452 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245294    2586 flags.go:64] FLAG: --cluster-dns="[]"
Jul 02 21:18:46.250452 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245298    2586 flags.go:64] FLAG: --cluster-domain=""
Jul 02 21:18:46.250452 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245301    2586 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 02 21:18:46.250452 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245304    2586 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 02 21:18:46.250452 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245307    2586 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 02 21:18:46.251063 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245311    2586 flags.go:64] FLAG: --container-log-max-files="5"
Jul 02 21:18:46.251063 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245314    2586 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 02 21:18:46.251063 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245317    2586 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 02 21:18:46.251063 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245320    2586 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 02 21:18:46.251063 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245324    2586 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 02 21:18:46.251063 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245327    2586 flags.go:64] FLAG: --contention-profiling="false"
Jul 02 21:18:46.251063 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245330    2586 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 02 21:18:46.251063 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245333    2586 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 02 21:18:46.251063 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245336    2586 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 02 21:18:46.251063 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245339    2586 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 02 21:18:46.251063 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245343    2586 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 02 21:18:46.251063 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245346    2586 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 02 21:18:46.251063 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245349    2586 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 02 21:18:46.251063 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245352    2586 flags.go:64] FLAG: --enable-load-reader="false"
Jul 02 21:18:46.251063 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245359    2586 flags.go:64] FLAG: --enable-server="true"
Jul 02 21:18:46.251063 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245362    2586 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 02 21:18:46.251063 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245367    2586 flags.go:64] FLAG: --event-burst="100"
Jul 02 21:18:46.251063 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245370    2586 flags.go:64] FLAG: --event-qps="50"
Jul 02 21:18:46.251063 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245373    2586 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 02 21:18:46.251063 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245376    2586 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 02 21:18:46.251063 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245379    2586 flags.go:64] FLAG: --eviction-hard=""
Jul 02 21:18:46.251063 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245383    2586 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 02 21:18:46.251063 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245385    2586 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 02 21:18:46.251063 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245388    2586 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 02 21:18:46.251063 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245392    2586 flags.go:64] FLAG: --eviction-soft=""
Jul 02 21:18:46.251669 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245394    2586 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 02 21:18:46.251669 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245397    2586 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 02 21:18:46.251669 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245400    2586 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 02 21:18:46.251669 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245403    2586 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 02 21:18:46.251669 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245406    2586 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 02 21:18:46.251669 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245411    2586 flags.go:64] FLAG: --fail-swap-on="true"
Jul 02 21:18:46.251669 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245413    2586 flags.go:64] FLAG: --feature-gates=""
Jul 02 21:18:46.251669 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245417    2586 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 02 21:18:46.251669 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245420    2586 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 02 21:18:46.251669 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245423    2586 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 02 21:18:46.251669 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245427    2586 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 02 21:18:46.251669 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245430    2586 flags.go:64] FLAG: --healthz-port="10248"
Jul 02 21:18:46.251669 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245433    2586 flags.go:64] FLAG: --help="false"
Jul 02 21:18:46.251669 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245435    2586 flags.go:64] FLAG: --hostname-override="ip-10-0-134-124.ec2.internal"
Jul 02 21:18:46.251669 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245439    2586 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 02 21:18:46.251669 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245441    2586 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 02 21:18:46.251669 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245444    2586 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 02 21:18:46.251669 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245448    2586 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 02 21:18:46.251669 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245451    2586 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 02 21:18:46.251669 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245454    2586 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 02 21:18:46.251669 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245457    2586 flags.go:64] FLAG: --image-service-endpoint=""
Jul 02 21:18:46.251669 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245461    2586 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 02 21:18:46.251669 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245464    2586 flags.go:64] FLAG: --kube-api-burst="100"
Jul 02 21:18:46.252692 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245467    2586 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 02 21:18:46.252692 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245470    2586 flags.go:64] FLAG: --kube-api-qps="50"
Jul 02 21:18:46.252692 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245473    2586 flags.go:64] FLAG: --kube-reserved=""
Jul 02 21:18:46.252692 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245476    2586 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 02 21:18:46.252692 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245479    2586 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 02 21:18:46.252692 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245482    2586 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 02 21:18:46.252692 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245485    2586 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 02 21:18:46.252692 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245487    2586 flags.go:64] FLAG: --lock-file=""
Jul 02 21:18:46.252692 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245490    2586 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 02 21:18:46.252692 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245493    2586 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 02 21:18:46.252692 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245496    2586 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 02 21:18:46.252692 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245501    2586 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 02 21:18:46.252692 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245504    2586 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 02 21:18:46.252692 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245507    2586 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 02 21:18:46.252692 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245514    2586 flags.go:64] FLAG: --logging-format="text"
Jul 02 21:18:46.252692 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245517    2586 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 02 21:18:46.252692 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245520    2586 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 02 21:18:46.252692 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245523    2586 flags.go:64] FLAG: --manifest-url=""
Jul 02 21:18:46.252692 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245526    2586 flags.go:64] FLAG: --manifest-url-header=""
Jul 02 21:18:46.252692 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245530    2586 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 02 21:18:46.252692 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245533    2586 flags.go:64] FLAG: --max-open-files="1000000"
Jul 02 21:18:46.252692 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245536    2586 flags.go:64] FLAG: --max-pods="110"
Jul 02 21:18:46.252692 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245539    2586 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 02 21:18:46.252692 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245542    2586 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 02 21:18:46.252692 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245545    2586 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 02 21:18:46.253685 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245548    2586 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 02 21:18:46.253685 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245551    2586 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 02 21:18:46.253685 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245553    2586 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 02 21:18:46.253685 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245556    2586 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 02 21:18:46.253685 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245563    2586 flags.go:64] FLAG: --node-status-max-images="50"
Jul 02 21:18:46.253685 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245568    2586 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 02 21:18:46.253685 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245571    2586 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 02 21:18:46.253685 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245574    2586 flags.go:64] FLAG: --pod-cidr=""
Jul 02 21:18:46.253685 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245577    2586 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 02 21:18:46.253685 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245582    2586 flags.go:64] FLAG: --pod-manifest-path=""
Jul 02 21:18:46.253685 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245585    2586 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 02 21:18:46.253685 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245588    2586 flags.go:64] FLAG: --pods-per-core="0"
Jul 02 21:18:46.253685 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245591    2586 flags.go:64] FLAG: --port="10250"
Jul 02 21:18:46.253685 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245594    2586 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 02 21:18:46.253685 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245596    2586 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0df00d5f1961506ba"
Jul 02 21:18:46.253685 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245600    2586 flags.go:64] FLAG: --qos-reserved=""
Jul 02 21:18:46.253685 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245602    2586 flags.go:64] FLAG: --read-only-port="10255"
Jul 02 21:18:46.253685 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245605    2586 flags.go:64] FLAG: --register-node="true"
Jul 02 21:18:46.253685 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245608    2586 flags.go:64] FLAG: --register-with-taints=""
Jul 02 21:18:46.253685 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245612    2586 flags.go:64] FLAG: --registry-burst="10"
Jul 02 21:18:46.253685 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245615    2586 flags.go:64] FLAG: --registry-qps="5"
Jul 02 21:18:46.253685 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245619    2586 flags.go:64] FLAG: --reserved-cpus=""
Jul 02 21:18:46.253685 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245622    2586 flags.go:64] FLAG: --reserved-memory=""
Jul 02 21:18:46.253685 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245626    2586 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 02 21:18:46.253685 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245629    2586 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 02 21:18:46.254498 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245632    2586 flags.go:64] FLAG: --rotate-certificates="false"
Jul 02 21:18:46.254498 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245635    2586 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 02 21:18:46.254498 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245637    2586 flags.go:64] FLAG: --runonce="false"
Jul 02 21:18:46.254498 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245640    2586 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 02 21:18:46.254498 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245643    2586 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 02 21:18:46.254498 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245646    2586 flags.go:64] FLAG: --seccomp-default="false"
Jul 02 21:18:46.254498 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245649    2586 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 02 21:18:46.254498 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245652    2586 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 02 21:18:46.254498 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245655    2586 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 02 21:18:46.254498 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245658    2586 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 02 21:18:46.254498 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245662    2586 flags.go:64] FLAG: --storage-driver-password="root"
Jul 02 21:18:46.254498 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245664    2586 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 02 21:18:46.254498 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245667    2586 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 02 21:18:46.254498 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245671    2586 flags.go:64] FLAG: --storage-driver-user="root"
Jul 02 21:18:46.254498 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245673    2586 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 02 21:18:46.254498 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245677    2586 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 02 21:18:46.254498 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245679    2586 flags.go:64] FLAG: --system-cgroups=""
Jul 02 21:18:46.254498 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245682    2586 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 02 21:18:46.254498 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245687    2586 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 02 21:18:46.254498 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245690    2586 flags.go:64] FLAG: --tls-cert-file=""
Jul 02 21:18:46.254498 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245693    2586 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 02 21:18:46.254498 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245696    2586 flags.go:64] FLAG: --tls-min-version=""
Jul 02 21:18:46.254498 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245699    2586 flags.go:64] FLAG: --tls-private-key-file=""
Jul 02 21:18:46.254498 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245702    2586 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 02 21:18:46.254498 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245705    2586 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 02 21:18:46.255123 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245707    2586 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 02 21:18:46.255123 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245710    2586 flags.go:64] FLAG: --v="2"
Jul 02 21:18:46.255123 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245714    2586 flags.go:64] FLAG: --version="false"
Jul 02 21:18:46.255123 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245719    2586 flags.go:64] FLAG: --vmodule=""
Jul 02 21:18:46.255123 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245723    2586 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 02 21:18:46.255123 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.245726    2586 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 02 21:18:46.255123 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245841    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 21:18:46.255123 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245844    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 21:18:46.255123 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245847    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 21:18:46.255123 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245850    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 21:18:46.255123 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245853    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 21:18:46.255123 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245856    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 21:18:46.255123 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245860    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 21:18:46.255123 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245863    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 21:18:46.255123 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245866    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 21:18:46.255123 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245868    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 21:18:46.255123 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245871    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 21:18:46.255123 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245873    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 21:18:46.255123 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245876    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 21:18:46.255123 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245878    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 21:18:46.255123 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245882    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 21:18:46.255688 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245884    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 21:18:46.255688 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245887    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 21:18:46.255688 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245889    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 21:18:46.255688 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245892    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 21:18:46.255688 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245895    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 21:18:46.255688 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245899    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 21:18:46.255688 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245901    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 21:18:46.255688 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245904    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 21:18:46.255688 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245907    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 21:18:46.255688 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245909    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 21:18:46.255688 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245912    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 21:18:46.255688 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245914    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 21:18:46.255688 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245916    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 21:18:46.255688 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245919    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 21:18:46.255688 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245923    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 21:18:46.255688 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245926    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 21:18:46.255688 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245929    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 21:18:46.255688 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245932    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 21:18:46.255688 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245934    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 21:18:46.255688 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245937    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 21:18:46.256169 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245939    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 21:18:46.256169 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245942    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 21:18:46.256169 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245944    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 21:18:46.256169 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245947    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 21:18:46.256169 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245950    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 21:18:46.256169 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245952    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 21:18:46.256169 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245955    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 21:18:46.256169 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245957    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 21:18:46.256169 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245959    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 21:18:46.256169 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245962    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 21:18:46.256169 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245965    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 21:18:46.256169 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245967    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 21:18:46.256169 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245970    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 21:18:46.256169 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245972    2586 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 21:18:46.256169 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245975    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 21:18:46.256169 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245977    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 21:18:46.256169 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245980    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 21:18:46.256169 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245983    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 21:18:46.256169 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245985    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 21:18:46.256169 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245987    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 21:18:46.256681 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245990    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 21:18:46.256681 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245992    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 21:18:46.256681 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245995    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 21:18:46.256681 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.245997    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 21:18:46.256681 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246000    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 21:18:46.256681 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246002    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 21:18:46.256681 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246007    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 21:18:46.256681 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246009    2586 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 21:18:46.256681 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246012    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 21:18:46.256681 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246015    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 21:18:46.256681 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246017    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 21:18:46.256681 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246021    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 21:18:46.256681 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246024    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 21:18:46.256681 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246027    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 21:18:46.256681 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246030    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 21:18:46.256681 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246033    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 21:18:46.256681 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246036    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 21:18:46.256681 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246038    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 21:18:46.256681 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246041    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 21:18:46.256681 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246044    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 21:18:46.257165 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246047    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 21:18:46.257165 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246049    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 21:18:46.257165 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246052    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 21:18:46.257165 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246054    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 21:18:46.257165 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246057    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 21:18:46.257165 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246059    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 21:18:46.257165 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246062    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 21:18:46.257165 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246065    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 21:18:46.257165 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246067    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 21:18:46.257165 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246070    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 21:18:46.257165 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246072    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 21:18:46.257165 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246074    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 21:18:46.257165 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246077    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 21:18:46.257165 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246079    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 21:18:46.257165 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246082    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 21:18:46.257165 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246084    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 21:18:46.257165 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246088    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 21:18:46.257165 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246091    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 21:18:46.257165 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246095    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 21:18:46.257655 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246098    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 21:18:46.257655 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246101    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 21:18:46.257655 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246103    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 21:18:46.257655 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246106    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 21:18:46.257655 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.246109    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 21:18:46.257655 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.246621    2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 21:18:46.257655 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.255392    2586 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 02 21:18:46.257655 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.255410    2586 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 02 21:18:46.257655 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.255436    2586 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 02 21:18:46.257655 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.255443    2586 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 02 21:18:46.257655 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255498    2586 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 21:18:46.257655 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255503    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 21:18:46.257655 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255507    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 21:18:46.257655 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255510    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 21:18:46.257655 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255513    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 21:18:46.258029 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255516    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 21:18:46.258029 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255519    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 21:18:46.258029 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255521    2586 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 21:18:46.258029 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255524    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 21:18:46.258029 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255527    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 21:18:46.258029 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255530    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 21:18:46.258029 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255533    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 21:18:46.258029 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255537    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 21:18:46.258029 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255539    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 21:18:46.258029 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255542    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 21:18:46.258029 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255544    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 21:18:46.258029 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255547    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 21:18:46.258029 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255550    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 21:18:46.258029 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255552    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 21:18:46.258029 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255555    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 21:18:46.258029 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255558    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 21:18:46.258029 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255561    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 21:18:46.258029 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255563    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 21:18:46.258029 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255566    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 21:18:46.258029 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255569    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 21:18:46.258603 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255571    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 21:18:46.258603 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255575    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 21:18:46.258603 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255577    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 21:18:46.258603 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255580    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 21:18:46.258603 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255583    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 21:18:46.258603 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255586    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 21:18:46.258603 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255589    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 21:18:46.258603 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255591    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 21:18:46.258603 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255594    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 21:18:46.258603 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255597    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 21:18:46.258603 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255599    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 21:18:46.258603 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255603    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 21:18:46.258603 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255607    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 21:18:46.258603 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255610    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 21:18:46.258603 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255613    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 21:18:46.258603 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255616    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 21:18:46.258603 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255619    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 21:18:46.258603 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255621    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 21:18:46.258603 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255624    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 21:18:46.259075 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255627    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 21:18:46.259075 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255630    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 21:18:46.259075 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255632    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 21:18:46.259075 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255635    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 21:18:46.259075 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255637    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 21:18:46.259075 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255640    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 21:18:46.259075 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255643    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 21:18:46.259075 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255646    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 21:18:46.259075 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255648    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 21:18:46.259075 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255651    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 21:18:46.259075 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255654    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 21:18:46.259075 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255656    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 21:18:46.259075 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255659    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 21:18:46.259075 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255661    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 21:18:46.259075 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255664    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 21:18:46.259075 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255666    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 21:18:46.259075 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255669    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 21:18:46.259075 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255672    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 21:18:46.259075 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255675    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 21:18:46.259075 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255678    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 21:18:46.259588 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255681    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 21:18:46.259588 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255683    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 21:18:46.259588 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255686    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 21:18:46.259588 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255689    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 21:18:46.259588 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255691    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 21:18:46.259588 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255696    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 21:18:46.259588 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255699    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 21:18:46.259588 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255704    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 21:18:46.259588 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255708    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 21:18:46.259588 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255711    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 21:18:46.259588 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255714    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 21:18:46.259588 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255717    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 21:18:46.259588 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255720    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 21:18:46.259588 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255723    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 21:18:46.259588 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255725    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 21:18:46.259588 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255729    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 21:18:46.259588 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255731    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 21:18:46.259588 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255734    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 21:18:46.259588 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255737    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 21:18:46.260039 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255740    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 21:18:46.260039 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255742    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 21:18:46.260039 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255745    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 21:18:46.260039 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255747    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 21:18:46.260039 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255750    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 21:18:46.260039 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255752    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 21:18:46.260039 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255755    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 21:18:46.260039 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255758    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 21:18:46.260039 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255760    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 21:18:46.260039 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255763    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 21:18:46.260039 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255767    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 21:18:46.260039 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255769    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 21:18:46.260039 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255772    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 21:18:46.260039 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255774    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 21:18:46.260039 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255777    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 21:18:46.260039 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255780    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 21:18:46.260441 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.255785    2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 21:18:46.260441 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255926    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 21:18:46.260441 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255932    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 21:18:46.260441 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255935    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 21:18:46.260441 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255938    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 21:18:46.260441 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255940    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 21:18:46.260441 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255943    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 21:18:46.260441 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255946    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 21:18:46.260441 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255948    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 21:18:46.260441 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255951    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 21:18:46.260441 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255953    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 21:18:46.260441 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255956    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 21:18:46.260441 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255958    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 21:18:46.260441 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255962    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 21:18:46.260441 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255964    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 21:18:46.260813 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255967    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 21:18:46.260813 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255969    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 21:18:46.260813 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255972    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 21:18:46.260813 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255974    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 21:18:46.260813 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255977    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 21:18:46.260813 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255979    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 21:18:46.260813 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255982    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 21:18:46.260813 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255984    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 21:18:46.260813 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255987    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 21:18:46.260813 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255989    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 21:18:46.260813 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255991    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 21:18:46.260813 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255994    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 21:18:46.260813 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.255997    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 21:18:46.260813 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256000    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 21:18:46.260813 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256003    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 21:18:46.260813 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256005    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 21:18:46.260813 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256008    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 21:18:46.260813 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256011    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 21:18:46.260813 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256013    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 21:18:46.260813 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256016    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 21:18:46.261320 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256018    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 21:18:46.261320 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256021    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 21:18:46.261320 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256024    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 21:18:46.261320 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256026    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 21:18:46.261320 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256029    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 21:18:46.261320 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256031    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 21:18:46.261320 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256034    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 21:18:46.261320 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256036    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 21:18:46.261320 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256039    2586 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 21:18:46.261320 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256042    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 21:18:46.261320 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256044    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 21:18:46.261320 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256049    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 21:18:46.261320 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256053    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 21:18:46.261320 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256056    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 21:18:46.261320 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256059    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 21:18:46.261320 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256062    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 21:18:46.261320 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256065    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 21:18:46.261320 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256067    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 21:18:46.261320 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256070    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 21:18:46.261320 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256073    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 21:18:46.261798 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256075    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 21:18:46.261798 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256077    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 21:18:46.261798 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256080    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 21:18:46.261798 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256083    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 21:18:46.261798 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256087    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 21:18:46.261798 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256089    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 21:18:46.261798 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256092    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 21:18:46.261798 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256095    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 21:18:46.261798 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256097    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 21:18:46.261798 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256100    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 21:18:46.261798 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256102    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 21:18:46.261798 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256105    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 21:18:46.261798 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256107    2586 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 21:18:46.261798 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256110    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 21:18:46.261798 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256112    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 21:18:46.261798 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256115    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 21:18:46.261798 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256117    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 21:18:46.261798 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256120    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 21:18:46.261798 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256122    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 21:18:46.261798 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256125    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 21:18:46.262363 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256127    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 21:18:46.262363 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256130    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 21:18:46.262363 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256132    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 21:18:46.262363 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256135    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 21:18:46.262363 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256139    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 21:18:46.262363 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256142    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 21:18:46.262363 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256145    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 21:18:46.262363 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256148    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 21:18:46.262363 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256151    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 21:18:46.262363 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256154    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 21:18:46.262363 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256157    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 21:18:46.262363 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256159    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 21:18:46.262363 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256163    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 21:18:46.262363 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256166    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 21:18:46.262363 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256170    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 21:18:46.262363 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256173    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 21:18:46.262363 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256175    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 21:18:46.262363 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256178    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 21:18:46.262363 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256181    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 21:18:46.262887 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256184    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 21:18:46.262887 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256186    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 21:18:46.262887 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256189    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 21:18:46.262887 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256192    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 21:18:46.262887 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256194    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 21:18:46.262887 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:46.256197    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 21:18:46.262887 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.256201    2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 21:18:46.262887 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.256327    2586 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 02 21:18:46.262887 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.258926    2586 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 02 21:18:46.262887 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.260025    2586 server.go:1019] "Starting client certificate rotation"
Jul 02 21:18:46.262887 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.260137    2586 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 21:18:46.262887 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.260170    2586 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 21:18:46.262887 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.262694    2586 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 21:18:46.264011 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.263987    2586 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 21:18:46.271679 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.271657    2586 log.go:25] "Validated CRI v1 runtime API"
Jul 02 21:18:46.274356 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.274337    2586 log.go:25] "Validated CRI v1 image API"
Jul 02 21:18:46.275323 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.275308    2586 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 02 21:18:46.277637 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.277616    2586 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ad2bb133-2145-41a3-86b5-9464f40d5c08:/dev/nvme0n1p3 b709a029-ac01-40a0-afd5-6a66a819f774:/dev/nvme0n1p4]
Jul 02 21:18:46.277705 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.277636    2586 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 02 21:18:46.282626 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.282494    2586 manager.go:216] Machine: {Timestamp:2026-07-02 21:18:46.281346177 +0000 UTC m=+0.261525564 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100969 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27ebd4db9bedb1faae93b9b93f5e84 SystemUUID:ec27ebd4-db9b-edb1-faae-93b9b93f5e84 BootID:b19493cd-c2f1-4c2d-980e-41b2167f1482 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:38:f0:2b:72:03 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:38:f0:2b:72:03 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:37:94:00:e2:cd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 02 21:18:46.282626 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.282621    2586 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 02 21:18:46.282778 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.282766    2586 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 02 21:18:46.283756 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.283724    2586 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 02 21:18:46.283924 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.283764    2586 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-124.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 02 21:18:46.284413 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.284401    2586 topology_manager.go:138] "Creating topology manager with none policy"
Jul 02 21:18:46.284452 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.284416    2586 container_manager_linux.go:309] "Creating device plugin manager"
Jul 02 21:18:46.284452 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.284424    2586 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 21:18:46.284452 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.284440    2586 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 21:18:46.285600 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.285588    2586 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 02 21:18:46.289395 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.289374    2586 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 21:18:46.289878 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.289864    2586 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 21:18:46.290024 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.290013    2586 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 02 21:18:46.290054 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.290046    2586 kubelet.go:486] "Attempting to sync node with API server"
Jul 02 21:18:46.290084 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.290057    2586 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 02 21:18:46.290084 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.290079    2586 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 02 21:18:46.290138 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.290090    2586 kubelet.go:398] "Adding apiserver pod source"
Jul 02 21:18:46.290138 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.290114    2586 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 02 21:18:46.290554 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.290540    2586 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 21:18:46.290622 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.290561    2586 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 21:18:46.293181 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.293159    2586 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 02 21:18:46.294988 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.294975    2586 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 02 21:18:46.295031 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.295005    2586 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 02 21:18:46.296144 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.296125    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 02 21:18:46.296180 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.296157    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 02 21:18:46.296180 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.296166    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 02 21:18:46.296180 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.296173    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 02 21:18:46.296253 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.296183    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 02 21:18:46.296253 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.296189    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 02 21:18:46.296253 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.296197    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 02 21:18:46.296253 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.296203    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 02 21:18:46.296253 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.296211    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 02 21:18:46.296253 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.296217    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 02 21:18:46.296253 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.296226    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 02 21:18:46.296253 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.296231    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 02 21:18:46.297257 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.297244    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 02 21:18:46.297257 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.297256    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 02 21:18:46.300710 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.300691    2586 server.go:1268] "Started kubelet"
Jul 02 21:18:46.300838 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.300808    2586 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 02 21:18:46.301000 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.300888    2586 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 02 21:18:46.301000 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.300971    2586 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 02 21:18:46.301458 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:46.301435    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 02 21:18:46.301577 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:46.301554    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-124.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 02 21:18:46.301706 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.301691    2586 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-124.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 02 21:18:46.302713 ip-10-0-134-124 systemd[1]: Started Kubernetes Kubelet.
Jul 02 21:18:46.302838 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.302784    2586 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 02 21:18:46.303702 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.303693    2586 server.go:310] "Adding debug handlers to kubelet server"
Jul 02 21:18:46.305252 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:46.305232    2586 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 02 21:18:46.305504 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:46.304767    2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-124.ec2.internal.18be95b978094da9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-124.ec2.internal,UID:ip-10-0-134-124.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-124.ec2.internal,},FirstTimestamp:2026-07-02 21:18:46.300659113 +0000 UTC m=+0.280838479,LastTimestamp:2026-07-02 21:18:46.300659113 +0000 UTC m=+0.280838479,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-124.ec2.internal,}"
Jul 02 21:18:46.307766 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.307748    2586 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 02 21:18:46.307853 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.307768    2586 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 02 21:18:46.307853 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:46.307836    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-124.ec2.internal\" not found"
Jul 02 21:18:46.308359 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.308328    2586 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 02 21:18:46.308359 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.308329    2586 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 02 21:18:46.308359 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.308356    2586 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 02 21:18:46.308514 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.308463    2586 reconstruct.go:97] "Volume reconstruction finished"
Jul 02 21:18:46.308514 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.308473    2586 reconciler.go:29] "Reconciler: start to sync state"
Jul 02 21:18:46.309392 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.309188    2586 factory.go:55] Registering systemd factory
Jul 02 21:18:46.309459 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.309396    2586 factory.go:223] Registration of the systemd container factory successfully
Jul 02 21:18:46.309633 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.309619    2586 factory.go:153] Registering CRI-O factory
Jul 02 21:18:46.309676 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.309637    2586 factory.go:223] Registration of the crio container factory successfully
Jul 02 21:18:46.309725 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.309716    2586 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 02 21:18:46.309756 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.309739    2586 factory.go:103] Registering Raw factory
Jul 02 21:18:46.309756 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.309752    2586 manager.go:1196] Started watching for new ooms in manager
Jul 02 21:18:46.310140 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.310130    2586 manager.go:349] Starting recovery of all containers
Jul 02 21:18:46.315151 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.315127    2586 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cdxj5"
Jul 02 21:18:46.318386 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:46.318357    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 02 21:18:46.318482 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:46.318450    2586 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-124.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 02 21:18:46.321211 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.321114    2586 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cdxj5"
Jul 02 21:18:46.322656 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.322639    2586 manager.go:354] Recovery completed
Jul 02 21:18:46.326966 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.326955    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 21:18:46.328809 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.328792    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 21:18:46.328891 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.328821    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 21:18:46.328891 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.328831    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasSufficientPID"
Jul 02 21:18:46.329343 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.329328    2586 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 02 21:18:46.329343 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.329343    2586 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 02 21:18:46.329497 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.329364    2586 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 21:18:46.329497 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.329344    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 21:18:46.330073 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.330054    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 21:18:46.330148 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.330088    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 21:18:46.330148 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.330103    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasSufficientPID"
Jul 02 21:18:46.331412 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.331396    2586 policy_none.go:49] "None policy: Start"
Jul 02 21:18:46.331468 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.331418    2586 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 02 21:18:46.331468 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.331434    2586 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 02 21:18:46.333504 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.333492    2586 policy_none.go:47] "Start"
Jul 02 21:18:46.377920 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.377906    2586 manager.go:337] "Starting Device Plugin manager"
Jul 02 21:18:46.377996 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:46.377941    2586 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 02 21:18:46.377996 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.377952    2586 server.go:87] "Starting device plugin registration server"
Jul 02 21:18:46.378264 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.378248    2586 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 02 21:18:46.378375 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.378263    2586 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 02 21:18:46.378375 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.378353    2586 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 02 21:18:46.378489 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.378480    2586 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 02 21:18:46.378489 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:46.378488    2586 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 02 21:18:46.379049 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:46.379028    2586 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 02 21:18:46.379136 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:46.379086    2586 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-124.ec2.internal\" not found"
Jul 02 21:18:48.014795 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.014746    2586 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 02 21:18:48.015991 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.015973    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 21:18:48.016088 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.016071    2586 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 02 21:18:48.016145 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.016128    2586 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 02 21:18:48.016198 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.016158    2586 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 02 21:18:48.016245 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:48.016206    2586 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 02 21:18:48.017099 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.017080    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 21:18:48.017206 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.017116    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 21:18:48.017206 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.017127    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasSufficientPID"
Jul 02 21:18:48.017206 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.017152    2586 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-124.ec2.internal"
Jul 02 21:18:48.019010 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.018993    2586 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 21:18:48.026203 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.026137    2586 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-124.ec2.internal"
Jul 02 21:18:48.026203 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:48.026162    2586 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-124.ec2.internal\": node \"ip-10-0-134-124.ec2.internal\" not found"
Jul 02 21:18:48.045350 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:48.045325    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-124.ec2.internal\" not found"
Jul 02 21:18:48.116361 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.116292    2586 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-124.ec2.internal"]
Jul 02 21:18:48.116539 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.116410    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 21:18:48.117972 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.117954    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 21:18:48.118051 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.117984    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 21:18:48.118051 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.117994    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasSufficientPID"
Jul 02 21:18:48.120367 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.120354    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 21:18:48.120480 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.120465    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal"
Jul 02 21:18:48.120520 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.120500    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 21:18:48.121102 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.121085    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 21:18:48.121212 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.121111    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 21:18:48.121212 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.121124    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasSufficientPID"
Jul 02 21:18:48.121212 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.121093    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 21:18:48.121212 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.121181    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 21:18:48.121212 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.121191    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasSufficientPID"
Jul 02 21:18:48.123513 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.123496    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-124.ec2.internal"
Jul 02 21:18:48.123617 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.123524    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 21:18:48.124218 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.124202    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 21:18:48.124294 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.124233    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 21:18:48.124294 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.124244    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasSufficientPID"
Jul 02 21:18:48.146242 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:48.146218    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-124.ec2.internal\" not found"
Jul 02 21:18:48.148426 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.148404    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3da92a8dc1fa784b367c822cbf212cbf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal\" (UID: \"3da92a8dc1fa784b367c822cbf212cbf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal"
Jul 02 21:18:48.148543 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.148432    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3da92a8dc1fa784b367c822cbf212cbf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal\" (UID: \"3da92a8dc1fa784b367c822cbf212cbf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal"
Jul 02 21:18:48.148543 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.148450    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/25c1e22383a6ef722c235e04d1247f6b-config\") pod \"kube-apiserver-proxy-ip-10-0-134-124.ec2.internal\" (UID: \"25c1e22383a6ef722c235e04d1247f6b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-124.ec2.internal"
Jul 02 21:18:48.152378 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:48.152359    2586 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-124.ec2.internal\" not found" node="ip-10-0-134-124.ec2.internal"
Jul 02 21:18:48.156812 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:48.156796    2586 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-124.ec2.internal\" not found" node="ip-10-0-134-124.ec2.internal"
Jul 02 21:18:48.247228 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:48.247193    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-124.ec2.internal\" not found"
Jul 02 21:18:48.249368 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.249348    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3da92a8dc1fa784b367c822cbf212cbf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal\" (UID: \"3da92a8dc1fa784b367c822cbf212cbf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal"
Jul 02 21:18:48.249440 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.249376    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3da92a8dc1fa784b367c822cbf212cbf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal\" (UID: \"3da92a8dc1fa784b367c822cbf212cbf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal"
Jul 02 21:18:48.249440 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.249394    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/25c1e22383a6ef722c235e04d1247f6b-config\") pod \"kube-apiserver-proxy-ip-10-0-134-124.ec2.internal\" (UID: \"25c1e22383a6ef722c235e04d1247f6b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-124.ec2.internal"
Jul 02 21:18:48.249440 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.249424    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/25c1e22383a6ef722c235e04d1247f6b-config\") pod \"kube-apiserver-proxy-ip-10-0-134-124.ec2.internal\" (UID: \"25c1e22383a6ef722c235e04d1247f6b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-124.ec2.internal"
Jul 02 21:18:48.249545 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.249442    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3da92a8dc1fa784b367c822cbf212cbf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal\" (UID: \"3da92a8dc1fa784b367c822cbf212cbf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal"
Jul 02 21:18:48.249545 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.249472    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3da92a8dc1fa784b367c822cbf212cbf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal\" (UID: \"3da92a8dc1fa784b367c822cbf212cbf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal"
Jul 02 21:18:48.348085 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:48.348000    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-124.ec2.internal\" not found"
Jul 02 21:18:48.448551 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:48.448515    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-124.ec2.internal\" not found"
Jul 02 21:18:48.455794 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.455779    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal"
Jul 02 21:18:48.460757 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.460700    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-124.ec2.internal"
Jul 02 21:18:48.548732 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:48.548692    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-124.ec2.internal\" not found"
Jul 02 21:18:48.649260 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:48.649172    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-124.ec2.internal\" not found"
Jul 02 21:18:48.678160 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.678133    2586 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 21:18:48.749522 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:48.749489    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-124.ec2.internal\" not found"
Jul 02 21:18:48.796791 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.796762    2586 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 02 21:18:48.797504 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.796898    2586 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 02 21:18:48.797504 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.796933    2586 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 02 21:18:48.842943 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.842917    2586 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 21:18:48.849820 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:48.849801    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-124.ec2.internal\" not found"
Jul 02 21:18:48.862093 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.862059    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-01 21:13:47 +0000 UTC" deadline="2027-12-14 00:32:20.584575749 +0000 UTC"
Jul 02 21:18:48.862093 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:48.862086    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12699h13m31.722492371s"
Jul 02 21:18:48.950594 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:48.950526    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-124.ec2.internal\" not found"
Jul 02 21:18:49.011274 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:49.011224    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25c1e22383a6ef722c235e04d1247f6b.slice/crio-c3dc86e3c41a477a704edf0ef3cec6cd39ad42911e19450c99e23db3717b84a0 WatchSource:0}: Error finding container c3dc86e3c41a477a704edf0ef3cec6cd39ad42911e19450c99e23db3717b84a0: Status 404 returned error can't find the container with id c3dc86e3c41a477a704edf0ef3cec6cd39ad42911e19450c99e23db3717b84a0
Jul 02 21:18:49.011578 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:49.011558    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3da92a8dc1fa784b367c822cbf212cbf.slice/crio-11c985c712a183b18c5baf8b34898b9435ce2ceda2d2fb7ecd89089092cec70c WatchSource:0}: Error finding container 11c985c712a183b18c5baf8b34898b9435ce2ceda2d2fb7ecd89089092cec70c: Status 404 returned error can't find the container with id 11c985c712a183b18c5baf8b34898b9435ce2ceda2d2fb7ecd89089092cec70c
Jul 02 21:18:49.015774 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.015758    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 21:18:49.019372 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.019328    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal" event={"ID":"3da92a8dc1fa784b367c822cbf212cbf","Type":"ContainerStarted","Data":"11c985c712a183b18c5baf8b34898b9435ce2ceda2d2fb7ecd89089092cec70c"}
Jul 02 21:18:49.020367 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.020345    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-124.ec2.internal" event={"ID":"25c1e22383a6ef722c235e04d1247f6b","Type":"ContainerStarted","Data":"c3dc86e3c41a477a704edf0ef3cec6cd39ad42911e19450c99e23db3717b84a0"}
Jul 02 21:18:49.051311 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:49.051270    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-124.ec2.internal\" not found"
Jul 02 21:18:49.151889 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:49.151835    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-124.ec2.internal\" not found"
Jul 02 21:18:49.252395 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:49.252312    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-124.ec2.internal\" not found"
Jul 02 21:18:49.326346 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.326322    2586 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 21:18:49.345877 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.345850    2586 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-124.ec2.internal"
Jul 02 21:18:49.359406 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.359387    2586 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 21:18:49.360125 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.360110    2586 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal"
Jul 02 21:18:49.388008 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.387982    2586 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 21:18:49.827824 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.827784    2586 apiserver.go:52] "Watching apiserver"
Jul 02 21:18:49.839467 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.839432    2586 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 02 21:18:49.839870 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.839834    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-5gfl7","kube-system/konnectivity-agent-xn9ss","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcvqp","openshift-cluster-node-tuning-operator/tuned-8j5v7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal","openshift-multus/multus-additional-cni-plugins-8tkxq","openshift-multus/multus-c56wq","openshift-multus/network-metrics-daemon-s5xr4","kube-system/kube-apiserver-proxy-ip-10-0-134-124.ec2.internal","openshift-dns/node-resolver-h77tf","openshift-image-registry/node-ca-6fpnv","openshift-network-diagnostics/network-check-target-d495m","openshift-network-operator/iptables-alerter-gmfs8","openshift-ovn-kubernetes/ovnkube-node-mvqc5"]
Jul 02 21:18:49.842357 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.842335    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.844498 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.844478    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6fpnv"
Jul 02 21:18:49.846146 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.846126    2586 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 02 21:18:49.846828 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.846454    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 02 21:18:49.846828 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.846549    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 02 21:18:49.846828 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.846456    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 02 21:18:49.846828 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.846771    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcvqp"
Jul 02 21:18:49.847425 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.847373    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 02 21:18:49.847425 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.847407    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 02 21:18:49.847557 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.847407    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 02 21:18:49.848057 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.848038    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 02 21:18:49.848824 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.848801    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:49.851177 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.851127    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:49.853014 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.852770    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wrcpd\""
Jul 02 21:18:49.853014 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.852879    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 02 21:18:49.853014 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.852883    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 02 21:18:49.853014 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.852903    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 02 21:18:49.853014 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.853013    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-cd42p\""
Jul 02 21:18:49.853299 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.853027    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hp9w7\""
Jul 02 21:18:49.853299 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.853042    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 02 21:18:49.853299 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.853015    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4rhwg\""
Jul 02 21:18:49.853705 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.853690    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8tkxq"
Jul 02 21:18:49.854133 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.854116    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 02 21:18:49.854211 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.854136    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 02 21:18:49.854738 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.854722    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b4ctk\""
Jul 02 21:18:49.855377 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.855362    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 02 21:18:49.856025 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.856008    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5xr4"
Jul 02 21:18:49.856107 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.856049    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 02 21:18:49.856107 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:49.856085    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5xr4" podUID="006d61c8-eb34-4336-a3d3-82a7a4fc2069"
Jul 02 21:18:49.856215 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.856110    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 02 21:18:49.856215 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.856009    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 02 21:18:49.856215 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.856009    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 02 21:18:49.857355 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.857332    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-host-run-k8s-cni-cncf-io\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.857446 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.857368    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-host-var-lib-cni-bin\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.857446 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.857396    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/12176f59-c61f-40f7-afce-c13616ba2f36-registration-dir\") pod \"aws-ebs-csi-driver-node-pcvqp\" (UID: \"12176f59-c61f-40f7-afce-c13616ba2f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcvqp"
Jul 02 21:18:49.857446 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.857421    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-cnibin\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.857609 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.857478    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-multus-cni-dir\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.857609 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.857505    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-etc-kubernetes\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.857609 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.857537    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh4ff\" (UniqueName: \"kubernetes.io/projected/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-kube-api-access-rh4ff\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.857609 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.857562    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75f700f7-60d3-4bf0-aef0-2052610d6cba-host\") pod \"node-ca-6fpnv\" (UID: \"75f700f7-60d3-4bf0-aef0-2052610d6cba\") " pod="openshift-image-registry/node-ca-6fpnv"
Jul 02 21:18:49.857609 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.857588    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/12176f59-c61f-40f7-afce-c13616ba2f36-etc-selinux\") pod \"aws-ebs-csi-driver-node-pcvqp\" (UID: \"12176f59-c61f-40f7-afce-c13616ba2f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcvqp"
Jul 02 21:18:49.857865 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.857611    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btrp4\" (UniqueName: \"kubernetes.io/projected/12176f59-c61f-40f7-afce-c13616ba2f36-kube-api-access-btrp4\") pod \"aws-ebs-csi-driver-node-pcvqp\" (UID: \"12176f59-c61f-40f7-afce-c13616ba2f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcvqp"
Jul 02 21:18:49.857865 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.857668    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-multus-daemon-config\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.857865 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.857686    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 02 21:18:49.857865 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.857697    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/75f700f7-60d3-4bf0-aef0-2052610d6cba-serviceca\") pod \"node-ca-6fpnv\" (UID: \"75f700f7-60d3-4bf0-aef0-2052610d6cba\") " pod="openshift-image-registry/node-ca-6fpnv"
Jul 02 21:18:49.857865 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.857723    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-hostroot\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.857865 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.857746    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-host-run-multus-certs\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.857865 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.857768    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-system-cni-dir\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.857865 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.857792    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-multus-socket-dir-parent\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.857865 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.857822    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-host-run-netns\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.857865 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.857854    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-host-var-lib-cni-multus\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.858341 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.857878    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7m2g\" (UniqueName: \"kubernetes.io/projected/75f700f7-60d3-4bf0-aef0-2052610d6cba-kube-api-access-p7m2g\") pod \"node-ca-6fpnv\" (UID: \"75f700f7-60d3-4bf0-aef0-2052610d6cba\") " pod="openshift-image-registry/node-ca-6fpnv"
Jul 02 21:18:49.858341 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.857901    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/12176f59-c61f-40f7-afce-c13616ba2f36-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pcvqp\" (UID: \"12176f59-c61f-40f7-afce-c13616ba2f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcvqp"
Jul 02 21:18:49.858341 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.857924    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/12176f59-c61f-40f7-afce-c13616ba2f36-device-dir\") pod \"aws-ebs-csi-driver-node-pcvqp\" (UID: \"12176f59-c61f-40f7-afce-c13616ba2f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcvqp"
Jul 02 21:18:49.858341 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.857952    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-cni-binary-copy\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.858341 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.857987    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-host-var-lib-kubelet\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.858341 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.858025    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-multus-conf-dir\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.858341 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.858060    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-os-release\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.858341 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.858095    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/12176f59-c61f-40f7-afce-c13616ba2f36-socket-dir\") pod \"aws-ebs-csi-driver-node-pcvqp\" (UID: \"12176f59-c61f-40f7-afce-c13616ba2f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcvqp"
Jul 02 21:18:49.858341 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.858124    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/12176f59-c61f-40f7-afce-c13616ba2f36-sys-fs\") pod \"aws-ebs-csi-driver-node-pcvqp\" (UID: \"12176f59-c61f-40f7-afce-c13616ba2f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcvqp"
Jul 02 21:18:49.858760 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.858401    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gmfs8"
Jul 02 21:18:49.861862 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.860912    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 02 21:18:49.861862 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.860924    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rv7kb\""
Jul 02 21:18:49.861862 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.861445    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 02 21:18:49.861862 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.861468    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kj6rg\""
Jul 02 21:18:49.861862 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.861681    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 02 21:18:49.861862 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.861804    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 02 21:18:49.862185 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.862058    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xn9ss"
Jul 02 21:18:49.864163 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.864144    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h77tf"
Jul 02 21:18:49.865083 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.865051    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8wk2k\""
Jul 02 21:18:49.865347 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.865135    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 02 21:18:49.865347 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.865315    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 02 21:18:49.867196 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.867180    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gfl7"
Jul 02 21:18:49.867292 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:49.867273    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gfl7" podUID="9c214277-f44b-4f90-be58-8c8078403fc2"
Jul 02 21:18:49.869627 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.869529    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d495m"
Jul 02 21:18:49.869627 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:49.869591    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d495m" podUID="45cb060e-3948-4b0e-b407-86ea9b6af872"
Jul 02 21:18:49.873751 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.873730    2586 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 21:18:49.873873 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.873855    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hq5bs\""
Jul 02 21:18:49.873977 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.873861    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 02 21:18:49.874035 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.873974    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 02 21:18:49.907757 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.907721    2586 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9tlpb"
Jul 02 21:18:49.923175 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.923147    2586 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9tlpb"
Jul 02 21:18:49.947313 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.947277    2586 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 02 21:18:49.958522 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.958480    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-host-run-k8s-cni-cncf-io\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.958522 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.958530    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/006d61c8-eb34-4336-a3d3-82a7a4fc2069-metrics-certs\") pod \"network-metrics-daemon-s5xr4\" (UID: \"006d61c8-eb34-4336-a3d3-82a7a4fc2069\") " pod="openshift-multus/network-metrics-daemon-s5xr4"
Jul 02 21:18:49.958755 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.958557    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1c79d16b-84a9-4c32-af87-3e104b416604-etc-sysconfig\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:49.958755 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.958576    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1c79d16b-84a9-4c32-af87-3e104b416604-etc-sysctl-d\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:49.958755 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.958597    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c79d16b-84a9-4c32-af87-3e104b416604-sys\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:49.958755 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.958571    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-host-run-k8s-cni-cncf-io\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.958755 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.958620    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-ovnkube-script-lib\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:49.958755 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.958679    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/12176f59-c61f-40f7-afce-c13616ba2f36-registration-dir\") pod \"aws-ebs-csi-driver-node-pcvqp\" (UID: \"12176f59-c61f-40f7-afce-c13616ba2f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcvqp"
Jul 02 21:18:49.958755 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.958714    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-cnibin\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.958755 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.958743    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce3cb284-71b0-4202-b141-7ad511895978-cni-binary-copy\") pod \"multus-additional-cni-plugins-8tkxq\" (UID: \"ce3cb284-71b0-4202-b141-7ad511895978\") " pod="openshift-multus/multus-additional-cni-plugins-8tkxq"
Jul 02 21:18:49.959066 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.958771    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ea0b0920-86bd-418f-8daf-ec2078dc58d5-agent-certs\") pod \"konnectivity-agent-xn9ss\" (UID: \"ea0b0920-86bd-418f-8daf-ec2078dc58d5\") " pod="kube-system/konnectivity-agent-xn9ss"
Jul 02 21:18:49.959066 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.958798    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-etc-kubernetes\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.959066 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.958801    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/12176f59-c61f-40f7-afce-c13616ba2f36-registration-dir\") pod \"aws-ebs-csi-driver-node-pcvqp\" (UID: \"12176f59-c61f-40f7-afce-c13616ba2f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcvqp"
Jul 02 21:18:49.959066 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.958805    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-cnibin\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.959066 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.958830    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh4ff\" (UniqueName: \"kubernetes.io/projected/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-kube-api-access-rh4ff\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.959066 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.958851    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-etc-kubernetes\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.959066 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.958858    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75f700f7-60d3-4bf0-aef0-2052610d6cba-host\") pod \"node-ca-6fpnv\" (UID: \"75f700f7-60d3-4bf0-aef0-2052610d6cba\") " pod="openshift-image-registry/node-ca-6fpnv"
Jul 02 21:18:49.959066 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.958884    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce3cb284-71b0-4202-b141-7ad511895978-cnibin\") pod \"multus-additional-cni-plugins-8tkxq\" (UID: \"ce3cb284-71b0-4202-b141-7ad511895978\") " pod="openshift-multus/multus-additional-cni-plugins-8tkxq"
Jul 02 21:18:49.959066 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.958910    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptgqs\" (UniqueName: \"kubernetes.io/projected/ce3cb284-71b0-4202-b141-7ad511895978-kube-api-access-ptgqs\") pod \"multus-additional-cni-plugins-8tkxq\" (UID: \"ce3cb284-71b0-4202-b141-7ad511895978\") " pod="openshift-multus/multus-additional-cni-plugins-8tkxq"
Jul 02 21:18:49.959066 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.958934    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75f700f7-60d3-4bf0-aef0-2052610d6cba-host\") pod \"node-ca-6fpnv\" (UID: \"75f700f7-60d3-4bf0-aef0-2052610d6cba\") " pod="openshift-image-registry/node-ca-6fpnv"
Jul 02 21:18:49.959066 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.958952    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thk5p\" (UniqueName: \"kubernetes.io/projected/45cb060e-3948-4b0e-b407-86ea9b6af872-kube-api-access-thk5p\") pod \"network-check-target-d495m\" (UID: \"45cb060e-3948-4b0e-b407-86ea9b6af872\") " pod="openshift-network-diagnostics/network-check-target-d495m"
Jul 02 21:18:49.959066 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.958980    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c79d16b-84a9-4c32-af87-3e104b416604-etc-kubernetes\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:49.959066 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959004    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-host-kubelet\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:49.959066 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959053    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ce3cb284-71b0-4202-b141-7ad511895978-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8tkxq\" (UID: \"ce3cb284-71b0-4202-b141-7ad511895978\") " pod="openshift-multus/multus-additional-cni-plugins-8tkxq"
Jul 02 21:18:49.959677 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959089    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wnqk\" (UniqueName: \"kubernetes.io/projected/83d2714f-a9bc-4830-b379-871a817b6f32-kube-api-access-5wnqk\") pod \"node-resolver-h77tf\" (UID: \"83d2714f-a9bc-4830-b379-871a817b6f32\") " pod="openshift-dns/node-resolver-h77tf"
Jul 02 21:18:49.959677 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959121    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c79d16b-84a9-4c32-af87-3e104b416604-var-lib-kubelet\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:49.959677 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959143    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c79d16b-84a9-4c32-af87-3e104b416604-tmp\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:49.959677 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959177    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-host-run-netns\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:49.959677 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959214    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-node-log\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:49.959677 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959260    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-host-run-ovn-kubernetes\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:49.959677 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959292    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-host-run-multus-certs\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.959677 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959318    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c79d16b-84a9-4c32-af87-3e104b416604-lib-modules\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:49.959677 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959344    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/732f8ea2-dc3e-426a-96a2-d8a7ba62d935-host-slash\") pod \"iptables-alerter-gmfs8\" (UID: \"732f8ea2-dc3e-426a-96a2-d8a7ba62d935\") " pod="openshift-network-operator/iptables-alerter-gmfs8"
Jul 02 21:18:49.959677 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959370    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-ovn-node-metrics-cert\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:49.959677 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959370    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-host-run-multus-certs\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.959677 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959406    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-host-run-netns\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.959677 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959436    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-host-var-lib-cni-multus\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.959677 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959461    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7m2g\" (UniqueName: \"kubernetes.io/projected/75f700f7-60d3-4bf0-aef0-2052610d6cba-kube-api-access-p7m2g\") pod \"node-ca-6fpnv\" (UID: \"75f700f7-60d3-4bf0-aef0-2052610d6cba\") " pod="openshift-image-registry/node-ca-6fpnv"
Jul 02 21:18:49.959677 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959486    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1c79d16b-84a9-4c32-af87-3e104b416604-host\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:49.959677 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959511    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-systemd-units\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:49.959677 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959518    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-host-var-lib-cni-multus\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.960395 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959535    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-host-cni-netd\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:49.960395 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959560    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/12176f59-c61f-40f7-afce-c13616ba2f36-device-dir\") pod \"aws-ebs-csi-driver-node-pcvqp\" (UID: \"12176f59-c61f-40f7-afce-c13616ba2f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcvqp"
Jul 02 21:18:49.960395 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959571    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-host-run-netns\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.960395 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959585    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ce3cb284-71b0-4202-b141-7ad511895978-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8tkxq\" (UID: \"ce3cb284-71b0-4202-b141-7ad511895978\") " pod="openshift-multus/multus-additional-cni-plugins-8tkxq"
Jul 02 21:18:49.960395 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959622    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-env-overrides\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:49.960395 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959691    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/12176f59-c61f-40f7-afce-c13616ba2f36-socket-dir\") pod \"aws-ebs-csi-driver-node-pcvqp\" (UID: \"12176f59-c61f-40f7-afce-c13616ba2f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcvqp"
Jul 02 21:18:49.960395 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959699    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/12176f59-c61f-40f7-afce-c13616ba2f36-device-dir\") pod \"aws-ebs-csi-driver-node-pcvqp\" (UID: \"12176f59-c61f-40f7-afce-c13616ba2f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcvqp"
Jul 02 21:18:49.960395 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959751    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/12176f59-c61f-40f7-afce-c13616ba2f36-sys-fs\") pod \"aws-ebs-csi-driver-node-pcvqp\" (UID: \"12176f59-c61f-40f7-afce-c13616ba2f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcvqp"
Jul 02 21:18:49.960395 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959796    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ce3cb284-71b0-4202-b141-7ad511895978-os-release\") pod \"multus-additional-cni-plugins-8tkxq\" (UID: \"ce3cb284-71b0-4202-b141-7ad511895978\") " pod="openshift-multus/multus-additional-cni-plugins-8tkxq"
Jul 02 21:18:49.960395 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959824    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1c79d16b-84a9-4c32-af87-3e104b416604-etc-tuned\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:49.960395 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959845    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/12176f59-c61f-40f7-afce-c13616ba2f36-socket-dir\") pod \"aws-ebs-csi-driver-node-pcvqp\" (UID: \"12176f59-c61f-40f7-afce-c13616ba2f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcvqp"
Jul 02 21:18:49.960395 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959855    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-etc-openvswitch\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:49.960395 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959890    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-host-var-lib-cni-bin\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.960395 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959899    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/12176f59-c61f-40f7-afce-c13616ba2f36-sys-fs\") pod \"aws-ebs-csi-driver-node-pcvqp\" (UID: \"12176f59-c61f-40f7-afce-c13616ba2f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcvqp"
Jul 02 21:18:49.960395 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959915    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c214277-f44b-4f90-be58-8c8078403fc2-dbus\") pod \"global-pull-secret-syncer-5gfl7\" (UID: \"9c214277-f44b-4f90-be58-8c8078403fc2\") " pod="kube-system/global-pull-secret-syncer-5gfl7"
Jul 02 21:18:49.960395 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959945    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c214277-f44b-4f90-be58-8c8078403fc2-original-pull-secret\") pod \"global-pull-secret-syncer-5gfl7\" (UID: \"9c214277-f44b-4f90-be58-8c8078403fc2\") " pod="kube-system/global-pull-secret-syncer-5gfl7"
Jul 02 21:18:49.960395 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959989    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-host-var-lib-cni-bin\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.961058 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.959993    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1c79d16b-84a9-4c32-af87-3e104b416604-etc-sysctl-conf\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:49.961058 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.960039    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1c79d16b-84a9-4c32-af87-3e104b416604-run\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:49.961058 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.960076    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-log-socket\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:49.961058 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.960118    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce3cb284-71b0-4202-b141-7ad511895978-system-cni-dir\") pod \"multus-additional-cni-plugins-8tkxq\" (UID: \"ce3cb284-71b0-4202-b141-7ad511895978\") " pod="openshift-multus/multus-additional-cni-plugins-8tkxq"
Jul 02 21:18:49.961058 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.960148    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-run-systemd\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:49.961058 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.960176    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-multus-cni-dir\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.961058 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.960200    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wtmb\" (UniqueName: \"kubernetes.io/projected/006d61c8-eb34-4336-a3d3-82a7a4fc2069-kube-api-access-9wtmb\") pod \"network-metrics-daemon-s5xr4\" (UID: \"006d61c8-eb34-4336-a3d3-82a7a4fc2069\") " pod="openshift-multus/network-metrics-daemon-s5xr4"
Jul 02 21:18:49.961058 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.960234    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/83d2714f-a9bc-4830-b379-871a817b6f32-hosts-file\") pod \"node-resolver-h77tf\" (UID: \"83d2714f-a9bc-4830-b379-871a817b6f32\") " pod="openshift-dns/node-resolver-h77tf"
Jul 02 21:18:49.961058 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.960279    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7m5d\" (UniqueName: \"kubernetes.io/projected/1c79d16b-84a9-4c32-af87-3e104b416604-kube-api-access-f7m5d\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:49.961058 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.960304    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-multus-cni-dir\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.961058 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.960323    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:49.961058 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.960361    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/12176f59-c61f-40f7-afce-c13616ba2f36-etc-selinux\") pod \"aws-ebs-csi-driver-node-pcvqp\" (UID: \"12176f59-c61f-40f7-afce-c13616ba2f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcvqp"
Jul 02 21:18:49.961058 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.960421    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btrp4\" (UniqueName: \"kubernetes.io/projected/12176f59-c61f-40f7-afce-c13616ba2f36-kube-api-access-btrp4\") pod \"aws-ebs-csi-driver-node-pcvqp\" (UID: \"12176f59-c61f-40f7-afce-c13616ba2f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcvqp"
Jul 02 21:18:49.961058 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.960434    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/12176f59-c61f-40f7-afce-c13616ba2f36-etc-selinux\") pod \"aws-ebs-csi-driver-node-pcvqp\" (UID: \"12176f59-c61f-40f7-afce-c13616ba2f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcvqp"
Jul 02 21:18:49.961058 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.960465    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-multus-daemon-config\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.961058 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.960517    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/75f700f7-60d3-4bf0-aef0-2052610d6cba-serviceca\") pod \"node-ca-6fpnv\" (UID: \"75f700f7-60d3-4bf0-aef0-2052610d6cba\") " pod="openshift-image-registry/node-ca-6fpnv"
Jul 02 21:18:49.961058 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.960585    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-host-cni-bin\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:49.961872 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.960620    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-hostroot\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.961872 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.960669    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce3cb284-71b0-4202-b141-7ad511895978-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8tkxq\" (UID: \"ce3cb284-71b0-4202-b141-7ad511895978\") " pod="openshift-multus/multus-additional-cni-plugins-8tkxq"
Jul 02 21:18:49.961872 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.960701    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx4v4\" (UniqueName: \"kubernetes.io/projected/732f8ea2-dc3e-426a-96a2-d8a7ba62d935-kube-api-access-dx4v4\") pod \"iptables-alerter-gmfs8\" (UID: \"732f8ea2-dc3e-426a-96a2-d8a7ba62d935\") " pod="openshift-network-operator/iptables-alerter-gmfs8"
Jul 02 21:18:49.961872 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.960725    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-hostroot\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.961872 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.960770    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-var-lib-openvswitch\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:49.961872 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.960796    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-run-ovn\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:49.961872 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.960842    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-system-cni-dir\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.961872 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.960868    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-multus-socket-dir-parent\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.961872 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.960914    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-system-cni-dir\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.961872 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.960934    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-multus-socket-dir-parent\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.961872 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.960939    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c214277-f44b-4f90-be58-8c8078403fc2-kubelet-config\") pod \"global-pull-secret-syncer-5gfl7\" (UID: \"9c214277-f44b-4f90-be58-8c8078403fc2\") " pod="kube-system/global-pull-secret-syncer-5gfl7"
Jul 02 21:18:49.961872 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.960981    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-run-openvswitch\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:49.961872 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.961011    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8t7w\" (UniqueName: \"kubernetes.io/projected/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-kube-api-access-m8t7w\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:49.961872 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.961029    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/75f700f7-60d3-4bf0-aef0-2052610d6cba-serviceca\") pod \"node-ca-6fpnv\" (UID: \"75f700f7-60d3-4bf0-aef0-2052610d6cba\") " pod="openshift-image-registry/node-ca-6fpnv"
Jul 02 21:18:49.961872 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.961040    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/12176f59-c61f-40f7-afce-c13616ba2f36-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pcvqp\" (UID: \"12176f59-c61f-40f7-afce-c13616ba2f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcvqp"
Jul 02 21:18:49.961872 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.961081    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-cni-binary-copy\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.961872 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.961099    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/12176f59-c61f-40f7-afce-c13616ba2f36-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pcvqp\" (UID: \"12176f59-c61f-40f7-afce-c13616ba2f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcvqp"
Jul 02 21:18:49.962458 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.961124    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-host-var-lib-kubelet\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.962458 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.961127    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-multus-daemon-config\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.962458 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.961245    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-multus-conf-dir\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.962458 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.961287    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-host-var-lib-kubelet\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.962458 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.961351    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/732f8ea2-dc3e-426a-96a2-d8a7ba62d935-iptables-alerter-script\") pod \"iptables-alerter-gmfs8\" (UID: \"732f8ea2-dc3e-426a-96a2-d8a7ba62d935\") " pod="openshift-network-operator/iptables-alerter-gmfs8"
Jul 02 21:18:49.962458 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.961357    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-multus-conf-dir\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.962458 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.961408    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-host-slash\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:49.962458 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.961444    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-ovnkube-config\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:49.962458 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.961473    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-os-release\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.962458 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.961497    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83d2714f-a9bc-4830-b379-871a817b6f32-tmp-dir\") pod \"node-resolver-h77tf\" (UID: \"83d2714f-a9bc-4830-b379-871a817b6f32\") " pod="openshift-dns/node-resolver-h77tf"
Jul 02 21:18:49.962458 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.961522    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1c79d16b-84a9-4c32-af87-3e104b416604-etc-modprobe-d\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:49.962458 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.961543    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1c79d16b-84a9-4c32-af87-3e104b416604-etc-systemd\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:49.962458 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.961565    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ea0b0920-86bd-418f-8daf-ec2078dc58d5-konnectivity-ca\") pod \"konnectivity-agent-xn9ss\" (UID: \"ea0b0920-86bd-418f-8daf-ec2078dc58d5\") " pod="kube-system/konnectivity-agent-xn9ss"
Jul 02 21:18:49.962458 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.961568    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-cni-binary-copy\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.962458 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.961707    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-os-release\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:49.968199 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.968160    2586 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 02 21:18:49.972189 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.971931    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btrp4\" (UniqueName: \"kubernetes.io/projected/12176f59-c61f-40f7-afce-c13616ba2f36-kube-api-access-btrp4\") pod \"aws-ebs-csi-driver-node-pcvqp\" (UID: \"12176f59-c61f-40f7-afce-c13616ba2f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcvqp"
Jul 02 21:18:49.972189 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.971930    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7m2g\" (UniqueName: \"kubernetes.io/projected/75f700f7-60d3-4bf0-aef0-2052610d6cba-kube-api-access-p7m2g\") pod \"node-ca-6fpnv\" (UID: \"75f700f7-60d3-4bf0-aef0-2052610d6cba\") " pod="openshift-image-registry/node-ca-6fpnv"
Jul 02 21:18:49.972387 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:49.972371    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh4ff\" (UniqueName: \"kubernetes.io/projected/e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a-kube-api-access-rh4ff\") pod \"multus-c56wq\" (UID: \"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a\") " pod="openshift-multus/multus-c56wq"
Jul 02 21:18:50.028359 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.028329    2586 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 21:18:50.062115 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062078    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c214277-f44b-4f90-be58-8c8078403fc2-original-pull-secret\") pod \"global-pull-secret-syncer-5gfl7\" (UID: \"9c214277-f44b-4f90-be58-8c8078403fc2\") " pod="kube-system/global-pull-secret-syncer-5gfl7"
Jul 02 21:18:50.062301 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062127    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1c79d16b-84a9-4c32-af87-3e104b416604-etc-sysctl-conf\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:50.062301 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062150    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1c79d16b-84a9-4c32-af87-3e104b416604-run\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:50.062301 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062176    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-log-socket\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.062301 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062201    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce3cb284-71b0-4202-b141-7ad511895978-system-cni-dir\") pod \"multus-additional-cni-plugins-8tkxq\" (UID: \"ce3cb284-71b0-4202-b141-7ad511895978\") " pod="openshift-multus/multus-additional-cni-plugins-8tkxq"
Jul 02 21:18:50.062301 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062225    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-run-systemd\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.062301 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062274    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wtmb\" (UniqueName: \"kubernetes.io/projected/006d61c8-eb34-4336-a3d3-82a7a4fc2069-kube-api-access-9wtmb\") pod \"network-metrics-daemon-s5xr4\" (UID: \"006d61c8-eb34-4336-a3d3-82a7a4fc2069\") " pod="openshift-multus/network-metrics-daemon-s5xr4"
Jul 02 21:18:50.062572 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062299    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/83d2714f-a9bc-4830-b379-871a817b6f32-hosts-file\") pod \"node-resolver-h77tf\" (UID: \"83d2714f-a9bc-4830-b379-871a817b6f32\") " pod="openshift-dns/node-resolver-h77tf"
Jul 02 21:18:50.062572 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062357    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7m5d\" (UniqueName: \"kubernetes.io/projected/1c79d16b-84a9-4c32-af87-3e104b416604-kube-api-access-f7m5d\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:50.062572 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062383    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.062572 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062412    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-host-cni-bin\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.062572 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062420    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-log-socket\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.062572 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062462    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce3cb284-71b0-4202-b141-7ad511895978-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8tkxq\" (UID: \"ce3cb284-71b0-4202-b141-7ad511895978\") " pod="openshift-multus/multus-additional-cni-plugins-8tkxq"
Jul 02 21:18:50.062572 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062472    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-host-cni-bin\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.062572 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062491    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1c79d16b-84a9-4c32-af87-3e104b416604-run\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:50.062572 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062499    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx4v4\" (UniqueName: \"kubernetes.io/projected/732f8ea2-dc3e-426a-96a2-d8a7ba62d935-kube-api-access-dx4v4\") pod \"iptables-alerter-gmfs8\" (UID: \"732f8ea2-dc3e-426a-96a2-d8a7ba62d935\") " pod="openshift-network-operator/iptables-alerter-gmfs8"
Jul 02 21:18:50.062572 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062500    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce3cb284-71b0-4202-b141-7ad511895978-system-cni-dir\") pod \"multus-additional-cni-plugins-8tkxq\" (UID: \"ce3cb284-71b0-4202-b141-7ad511895978\") " pod="openshift-multus/multus-additional-cni-plugins-8tkxq"
Jul 02 21:18:50.062572 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062527    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-run-systemd\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.062572 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062535    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-var-lib-openvswitch\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.062572 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062560    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-run-ovn\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.062572 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062566    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1c79d16b-84a9-4c32-af87-3e104b416604-etc-sysctl-conf\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:50.063303 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062584    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/83d2714f-a9bc-4830-b379-871a817b6f32-hosts-file\") pod \"node-resolver-h77tf\" (UID: \"83d2714f-a9bc-4830-b379-871a817b6f32\") " pod="openshift-dns/node-resolver-h77tf"
Jul 02 21:18:50.063303 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062586    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c214277-f44b-4f90-be58-8c8078403fc2-kubelet-config\") pod \"global-pull-secret-syncer-5gfl7\" (UID: \"9c214277-f44b-4f90-be58-8c8078403fc2\") " pod="kube-system/global-pull-secret-syncer-5gfl7"
Jul 02 21:18:50.063303 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062620    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-run-ovn\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.063303 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062622    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-run-openvswitch\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.063303 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062641    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c214277-f44b-4f90-be58-8c8078403fc2-kubelet-config\") pod \"global-pull-secret-syncer-5gfl7\" (UID: \"9c214277-f44b-4f90-be58-8c8078403fc2\") " pod="kube-system/global-pull-secret-syncer-5gfl7"
Jul 02 21:18:50.063303 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062669    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8t7w\" (UniqueName: \"kubernetes.io/projected/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-kube-api-access-m8t7w\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.063303 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062674    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-var-lib-openvswitch\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.063303 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062701    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce3cb284-71b0-4202-b141-7ad511895978-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8tkxq\" (UID: \"ce3cb284-71b0-4202-b141-7ad511895978\") " pod="openshift-multus/multus-additional-cni-plugins-8tkxq"
Jul 02 21:18:50.063303 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062709    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-run-openvswitch\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.063303 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062714    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.063303 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062702    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/732f8ea2-dc3e-426a-96a2-d8a7ba62d935-iptables-alerter-script\") pod \"iptables-alerter-gmfs8\" (UID: \"732f8ea2-dc3e-426a-96a2-d8a7ba62d935\") " pod="openshift-network-operator/iptables-alerter-gmfs8"
Jul 02 21:18:50.063303 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062753    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-host-slash\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.063303 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062778    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-ovnkube-config\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.063303 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062802    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83d2714f-a9bc-4830-b379-871a817b6f32-tmp-dir\") pod \"node-resolver-h77tf\" (UID: \"83d2714f-a9bc-4830-b379-871a817b6f32\") " pod="openshift-dns/node-resolver-h77tf"
Jul 02 21:18:50.063303 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062817    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-host-slash\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.063303 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062823    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1c79d16b-84a9-4c32-af87-3e104b416604-etc-modprobe-d\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:50.063303 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062853    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1c79d16b-84a9-4c32-af87-3e104b416604-etc-systemd\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:50.063912 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062879    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ea0b0920-86bd-418f-8daf-ec2078dc58d5-konnectivity-ca\") pod \"konnectivity-agent-xn9ss\" (UID: \"ea0b0920-86bd-418f-8daf-ec2078dc58d5\") " pod="kube-system/konnectivity-agent-xn9ss"
Jul 02 21:18:50.063912 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:50.062778    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 21:18:50.063912 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062931    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1c79d16b-84a9-4c32-af87-3e104b416604-etc-modprobe-d\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:50.063912 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:50.063004    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:18:50.063912 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:50.063020    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c214277-f44b-4f90-be58-8c8078403fc2-original-pull-secret podName:9c214277-f44b-4f90-be58-8c8078403fc2 nodeName:}" failed. No retries permitted until 2026-07-02 21:18:50.562976043 +0000 UTC m=+3.005776463 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c214277-f44b-4f90-be58-8c8078403fc2-original-pull-secret") pod "global-pull-secret-syncer-5gfl7" (UID: "9c214277-f44b-4f90-be58-8c8078403fc2") : object "kube-system"/"original-pull-secret" not registered
Jul 02 21:18:50.063912 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:50.063049    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/006d61c8-eb34-4336-a3d3-82a7a4fc2069-metrics-certs podName:006d61c8-eb34-4336-a3d3-82a7a4fc2069 nodeName:}" failed. No retries permitted until 2026-07-02 21:18:50.563034348 +0000 UTC m=+3.005834757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/006d61c8-eb34-4336-a3d3-82a7a4fc2069-metrics-certs") pod "network-metrics-daemon-s5xr4" (UID: "006d61c8-eb34-4336-a3d3-82a7a4fc2069") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:18:50.063912 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.062908    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/006d61c8-eb34-4336-a3d3-82a7a4fc2069-metrics-certs\") pod \"network-metrics-daemon-s5xr4\" (UID: \"006d61c8-eb34-4336-a3d3-82a7a4fc2069\") " pod="openshift-multus/network-metrics-daemon-s5xr4"
Jul 02 21:18:50.063912 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.063078    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1c79d16b-84a9-4c32-af87-3e104b416604-etc-sysconfig\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:50.063912 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.063102    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1c79d16b-84a9-4c32-af87-3e104b416604-etc-sysctl-d\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:50.063912 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.063124    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c79d16b-84a9-4c32-af87-3e104b416604-sys\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:50.063912 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.063157    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-ovnkube-script-lib\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.063912 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.063185    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce3cb284-71b0-4202-b141-7ad511895978-cni-binary-copy\") pod \"multus-additional-cni-plugins-8tkxq\" (UID: \"ce3cb284-71b0-4202-b141-7ad511895978\") " pod="openshift-multus/multus-additional-cni-plugins-8tkxq"
Jul 02 21:18:50.063912 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.063215    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ea0b0920-86bd-418f-8daf-ec2078dc58d5-agent-certs\") pod \"konnectivity-agent-xn9ss\" (UID: \"ea0b0920-86bd-418f-8daf-ec2078dc58d5\") " pod="kube-system/konnectivity-agent-xn9ss"
Jul 02 21:18:50.063912 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.063231    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c79d16b-84a9-4c32-af87-3e104b416604-sys\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:50.063912 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.063245    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce3cb284-71b0-4202-b141-7ad511895978-cnibin\") pod \"multus-additional-cni-plugins-8tkxq\" (UID: \"ce3cb284-71b0-4202-b141-7ad511895978\") " pod="openshift-multus/multus-additional-cni-plugins-8tkxq"
Jul 02 21:18:50.063912 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.063271    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptgqs\" (UniqueName: \"kubernetes.io/projected/ce3cb284-71b0-4202-b141-7ad511895978-kube-api-access-ptgqs\") pod \"multus-additional-cni-plugins-8tkxq\" (UID: \"ce3cb284-71b0-4202-b141-7ad511895978\") " pod="openshift-multus/multus-additional-cni-plugins-8tkxq"
Jul 02 21:18:50.063912 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.063287    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1c79d16b-84a9-4c32-af87-3e104b416604-etc-sysctl-d\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:50.064739 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.063297    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thk5p\" (UniqueName: \"kubernetes.io/projected/45cb060e-3948-4b0e-b407-86ea9b6af872-kube-api-access-thk5p\") pod \"network-check-target-d495m\" (UID: \"45cb060e-3948-4b0e-b407-86ea9b6af872\") " pod="openshift-network-diagnostics/network-check-target-d495m"
Jul 02 21:18:50.064739 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.063321    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c79d16b-84a9-4c32-af87-3e104b416604-etc-kubernetes\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:50.064739 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.063338    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1c79d16b-84a9-4c32-af87-3e104b416604-etc-systemd\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:50.064739 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.063343    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-host-kubelet\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.064739 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.063371    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ce3cb284-71b0-4202-b141-7ad511895978-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8tkxq\" (UID: \"ce3cb284-71b0-4202-b141-7ad511895978\") " pod="openshift-multus/multus-additional-cni-plugins-8tkxq"
Jul 02 21:18:50.064739 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.063385    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1c79d16b-84a9-4c32-af87-3e104b416604-etc-sysconfig\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:50.064739 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.063398    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wnqk\" (UniqueName: \"kubernetes.io/projected/83d2714f-a9bc-4830-b379-871a817b6f32-kube-api-access-5wnqk\") pod \"node-resolver-h77tf\" (UID: \"83d2714f-a9bc-4830-b379-871a817b6f32\") " pod="openshift-dns/node-resolver-h77tf"
Jul 02 21:18:50.064739 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.063422    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c79d16b-84a9-4c32-af87-3e104b416604-var-lib-kubelet\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:50.064739 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.063445    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c79d16b-84a9-4c32-af87-3e104b416604-tmp\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:50.064739 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.063470    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-host-run-netns\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.064739 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.063481    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ea0b0920-86bd-418f-8daf-ec2078dc58d5-konnectivity-ca\") pod \"konnectivity-agent-xn9ss\" (UID: \"ea0b0920-86bd-418f-8daf-ec2078dc58d5\") " pod="kube-system/konnectivity-agent-xn9ss"
Jul 02 21:18:50.064739 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.063493    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-node-log\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.064739 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.063530    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-host-run-ovn-kubernetes\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.064739 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.063536    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-node-log\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.064739 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.063184    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83d2714f-a9bc-4830-b379-871a817b6f32-tmp-dir\") pod \"node-resolver-h77tf\" (UID: \"83d2714f-a9bc-4830-b379-871a817b6f32\") " pod="openshift-dns/node-resolver-h77tf"
Jul 02 21:18:50.064739 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.063592    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-host-run-ovn-kubernetes\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.064739 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.063634    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-host-kubelet\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.065486 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.063702    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c79d16b-84a9-4c32-af87-3e104b416604-etc-kubernetes\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:50.065486 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.063816    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-host-run-netns\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.065486 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.063853    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c79d16b-84a9-4c32-af87-3e104b416604-lib-modules\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:50.065486 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.063885    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/732f8ea2-dc3e-426a-96a2-d8a7ba62d935-host-slash\") pod \"iptables-alerter-gmfs8\" (UID: \"732f8ea2-dc3e-426a-96a2-d8a7ba62d935\") " pod="openshift-network-operator/iptables-alerter-gmfs8"
Jul 02 21:18:50.065486 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.063910    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-ovn-node-metrics-cert\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.065486 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.063908    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-ovnkube-script-lib\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.065486 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.063909    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce3cb284-71b0-4202-b141-7ad511895978-cni-binary-copy\") pod \"multus-additional-cni-plugins-8tkxq\" (UID: \"ce3cb284-71b0-4202-b141-7ad511895978\") " pod="openshift-multus/multus-additional-cni-plugins-8tkxq"
Jul 02 21:18:50.065486 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.063975    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c79d16b-84a9-4c32-af87-3e104b416604-var-lib-kubelet\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:50.065486 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.063979    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/732f8ea2-dc3e-426a-96a2-d8a7ba62d935-host-slash\") pod \"iptables-alerter-gmfs8\" (UID: \"732f8ea2-dc3e-426a-96a2-d8a7ba62d935\") " pod="openshift-network-operator/iptables-alerter-gmfs8"
Jul 02 21:18:50.065486 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.064019    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1c79d16b-84a9-4c32-af87-3e104b416604-host\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:50.065486 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.064047    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/732f8ea2-dc3e-426a-96a2-d8a7ba62d935-iptables-alerter-script\") pod \"iptables-alerter-gmfs8\" (UID: \"732f8ea2-dc3e-426a-96a2-d8a7ba62d935\") " pod="openshift-network-operator/iptables-alerter-gmfs8"
Jul 02 21:18:50.065486 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.064076    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-systemd-units\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.065486 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.064080    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c79d16b-84a9-4c32-af87-3e104b416604-lib-modules\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:50.065486 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.064103    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce3cb284-71b0-4202-b141-7ad511895978-cnibin\") pod \"multus-additional-cni-plugins-8tkxq\" (UID: \"ce3cb284-71b0-4202-b141-7ad511895978\") " pod="openshift-multus/multus-additional-cni-plugins-8tkxq"
Jul 02 21:18:50.065486 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.064104    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-host-cni-netd\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.065486 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.064123    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-systemd-units\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.065486 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.064137    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-host-cni-netd\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.066288 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.064146    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ce3cb284-71b0-4202-b141-7ad511895978-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8tkxq\" (UID: \"ce3cb284-71b0-4202-b141-7ad511895978\") " pod="openshift-multus/multus-additional-cni-plugins-8tkxq"
Jul 02 21:18:50.066288 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.064174    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-env-overrides\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.066288 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.064182    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ce3cb284-71b0-4202-b141-7ad511895978-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8tkxq\" (UID: \"ce3cb284-71b0-4202-b141-7ad511895978\") " pod="openshift-multus/multus-additional-cni-plugins-8tkxq"
Jul 02 21:18:50.066288 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.064199    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ce3cb284-71b0-4202-b141-7ad511895978-os-release\") pod \"multus-additional-cni-plugins-8tkxq\" (UID: \"ce3cb284-71b0-4202-b141-7ad511895978\") " pod="openshift-multus/multus-additional-cni-plugins-8tkxq"
Jul 02 21:18:50.066288 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.064137    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1c79d16b-84a9-4c32-af87-3e104b416604-host\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:50.066288 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.064239    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1c79d16b-84a9-4c32-af87-3e104b416604-etc-tuned\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:50.066288 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.064270    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-etc-openvswitch\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.066288 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.064302    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c214277-f44b-4f90-be58-8c8078403fc2-dbus\") pod \"global-pull-secret-syncer-5gfl7\" (UID: \"9c214277-f44b-4f90-be58-8c8078403fc2\") " pod="kube-system/global-pull-secret-syncer-5gfl7"
Jul 02 21:18:50.066288 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.064314    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ce3cb284-71b0-4202-b141-7ad511895978-os-release\") pod \"multus-additional-cni-plugins-8tkxq\" (UID: \"ce3cb284-71b0-4202-b141-7ad511895978\") " pod="openshift-multus/multus-additional-cni-plugins-8tkxq"
Jul 02 21:18:50.066288 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.064380    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-etc-openvswitch\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.066288 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.064485    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c214277-f44b-4f90-be58-8c8078403fc2-dbus\") pod \"global-pull-secret-syncer-5gfl7\" (UID: \"9c214277-f44b-4f90-be58-8c8078403fc2\") " pod="kube-system/global-pull-secret-syncer-5gfl7"
Jul 02 21:18:50.066288 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.064605    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-env-overrides\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.066288 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.064617    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ce3cb284-71b0-4202-b141-7ad511895978-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8tkxq\" (UID: \"ce3cb284-71b0-4202-b141-7ad511895978\") " pod="openshift-multus/multus-additional-cni-plugins-8tkxq"
Jul 02 21:18:50.066288 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.064607    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-ovnkube-config\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.067001 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.066817    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1c79d16b-84a9-4c32-af87-3e104b416604-etc-tuned\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:50.067562 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.067511    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-ovn-node-metrics-cert\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.068091 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.068046    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ea0b0920-86bd-418f-8daf-ec2078dc58d5-agent-certs\") pod \"konnectivity-agent-xn9ss\" (UID: \"ea0b0920-86bd-418f-8daf-ec2078dc58d5\") " pod="kube-system/konnectivity-agent-xn9ss"
Jul 02 21:18:50.068258 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.068228    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c79d16b-84a9-4c32-af87-3e104b416604-tmp\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:50.077316 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:50.077294    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 21:18:50.077419 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:50.077322    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 21:18:50.077419 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:50.077335    2586 projected.go:196] Error preparing data for projected volume kube-api-access-thk5p for pod openshift-network-diagnostics/network-check-target-d495m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:18:50.077419 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:50.077391    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45cb060e-3948-4b0e-b407-86ea9b6af872-kube-api-access-thk5p podName:45cb060e-3948-4b0e-b407-86ea9b6af872 nodeName:}" failed. No retries permitted until 2026-07-02 21:18:50.577374061 +0000 UTC m=+3.020174484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-thk5p" (UniqueName: "kubernetes.io/projected/45cb060e-3948-4b0e-b407-86ea9b6af872-kube-api-access-thk5p") pod "network-check-target-d495m" (UID: "45cb060e-3948-4b0e-b407-86ea9b6af872") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:18:50.078401 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.078329    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8t7w\" (UniqueName: \"kubernetes.io/projected/da774dac-c1e6-4f51-b69e-95a97f5bf0f1-kube-api-access-m8t7w\") pod \"ovnkube-node-mvqc5\" (UID: \"da774dac-c1e6-4f51-b69e-95a97f5bf0f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.078695 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.078680    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptgqs\" (UniqueName: \"kubernetes.io/projected/ce3cb284-71b0-4202-b141-7ad511895978-kube-api-access-ptgqs\") pod \"multus-additional-cni-plugins-8tkxq\" (UID: \"ce3cb284-71b0-4202-b141-7ad511895978\") " pod="openshift-multus/multus-additional-cni-plugins-8tkxq"
Jul 02 21:18:50.078903 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.078883    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wnqk\" (UniqueName: \"kubernetes.io/projected/83d2714f-a9bc-4830-b379-871a817b6f32-kube-api-access-5wnqk\") pod \"node-resolver-h77tf\" (UID: \"83d2714f-a9bc-4830-b379-871a817b6f32\") " pod="openshift-dns/node-resolver-h77tf"
Jul 02 21:18:50.079776 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.079757    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx4v4\" (UniqueName: \"kubernetes.io/projected/732f8ea2-dc3e-426a-96a2-d8a7ba62d935-kube-api-access-dx4v4\") pod \"iptables-alerter-gmfs8\" (UID: \"732f8ea2-dc3e-426a-96a2-d8a7ba62d935\") " pod="openshift-network-operator/iptables-alerter-gmfs8"
Jul 02 21:18:50.080065 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.080044    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wtmb\" (UniqueName: \"kubernetes.io/projected/006d61c8-eb34-4336-a3d3-82a7a4fc2069-kube-api-access-9wtmb\") pod \"network-metrics-daemon-s5xr4\" (UID: \"006d61c8-eb34-4336-a3d3-82a7a4fc2069\") " pod="openshift-multus/network-metrics-daemon-s5xr4"
Jul 02 21:18:50.083603 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.083584    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7m5d\" (UniqueName: \"kubernetes.io/projected/1c79d16b-84a9-4c32-af87-3e104b416604-kube-api-access-f7m5d\") pod \"tuned-8j5v7\" (UID: \"1c79d16b-84a9-4c32-af87-3e104b416604\") " pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:50.160980 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.160940    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c56wq"
Jul 02 21:18:50.167207 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.167182    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6fpnv"
Jul 02 21:18:50.173854 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.173837    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcvqp"
Jul 02 21:18:50.180352 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.180333    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8j5v7"
Jul 02 21:18:50.185992 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.185975    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:18:50.194643 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.194620    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8tkxq"
Jul 02 21:18:50.202335 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.202313    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gmfs8"
Jul 02 21:18:50.208067 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.208048    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xn9ss"
Jul 02 21:18:50.215708 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.215688    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h77tf"
Jul 02 21:18:50.568255 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.568216    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/006d61c8-eb34-4336-a3d3-82a7a4fc2069-metrics-certs\") pod \"network-metrics-daemon-s5xr4\" (UID: \"006d61c8-eb34-4336-a3d3-82a7a4fc2069\") " pod="openshift-multus/network-metrics-daemon-s5xr4"
Jul 02 21:18:50.568450 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.568311    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c214277-f44b-4f90-be58-8c8078403fc2-original-pull-secret\") pod \"global-pull-secret-syncer-5gfl7\" (UID: \"9c214277-f44b-4f90-be58-8c8078403fc2\") " pod="kube-system/global-pull-secret-syncer-5gfl7"
Jul 02 21:18:50.568450 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:50.568389    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:18:50.568450 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:50.568402    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 21:18:50.568577 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:50.568457    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c214277-f44b-4f90-be58-8c8078403fc2-original-pull-secret podName:9c214277-f44b-4f90-be58-8c8078403fc2 nodeName:}" failed. No retries permitted until 2026-07-02 21:18:51.568442234 +0000 UTC m=+4.011242635 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c214277-f44b-4f90-be58-8c8078403fc2-original-pull-secret") pod "global-pull-secret-syncer-5gfl7" (UID: "9c214277-f44b-4f90-be58-8c8078403fc2") : object "kube-system"/"original-pull-secret" not registered
Jul 02 21:18:50.568577 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:50.568477    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/006d61c8-eb34-4336-a3d3-82a7a4fc2069-metrics-certs podName:006d61c8-eb34-4336-a3d3-82a7a4fc2069 nodeName:}" failed. No retries permitted until 2026-07-02 21:18:51.568468182 +0000 UTC m=+4.011268596 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/006d61c8-eb34-4336-a3d3-82a7a4fc2069-metrics-certs") pod "network-metrics-daemon-s5xr4" (UID: "006d61c8-eb34-4336-a3d3-82a7a4fc2069") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:18:50.669035 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.669002    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thk5p\" (UniqueName: \"kubernetes.io/projected/45cb060e-3948-4b0e-b407-86ea9b6af872-kube-api-access-thk5p\") pod \"network-check-target-d495m\" (UID: \"45cb060e-3948-4b0e-b407-86ea9b6af872\") " pod="openshift-network-diagnostics/network-check-target-d495m"
Jul 02 21:18:50.669227 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:50.669206    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 21:18:50.669282 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:50.669238    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 21:18:50.669282 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:50.669251    2586 projected.go:196] Error preparing data for projected volume kube-api-access-thk5p for pod openshift-network-diagnostics/network-check-target-d495m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:18:50.669342 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:50.669323    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45cb060e-3948-4b0e-b407-86ea9b6af872-kube-api-access-thk5p podName:45cb060e-3948-4b0e-b407-86ea9b6af872 nodeName:}" failed. No retries permitted until 2026-07-02 21:18:51.669302887 +0000 UTC m=+4.112103292 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-thk5p" (UniqueName: "kubernetes.io/projected/45cb060e-3948-4b0e-b407-86ea9b6af872-kube-api-access-thk5p") pod "network-check-target-d495m" (UID: "45cb060e-3948-4b0e-b407-86ea9b6af872") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:18:50.786546 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.786511    2586 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 21:18:50.926806 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.926725    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 21:13:49 +0000 UTC" deadline="2028-01-24 16:17:06.333346299 +0000 UTC"
Jul 02 21:18:50.926806 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:50.926764    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13698h58m15.406585586s"
Jul 02 21:18:50.979728 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:50.979563    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea0b0920_86bd_418f_8daf_ec2078dc58d5.slice/crio-c8b22d9aafe9384436e8abf894f6542729380dbfaa64c10777bc99c9ecb10a9d WatchSource:0}: Error finding container c8b22d9aafe9384436e8abf894f6542729380dbfaa64c10777bc99c9ecb10a9d: Status 404 returned error can't find the container with id c8b22d9aafe9384436e8abf894f6542729380dbfaa64c10777bc99c9ecb10a9d
Jul 02 21:18:50.982839 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:50.982801    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode26f8e6a_05cd_4164_bc55_6f44d1c8bf8a.slice/crio-f7a068adc6000b6af8d9855937fa9826407fb5f74b983059ef50fb540019c595 WatchSource:0}: Error finding container f7a068adc6000b6af8d9855937fa9826407fb5f74b983059ef50fb540019c595: Status 404 returned error can't find the container with id f7a068adc6000b6af8d9855937fa9826407fb5f74b983059ef50fb540019c595
Jul 02 21:18:50.984809 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:50.984782    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12176f59_c61f_40f7_afce_c13616ba2f36.slice/crio-e7fcd116eeb3f64159987c4cd0c3090e87e45f57aca00704b46c308af1cb5441 WatchSource:0}: Error finding container e7fcd116eeb3f64159987c4cd0c3090e87e45f57aca00704b46c308af1cb5441: Status 404 returned error can't find the container with id e7fcd116eeb3f64159987c4cd0c3090e87e45f57aca00704b46c308af1cb5441
Jul 02 21:18:50.985386 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:50.985364    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce3cb284_71b0_4202_b141_7ad511895978.slice/crio-0528f98552dcfb919b04b056c0129fc90b2ce051c669c1676cbd63b3b3943aa3 WatchSource:0}: Error finding container 0528f98552dcfb919b04b056c0129fc90b2ce051c669c1676cbd63b3b3943aa3: Status 404 returned error can't find the container with id 0528f98552dcfb919b04b056c0129fc90b2ce051c669c1676cbd63b3b3943aa3
Jul 02 21:18:50.986569 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:50.986375    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83d2714f_a9bc_4830_b379_871a817b6f32.slice/crio-98920c03f866c79fffa0da1d7a01ddda24fc310c98de538011b3fb908eb213cf WatchSource:0}: Error finding container 98920c03f866c79fffa0da1d7a01ddda24fc310c98de538011b3fb908eb213cf: Status 404 returned error can't find the container with id 98920c03f866c79fffa0da1d7a01ddda24fc310c98de538011b3fb908eb213cf
Jul 02 21:18:50.987551 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:50.987529    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f700f7_60d3_4bf0_aef0_2052610d6cba.slice/crio-564963c9e972362d8504338191e513a7c9f420021d5b3335952f11ddf84cd1cf WatchSource:0}: Error finding container 564963c9e972362d8504338191e513a7c9f420021d5b3335952f11ddf84cd1cf: Status 404 returned error can't find the container with id 564963c9e972362d8504338191e513a7c9f420021d5b3335952f11ddf84cd1cf
Jul 02 21:18:50.990725 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:18:50.989272    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c79d16b_84a9_4c32_af87_3e104b416604.slice/crio-9401542b2696e2b500e9e49578ed8e461cb37cd449c240663250f4f8c87b1d1b WatchSource:0}: Error finding container 9401542b2696e2b500e9e49578ed8e461cb37cd449c240663250f4f8c87b1d1b: Status 404 returned error can't find the container with id 9401542b2696e2b500e9e49578ed8e461cb37cd449c240663250f4f8c87b1d1b
Jul 02 21:18:51.027812 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:51.027772    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5" event={"ID":"da774dac-c1e6-4f51-b69e-95a97f5bf0f1","Type":"ContainerStarted","Data":"2d7ed9c8b2bc121d2d85eb77fe8be6009c852eb2ab3b3ff0e3f61569ca56c8f6"}
Jul 02 21:18:51.029258 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:51.029186    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gmfs8" event={"ID":"732f8ea2-dc3e-426a-96a2-d8a7ba62d935","Type":"ContainerStarted","Data":"a0ae13fc72c6bfbce86e0f29f8dee3deeca3759a454352742ab7a291d87e1a3a"}
Jul 02 21:18:51.030550 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:51.030518    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8tkxq" event={"ID":"ce3cb284-71b0-4202-b141-7ad511895978","Type":"ContainerStarted","Data":"0528f98552dcfb919b04b056c0129fc90b2ce051c669c1676cbd63b3b3943aa3"}
Jul 02 21:18:51.032074 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:51.032044    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c56wq" event={"ID":"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a","Type":"ContainerStarted","Data":"f7a068adc6000b6af8d9855937fa9826407fb5f74b983059ef50fb540019c595"}
Jul 02 21:18:51.033135 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:51.033109    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xn9ss" event={"ID":"ea0b0920-86bd-418f-8daf-ec2078dc58d5","Type":"ContainerStarted","Data":"c8b22d9aafe9384436e8abf894f6542729380dbfaa64c10777bc99c9ecb10a9d"}
Jul 02 21:18:51.034297 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:51.034275    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8j5v7" event={"ID":"1c79d16b-84a9-4c32-af87-3e104b416604","Type":"ContainerStarted","Data":"9401542b2696e2b500e9e49578ed8e461cb37cd449c240663250f4f8c87b1d1b"}
Jul 02 21:18:51.035433 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:51.035414    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6fpnv" event={"ID":"75f700f7-60d3-4bf0-aef0-2052610d6cba","Type":"ContainerStarted","Data":"564963c9e972362d8504338191e513a7c9f420021d5b3335952f11ddf84cd1cf"}
Jul 02 21:18:51.039144 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:51.039123    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h77tf" event={"ID":"83d2714f-a9bc-4830-b379-871a817b6f32","Type":"ContainerStarted","Data":"98920c03f866c79fffa0da1d7a01ddda24fc310c98de538011b3fb908eb213cf"}
Jul 02 21:18:51.042476 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:51.042382    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcvqp" event={"ID":"12176f59-c61f-40f7-afce-c13616ba2f36","Type":"ContainerStarted","Data":"e7fcd116eeb3f64159987c4cd0c3090e87e45f57aca00704b46c308af1cb5441"}
Jul 02 21:18:51.577223 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:51.576319    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/006d61c8-eb34-4336-a3d3-82a7a4fc2069-metrics-certs\") pod \"network-metrics-daemon-s5xr4\" (UID: \"006d61c8-eb34-4336-a3d3-82a7a4fc2069\") " pod="openshift-multus/network-metrics-daemon-s5xr4"
Jul 02 21:18:51.577223 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:51.576401    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c214277-f44b-4f90-be58-8c8078403fc2-original-pull-secret\") pod \"global-pull-secret-syncer-5gfl7\" (UID: \"9c214277-f44b-4f90-be58-8c8078403fc2\") " pod="kube-system/global-pull-secret-syncer-5gfl7"
Jul 02 21:18:51.577223 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:51.576537    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 21:18:51.577223 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:51.576601    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c214277-f44b-4f90-be58-8c8078403fc2-original-pull-secret podName:9c214277-f44b-4f90-be58-8c8078403fc2 nodeName:}" failed. No retries permitted until 2026-07-02 21:18:53.576582449 +0000 UTC m=+6.019382873 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c214277-f44b-4f90-be58-8c8078403fc2-original-pull-secret") pod "global-pull-secret-syncer-5gfl7" (UID: "9c214277-f44b-4f90-be58-8c8078403fc2") : object "kube-system"/"original-pull-secret" not registered
Jul 02 21:18:51.577223 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:51.577081    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:18:51.577223 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:51.577139    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/006d61c8-eb34-4336-a3d3-82a7a4fc2069-metrics-certs podName:006d61c8-eb34-4336-a3d3-82a7a4fc2069 nodeName:}" failed. No retries permitted until 2026-07-02 21:18:53.577120866 +0000 UTC m=+6.019921285 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/006d61c8-eb34-4336-a3d3-82a7a4fc2069-metrics-certs") pod "network-metrics-daemon-s5xr4" (UID: "006d61c8-eb34-4336-a3d3-82a7a4fc2069") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:18:51.678024 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:51.677337    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thk5p\" (UniqueName: \"kubernetes.io/projected/45cb060e-3948-4b0e-b407-86ea9b6af872-kube-api-access-thk5p\") pod \"network-check-target-d495m\" (UID: \"45cb060e-3948-4b0e-b407-86ea9b6af872\") " pod="openshift-network-diagnostics/network-check-target-d495m"
Jul 02 21:18:51.678024 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:51.677544    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 21:18:51.678024 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:51.677567    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 21:18:51.678024 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:51.677580    2586 projected.go:196] Error preparing data for projected volume kube-api-access-thk5p for pod openshift-network-diagnostics/network-check-target-d495m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:18:51.678024 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:51.677640    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45cb060e-3948-4b0e-b407-86ea9b6af872-kube-api-access-thk5p podName:45cb060e-3948-4b0e-b407-86ea9b6af872 nodeName:}" failed. No retries permitted until 2026-07-02 21:18:53.677622643 +0000 UTC m=+6.120423062 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-thk5p" (UniqueName: "kubernetes.io/projected/45cb060e-3948-4b0e-b407-86ea9b6af872-kube-api-access-thk5p") pod "network-check-target-d495m" (UID: "45cb060e-3948-4b0e-b407-86ea9b6af872") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:18:51.927764 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:51.927597    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 21:13:49 +0000 UTC" deadline="2028-04-18 13:06:07.960905012 +0000 UTC"
Jul 02 21:18:51.927764 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:51.927675    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15735h47m16.033235272s"
Jul 02 21:18:52.018569 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:52.017803    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5xr4"
Jul 02 21:18:52.018569 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:52.017954    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5xr4" podUID="006d61c8-eb34-4336-a3d3-82a7a4fc2069"
Jul 02 21:18:52.018569 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:52.018406    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gfl7"
Jul 02 21:18:52.018569 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:52.018511    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gfl7" podUID="9c214277-f44b-4f90-be58-8c8078403fc2"
Jul 02 21:18:52.019174 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:52.019033    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d495m"
Jul 02 21:18:52.019174 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:52.019132    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d495m" podUID="45cb060e-3948-4b0e-b407-86ea9b6af872"
Jul 02 21:18:52.055479 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:52.055384    2586 generic.go:356] "Generic (PLEG): container finished" podID="3da92a8dc1fa784b367c822cbf212cbf" containerID="edf7193593037371dec38dc3a0cf0a15434e273aabb573970f858066aa495d4e" exitCode=0
Jul 02 21:18:52.055608 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:52.055479    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal" event={"ID":"3da92a8dc1fa784b367c822cbf212cbf","Type":"ContainerDied","Data":"edf7193593037371dec38dc3a0cf0a15434e273aabb573970f858066aa495d4e"}
Jul 02 21:18:52.067582 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:52.067499    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-124.ec2.internal" event={"ID":"25c1e22383a6ef722c235e04d1247f6b","Type":"ContainerStarted","Data":"3547568941f9c42fb72ddc0952019be1a6cd4d676c8e78285e45aa8a0d442e66"}
Jul 02 21:18:53.085142 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:53.085082    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal" event={"ID":"3da92a8dc1fa784b367c822cbf212cbf","Type":"ContainerStarted","Data":"93f7e14cd6355c8278f072aa51dacf9910982b3ab1f302df82f5154552fbfafd"}
Jul 02 21:18:53.101221 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:53.099841    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-124.ec2.internal" podStartSLOduration=4.099817042 podStartE2EDuration="4.099817042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:18:52.093335083 +0000 UTC m=+4.536135507" watchObservedRunningTime="2026-07-02 21:18:53.099817042 +0000 UTC m=+5.542617457"
Jul 02 21:18:53.101221 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:53.100676    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal" podStartSLOduration=4.100663024 podStartE2EDuration="4.100663024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:18:53.100111311 +0000 UTC m=+5.542911735" watchObservedRunningTime="2026-07-02 21:18:53.100663024 +0000 UTC m=+5.543463442"
Jul 02 21:18:53.591788 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:53.591751    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c214277-f44b-4f90-be58-8c8078403fc2-original-pull-secret\") pod \"global-pull-secret-syncer-5gfl7\" (UID: \"9c214277-f44b-4f90-be58-8c8078403fc2\") " pod="kube-system/global-pull-secret-syncer-5gfl7"
Jul 02 21:18:53.591988 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:53.591826    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/006d61c8-eb34-4336-a3d3-82a7a4fc2069-metrics-certs\") pod \"network-metrics-daemon-s5xr4\" (UID: \"006d61c8-eb34-4336-a3d3-82a7a4fc2069\") " pod="openshift-multus/network-metrics-daemon-s5xr4"
Jul 02 21:18:53.591988 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:53.591960    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:18:53.592111 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:53.592037    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/006d61c8-eb34-4336-a3d3-82a7a4fc2069-metrics-certs podName:006d61c8-eb34-4336-a3d3-82a7a4fc2069 nodeName:}" failed. No retries permitted until 2026-07-02 21:18:57.59201807 +0000 UTC m=+10.034818474 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/006d61c8-eb34-4336-a3d3-82a7a4fc2069-metrics-certs") pod "network-metrics-daemon-s5xr4" (UID: "006d61c8-eb34-4336-a3d3-82a7a4fc2069") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:18:53.592465 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:53.592439    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 21:18:53.592548 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:53.592498    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c214277-f44b-4f90-be58-8c8078403fc2-original-pull-secret podName:9c214277-f44b-4f90-be58-8c8078403fc2 nodeName:}" failed. No retries permitted until 2026-07-02 21:18:57.592482997 +0000 UTC m=+10.035283412 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c214277-f44b-4f90-be58-8c8078403fc2-original-pull-secret") pod "global-pull-secret-syncer-5gfl7" (UID: "9c214277-f44b-4f90-be58-8c8078403fc2") : object "kube-system"/"original-pull-secret" not registered
Jul 02 21:18:53.693002 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:53.692872    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thk5p\" (UniqueName: \"kubernetes.io/projected/45cb060e-3948-4b0e-b407-86ea9b6af872-kube-api-access-thk5p\") pod \"network-check-target-d495m\" (UID: \"45cb060e-3948-4b0e-b407-86ea9b6af872\") " pod="openshift-network-diagnostics/network-check-target-d495m"
Jul 02 21:18:53.698965 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:53.697850    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 21:18:53.698965 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:53.697901    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 21:18:53.698965 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:53.697926    2586 projected.go:196] Error preparing data for projected volume kube-api-access-thk5p for pod openshift-network-diagnostics/network-check-target-d495m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:18:53.698965 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:53.698434    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45cb060e-3948-4b0e-b407-86ea9b6af872-kube-api-access-thk5p podName:45cb060e-3948-4b0e-b407-86ea9b6af872 nodeName:}" failed. No retries permitted until 2026-07-02 21:18:57.698403778 +0000 UTC m=+10.141204183 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-thk5p" (UniqueName: "kubernetes.io/projected/45cb060e-3948-4b0e-b407-86ea9b6af872-kube-api-access-thk5p") pod "network-check-target-d495m" (UID: "45cb060e-3948-4b0e-b407-86ea9b6af872") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:18:54.019095 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:54.019060    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5xr4"
Jul 02 21:18:54.019302 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:54.019213    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5xr4" podUID="006d61c8-eb34-4336-a3d3-82a7a4fc2069"
Jul 02 21:18:54.019302 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:54.019285    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gfl7"
Jul 02 21:18:54.019429 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:54.019358    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gfl7" podUID="9c214277-f44b-4f90-be58-8c8078403fc2"
Jul 02 21:18:54.019429 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:54.019407    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d495m"
Jul 02 21:18:54.019534 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:54.019466    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d495m" podUID="45cb060e-3948-4b0e-b407-86ea9b6af872"
Jul 02 21:18:56.016778 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:56.016742    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gfl7"
Jul 02 21:18:56.017232 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:56.016888    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gfl7" podUID="9c214277-f44b-4f90-be58-8c8078403fc2"
Jul 02 21:18:56.017232 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:56.016742    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5xr4"
Jul 02 21:18:56.017232 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:56.017007    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5xr4" podUID="006d61c8-eb34-4336-a3d3-82a7a4fc2069"
Jul 02 21:18:56.017232 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:56.016742    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d495m"
Jul 02 21:18:56.017232 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:56.017120    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d495m" podUID="45cb060e-3948-4b0e-b407-86ea9b6af872"
Jul 02 21:18:57.629901 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:57.629859    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/006d61c8-eb34-4336-a3d3-82a7a4fc2069-metrics-certs\") pod \"network-metrics-daemon-s5xr4\" (UID: \"006d61c8-eb34-4336-a3d3-82a7a4fc2069\") " pod="openshift-multus/network-metrics-daemon-s5xr4"
Jul 02 21:18:57.630393 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:57.629949    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c214277-f44b-4f90-be58-8c8078403fc2-original-pull-secret\") pod \"global-pull-secret-syncer-5gfl7\" (UID: \"9c214277-f44b-4f90-be58-8c8078403fc2\") " pod="kube-system/global-pull-secret-syncer-5gfl7"
Jul 02 21:18:57.630393 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:57.630055    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:18:57.630393 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:57.630132    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/006d61c8-eb34-4336-a3d3-82a7a4fc2069-metrics-certs podName:006d61c8-eb34-4336-a3d3-82a7a4fc2069 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:05.63011179 +0000 UTC m=+18.072912196 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/006d61c8-eb34-4336-a3d3-82a7a4fc2069-metrics-certs") pod "network-metrics-daemon-s5xr4" (UID: "006d61c8-eb34-4336-a3d3-82a7a4fc2069") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:18:57.630574 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:57.630058    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 21:18:57.630574 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:57.630538    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c214277-f44b-4f90-be58-8c8078403fc2-original-pull-secret podName:9c214277-f44b-4f90-be58-8c8078403fc2 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:05.630520077 +0000 UTC m=+18.073320497 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c214277-f44b-4f90-be58-8c8078403fc2-original-pull-secret") pod "global-pull-secret-syncer-5gfl7" (UID: "9c214277-f44b-4f90-be58-8c8078403fc2") : object "kube-system"/"original-pull-secret" not registered
Jul 02 21:18:57.730836 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:57.730798    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thk5p\" (UniqueName: \"kubernetes.io/projected/45cb060e-3948-4b0e-b407-86ea9b6af872-kube-api-access-thk5p\") pod \"network-check-target-d495m\" (UID: \"45cb060e-3948-4b0e-b407-86ea9b6af872\") " pod="openshift-network-diagnostics/network-check-target-d495m"
Jul 02 21:18:57.731069 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:57.731030    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 21:18:57.731069 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:57.731065    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 21:18:57.731255 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:57.731081    2586 projected.go:196] Error preparing data for projected volume kube-api-access-thk5p for pod openshift-network-diagnostics/network-check-target-d495m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:18:57.731255 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:57.731158    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45cb060e-3948-4b0e-b407-86ea9b6af872-kube-api-access-thk5p podName:45cb060e-3948-4b0e-b407-86ea9b6af872 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:05.731135528 +0000 UTC m=+18.173935947 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-thk5p" (UniqueName: "kubernetes.io/projected/45cb060e-3948-4b0e-b407-86ea9b6af872-kube-api-access-thk5p") pod "network-check-target-d495m" (UID: "45cb060e-3948-4b0e-b407-86ea9b6af872") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:18:58.017940 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:58.017800    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5xr4"
Jul 02 21:18:58.018135 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:58.017958    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5xr4" podUID="006d61c8-eb34-4336-a3d3-82a7a4fc2069"
Jul 02 21:18:58.018364 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:58.018338    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gfl7"
Jul 02 21:18:58.018478 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:58.018456    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gfl7" podUID="9c214277-f44b-4f90-be58-8c8078403fc2"
Jul 02 21:18:58.018552 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:18:58.018533    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d495m"
Jul 02 21:18:58.018639 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:18:58.018620    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d495m" podUID="45cb060e-3948-4b0e-b407-86ea9b6af872"
Jul 02 21:19:00.016760 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:00.016574    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gfl7"
Jul 02 21:19:00.016760 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:00.016622    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d495m"
Jul 02 21:19:00.016760 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:00.016576    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5xr4"
Jul 02 21:19:00.016760 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:00.016728    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gfl7" podUID="9c214277-f44b-4f90-be58-8c8078403fc2"
Jul 02 21:19:00.017360 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:00.016849    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5xr4" podUID="006d61c8-eb34-4336-a3d3-82a7a4fc2069"
Jul 02 21:19:00.017360 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:00.016943    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d495m" podUID="45cb060e-3948-4b0e-b407-86ea9b6af872"
Jul 02 21:19:02.016779 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:02.016499    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5xr4"
Jul 02 21:19:02.017233 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:02.016499    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gfl7"
Jul 02 21:19:02.017233 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:02.016539    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d495m"
Jul 02 21:19:02.017233 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:02.016900    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5xr4" podUID="006d61c8-eb34-4336-a3d3-82a7a4fc2069"
Jul 02 21:19:02.017233 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:02.017014    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d495m" podUID="45cb060e-3948-4b0e-b407-86ea9b6af872"
Jul 02 21:19:02.017233 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:02.017111    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gfl7" podUID="9c214277-f44b-4f90-be58-8c8078403fc2"
Jul 02 21:19:04.017176 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:04.017141    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d495m"
Jul 02 21:19:04.017616 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:04.017137    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5xr4"
Jul 02 21:19:04.017616 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:04.017271    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d495m" podUID="45cb060e-3948-4b0e-b407-86ea9b6af872"
Jul 02 21:19:04.017616 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:04.017367    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5xr4" podUID="006d61c8-eb34-4336-a3d3-82a7a4fc2069"
Jul 02 21:19:04.017616 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:04.017141    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gfl7"
Jul 02 21:19:04.017616 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:04.017493    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gfl7" podUID="9c214277-f44b-4f90-be58-8c8078403fc2"
Jul 02 21:19:05.689899 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:05.689869    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c214277-f44b-4f90-be58-8c8078403fc2-original-pull-secret\") pod \"global-pull-secret-syncer-5gfl7\" (UID: \"9c214277-f44b-4f90-be58-8c8078403fc2\") " pod="kube-system/global-pull-secret-syncer-5gfl7"
Jul 02 21:19:05.690410 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:05.689932    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/006d61c8-eb34-4336-a3d3-82a7a4fc2069-metrics-certs\") pod \"network-metrics-daemon-s5xr4\" (UID: \"006d61c8-eb34-4336-a3d3-82a7a4fc2069\") " pod="openshift-multus/network-metrics-daemon-s5xr4"
Jul 02 21:19:05.690410 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:05.690038    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 21:19:05.690410 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:05.690061    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:19:05.690410 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:05.690125    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c214277-f44b-4f90-be58-8c8078403fc2-original-pull-secret podName:9c214277-f44b-4f90-be58-8c8078403fc2 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:21.690102595 +0000 UTC m=+34.132903015 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c214277-f44b-4f90-be58-8c8078403fc2-original-pull-secret") pod "global-pull-secret-syncer-5gfl7" (UID: "9c214277-f44b-4f90-be58-8c8078403fc2") : object "kube-system"/"original-pull-secret" not registered
Jul 02 21:19:05.690410 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:05.690150    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/006d61c8-eb34-4336-a3d3-82a7a4fc2069-metrics-certs podName:006d61c8-eb34-4336-a3d3-82a7a4fc2069 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:21.690134198 +0000 UTC m=+34.132934612 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/006d61c8-eb34-4336-a3d3-82a7a4fc2069-metrics-certs") pod "network-metrics-daemon-s5xr4" (UID: "006d61c8-eb34-4336-a3d3-82a7a4fc2069") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:19:05.790918 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:05.790883    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thk5p\" (UniqueName: \"kubernetes.io/projected/45cb060e-3948-4b0e-b407-86ea9b6af872-kube-api-access-thk5p\") pod \"network-check-target-d495m\" (UID: \"45cb060e-3948-4b0e-b407-86ea9b6af872\") " pod="openshift-network-diagnostics/network-check-target-d495m"
Jul 02 21:19:05.791082 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:05.791062    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 21:19:05.791133 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:05.791088    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 21:19:05.791133 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:05.791098    2586 projected.go:196] Error preparing data for projected volume kube-api-access-thk5p for pod openshift-network-diagnostics/network-check-target-d495m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:19:05.791229 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:05.791155    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45cb060e-3948-4b0e-b407-86ea9b6af872-kube-api-access-thk5p podName:45cb060e-3948-4b0e-b407-86ea9b6af872 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:21.791135858 +0000 UTC m=+34.233936262 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-thk5p" (UniqueName: "kubernetes.io/projected/45cb060e-3948-4b0e-b407-86ea9b6af872-kube-api-access-thk5p") pod "network-check-target-d495m" (UID: "45cb060e-3948-4b0e-b407-86ea9b6af872") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:19:06.017314 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:06.017227    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gfl7"
Jul 02 21:19:06.017314 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:06.017264    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d495m"
Jul 02 21:19:06.017512 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:06.017228    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5xr4"
Jul 02 21:19:06.017512 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:06.017373    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gfl7" podUID="9c214277-f44b-4f90-be58-8c8078403fc2"
Jul 02 21:19:06.017512 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:06.017471    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5xr4" podUID="006d61c8-eb34-4336-a3d3-82a7a4fc2069"
Jul 02 21:19:06.017642 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:06.017556    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d495m" podUID="45cb060e-3948-4b0e-b407-86ea9b6af872"
Jul 02 21:19:08.017922 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:08.017882    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5xr4"
Jul 02 21:19:08.018392 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:08.017977    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gfl7"
Jul 02 21:19:08.018392 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:08.018000    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5xr4" podUID="006d61c8-eb34-4336-a3d3-82a7a4fc2069"
Jul 02 21:19:08.018392 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:08.018086    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gfl7" podUID="9c214277-f44b-4f90-be58-8c8078403fc2"
Jul 02 21:19:08.018392 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:08.018096    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d495m"
Jul 02 21:19:08.018392 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:08.018174    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d495m" podUID="45cb060e-3948-4b0e-b407-86ea9b6af872"
Jul 02 21:19:10.017710 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:10.017458    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gfl7"
Jul 02 21:19:10.018394 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:10.017459    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5xr4"
Jul 02 21:19:10.018394 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:10.017817    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gfl7" podUID="9c214277-f44b-4f90-be58-8c8078403fc2"
Jul 02 21:19:10.018394 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:10.017922    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5xr4" podUID="006d61c8-eb34-4336-a3d3-82a7a4fc2069"
Jul 02 21:19:10.018394 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:10.017459    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d495m"
Jul 02 21:19:10.018394 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:10.018058    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d495m" podUID="45cb060e-3948-4b0e-b407-86ea9b6af872"
Jul 02 21:19:10.118595 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:10.118564    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5" event={"ID":"da774dac-c1e6-4f51-b69e-95a97f5bf0f1","Type":"ContainerStarted","Data":"37d71d973b360162c3035cdda2df4d1d5e461d8d8303f7de9098631b9f1607b3"}
Jul 02 21:19:10.118774 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:10.118665    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5" event={"ID":"da774dac-c1e6-4f51-b69e-95a97f5bf0f1","Type":"ContainerStarted","Data":"0f63e9a174e417f9c7e3178d28b0c5bc58b9d671d4f7d7063ed51d7be68be8ed"}
Jul 02 21:19:10.118774 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:10.118716    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5" event={"ID":"da774dac-c1e6-4f51-b69e-95a97f5bf0f1","Type":"ContainerStarted","Data":"212dd3e8cc6bd05951fa47bd16a9d00be2311cc1496fbb988a3784565b366c63"}
Jul 02 21:19:10.118774 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:10.118759    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5" event={"ID":"da774dac-c1e6-4f51-b69e-95a97f5bf0f1","Type":"ContainerStarted","Data":"df864f65ac811ca50358aea81af8b100b71d747e4dcf7ec11c24fd1412d51fcc"}
Jul 02 21:19:10.120155 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:10.120126    2586 generic.go:356] "Generic (PLEG): container finished" podID="ce3cb284-71b0-4202-b141-7ad511895978" containerID="4e35784e70e58b98308f5e54d085e32746000c0df4fedd646fb644465fc6f09e" exitCode=0
Jul 02 21:19:10.120270 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:10.120223    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8tkxq" event={"ID":"ce3cb284-71b0-4202-b141-7ad511895978","Type":"ContainerDied","Data":"4e35784e70e58b98308f5e54d085e32746000c0df4fedd646fb644465fc6f09e"}
Jul 02 21:19:10.123807 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:10.123744    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c56wq" event={"ID":"e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a","Type":"ContainerStarted","Data":"8edfe3e9049772b21c7ee393bedf5a1b84b10fe847c4aac7d7690146944c28ac"}
Jul 02 21:19:10.125542 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:10.125490    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xn9ss" event={"ID":"ea0b0920-86bd-418f-8daf-ec2078dc58d5","Type":"ContainerStarted","Data":"c24e9e97c40e386a62dacdfddaee678d8cac92d6f9ebe2d36705ae7a23b82c2c"}
Jul 02 21:19:10.127215 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:10.127180    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8j5v7" event={"ID":"1c79d16b-84a9-4c32-af87-3e104b416604","Type":"ContainerStarted","Data":"9054d8bb7d239bc607b6f7f33802723e0df0e3810cd1f7c92dd232b61a1ab797"}
Jul 02 21:19:10.128608 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:10.128586    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6fpnv" event={"ID":"75f700f7-60d3-4bf0-aef0-2052610d6cba","Type":"ContainerStarted","Data":"b63afe0a818dedbbfa4890862cb956b35388beb6b97a92f15e6a6a3615d2ec8f"}
Jul 02 21:19:10.129881 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:10.129863    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h77tf" event={"ID":"83d2714f-a9bc-4830-b379-871a817b6f32","Type":"ContainerStarted","Data":"84c973778ae72e0deb6954b479002c53a6d29489d3acd3af2cfc9a7418bb1b35"}
Jul 02 21:19:10.131239 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:10.131218    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcvqp" event={"ID":"12176f59-c61f-40f7-afce-c13616ba2f36","Type":"ContainerStarted","Data":"e1e557f974a08ab070128e86c8feaf81368b64a30aa4e4ebdeb27897ac021665"}
Jul 02 21:19:10.163124 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:10.163080    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-c56wq" podStartSLOduration=3.535862004 podStartE2EDuration="22.163066217s" totalImagesPullingTime="18.627204213s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:18:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:10.162573036 +0000 UTC m=+22.605373458" watchObservedRunningTime="2026-07-02 21:19:10.163066217 +0000 UTC m=+22.605866638"
Jul 02 21:19:10.192954 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:10.192910    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xn9ss" podStartSLOduration=8.113962685 podStartE2EDuration="22.192895662s" totalImagesPullingTime="14.078932977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:18:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:10.177369632 +0000 UTC m=+22.620170056" watchObservedRunningTime="2026-07-02 21:19:10.192895662 +0000 UTC m=+22.635696083"
Jul 02 21:19:10.193306 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:10.193277    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-h77tf" podStartSLOduration=3.620350181 podStartE2EDuration="22.193271254s" totalImagesPullingTime="18.572921073s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:18:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:10.192623325 +0000 UTC m=+22.635423748" watchObservedRunningTime="2026-07-02 21:19:10.193271254 +0000 UTC m=+22.636071675"
Jul 02 21:19:10.208045 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:10.207987    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6fpnv" podStartSLOduration=3.636881766 podStartE2EDuration="22.20797059s" totalImagesPullingTime="18.571088824s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:18:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:10.20745588 +0000 UTC m=+22.650256315" watchObservedRunningTime="2026-07-02 21:19:10.20797059 +0000 UTC m=+22.650770991"
Jul 02 21:19:10.227853 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:10.227796    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8j5v7" podStartSLOduration=3.648865387 podStartE2EDuration="22.227781736s" totalImagesPullingTime="18.578916349s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:18:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:10.227604126 +0000 UTC m=+22.670404549" watchObservedRunningTime="2026-07-02 21:19:10.227781736 +0000 UTC m=+22.670582157"
Jul 02 21:19:11.072854 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:11.072700    2586 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 02 21:19:11.137688 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:11.137635    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5" event={"ID":"da774dac-c1e6-4f51-b69e-95a97f5bf0f1","Type":"ContainerStarted","Data":"360276dd18e1d79a488e51a9b77f3da881c55a6034fc555ef0af3e882a841c6e"}
Jul 02 21:19:11.137873 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:11.137784    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5" event={"ID":"da774dac-c1e6-4f51-b69e-95a97f5bf0f1","Type":"ContainerStarted","Data":"8ab834911d9e77eb2ffaa1a41e9a5606af8e72cc074882f0d1c5a7e5a6e5947f"}
Jul 02 21:19:11.139304 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:11.139278    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gmfs8" event={"ID":"732f8ea2-dc3e-426a-96a2-d8a7ba62d935","Type":"ContainerStarted","Data":"3261531fb63490b9b4de9c8b9159a5a6ffd425c6a8adb34f49f8ef29f2c90fd8"}
Jul 02 21:19:11.141337 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:11.141289    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcvqp" event={"ID":"12176f59-c61f-40f7-afce-c13616ba2f36","Type":"ContainerStarted","Data":"ceeef0f90dd403dd8bce83c48efd5be37bcae1498baf0182c10db4b28329f6b8"}
Jul 02 21:19:11.156126 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:11.156080    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gmfs8" podStartSLOduration=4.588778342 podStartE2EDuration="23.156062154s" totalImagesPullingTime="18.567283812s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:18:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:11.155609363 +0000 UTC m=+23.598409785" watchObservedRunningTime="2026-07-02 21:19:11.156062154 +0000 UTC m=+23.598862577"
Jul 02 21:19:11.930711 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:11.930570    2586 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-02T21:19:11.072851601Z","UUID":"0e058025-b934-4fe7-96de-2aeebad013f5","Handler":null,"Name":"","Endpoint":""}
Jul 02 21:19:11.932443 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:11.932416    2586 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 02 21:19:11.932598 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:11.932463    2586 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 02 21:19:12.017374 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:12.017330    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5xr4"
Jul 02 21:19:12.017374 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:12.017372    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d495m"
Jul 02 21:19:12.017623 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:12.017341    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gfl7"
Jul 02 21:19:12.017623 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:12.017477    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5xr4" podUID="006d61c8-eb34-4336-a3d3-82a7a4fc2069"
Jul 02 21:19:12.017623 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:12.017565    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gfl7" podUID="9c214277-f44b-4f90-be58-8c8078403fc2"
Jul 02 21:19:12.017841 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:12.017638    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d495m" podUID="45cb060e-3948-4b0e-b407-86ea9b6af872"
Jul 02 21:19:12.992515 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:12.992477    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xn9ss"
Jul 02 21:19:12.993318 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:12.993299    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xn9ss"
Jul 02 21:19:13.150446 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:13.150407    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5" event={"ID":"da774dac-c1e6-4f51-b69e-95a97f5bf0f1","Type":"ContainerStarted","Data":"a047ff9cee2711ccc86b91380b1e9d54a46af44530e5eb2037d9262d8d8a7e32"}
Jul 02 21:19:13.152705 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:13.152669    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcvqp" event={"ID":"12176f59-c61f-40f7-afce-c13616ba2f36","Type":"ContainerStarted","Data":"b75363d1a5f5579353216dbc20914f06462ff171bdfb1548bb6efe52a148425f"}
Jul 02 21:19:13.152975 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:13.152941    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xn9ss"
Jul 02 21:19:13.153423 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:13.153405    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xn9ss"
Jul 02 21:19:13.195345 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:13.195289    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcvqp" podStartSLOduration=4.419409882 podStartE2EDuration="25.195272197s" totalImagesPullingTime="20.775862315s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:18:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:13.17377576 +0000 UTC m=+25.616576203" watchObservedRunningTime="2026-07-02 21:19:13.195272197 +0000 UTC m=+25.638072680"
Jul 02 21:19:14.016522 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:14.016487    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5xr4"
Jul 02 21:19:14.017116 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:14.016487    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d495m"
Jul 02 21:19:14.017116 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:14.016616    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5xr4" podUID="006d61c8-eb34-4336-a3d3-82a7a4fc2069"
Jul 02 21:19:14.017116 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:14.016685    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d495m" podUID="45cb060e-3948-4b0e-b407-86ea9b6af872"
Jul 02 21:19:14.017116 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:14.016488    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gfl7"
Jul 02 21:19:14.017116 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:14.016775    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gfl7" podUID="9c214277-f44b-4f90-be58-8c8078403fc2"
Jul 02 21:19:16.017053 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:16.016855    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5xr4"
Jul 02 21:19:16.017483 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:16.016856    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gfl7"
Jul 02 21:19:16.017483 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:16.017141    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5xr4" podUID="006d61c8-eb34-4336-a3d3-82a7a4fc2069"
Jul 02 21:19:16.017483 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:16.016870    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d495m"
Jul 02 21:19:16.017483 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:16.017233    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gfl7" podUID="9c214277-f44b-4f90-be58-8c8078403fc2"
Jul 02 21:19:16.017483 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:16.017282    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d495m" podUID="45cb060e-3948-4b0e-b407-86ea9b6af872"
Jul 02 21:19:16.163090 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:16.163003    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5" event={"ID":"da774dac-c1e6-4f51-b69e-95a97f5bf0f1","Type":"ContainerStarted","Data":"583b9906ab080dc5ff71098043f4b087e4d756a722ce56574d444c963509b5bc"}
Jul 02 21:19:16.163276 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:16.163206    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:19:16.164946 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:16.164920    2586 generic.go:356] "Generic (PLEG): container finished" podID="ce3cb284-71b0-4202-b141-7ad511895978" containerID="d0fc1f4a31cac33620ddfcadd66a8e71f9eb501faec0dd219e352517c1b65630" exitCode=0
Jul 02 21:19:16.165059 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:16.164949    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8tkxq" event={"ID":"ce3cb284-71b0-4202-b141-7ad511895978","Type":"ContainerDied","Data":"d0fc1f4a31cac33620ddfcadd66a8e71f9eb501faec0dd219e352517c1b65630"}
Jul 02 21:19:16.179750 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:16.179729    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:19:16.197847 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:16.197801    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5" podStartSLOduration=9.521251427 podStartE2EDuration="28.197788524s" totalImagesPullingTime="18.676537097s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:18:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:16.196015126 +0000 UTC m=+28.638815551" watchObservedRunningTime="2026-07-02 21:19:16.197788524 +0000 UTC m=+28.640588945"
Jul 02 21:19:17.169015 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:17.168771    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8tkxq" event={"ID":"ce3cb284-71b0-4202-b141-7ad511895978","Type":"ContainerStarted","Data":"55e0f10beaba728e439ca17a8923f2fd0d20dc0344b289d3252cd2bf577a485d"}
Jul 02 21:19:17.169488 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:17.169469    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:19:17.169540 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:17.169532    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:19:17.183330 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:17.183307    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:19:17.480535 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:17.480507    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s5xr4"]
Jul 02 21:19:17.480728 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:17.480679    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5xr4"
Jul 02 21:19:17.480793 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:17.480775    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5xr4" podUID="006d61c8-eb34-4336-a3d3-82a7a4fc2069"
Jul 02 21:19:17.481113 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:17.481095    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-d495m"]
Jul 02 21:19:17.481186 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:17.481176    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d495m"
Jul 02 21:19:17.481260 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:17.481244    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d495m" podUID="45cb060e-3948-4b0e-b407-86ea9b6af872"
Jul 02 21:19:17.484921 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:17.484896    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5gfl7"]
Jul 02 21:19:17.485049 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:17.484989    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gfl7"
Jul 02 21:19:17.485096 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:17.485070    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gfl7" podUID="9c214277-f44b-4f90-be58-8c8078403fc2"
Jul 02 21:19:18.173342 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:18.173308    2586 generic.go:356] "Generic (PLEG): container finished" podID="ce3cb284-71b0-4202-b141-7ad511895978" containerID="55e0f10beaba728e439ca17a8923f2fd0d20dc0344b289d3252cd2bf577a485d" exitCode=0
Jul 02 21:19:18.174382 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:18.173395    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8tkxq" event={"ID":"ce3cb284-71b0-4202-b141-7ad511895978","Type":"ContainerDied","Data":"55e0f10beaba728e439ca17a8923f2fd0d20dc0344b289d3252cd2bf577a485d"}
Jul 02 21:19:19.017406 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:19.017048    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5xr4"
Jul 02 21:19:19.017406 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:19.017188    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5xr4" podUID="006d61c8-eb34-4336-a3d3-82a7a4fc2069"
Jul 02 21:19:19.017406 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:19.017340    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d495m"
Jul 02 21:19:19.017733 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:19.017538    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d495m" podUID="45cb060e-3948-4b0e-b407-86ea9b6af872"
Jul 02 21:19:19.017733 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:19.017599    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gfl7"
Jul 02 21:19:19.017838 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:19.017795    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gfl7" podUID="9c214277-f44b-4f90-be58-8c8078403fc2"
Jul 02 21:19:19.177362 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:19.177330    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8tkxq" event={"ID":"ce3cb284-71b0-4202-b141-7ad511895978","Type":"ContainerStarted","Data":"98d88afc460632e19e69d28f96dd184bd0e8b8ab85f76080dc0e36819e1df3e1"}
Jul 02 21:19:20.181716 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:20.181679    2586 generic.go:356] "Generic (PLEG): container finished" podID="ce3cb284-71b0-4202-b141-7ad511895978" containerID="98d88afc460632e19e69d28f96dd184bd0e8b8ab85f76080dc0e36819e1df3e1" exitCode=0
Jul 02 21:19:20.182089 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:20.181725    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8tkxq" event={"ID":"ce3cb284-71b0-4202-b141-7ad511895978","Type":"ContainerDied","Data":"98d88afc460632e19e69d28f96dd184bd0e8b8ab85f76080dc0e36819e1df3e1"}
Jul 02 21:19:21.016743 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:21.016511    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5xr4"
Jul 02 21:19:21.016945 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:21.016538    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gfl7"
Jul 02 21:19:21.016945 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:21.016870    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5xr4" podUID="006d61c8-eb34-4336-a3d3-82a7a4fc2069"
Jul 02 21:19:21.016945 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:21.016541    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d495m"
Jul 02 21:19:21.016945 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:21.016919    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gfl7" podUID="9c214277-f44b-4f90-be58-8c8078403fc2"
Jul 02 21:19:21.017179 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:21.017037    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d495m" podUID="45cb060e-3948-4b0e-b407-86ea9b6af872"
Jul 02 21:19:21.706140 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:21.706087    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c214277-f44b-4f90-be58-8c8078403fc2-original-pull-secret\") pod \"global-pull-secret-syncer-5gfl7\" (UID: \"9c214277-f44b-4f90-be58-8c8078403fc2\") " pod="kube-system/global-pull-secret-syncer-5gfl7"
Jul 02 21:19:21.706626 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:21.706169    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/006d61c8-eb34-4336-a3d3-82a7a4fc2069-metrics-certs\") pod \"network-metrics-daemon-s5xr4\" (UID: \"006d61c8-eb34-4336-a3d3-82a7a4fc2069\") " pod="openshift-multus/network-metrics-daemon-s5xr4"
Jul 02 21:19:21.706626 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:21.706244    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 21:19:21.706626 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:21.706313    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c214277-f44b-4f90-be58-8c8078403fc2-original-pull-secret podName:9c214277-f44b-4f90-be58-8c8078403fc2 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:53.706293406 +0000 UTC m=+66.149093823 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c214277-f44b-4f90-be58-8c8078403fc2-original-pull-secret") pod "global-pull-secret-syncer-5gfl7" (UID: "9c214277-f44b-4f90-be58-8c8078403fc2") : object "kube-system"/"original-pull-secret" not registered
Jul 02 21:19:21.706626 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:21.706324    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:19:21.706626 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:21.706393    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/006d61c8-eb34-4336-a3d3-82a7a4fc2069-metrics-certs podName:006d61c8-eb34-4336-a3d3-82a7a4fc2069 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:53.706378049 +0000 UTC m=+66.149178452 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/006d61c8-eb34-4336-a3d3-82a7a4fc2069-metrics-certs") pod "network-metrics-daemon-s5xr4" (UID: "006d61c8-eb34-4336-a3d3-82a7a4fc2069") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:19:21.807135 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:21.807095    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thk5p\" (UniqueName: \"kubernetes.io/projected/45cb060e-3948-4b0e-b407-86ea9b6af872-kube-api-access-thk5p\") pod \"network-check-target-d495m\" (UID: \"45cb060e-3948-4b0e-b407-86ea9b6af872\") " pod="openshift-network-diagnostics/network-check-target-d495m"
Jul 02 21:19:21.807340 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:21.807300    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 21:19:21.807340 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:21.807325    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 21:19:21.807340 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:21.807338    2586 projected.go:196] Error preparing data for projected volume kube-api-access-thk5p for pod openshift-network-diagnostics/network-check-target-d495m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:19:21.807487 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:21.807402    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45cb060e-3948-4b0e-b407-86ea9b6af872-kube-api-access-thk5p podName:45cb060e-3948-4b0e-b407-86ea9b6af872 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:53.8073877 +0000 UTC m=+66.250188100 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-thk5p" (UniqueName: "kubernetes.io/projected/45cb060e-3948-4b0e-b407-86ea9b6af872-kube-api-access-thk5p") pod "network-check-target-d495m" (UID: "45cb060e-3948-4b0e-b407-86ea9b6af872") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:19:22.338324 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:22.338241    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeReady"
Jul 02 21:19:22.338483 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:22.338390    2586 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 02 21:19:22.399219 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:22.399183    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-w5w24"]
Jul 02 21:19:22.402974 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:22.402949    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mvrrr"]
Jul 02 21:19:22.403132 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:22.403047    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w5w24"
Jul 02 21:19:22.406177 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:22.406124    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 02 21:19:22.406177 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:22.406168    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sklqf\""
Jul 02 21:19:22.406368 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:22.406177    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 02 21:19:22.406891 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:22.406872    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mvrrr"
Jul 02 21:19:22.409778 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:22.409708    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 02 21:19:22.409887 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:22.409779    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-z7htl\""
Jul 02 21:19:22.409887 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:22.409785    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 02 21:19:22.410024 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:22.409913    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 02 21:19:22.415308 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:22.415286    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mvrrr"]
Jul 02 21:19:22.417310 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:22.417263    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w5w24"]
Jul 02 21:19:22.512987 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:22.512942    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxx8k\" (UniqueName: \"kubernetes.io/projected/d2c34410-8c0a-4d13-8864-575ba506997d-kube-api-access-dxx8k\") pod \"dns-default-w5w24\" (UID: \"d2c34410-8c0a-4d13-8864-575ba506997d\") " pod="openshift-dns/dns-default-w5w24"
Jul 02 21:19:22.513175 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:22.513007    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d2c34410-8c0a-4d13-8864-575ba506997d-config-volume\") pod \"dns-default-w5w24\" (UID: \"d2c34410-8c0a-4d13-8864-575ba506997d\") " pod="openshift-dns/dns-default-w5w24"
Jul 02 21:19:22.513175 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:22.513031    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvdm7\" (UniqueName: \"kubernetes.io/projected/2fc74b27-fbce-4841-b23e-100f5fffdb1b-kube-api-access-pvdm7\") pod \"ingress-canary-mvrrr\" (UID: \"2fc74b27-fbce-4841-b23e-100f5fffdb1b\") " pod="openshift-ingress-canary/ingress-canary-mvrrr"
Jul 02 21:19:22.513175 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:22.513052    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2c34410-8c0a-4d13-8864-575ba506997d-tmp-dir\") pod \"dns-default-w5w24\" (UID: \"d2c34410-8c0a-4d13-8864-575ba506997d\") " pod="openshift-dns/dns-default-w5w24"
Jul 02 21:19:22.513175 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:22.513069    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2fc74b27-fbce-4841-b23e-100f5fffdb1b-cert\") pod \"ingress-canary-mvrrr\" (UID: \"2fc74b27-fbce-4841-b23e-100f5fffdb1b\") " pod="openshift-ingress-canary/ingress-canary-mvrrr"
Jul 02 21:19:22.513175 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:22.513090    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2c34410-8c0a-4d13-8864-575ba506997d-metrics-tls\") pod \"dns-default-w5w24\" (UID: \"d2c34410-8c0a-4d13-8864-575ba506997d\") " pod="openshift-dns/dns-default-w5w24"
Jul 02 21:19:22.614313 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:22.614228    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d2c34410-8c0a-4d13-8864-575ba506997d-config-volume\") pod \"dns-default-w5w24\" (UID: \"d2c34410-8c0a-4d13-8864-575ba506997d\") " pod="openshift-dns/dns-default-w5w24"
Jul 02 21:19:22.614313 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:22.614264    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvdm7\" (UniqueName: \"kubernetes.io/projected/2fc74b27-fbce-4841-b23e-100f5fffdb1b-kube-api-access-pvdm7\") pod \"ingress-canary-mvrrr\" (UID: \"2fc74b27-fbce-4841-b23e-100f5fffdb1b\") " pod="openshift-ingress-canary/ingress-canary-mvrrr"
Jul 02 21:19:22.614313 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:22.614288    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2c34410-8c0a-4d13-8864-575ba506997d-tmp-dir\") pod \"dns-default-w5w24\" (UID: \"d2c34410-8c0a-4d13-8864-575ba506997d\") " pod="openshift-dns/dns-default-w5w24"
Jul 02 21:19:22.614576 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:22.614317    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2fc74b27-fbce-4841-b23e-100f5fffdb1b-cert\") pod \"ingress-canary-mvrrr\" (UID: \"2fc74b27-fbce-4841-b23e-100f5fffdb1b\") " pod="openshift-ingress-canary/ingress-canary-mvrrr"
Jul 02 21:19:22.614576 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:22.614342    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2c34410-8c0a-4d13-8864-575ba506997d-metrics-tls\") pod \"dns-default-w5w24\" (UID: \"d2c34410-8c0a-4d13-8864-575ba506997d\") " pod="openshift-dns/dns-default-w5w24"
Jul 02 21:19:22.614576 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:22.614423    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxx8k\" (UniqueName: \"kubernetes.io/projected/d2c34410-8c0a-4d13-8864-575ba506997d-kube-api-access-dxx8k\") pod \"dns-default-w5w24\" (UID: \"d2c34410-8c0a-4d13-8864-575ba506997d\") " pod="openshift-dns/dns-default-w5w24"
Jul 02 21:19:22.614845 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:22.614816    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d2c34410-8c0a-4d13-8864-575ba506997d-config-volume\") pod \"dns-default-w5w24\" (UID: \"d2c34410-8c0a-4d13-8864-575ba506997d\") " pod="openshift-dns/dns-default-w5w24"
Jul 02 21:19:22.614984 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:22.614848    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2c34410-8c0a-4d13-8864-575ba506997d-tmp-dir\") pod \"dns-default-w5w24\" (UID: \"d2c34410-8c0a-4d13-8864-575ba506997d\") " pod="openshift-dns/dns-default-w5w24"
Jul 02 21:19:22.614984 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:22.614928    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 21:19:22.615072 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:22.614990    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2c34410-8c0a-4d13-8864-575ba506997d-metrics-tls podName:d2c34410-8c0a-4d13-8864-575ba506997d nodeName:}" failed. No retries permitted until 2026-07-02 21:19:23.114971051 +0000 UTC m=+35.557771452 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d2c34410-8c0a-4d13-8864-575ba506997d-metrics-tls") pod "dns-default-w5w24" (UID: "d2c34410-8c0a-4d13-8864-575ba506997d") : secret "dns-default-metrics-tls" not found
Jul 02 21:19:22.619494 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:22.619461    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2fc74b27-fbce-4841-b23e-100f5fffdb1b-cert\") pod \"ingress-canary-mvrrr\" (UID: \"2fc74b27-fbce-4841-b23e-100f5fffdb1b\") " pod="openshift-ingress-canary/ingress-canary-mvrrr"
Jul 02 21:19:22.624018 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:22.623987    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxx8k\" (UniqueName: \"kubernetes.io/projected/d2c34410-8c0a-4d13-8864-575ba506997d-kube-api-access-dxx8k\") pod \"dns-default-w5w24\" (UID: \"d2c34410-8c0a-4d13-8864-575ba506997d\") " pod="openshift-dns/dns-default-w5w24"
Jul 02 21:19:22.624178 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:22.624064    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvdm7\" (UniqueName: \"kubernetes.io/projected/2fc74b27-fbce-4841-b23e-100f5fffdb1b-kube-api-access-pvdm7\") pod \"ingress-canary-mvrrr\" (UID: \"2fc74b27-fbce-4841-b23e-100f5fffdb1b\") " pod="openshift-ingress-canary/ingress-canary-mvrrr"
Jul 02 21:19:22.724196 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:22.724159    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mvrrr"
Jul 02 21:19:22.878710 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:22.878569    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mvrrr"]
Jul 02 21:19:22.883151 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:19:22.883124    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc74b27_fbce_4841_b23e_100f5fffdb1b.slice/crio-04f71d3b922ba509c4dcfda78f0c977fad8ba686cd26f076e47d7a4b9aa56a3c WatchSource:0}: Error finding container 04f71d3b922ba509c4dcfda78f0c977fad8ba686cd26f076e47d7a4b9aa56a3c: Status 404 returned error can't find the container with id 04f71d3b922ba509c4dcfda78f0c977fad8ba686cd26f076e47d7a4b9aa56a3c
Jul 02 21:19:23.016894 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:23.016859    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gfl7"
Jul 02 21:19:23.017088 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:23.016909    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5xr4"
Jul 02 21:19:23.017088 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:23.016983    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d495m"
Jul 02 21:19:23.021285 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:23.021258    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6k252\""
Jul 02 21:19:23.021285 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:23.021274    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 21:19:23.021285 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:23.021257    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 21:19:23.021563 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:23.021281    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 21:19:23.021563 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:23.021281    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bjnbk\""
Jul 02 21:19:23.021563 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:23.021258    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 21:19:23.118242 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:23.118209    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2c34410-8c0a-4d13-8864-575ba506997d-metrics-tls\") pod \"dns-default-w5w24\" (UID: \"d2c34410-8c0a-4d13-8864-575ba506997d\") " pod="openshift-dns/dns-default-w5w24"
Jul 02 21:19:23.118412 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:23.118352    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 21:19:23.118503 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:23.118414    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2c34410-8c0a-4d13-8864-575ba506997d-metrics-tls podName:d2c34410-8c0a-4d13-8864-575ba506997d nodeName:}" failed. No retries permitted until 2026-07-02 21:19:24.118396298 +0000 UTC m=+36.561196698 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d2c34410-8c0a-4d13-8864-575ba506997d-metrics-tls") pod "dns-default-w5w24" (UID: "d2c34410-8c0a-4d13-8864-575ba506997d") : secret "dns-default-metrics-tls" not found
Jul 02 21:19:23.190018 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:23.189977    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mvrrr" event={"ID":"2fc74b27-fbce-4841-b23e-100f5fffdb1b","Type":"ContainerStarted","Data":"04f71d3b922ba509c4dcfda78f0c977fad8ba686cd26f076e47d7a4b9aa56a3c"}
Jul 02 21:19:24.127179 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:24.127127    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2c34410-8c0a-4d13-8864-575ba506997d-metrics-tls\") pod \"dns-default-w5w24\" (UID: \"d2c34410-8c0a-4d13-8864-575ba506997d\") " pod="openshift-dns/dns-default-w5w24"
Jul 02 21:19:24.127627 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:24.127324    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 21:19:24.127627 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:24.127414    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2c34410-8c0a-4d13-8864-575ba506997d-metrics-tls podName:d2c34410-8c0a-4d13-8864-575ba506997d nodeName:}" failed. No retries permitted until 2026-07-02 21:19:26.127394742 +0000 UTC m=+38.570195142 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d2c34410-8c0a-4d13-8864-575ba506997d-metrics-tls") pod "dns-default-w5w24" (UID: "d2c34410-8c0a-4d13-8864-575ba506997d") : secret "dns-default-metrics-tls" not found
Jul 02 21:19:25.198753 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:25.198494    2586 generic.go:356] "Generic (PLEG): container finished" podID="ce3cb284-71b0-4202-b141-7ad511895978" containerID="bfc8a22f4e0ce67e8c085b5e80e5795512aa0002d6a7ae14adad6680d2d45ca0" exitCode=0
Jul 02 21:19:25.198753 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:25.198585    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8tkxq" event={"ID":"ce3cb284-71b0-4202-b141-7ad511895978","Type":"ContainerDied","Data":"bfc8a22f4e0ce67e8c085b5e80e5795512aa0002d6a7ae14adad6680d2d45ca0"}
Jul 02 21:19:26.144921 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:26.144885    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2c34410-8c0a-4d13-8864-575ba506997d-metrics-tls\") pod \"dns-default-w5w24\" (UID: \"d2c34410-8c0a-4d13-8864-575ba506997d\") " pod="openshift-dns/dns-default-w5w24"
Jul 02 21:19:26.145123 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:26.145017    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 21:19:26.145123 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:26.145099    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2c34410-8c0a-4d13-8864-575ba506997d-metrics-tls podName:d2c34410-8c0a-4d13-8864-575ba506997d nodeName:}" failed. No retries permitted until 2026-07-02 21:19:30.145079166 +0000 UTC m=+42.587879582 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d2c34410-8c0a-4d13-8864-575ba506997d-metrics-tls") pod "dns-default-w5w24" (UID: "d2c34410-8c0a-4d13-8864-575ba506997d") : secret "dns-default-metrics-tls" not found
Jul 02 21:19:26.205309 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:26.205269    2586 generic.go:356] "Generic (PLEG): container finished" podID="ce3cb284-71b0-4202-b141-7ad511895978" containerID="3816f4d6cc79be0f79c84b972deb34cddf86413bebe3741e86771286f45a22ef" exitCode=0
Jul 02 21:19:26.205950 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:26.205315    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8tkxq" event={"ID":"ce3cb284-71b0-4202-b141-7ad511895978","Type":"ContainerDied","Data":"3816f4d6cc79be0f79c84b972deb34cddf86413bebe3741e86771286f45a22ef"}
Jul 02 21:19:27.209386 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:27.209349    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mvrrr" event={"ID":"2fc74b27-fbce-4841-b23e-100f5fffdb1b","Type":"ContainerStarted","Data":"8e505c3ca3ab0c008bfc2ffd43f99dad0f3577dc2e3ace6cf382afd5bf54844e"}
Jul 02 21:19:27.212775 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:27.212743    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8tkxq" event={"ID":"ce3cb284-71b0-4202-b141-7ad511895978","Type":"ContainerStarted","Data":"38f464968fd1be8093bde6113f52c0978f1e166f51e3d7c8de6a2fe24e8623a6"}
Jul 02 21:19:27.227212 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:27.227161    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mvrrr" podStartSLOduration=1.262356168 podStartE2EDuration="5.227144034s" totalImagesPullingTime="3.964787866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:19:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:27.226854171 +0000 UTC m=+39.669654585" watchObservedRunningTime="2026-07-02 21:19:27.227144034 +0000 UTC m=+39.669944487"
Jul 02 21:19:27.253695 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:27.253281    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8tkxq" podStartSLOduration=9.449166963 podStartE2EDuration="39.253260915s" totalImagesPullingTime="29.804093952s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:18:48 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:27.250360084 +0000 UTC m=+39.693160508" watchObservedRunningTime="2026-07-02 21:19:27.253260915 +0000 UTC m=+39.696061338"
Jul 02 21:19:30.177700 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:30.177637    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2c34410-8c0a-4d13-8864-575ba506997d-metrics-tls\") pod \"dns-default-w5w24\" (UID: \"d2c34410-8c0a-4d13-8864-575ba506997d\") " pod="openshift-dns/dns-default-w5w24"
Jul 02 21:19:30.178050 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:30.177797    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 21:19:30.178050 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:30.177859    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2c34410-8c0a-4d13-8864-575ba506997d-metrics-tls podName:d2c34410-8c0a-4d13-8864-575ba506997d nodeName:}" failed. No retries permitted until 2026-07-02 21:19:38.177843614 +0000 UTC m=+50.620644019 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d2c34410-8c0a-4d13-8864-575ba506997d-metrics-tls") pod "dns-default-w5w24" (UID: "d2c34410-8c0a-4d13-8864-575ba506997d") : secret "dns-default-metrics-tls" not found
Jul 02 21:19:38.228845 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:38.228800    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2c34410-8c0a-4d13-8864-575ba506997d-metrics-tls\") pod \"dns-default-w5w24\" (UID: \"d2c34410-8c0a-4d13-8864-575ba506997d\") " pod="openshift-dns/dns-default-w5w24"
Jul 02 21:19:38.229287 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:38.228991    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 21:19:38.229287 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:38.229080    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2c34410-8c0a-4d13-8864-575ba506997d-metrics-tls podName:d2c34410-8c0a-4d13-8864-575ba506997d nodeName:}" failed. No retries permitted until 2026-07-02 21:19:54.229062804 +0000 UTC m=+66.671863203 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d2c34410-8c0a-4d13-8864-575ba506997d-metrics-tls") pod "dns-default-w5w24" (UID: "d2c34410-8c0a-4d13-8864-575ba506997d") : secret "dns-default-metrics-tls" not found
Jul 02 21:19:49.194113 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:49.194083    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mvqc5"
Jul 02 21:19:50.449890 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.449853    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xzswr"]
Jul 02 21:19:50.452570 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.452544    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xzswr"
Jul 02 21:19:50.455809 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.455786    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 02 21:19:50.455908 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.455853    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-csmrv\""
Jul 02 21:19:50.456861 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.456846    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 02 21:19:50.456928 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.456878    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 02 21:19:50.456928 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.456897    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 02 21:19:50.478491 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.478469    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-69f78b9dc6-mvztz"]
Jul 02 21:19:50.480403 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.480384    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz"
Jul 02 21:19:50.480540 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.480517    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xzswr"]
Jul 02 21:19:50.483442 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.483426    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 02 21:19:50.483550 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.483522    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 02 21:19:50.483643 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.483627    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 02 21:19:50.483718 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.483677    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8l5qh\""
Jul 02 21:19:50.498948 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.498918    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 02 21:19:50.507985 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.507959    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69f78b9dc6-mvztz"]
Jul 02 21:19:50.616247 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.616204    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d2dfa800-1ffa-44d7-8dce-dd857531d58e-config\") pod \"service-ca-operator-5d46d9ffd4-xzswr\" (UID: \"d2dfa800-1ffa-44d7-8dce-dd857531d58e\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xzswr"
Jul 02 21:19:50.616247 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.616249    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d30ee675-695d-42ad-9932-248d4adde3e3-trusted-ca\") pod \"image-registry-69f78b9dc6-mvztz\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") " pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz"
Jul 02 21:19:50.616486 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.616317    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d30ee675-695d-42ad-9932-248d4adde3e3-bound-sa-token\") pod \"image-registry-69f78b9dc6-mvztz\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") " pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz"
Jul 02 21:19:50.616486 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.616375    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d30ee675-695d-42ad-9932-248d4adde3e3-registry-tls\") pod \"image-registry-69f78b9dc6-mvztz\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") " pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz"
Jul 02 21:19:50.616486 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.616401    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x425b\" (UniqueName: \"kubernetes.io/projected/d2dfa800-1ffa-44d7-8dce-dd857531d58e-kube-api-access-x425b\") pod \"service-ca-operator-5d46d9ffd4-xzswr\" (UID: \"d2dfa800-1ffa-44d7-8dce-dd857531d58e\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xzswr"
Jul 02 21:19:50.616486 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.616423    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d30ee675-695d-42ad-9932-248d4adde3e3-image-registry-private-configuration\") pod \"image-registry-69f78b9dc6-mvztz\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") " pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz"
Jul 02 21:19:50.616486 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.616440    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d2dfa800-1ffa-44d7-8dce-dd857531d58e-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-xzswr\" (UID: \"d2dfa800-1ffa-44d7-8dce-dd857531d58e\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xzswr"
Jul 02 21:19:50.616486 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.616483    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d30ee675-695d-42ad-9932-248d4adde3e3-installation-pull-secrets\") pod \"image-registry-69f78b9dc6-mvztz\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") " pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz"
Jul 02 21:19:50.616746 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.616521    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg4kn\" (UniqueName: \"kubernetes.io/projected/d30ee675-695d-42ad-9932-248d4adde3e3-kube-api-access-mg4kn\") pod \"image-registry-69f78b9dc6-mvztz\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") " pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz"
Jul 02 21:19:50.616746 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.616568    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d30ee675-695d-42ad-9932-248d4adde3e3-registry-certificates\") pod \"image-registry-69f78b9dc6-mvztz\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") " pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz"
Jul 02 21:19:50.616746 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.616602    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d30ee675-695d-42ad-9932-248d4adde3e3-ca-trust-extracted\") pod \"image-registry-69f78b9dc6-mvztz\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") " pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz"
Jul 02 21:19:50.651858 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.650422    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-c7tm5"]
Jul 02 21:19:50.653021 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.653003    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-c7tm5"
Jul 02 21:19:50.656120 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.656101    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 02 21:19:50.656680 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.656642    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-6fgsc\""
Jul 02 21:19:50.656770 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.656715    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 02 21:19:50.657004 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.656991    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 02 21:19:50.658795 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.658739    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 02 21:19:50.667681 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.667640    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-cr85g"]
Jul 02 21:19:50.669870 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.669852    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-c7tm5"]
Jul 02 21:19:50.669961 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.669929    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-cr85g"
Jul 02 21:19:50.672351 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.672334    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-slbs7\""
Jul 02 21:19:50.672526 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.672513    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 02 21:19:50.672612 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.672594    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 02 21:19:50.686251 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.686230    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-cr85g"]
Jul 02 21:19:50.717983 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.717891    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x425b\" (UniqueName: \"kubernetes.io/projected/d2dfa800-1ffa-44d7-8dce-dd857531d58e-kube-api-access-x425b\") pod \"service-ca-operator-5d46d9ffd4-xzswr\" (UID: \"d2dfa800-1ffa-44d7-8dce-dd857531d58e\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xzswr"
Jul 02 21:19:50.717983 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.717942    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d30ee675-695d-42ad-9932-248d4adde3e3-image-registry-private-configuration\") pod \"image-registry-69f78b9dc6-mvztz\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") " pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz"
Jul 02 21:19:50.717983 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.717964    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d2dfa800-1ffa-44d7-8dce-dd857531d58e-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-xzswr\" (UID: \"d2dfa800-1ffa-44d7-8dce-dd857531d58e\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xzswr"
Jul 02 21:19:50.718269 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.718009    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d30ee675-695d-42ad-9932-248d4adde3e3-installation-pull-secrets\") pod \"image-registry-69f78b9dc6-mvztz\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") " pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz"
Jul 02 21:19:50.718269 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.718031    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg4kn\" (UniqueName: \"kubernetes.io/projected/d30ee675-695d-42ad-9932-248d4adde3e3-kube-api-access-mg4kn\") pod \"image-registry-69f78b9dc6-mvztz\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") " pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz"
Jul 02 21:19:50.718269 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.718051    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d30ee675-695d-42ad-9932-248d4adde3e3-registry-certificates\") pod \"image-registry-69f78b9dc6-mvztz\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") " pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz"
Jul 02 21:19:50.718269 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.718097    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d30ee675-695d-42ad-9932-248d4adde3e3-ca-trust-extracted\") pod \"image-registry-69f78b9dc6-mvztz\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") " pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz"
Jul 02 21:19:50.718269 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.718152    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d2dfa800-1ffa-44d7-8dce-dd857531d58e-config\") pod \"service-ca-operator-5d46d9ffd4-xzswr\" (UID: \"d2dfa800-1ffa-44d7-8dce-dd857531d58e\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xzswr"
Jul 02 21:19:50.718269 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.718186    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d30ee675-695d-42ad-9932-248d4adde3e3-trusted-ca\") pod \"image-registry-69f78b9dc6-mvztz\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") " pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz"
Jul 02 21:19:50.718269 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.718238    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d30ee675-695d-42ad-9932-248d4adde3e3-bound-sa-token\") pod \"image-registry-69f78b9dc6-mvztz\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") " pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz"
Jul 02 21:19:50.718597 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.718281    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d30ee675-695d-42ad-9932-248d4adde3e3-registry-tls\") pod \"image-registry-69f78b9dc6-mvztz\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") " pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz"
Jul 02 21:19:50.718597 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:50.718395    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 21:19:50.718597 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:50.718414    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69f78b9dc6-mvztz: secret "image-registry-tls" not found
Jul 02 21:19:50.718597 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:50.718484    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d30ee675-695d-42ad-9932-248d4adde3e3-registry-tls podName:d30ee675-695d-42ad-9932-248d4adde3e3 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:51.21846386 +0000 UTC m=+63.661264276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d30ee675-695d-42ad-9932-248d4adde3e3-registry-tls") pod "image-registry-69f78b9dc6-mvztz" (UID: "d30ee675-695d-42ad-9932-248d4adde3e3") : secret "image-registry-tls" not found
Jul 02 21:19:50.718845 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.718820    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d2dfa800-1ffa-44d7-8dce-dd857531d58e-config\") pod \"service-ca-operator-5d46d9ffd4-xzswr\" (UID: \"d2dfa800-1ffa-44d7-8dce-dd857531d58e\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xzswr"
Jul 02 21:19:50.719120 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.719096    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d30ee675-695d-42ad-9932-248d4adde3e3-ca-trust-extracted\") pod \"image-registry-69f78b9dc6-mvztz\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") " pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz"
Jul 02 21:19:50.719205 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.719186    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d30ee675-695d-42ad-9932-248d4adde3e3-registry-certificates\") pod \"image-registry-69f78b9dc6-mvztz\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") " pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz"
Jul 02 21:19:50.719683 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.719639    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d30ee675-695d-42ad-9932-248d4adde3e3-trusted-ca\") pod \"image-registry-69f78b9dc6-mvztz\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") " pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz"
Jul 02 21:19:50.721543 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.721504    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d2dfa800-1ffa-44d7-8dce-dd857531d58e-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-xzswr\" (UID: \"d2dfa800-1ffa-44d7-8dce-dd857531d58e\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xzswr"
Jul 02 21:19:50.721707 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.721688    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d30ee675-695d-42ad-9932-248d4adde3e3-installation-pull-secrets\") pod \"image-registry-69f78b9dc6-mvztz\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") " pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz"
Jul 02 21:19:50.721771 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.721714    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d30ee675-695d-42ad-9932-248d4adde3e3-image-registry-private-configuration\") pod \"image-registry-69f78b9dc6-mvztz\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") " pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz"
Jul 02 21:19:50.729495 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.729456    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x425b\" (UniqueName: \"kubernetes.io/projected/d2dfa800-1ffa-44d7-8dce-dd857531d58e-kube-api-access-x425b\") pod \"service-ca-operator-5d46d9ffd4-xzswr\" (UID: \"d2dfa800-1ffa-44d7-8dce-dd857531d58e\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xzswr"
Jul 02 21:19:50.729819 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.729800    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d30ee675-695d-42ad-9932-248d4adde3e3-bound-sa-token\") pod \"image-registry-69f78b9dc6-mvztz\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") " pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz"
Jul 02 21:19:50.730114 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.730090    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg4kn\" (UniqueName: \"kubernetes.io/projected/d30ee675-695d-42ad-9932-248d4adde3e3-kube-api-access-mg4kn\") pod \"image-registry-69f78b9dc6-mvztz\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") " pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz"
Jul 02 21:19:50.760937 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.760910    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xzswr"
Jul 02 21:19:50.819579 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.819539    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b321922d-a68a-43a5-89bb-fd49e22371c3-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-cr85g\" (UID: \"b321922d-a68a-43a5-89bb-fd49e22371c3\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-cr85g"
Jul 02 21:19:50.819766 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.819594    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b77428f-b316-4a33-9397-b3227211e3e3-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-c7tm5\" (UID: \"1b77428f-b316-4a33-9397-b3227211e3e3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-c7tm5"
Jul 02 21:19:50.819766 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.819695    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b77428f-b316-4a33-9397-b3227211e3e3-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-c7tm5\" (UID: \"1b77428f-b316-4a33-9397-b3227211e3e3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-c7tm5"
Jul 02 21:19:50.819766 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.819735    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt6rv\" (UniqueName: \"kubernetes.io/projected/1b77428f-b316-4a33-9397-b3227211e3e3-kube-api-access-gt6rv\") pod \"kube-storage-version-migrator-operator-846bf668f7-c7tm5\" (UID: \"1b77428f-b316-4a33-9397-b3227211e3e3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-c7tm5"
Jul 02 21:19:50.819931 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.819786    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b321922d-a68a-43a5-89bb-fd49e22371c3-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-cr85g\" (UID: \"b321922d-a68a-43a5-89bb-fd49e22371c3\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-cr85g"
Jul 02 21:19:50.888399 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.888366    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xzswr"]
Jul 02 21:19:50.891309 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:19:50.891279    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2dfa800_1ffa_44d7_8dce_dd857531d58e.slice/crio-9bbea3d6133d8d89696b41b8e2f2e6913cc793763e831677823deab66a8c48b7 WatchSource:0}: Error finding container 9bbea3d6133d8d89696b41b8e2f2e6913cc793763e831677823deab66a8c48b7: Status 404 returned error can't find the container with id 9bbea3d6133d8d89696b41b8e2f2e6913cc793763e831677823deab66a8c48b7
Jul 02 21:19:50.920412 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.920369    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b321922d-a68a-43a5-89bb-fd49e22371c3-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-cr85g\" (UID: \"b321922d-a68a-43a5-89bb-fd49e22371c3\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-cr85g"
Jul 02 21:19:50.920412 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.920413    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b77428f-b316-4a33-9397-b3227211e3e3-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-c7tm5\" (UID: \"1b77428f-b316-4a33-9397-b3227211e3e3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-c7tm5"
Jul 02 21:19:50.920682 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.920435    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b77428f-b316-4a33-9397-b3227211e3e3-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-c7tm5\" (UID: \"1b77428f-b316-4a33-9397-b3227211e3e3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-c7tm5"
Jul 02 21:19:50.920682 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.920468    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt6rv\" (UniqueName: \"kubernetes.io/projected/1b77428f-b316-4a33-9397-b3227211e3e3-kube-api-access-gt6rv\") pod \"kube-storage-version-migrator-operator-846bf668f7-c7tm5\" (UID: \"1b77428f-b316-4a33-9397-b3227211e3e3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-c7tm5"
Jul 02 21:19:50.920682 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.920514    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b321922d-a68a-43a5-89bb-fd49e22371c3-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-cr85g\" (UID: \"b321922d-a68a-43a5-89bb-fd49e22371c3\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-cr85g"
Jul 02 21:19:50.920838 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:50.920699    2586 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 02 21:19:50.920838 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:50.920779    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b321922d-a68a-43a5-89bb-fd49e22371c3-networking-console-plugin-cert podName:b321922d-a68a-43a5-89bb-fd49e22371c3 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:51.420758713 +0000 UTC m=+63.863559112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b321922d-a68a-43a5-89bb-fd49e22371c3-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-cr85g" (UID: "b321922d-a68a-43a5-89bb-fd49e22371c3") : secret "networking-console-plugin-cert" not found
Jul 02 21:19:50.921130 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.921110    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b321922d-a68a-43a5-89bb-fd49e22371c3-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-cr85g\" (UID: \"b321922d-a68a-43a5-89bb-fd49e22371c3\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-cr85g"
Jul 02 21:19:50.921186 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.921165    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b77428f-b316-4a33-9397-b3227211e3e3-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-c7tm5\" (UID: \"1b77428f-b316-4a33-9397-b3227211e3e3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-c7tm5"
Jul 02 21:19:50.922593 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.922575    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b77428f-b316-4a33-9397-b3227211e3e3-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-c7tm5\" (UID: \"1b77428f-b316-4a33-9397-b3227211e3e3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-c7tm5"
Jul 02 21:19:50.929964 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.929935    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt6rv\" (UniqueName: \"kubernetes.io/projected/1b77428f-b316-4a33-9397-b3227211e3e3-kube-api-access-gt6rv\") pod \"kube-storage-version-migrator-operator-846bf668f7-c7tm5\" (UID: \"1b77428f-b316-4a33-9397-b3227211e3e3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-c7tm5"
Jul 02 21:19:50.961591 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:50.961566    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-c7tm5"
Jul 02 21:19:51.086847 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:51.086709    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-c7tm5"]
Jul 02 21:19:51.089444 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:19:51.089417    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b77428f_b316_4a33_9397_b3227211e3e3.slice/crio-cd8788405d59beb850b12dd10acac12b69e1c8ecff6ed06a22f744cb83b3242a WatchSource:0}: Error finding container cd8788405d59beb850b12dd10acac12b69e1c8ecff6ed06a22f744cb83b3242a: Status 404 returned error can't find the container with id cd8788405d59beb850b12dd10acac12b69e1c8ecff6ed06a22f744cb83b3242a
Jul 02 21:19:51.222924 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:51.222826    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d30ee675-695d-42ad-9932-248d4adde3e3-registry-tls\") pod \"image-registry-69f78b9dc6-mvztz\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") " pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz"
Jul 02 21:19:51.223068 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:51.222948    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 21:19:51.223068 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:51.222961    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69f78b9dc6-mvztz: secret "image-registry-tls" not found
Jul 02 21:19:51.223068 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:51.223013    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d30ee675-695d-42ad-9932-248d4adde3e3-registry-tls podName:d30ee675-695d-42ad-9932-248d4adde3e3 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:52.222999791 +0000 UTC m=+64.665800191 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d30ee675-695d-42ad-9932-248d4adde3e3-registry-tls") pod "image-registry-69f78b9dc6-mvztz" (UID: "d30ee675-695d-42ad-9932-248d4adde3e3") : secret "image-registry-tls" not found
Jul 02 21:19:51.270551 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:51.270513    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-c7tm5" event={"ID":"1b77428f-b316-4a33-9397-b3227211e3e3","Type":"ContainerStarted","Data":"cd8788405d59beb850b12dd10acac12b69e1c8ecff6ed06a22f744cb83b3242a"}
Jul 02 21:19:51.271601 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:51.271575    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xzswr" event={"ID":"d2dfa800-1ffa-44d7-8dce-dd857531d58e","Type":"ContainerStarted","Data":"9bbea3d6133d8d89696b41b8e2f2e6913cc793763e831677823deab66a8c48b7"}
Jul 02 21:19:51.423998 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:51.423960    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b321922d-a68a-43a5-89bb-fd49e22371c3-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-cr85g\" (UID: \"b321922d-a68a-43a5-89bb-fd49e22371c3\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-cr85g"
Jul 02 21:19:51.424209 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:51.424149    2586 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 02 21:19:51.424273 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:51.424238    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b321922d-a68a-43a5-89bb-fd49e22371c3-networking-console-plugin-cert podName:b321922d-a68a-43a5-89bb-fd49e22371c3 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:52.424216294 +0000 UTC m=+64.867016713 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b321922d-a68a-43a5-89bb-fd49e22371c3-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-cr85g" (UID: "b321922d-a68a-43a5-89bb-fd49e22371c3") : secret "networking-console-plugin-cert" not found
Jul 02 21:19:52.231574 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:52.231532    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d30ee675-695d-42ad-9932-248d4adde3e3-registry-tls\") pod \"image-registry-69f78b9dc6-mvztz\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") " pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz"
Jul 02 21:19:52.232157 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:52.231749    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 21:19:52.232157 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:52.231774    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69f78b9dc6-mvztz: secret "image-registry-tls" not found
Jul 02 21:19:52.232157 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:52.231851    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d30ee675-695d-42ad-9932-248d4adde3e3-registry-tls podName:d30ee675-695d-42ad-9932-248d4adde3e3 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:54.231826851 +0000 UTC m=+66.674627265 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d30ee675-695d-42ad-9932-248d4adde3e3-registry-tls") pod "image-registry-69f78b9dc6-mvztz" (UID: "d30ee675-695d-42ad-9932-248d4adde3e3") : secret "image-registry-tls" not found
Jul 02 21:19:52.433378 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:52.433341    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b321922d-a68a-43a5-89bb-fd49e22371c3-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-cr85g\" (UID: \"b321922d-a68a-43a5-89bb-fd49e22371c3\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-cr85g"
Jul 02 21:19:52.433558 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:52.433521    2586 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 02 21:19:52.433610 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:52.433602    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b321922d-a68a-43a5-89bb-fd49e22371c3-networking-console-plugin-cert podName:b321922d-a68a-43a5-89bb-fd49e22371c3 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:54.433575175 +0000 UTC m=+66.876375576 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b321922d-a68a-43a5-89bb-fd49e22371c3-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-cr85g" (UID: "b321922d-a68a-43a5-89bb-fd49e22371c3") : secret "networking-console-plugin-cert" not found
Jul 02 21:19:53.743964 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:53.743914    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c214277-f44b-4f90-be58-8c8078403fc2-original-pull-secret\") pod \"global-pull-secret-syncer-5gfl7\" (UID: \"9c214277-f44b-4f90-be58-8c8078403fc2\") " pod="kube-system/global-pull-secret-syncer-5gfl7"
Jul 02 21:19:53.744426 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:53.744112    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/006d61c8-eb34-4336-a3d3-82a7a4fc2069-metrics-certs\") pod \"network-metrics-daemon-s5xr4\" (UID: \"006d61c8-eb34-4336-a3d3-82a7a4fc2069\") " pod="openshift-multus/network-metrics-daemon-s5xr4"
Jul 02 21:19:53.746862 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:53.746840    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 21:19:53.746952 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:53.746923    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 21:19:53.754400 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:53.754382    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 02 21:19:53.754459 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:53.754447    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/006d61c8-eb34-4336-a3d3-82a7a4fc2069-metrics-certs podName:006d61c8-eb34-4336-a3d3-82a7a4fc2069 nodeName:}" failed. No retries permitted until 2026-07-02 21:20:57.754425164 +0000 UTC m=+130.197225573 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/006d61c8-eb34-4336-a3d3-82a7a4fc2069-metrics-certs") pod "network-metrics-daemon-s5xr4" (UID: "006d61c8-eb34-4336-a3d3-82a7a4fc2069") : secret "metrics-daemon-secret" not found
Jul 02 21:19:53.756457 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:53.756430    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c214277-f44b-4f90-be58-8c8078403fc2-original-pull-secret\") pod \"global-pull-secret-syncer-5gfl7\" (UID: \"9c214277-f44b-4f90-be58-8c8078403fc2\") " pod="kube-system/global-pull-secret-syncer-5gfl7"
Jul 02 21:19:53.845538 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:53.845506    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thk5p\" (UniqueName: \"kubernetes.io/projected/45cb060e-3948-4b0e-b407-86ea9b6af872-kube-api-access-thk5p\") pod \"network-check-target-d495m\" (UID: \"45cb060e-3948-4b0e-b407-86ea9b6af872\") " pod="openshift-network-diagnostics/network-check-target-d495m"
Jul 02 21:19:53.848575 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:53.848551    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 21:19:53.858885 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:53.858863    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 21:19:53.869369 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:53.869341    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thk5p\" (UniqueName: \"kubernetes.io/projected/45cb060e-3948-4b0e-b407-86ea9b6af872-kube-api-access-thk5p\") pod \"network-check-target-d495m\" (UID: \"45cb060e-3948-4b0e-b407-86ea9b6af872\") " pod="openshift-network-diagnostics/network-check-target-d495m"
Jul 02 21:19:53.929922 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:53.929882    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gfl7"
Jul 02 21:19:53.940055 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:53.940027    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bjnbk\""
Jul 02 21:19:53.947790 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:53.947765    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d495m"
Jul 02 21:19:54.081366 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:54.081335    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5gfl7"]
Jul 02 21:19:54.084153 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:19:54.084113    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c214277_f44b_4f90_be58_8c8078403fc2.slice/crio-9f285f090e923036b8ac9ff4122d5e22d31be2fb10e17407103ff8b0dfc46b71 WatchSource:0}: Error finding container 9f285f090e923036b8ac9ff4122d5e22d31be2fb10e17407103ff8b0dfc46b71: Status 404 returned error can't find the container with id 9f285f090e923036b8ac9ff4122d5e22d31be2fb10e17407103ff8b0dfc46b71
Jul 02 21:19:54.097410 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:54.097223    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-d495m"]
Jul 02 21:19:54.099864 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:19:54.099835    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45cb060e_3948_4b0e_b407_86ea9b6af872.slice/crio-1e3f5d0f1c4281b078b199da9c759b9398517939410e4a4525077ea302d09ffb WatchSource:0}: Error finding container 1e3f5d0f1c4281b078b199da9c759b9398517939410e4a4525077ea302d09ffb: Status 404 returned error can't find the container with id 1e3f5d0f1c4281b078b199da9c759b9398517939410e4a4525077ea302d09ffb
Jul 02 21:19:54.249463 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:54.249428    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2c34410-8c0a-4d13-8864-575ba506997d-metrics-tls\") pod \"dns-default-w5w24\" (UID: \"d2c34410-8c0a-4d13-8864-575ba506997d\") " pod="openshift-dns/dns-default-w5w24"
Jul 02 21:19:54.249463 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:54.249469    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d30ee675-695d-42ad-9932-248d4adde3e3-registry-tls\") pod \"image-registry-69f78b9dc6-mvztz\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") " pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz"
Jul 02 21:19:54.249695 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:54.249604    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 21:19:54.249695 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:54.249684    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2c34410-8c0a-4d13-8864-575ba506997d-metrics-tls podName:d2c34410-8c0a-4d13-8864-575ba506997d nodeName:}" failed. No retries permitted until 2026-07-02 21:20:26.249669175 +0000 UTC m=+98.692469578 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d2c34410-8c0a-4d13-8864-575ba506997d-metrics-tls") pod "dns-default-w5w24" (UID: "d2c34410-8c0a-4d13-8864-575ba506997d") : secret "dns-default-metrics-tls" not found
Jul 02 21:19:54.249780 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:54.249608    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 21:19:54.249780 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:54.249716    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69f78b9dc6-mvztz: secret "image-registry-tls" not found
Jul 02 21:19:54.249780 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:54.249757    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d30ee675-695d-42ad-9932-248d4adde3e3-registry-tls podName:d30ee675-695d-42ad-9932-248d4adde3e3 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:58.24974509 +0000 UTC m=+70.692545505 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d30ee675-695d-42ad-9932-248d4adde3e3-registry-tls") pod "image-registry-69f78b9dc6-mvztz" (UID: "d30ee675-695d-42ad-9932-248d4adde3e3") : secret "image-registry-tls" not found
Jul 02 21:19:54.281080 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:54.281001    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5gfl7" event={"ID":"9c214277-f44b-4f90-be58-8c8078403fc2","Type":"ContainerStarted","Data":"9f285f090e923036b8ac9ff4122d5e22d31be2fb10e17407103ff8b0dfc46b71"}
Jul 02 21:19:54.282581 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:54.282550    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xzswr" event={"ID":"d2dfa800-1ffa-44d7-8dce-dd857531d58e","Type":"ContainerStarted","Data":"d78b7bbab21860ae93efa7c60ee2010a94d58924008850440e46b39be90df010"}
Jul 02 21:19:54.284113 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:54.284090    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-c7tm5" event={"ID":"1b77428f-b316-4a33-9397-b3227211e3e3","Type":"ContainerStarted","Data":"93c44b9896ed68768f50681bbaa92cb302aec9bb40793e852eada7de7e6d2cb6"}
Jul 02 21:19:54.285276 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:54.285252    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-d495m" event={"ID":"45cb060e-3948-4b0e-b407-86ea9b6af872","Type":"ContainerStarted","Data":"1e3f5d0f1c4281b078b199da9c759b9398517939410e4a4525077ea302d09ffb"}
Jul 02 21:19:54.299713 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:54.299640    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xzswr" podStartSLOduration=1.830497799 podStartE2EDuration="4.299623297s" totalImagesPullingTime="2.469125498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:19:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:54.298862552 +0000 UTC m=+66.741662997" watchObservedRunningTime="2026-07-02 21:19:54.299623297 +0000 UTC m=+66.742423710"
Jul 02 21:19:54.318593 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:54.318535    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-c7tm5" podStartSLOduration=2.045780485 podStartE2EDuration="4.318510055s" totalImagesPullingTime="2.27272957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:19:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:54.317585783 +0000 UTC m=+66.760386205" watchObservedRunningTime="2026-07-02 21:19:54.318510055 +0000 UTC m=+66.761310476"
Jul 02 21:19:54.451015 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:54.450972    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b321922d-a68a-43a5-89bb-fd49e22371c3-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-cr85g\" (UID: \"b321922d-a68a-43a5-89bb-fd49e22371c3\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-cr85g"
Jul 02 21:19:54.451204 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:54.451112    2586 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 02 21:19:54.451204 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:54.451174    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b321922d-a68a-43a5-89bb-fd49e22371c3-networking-console-plugin-cert podName:b321922d-a68a-43a5-89bb-fd49e22371c3 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:58.45115699 +0000 UTC m=+70.893957392 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b321922d-a68a-43a5-89bb-fd49e22371c3-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-cr85g" (UID: "b321922d-a68a-43a5-89bb-fd49e22371c3") : secret "networking-console-plugin-cert" not found
Jul 02 21:19:54.640003 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:54.639846    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dldh9"]
Jul 02 21:19:54.644332 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:54.643214    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dldh9"
Jul 02 21:19:54.647155 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:54.646968    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 02 21:19:54.655559 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:54.655531    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 02 21:19:54.656882 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:54.656864    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 02 21:19:54.657119 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:54.657103    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 02 21:19:54.657179 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:54.657104    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zd4gb\""
Jul 02 21:19:54.666122 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:54.666096    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dldh9"]
Jul 02 21:19:54.753719 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:54.753683    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9103d7e-3bea-4efc-ab49-8790e23e383a-data-volume\") pod \"insights-runtime-extractor-dldh9\" (UID: \"d9103d7e-3bea-4efc-ab49-8790e23e383a\") " pod="openshift-insights/insights-runtime-extractor-dldh9"
Jul 02 21:19:54.754159 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:54.753765    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9103d7e-3bea-4efc-ab49-8790e23e383a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dldh9\" (UID: \"d9103d7e-3bea-4efc-ab49-8790e23e383a\") " pod="openshift-insights/insights-runtime-extractor-dldh9"
Jul 02 21:19:54.754159 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:54.753796    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stnqn\" (UniqueName: \"kubernetes.io/projected/d9103d7e-3bea-4efc-ab49-8790e23e383a-kube-api-access-stnqn\") pod \"insights-runtime-extractor-dldh9\" (UID: \"d9103d7e-3bea-4efc-ab49-8790e23e383a\") " pod="openshift-insights/insights-runtime-extractor-dldh9"
Jul 02 21:19:54.754159 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:54.753840    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d9103d7e-3bea-4efc-ab49-8790e23e383a-crio-socket\") pod \"insights-runtime-extractor-dldh9\" (UID: \"d9103d7e-3bea-4efc-ab49-8790e23e383a\") " pod="openshift-insights/insights-runtime-extractor-dldh9"
Jul 02 21:19:54.754159 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:54.753935    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d9103d7e-3bea-4efc-ab49-8790e23e383a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dldh9\" (UID: \"d9103d7e-3bea-4efc-ab49-8790e23e383a\") " pod="openshift-insights/insights-runtime-extractor-dldh9"
Jul 02 21:19:54.854969 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:54.854808    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9103d7e-3bea-4efc-ab49-8790e23e383a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dldh9\" (UID: \"d9103d7e-3bea-4efc-ab49-8790e23e383a\") " pod="openshift-insights/insights-runtime-extractor-dldh9"
Jul 02 21:19:54.854969 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:54.854857    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stnqn\" (UniqueName: \"kubernetes.io/projected/d9103d7e-3bea-4efc-ab49-8790e23e383a-kube-api-access-stnqn\") pod \"insights-runtime-extractor-dldh9\" (UID: \"d9103d7e-3bea-4efc-ab49-8790e23e383a\") " pod="openshift-insights/insights-runtime-extractor-dldh9"
Jul 02 21:19:54.854969 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:54.854902    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d9103d7e-3bea-4efc-ab49-8790e23e383a-crio-socket\") pod \"insights-runtime-extractor-dldh9\" (UID: \"d9103d7e-3bea-4efc-ab49-8790e23e383a\") " pod="openshift-insights/insights-runtime-extractor-dldh9"
Jul 02 21:19:54.854969 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:54.854957    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d9103d7e-3bea-4efc-ab49-8790e23e383a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dldh9\" (UID: \"d9103d7e-3bea-4efc-ab49-8790e23e383a\") " pod="openshift-insights/insights-runtime-extractor-dldh9"
Jul 02 21:19:54.855306 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:54.855021    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9103d7e-3bea-4efc-ab49-8790e23e383a-data-volume\") pod \"insights-runtime-extractor-dldh9\" (UID: \"d9103d7e-3bea-4efc-ab49-8790e23e383a\") " pod="openshift-insights/insights-runtime-extractor-dldh9"
Jul 02 21:19:54.855493 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:54.855470    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 21:19:54.855578 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:54.855555    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9103d7e-3bea-4efc-ab49-8790e23e383a-insights-runtime-extractor-tls podName:d9103d7e-3bea-4efc-ab49-8790e23e383a nodeName:}" failed. No retries permitted until 2026-07-02 21:19:55.355525723 +0000 UTC m=+67.798326127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d9103d7e-3bea-4efc-ab49-8790e23e383a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dldh9" (UID: "d9103d7e-3bea-4efc-ab49-8790e23e383a") : secret "insights-runtime-extractor-tls" not found
Jul 02 21:19:54.855705 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:54.855683    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9103d7e-3bea-4efc-ab49-8790e23e383a-data-volume\") pod \"insights-runtime-extractor-dldh9\" (UID: \"d9103d7e-3bea-4efc-ab49-8790e23e383a\") " pod="openshift-insights/insights-runtime-extractor-dldh9"
Jul 02 21:19:54.855796 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:54.855776    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d9103d7e-3bea-4efc-ab49-8790e23e383a-crio-socket\") pod \"insights-runtime-extractor-dldh9\" (UID: \"d9103d7e-3bea-4efc-ab49-8790e23e383a\") " pod="openshift-insights/insights-runtime-extractor-dldh9"
Jul 02 21:19:54.855956 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:54.855930    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 21:19:54.856014 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:54.856007    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d9103d7e-3bea-4efc-ab49-8790e23e383a-kube-rbac-proxy-cm podName:d9103d7e-3bea-4efc-ab49-8790e23e383a nodeName:}" failed. No retries permitted until 2026-07-02 21:19:55.355991323 +0000 UTC m=+67.798791743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d9103d7e-3bea-4efc-ab49-8790e23e383a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dldh9" (UID: "d9103d7e-3bea-4efc-ab49-8790e23e383a") : configmap "kube-rbac-proxy" not found
Jul 02 21:19:54.868293 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:54.868241    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stnqn\" (UniqueName: \"kubernetes.io/projected/d9103d7e-3bea-4efc-ab49-8790e23e383a-kube-api-access-stnqn\") pod \"insights-runtime-extractor-dldh9\" (UID: \"d9103d7e-3bea-4efc-ab49-8790e23e383a\") " pod="openshift-insights/insights-runtime-extractor-dldh9"
Jul 02 21:19:55.358954 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:55.358911    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9103d7e-3bea-4efc-ab49-8790e23e383a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dldh9\" (UID: \"d9103d7e-3bea-4efc-ab49-8790e23e383a\") " pod="openshift-insights/insights-runtime-extractor-dldh9"
Jul 02 21:19:55.359151 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:55.359080    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 21:19:55.359223 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:55.359149    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 21:19:55.359223 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:55.359159    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9103d7e-3bea-4efc-ab49-8790e23e383a-insights-runtime-extractor-tls podName:d9103d7e-3bea-4efc-ab49-8790e23e383a nodeName:}" failed. No retries permitted until 2026-07-02 21:19:56.359142468 +0000 UTC m=+68.801942868 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d9103d7e-3bea-4efc-ab49-8790e23e383a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dldh9" (UID: "d9103d7e-3bea-4efc-ab49-8790e23e383a") : secret "insights-runtime-extractor-tls" not found
Jul 02 21:19:55.359223 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:55.359202    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d9103d7e-3bea-4efc-ab49-8790e23e383a-kube-rbac-proxy-cm podName:d9103d7e-3bea-4efc-ab49-8790e23e383a nodeName:}" failed. No retries permitted until 2026-07-02 21:19:56.359186393 +0000 UTC m=+68.801986798 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d9103d7e-3bea-4efc-ab49-8790e23e383a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dldh9" (UID: "d9103d7e-3bea-4efc-ab49-8790e23e383a") : configmap "kube-rbac-proxy" not found
Jul 02 21:19:55.359223 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:55.359089    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d9103d7e-3bea-4efc-ab49-8790e23e383a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dldh9\" (UID: \"d9103d7e-3bea-4efc-ab49-8790e23e383a\") " pod="openshift-insights/insights-runtime-extractor-dldh9"
Jul 02 21:19:55.506966 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:55.506909    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-pxlmv"]
Jul 02 21:19:55.522558 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:55.522521    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-pxlmv"
Jul 02 21:19:55.526718 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:55.526639    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-xcdjt\""
Jul 02 21:19:55.526718 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:55.526640    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 02 21:19:55.526953 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:55.526639    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 02 21:19:55.538632 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:55.538593    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-pxlmv"]
Jul 02 21:19:55.662088 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:55.662002    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsk4g\" (UniqueName: \"kubernetes.io/projected/01ce737f-793f-48cd-be14-aa4431880a61-kube-api-access-jsk4g\") pod \"migrator-d57d58cc9-pxlmv\" (UID: \"01ce737f-793f-48cd-be14-aa4431880a61\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-pxlmv"
Jul 02 21:19:55.762386 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:55.762350    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsk4g\" (UniqueName: \"kubernetes.io/projected/01ce737f-793f-48cd-be14-aa4431880a61-kube-api-access-jsk4g\") pod \"migrator-d57d58cc9-pxlmv\" (UID: \"01ce737f-793f-48cd-be14-aa4431880a61\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-pxlmv"
Jul 02 21:19:55.772701 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:55.772673    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsk4g\" (UniqueName: \"kubernetes.io/projected/01ce737f-793f-48cd-be14-aa4431880a61-kube-api-access-jsk4g\") pod \"migrator-d57d58cc9-pxlmv\" (UID: \"01ce737f-793f-48cd-be14-aa4431880a61\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-pxlmv"
Jul 02 21:19:55.835264 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:55.835201    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-pxlmv"
Jul 02 21:19:56.013385 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:56.013335    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-pxlmv"]
Jul 02 21:19:56.017018 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:19:56.016974    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ce737f_793f_48cd_be14_aa4431880a61.slice/crio-20bcdf62d4aeb3062a95e7a8e81108a3ceaacecbada606741319454346a1223b WatchSource:0}: Error finding container 20bcdf62d4aeb3062a95e7a8e81108a3ceaacecbada606741319454346a1223b: Status 404 returned error can't find the container with id 20bcdf62d4aeb3062a95e7a8e81108a3ceaacecbada606741319454346a1223b
Jul 02 21:19:56.293396 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:56.293357    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-pxlmv" event={"ID":"01ce737f-793f-48cd-be14-aa4431880a61","Type":"ContainerStarted","Data":"20bcdf62d4aeb3062a95e7a8e81108a3ceaacecbada606741319454346a1223b"}
Jul 02 21:19:56.366500 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:56.366459    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9103d7e-3bea-4efc-ab49-8790e23e383a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dldh9\" (UID: \"d9103d7e-3bea-4efc-ab49-8790e23e383a\") " pod="openshift-insights/insights-runtime-extractor-dldh9"
Jul 02 21:19:56.366701 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:56.366589    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d9103d7e-3bea-4efc-ab49-8790e23e383a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dldh9\" (UID: \"d9103d7e-3bea-4efc-ab49-8790e23e383a\") " pod="openshift-insights/insights-runtime-extractor-dldh9"
Jul 02 21:19:56.366701 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:56.366629    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 21:19:56.366805 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:56.366720    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 21:19:56.366805 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:56.366731    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9103d7e-3bea-4efc-ab49-8790e23e383a-insights-runtime-extractor-tls podName:d9103d7e-3bea-4efc-ab49-8790e23e383a nodeName:}" failed. No retries permitted until 2026-07-02 21:19:58.366709903 +0000 UTC m=+70.809510323 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d9103d7e-3bea-4efc-ab49-8790e23e383a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dldh9" (UID: "d9103d7e-3bea-4efc-ab49-8790e23e383a") : secret "insights-runtime-extractor-tls" not found
Jul 02 21:19:56.366805 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:56.366786    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d9103d7e-3bea-4efc-ab49-8790e23e383a-kube-rbac-proxy-cm podName:d9103d7e-3bea-4efc-ab49-8790e23e383a nodeName:}" failed. No retries permitted until 2026-07-02 21:19:58.366772606 +0000 UTC m=+70.809573006 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d9103d7e-3bea-4efc-ab49-8790e23e383a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dldh9" (UID: "d9103d7e-3bea-4efc-ab49-8790e23e383a") : configmap "kube-rbac-proxy" not found
Jul 02 21:19:56.517170 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:56.517058    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h77tf_83d2714f-a9bc-4830-b379-871a817b6f32/dns-node-resolver/0.log"
Jul 02 21:19:57.317518 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:57.317490    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6fpnv_75f700f7-60d3-4bf0-aef0-2052610d6cba/node-ca/0.log"
Jul 02 21:19:58.284330 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:58.284290    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d30ee675-695d-42ad-9932-248d4adde3e3-registry-tls\") pod \"image-registry-69f78b9dc6-mvztz\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") " pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz"
Jul 02 21:19:58.284536 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:58.284474    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 21:19:58.284536 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:58.284499    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69f78b9dc6-mvztz: secret "image-registry-tls" not found
Jul 02 21:19:58.284628 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:58.284570    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d30ee675-695d-42ad-9932-248d4adde3e3-registry-tls podName:d30ee675-695d-42ad-9932-248d4adde3e3 nodeName:}" failed. No retries permitted until 2026-07-02 21:20:06.284546673 +0000 UTC m=+78.727347082 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d30ee675-695d-42ad-9932-248d4adde3e3-registry-tls") pod "image-registry-69f78b9dc6-mvztz" (UID: "d30ee675-695d-42ad-9932-248d4adde3e3") : secret "image-registry-tls" not found
Jul 02 21:19:58.320036 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:58.319982    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mvrrr_2fc74b27-fbce-4841-b23e-100f5fffdb1b/serve-healthcheck-canary/0.log"
Jul 02 21:19:58.385324 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:58.385290    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9103d7e-3bea-4efc-ab49-8790e23e383a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dldh9\" (UID: \"d9103d7e-3bea-4efc-ab49-8790e23e383a\") " pod="openshift-insights/insights-runtime-extractor-dldh9"
Jul 02 21:19:58.385486 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:58.385372    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d9103d7e-3bea-4efc-ab49-8790e23e383a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dldh9\" (UID: \"d9103d7e-3bea-4efc-ab49-8790e23e383a\") " pod="openshift-insights/insights-runtime-extractor-dldh9"
Jul 02 21:19:58.385486 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:58.385430    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 21:19:58.385486 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:58.385469    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 21:19:58.385616 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:58.385500    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9103d7e-3bea-4efc-ab49-8790e23e383a-insights-runtime-extractor-tls podName:d9103d7e-3bea-4efc-ab49-8790e23e383a nodeName:}" failed. No retries permitted until 2026-07-02 21:20:02.385482274 +0000 UTC m=+74.828282691 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d9103d7e-3bea-4efc-ab49-8790e23e383a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dldh9" (UID: "d9103d7e-3bea-4efc-ab49-8790e23e383a") : secret "insights-runtime-extractor-tls" not found
Jul 02 21:19:58.385616 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:58.385518    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d9103d7e-3bea-4efc-ab49-8790e23e383a-kube-rbac-proxy-cm podName:d9103d7e-3bea-4efc-ab49-8790e23e383a nodeName:}" failed. No retries permitted until 2026-07-02 21:20:02.385509482 +0000 UTC m=+74.828309881 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d9103d7e-3bea-4efc-ab49-8790e23e383a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dldh9" (UID: "d9103d7e-3bea-4efc-ab49-8790e23e383a") : configmap "kube-rbac-proxy" not found
Jul 02 21:19:58.485931 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:58.485893    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b321922d-a68a-43a5-89bb-fd49e22371c3-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-cr85g\" (UID: \"b321922d-a68a-43a5-89bb-fd49e22371c3\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-cr85g"
Jul 02 21:19:58.486115 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:58.486021    2586 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 02 21:19:58.486115 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:19:58.486101    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b321922d-a68a-43a5-89bb-fd49e22371c3-networking-console-plugin-cert podName:b321922d-a68a-43a5-89bb-fd49e22371c3 nodeName:}" failed. No retries permitted until 2026-07-02 21:20:06.486081079 +0000 UTC m=+78.928881493 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b321922d-a68a-43a5-89bb-fd49e22371c3-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-cr85g" (UID: "b321922d-a68a-43a5-89bb-fd49e22371c3") : secret "networking-console-plugin-cert" not found
Jul 02 21:19:59.304727 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:59.304606    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-d495m" event={"ID":"45cb060e-3948-4b0e-b407-86ea9b6af872","Type":"ContainerStarted","Data":"ff735943b70e351ea4372ae6ce0717255a082d9e0ac858771c744d214528f715"}
Jul 02 21:19:59.304727 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:59.304710    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-d495m"
Jul 02 21:19:59.306152 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:59.306123    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5gfl7" event={"ID":"9c214277-f44b-4f90-be58-8c8078403fc2","Type":"ContainerStarted","Data":"1930ad5be9e3d807a5645cd672ed937d21e4592322fbdcaaab7a0210566fd07c"}
Jul 02 21:19:59.307787 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:59.307765    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-pxlmv" event={"ID":"01ce737f-793f-48cd-be14-aa4431880a61","Type":"ContainerStarted","Data":"4467a45e72840a57a549adc4b4dd0d452165c6a1449caefc4035f93206392a60"}
Jul 02 21:19:59.307884 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:59.307803    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-pxlmv" event={"ID":"01ce737f-793f-48cd-be14-aa4431880a61","Type":"ContainerStarted","Data":"bbd2bc745a10404b38244174bb41628c6ac8b74912ce6569206ba515b57ed3b8"}
Jul 02 21:19:59.324292 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:59.324241    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-d495m" podStartSLOduration=66.474353485 podStartE2EDuration="1m11.32422185s" totalImagesPullingTime="4.849868365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:18:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:59.324142302 +0000 UTC m=+71.766942742" watchObservedRunningTime="2026-07-02 21:19:59.32422185 +0000 UTC m=+71.767022274"
Jul 02 21:19:59.349037 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:59.347197    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-pxlmv" podStartSLOduration=1.365941721 podStartE2EDuration="4.34717687s" totalImagesPullingTime="2.981235149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:59.3458734 +0000 UTC m=+71.788673842" watchObservedRunningTime="2026-07-02 21:19:59.34717687 +0000 UTC m=+71.789977294"
Jul 02 21:19:59.364161 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:19:59.364124    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5gfl7" podStartSLOduration=66.498373711 podStartE2EDuration="1m11.364112226s" totalImagesPullingTime="4.865738515s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:18:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:59.364089981 +0000 UTC m=+71.806890404" watchObservedRunningTime="2026-07-02 21:19:59.364112226 +0000 UTC m=+71.806912648"
Jul 02 21:20:02.417982 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:02.417945    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9103d7e-3bea-4efc-ab49-8790e23e383a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dldh9\" (UID: \"d9103d7e-3bea-4efc-ab49-8790e23e383a\") " pod="openshift-insights/insights-runtime-extractor-dldh9"
Jul 02 21:20:02.418467 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:20:02.418083    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 21:20:02.418467 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:20:02.418130    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 21:20:02.418467 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:20:02.418145    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9103d7e-3bea-4efc-ab49-8790e23e383a-insights-runtime-extractor-tls podName:d9103d7e-3bea-4efc-ab49-8790e23e383a nodeName:}" failed. No retries permitted until 2026-07-02 21:20:10.418129218 +0000 UTC m=+82.860929618 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d9103d7e-3bea-4efc-ab49-8790e23e383a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dldh9" (UID: "d9103d7e-3bea-4efc-ab49-8790e23e383a") : secret "insights-runtime-extractor-tls" not found
Jul 02 21:20:02.418467 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:02.418085    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d9103d7e-3bea-4efc-ab49-8790e23e383a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dldh9\" (UID: \"d9103d7e-3bea-4efc-ab49-8790e23e383a\") " pod="openshift-insights/insights-runtime-extractor-dldh9"
Jul 02 21:20:02.418467 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:20:02.418182    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d9103d7e-3bea-4efc-ab49-8790e23e383a-kube-rbac-proxy-cm podName:d9103d7e-3bea-4efc-ab49-8790e23e383a nodeName:}" failed. No retries permitted until 2026-07-02 21:20:10.418170161 +0000 UTC m=+82.860970566 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d9103d7e-3bea-4efc-ab49-8790e23e383a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dldh9" (UID: "d9103d7e-3bea-4efc-ab49-8790e23e383a") : configmap "kube-rbac-proxy" not found
Jul 02 21:20:06.343610 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:06.343573    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d30ee675-695d-42ad-9932-248d4adde3e3-registry-tls\") pod \"image-registry-69f78b9dc6-mvztz\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") " pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz"
Jul 02 21:20:06.345939 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:06.345918    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d30ee675-695d-42ad-9932-248d4adde3e3-registry-tls\") pod \"image-registry-69f78b9dc6-mvztz\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") " pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz"
Jul 02 21:20:06.388886 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:06.388866    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz"
Jul 02 21:20:06.520908 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:06.520876    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69f78b9dc6-mvztz"]
Jul 02 21:20:06.524525 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:20:06.524496    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd30ee675_695d_42ad_9932_248d4adde3e3.slice/crio-2ec2597f6ff4133fbadf4b3d3fce41d67b528dadadf57793243280df49061266 WatchSource:0}: Error finding container 2ec2597f6ff4133fbadf4b3d3fce41d67b528dadadf57793243280df49061266: Status 404 returned error can't find the container with id 2ec2597f6ff4133fbadf4b3d3fce41d67b528dadadf57793243280df49061266
Jul 02 21:20:06.544589 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:06.544563    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b321922d-a68a-43a5-89bb-fd49e22371c3-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-cr85g\" (UID: \"b321922d-a68a-43a5-89bb-fd49e22371c3\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-cr85g"
Jul 02 21:20:06.544721 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:20:06.544702    2586 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 02 21:20:06.544793 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:20:06.544783    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b321922d-a68a-43a5-89bb-fd49e22371c3-networking-console-plugin-cert podName:b321922d-a68a-43a5-89bb-fd49e22371c3 nodeName:}" failed. No retries permitted until 2026-07-02 21:20:22.54476652 +0000 UTC m=+94.987566923 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b321922d-a68a-43a5-89bb-fd49e22371c3-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-cr85g" (UID: "b321922d-a68a-43a5-89bb-fd49e22371c3") : secret "networking-console-plugin-cert" not found
Jul 02 21:20:07.336954 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:07.336921    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz" event={"ID":"d30ee675-695d-42ad-9932-248d4adde3e3","Type":"ContainerStarted","Data":"d9d2870fa6b143206660eb46f26f3caf2a07a4872f79623cfeeb870e090e631e"}
Jul 02 21:20:07.337138 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:07.336971    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz" event={"ID":"d30ee675-695d-42ad-9932-248d4adde3e3","Type":"ContainerStarted","Data":"2ec2597f6ff4133fbadf4b3d3fce41d67b528dadadf57793243280df49061266"}
Jul 02 21:20:07.337138 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:07.337052    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz"
Jul 02 21:20:07.367551 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:07.367508    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz" podStartSLOduration=17.367490646 podStartE2EDuration="17.367490646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:07.365780226 +0000 UTC m=+79.808580645" watchObservedRunningTime="2026-07-02 21:20:07.367490646 +0000 UTC m=+79.810291068"
Jul 02 21:20:10.473578 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:10.473440    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d9103d7e-3bea-4efc-ab49-8790e23e383a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dldh9\" (UID: \"d9103d7e-3bea-4efc-ab49-8790e23e383a\") " pod="openshift-insights/insights-runtime-extractor-dldh9"
Jul 02 21:20:10.473578 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:10.473531    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9103d7e-3bea-4efc-ab49-8790e23e383a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dldh9\" (UID: \"d9103d7e-3bea-4efc-ab49-8790e23e383a\") " pod="openshift-insights/insights-runtime-extractor-dldh9"
Jul 02 21:20:10.474083 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:20:10.473686    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 21:20:10.474083 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:20:10.473760    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9103d7e-3bea-4efc-ab49-8790e23e383a-insights-runtime-extractor-tls podName:d9103d7e-3bea-4efc-ab49-8790e23e383a nodeName:}" failed. No retries permitted until 2026-07-02 21:20:26.473741759 +0000 UTC m=+98.916542160 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d9103d7e-3bea-4efc-ab49-8790e23e383a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dldh9" (UID: "d9103d7e-3bea-4efc-ab49-8790e23e383a") : secret "insights-runtime-extractor-tls" not found
Jul 02 21:20:10.474083 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:10.473939    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d9103d7e-3bea-4efc-ab49-8790e23e383a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dldh9\" (UID: \"d9103d7e-3bea-4efc-ab49-8790e23e383a\") " pod="openshift-insights/insights-runtime-extractor-dldh9"
Jul 02 21:20:20.340562 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.340527    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88bcd87c-vl24v"]
Jul 02 21:20:20.345467 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.345447    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88bcd87c-vl24v"
Jul 02 21:20:20.348149 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.348129    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69f78b9dc6-mvztz"]
Jul 02 21:20:20.351441 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.351420    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 02 21:20:20.351588 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.351472    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 02 21:20:20.351681 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.351487    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 02 21:20:20.351681 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.351497    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 02 21:20:20.370684 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.370660    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88bcd87c-vl24v"]
Jul 02 21:20:20.444833 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.444724    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5cc4899c7b-qpxzm"]
Jul 02 21:20:20.449023 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.448997    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cc4899c7b-qpxzm"
Jul 02 21:20:20.452713 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.452690    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkwsq\" (UniqueName: \"kubernetes.io/projected/3d6825f1-0e44-4318-b8b6-885c93a68a8f-kube-api-access-qkwsq\") pod \"klusterlet-addon-workmgr-5b88bcd87c-vl24v\" (UID: \"3d6825f1-0e44-4318-b8b6-885c93a68a8f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88bcd87c-vl24v"
Jul 02 21:20:20.452832 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.452731    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d6825f1-0e44-4318-b8b6-885c93a68a8f-tmp\") pod \"klusterlet-addon-workmgr-5b88bcd87c-vl24v\" (UID: \"3d6825f1-0e44-4318-b8b6-885c93a68a8f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88bcd87c-vl24v"
Jul 02 21:20:20.452832 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.452824    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3d6825f1-0e44-4318-b8b6-885c93a68a8f-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b88bcd87c-vl24v\" (UID: \"3d6825f1-0e44-4318-b8b6-885c93a68a8f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88bcd87c-vl24v"
Jul 02 21:20:20.491355 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.491330    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cc4899c7b-qpxzm"]
Jul 02 21:20:20.553950 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.553914    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mw6p\" (UniqueName: \"kubernetes.io/projected/2ee94142-64e7-46b3-a91a-e362146cc451-kube-api-access-6mw6p\") pod \"image-registry-5cc4899c7b-qpxzm\" (UID: \"2ee94142-64e7-46b3-a91a-e362146cc451\") " pod="openshift-image-registry/image-registry-5cc4899c7b-qpxzm"
Jul 02 21:20:20.554147 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.553968    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2ee94142-64e7-46b3-a91a-e362146cc451-registry-certificates\") pod \"image-registry-5cc4899c7b-qpxzm\" (UID: \"2ee94142-64e7-46b3-a91a-e362146cc451\") " pod="openshift-image-registry/image-registry-5cc4899c7b-qpxzm"
Jul 02 21:20:20.554147 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.554028    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkwsq\" (UniqueName: \"kubernetes.io/projected/3d6825f1-0e44-4318-b8b6-885c93a68a8f-kube-api-access-qkwsq\") pod \"klusterlet-addon-workmgr-5b88bcd87c-vl24v\" (UID: \"3d6825f1-0e44-4318-b8b6-885c93a68a8f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88bcd87c-vl24v"
Jul 02 21:20:20.554147 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.554051    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2ee94142-64e7-46b3-a91a-e362146cc451-trusted-ca\") pod \"image-registry-5cc4899c7b-qpxzm\" (UID: \"2ee94142-64e7-46b3-a91a-e362146cc451\") " pod="openshift-image-registry/image-registry-5cc4899c7b-qpxzm"
Jul 02 21:20:20.554147 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.554083    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d6825f1-0e44-4318-b8b6-885c93a68a8f-tmp\") pod \"klusterlet-addon-workmgr-5b88bcd87c-vl24v\" (UID: \"3d6825f1-0e44-4318-b8b6-885c93a68a8f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88bcd87c-vl24v"
Jul 02 21:20:20.554147 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.554107    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2ee94142-64e7-46b3-a91a-e362146cc451-ca-trust-extracted\") pod \"image-registry-5cc4899c7b-qpxzm\" (UID: \"2ee94142-64e7-46b3-a91a-e362146cc451\") " pod="openshift-image-registry/image-registry-5cc4899c7b-qpxzm"
Jul 02 21:20:20.554326 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.554199    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ee94142-64e7-46b3-a91a-e362146cc451-bound-sa-token\") pod \"image-registry-5cc4899c7b-qpxzm\" (UID: \"2ee94142-64e7-46b3-a91a-e362146cc451\") " pod="openshift-image-registry/image-registry-5cc4899c7b-qpxzm"
Jul 02 21:20:20.554326 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.554230    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3d6825f1-0e44-4318-b8b6-885c93a68a8f-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b88bcd87c-vl24v\" (UID: \"3d6825f1-0e44-4318-b8b6-885c93a68a8f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88bcd87c-vl24v"
Jul 02 21:20:20.554326 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.554247    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ee94142-64e7-46b3-a91a-e362146cc451-registry-tls\") pod \"image-registry-5cc4899c7b-qpxzm\" (UID: \"2ee94142-64e7-46b3-a91a-e362146cc451\") " pod="openshift-image-registry/image-registry-5cc4899c7b-qpxzm"
Jul 02 21:20:20.554477 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.554390    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2ee94142-64e7-46b3-a91a-e362146cc451-image-registry-private-configuration\") pod \"image-registry-5cc4899c7b-qpxzm\" (UID: \"2ee94142-64e7-46b3-a91a-e362146cc451\") " pod="openshift-image-registry/image-registry-5cc4899c7b-qpxzm"
Jul 02 21:20:20.554477 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.554432    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2ee94142-64e7-46b3-a91a-e362146cc451-installation-pull-secrets\") pod \"image-registry-5cc4899c7b-qpxzm\" (UID: \"2ee94142-64e7-46b3-a91a-e362146cc451\") " pod="openshift-image-registry/image-registry-5cc4899c7b-qpxzm"
Jul 02 21:20:20.554477 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.554456    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d6825f1-0e44-4318-b8b6-885c93a68a8f-tmp\") pod \"klusterlet-addon-workmgr-5b88bcd87c-vl24v\" (UID: \"3d6825f1-0e44-4318-b8b6-885c93a68a8f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88bcd87c-vl24v"
Jul 02 21:20:20.556588 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.556566    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3d6825f1-0e44-4318-b8b6-885c93a68a8f-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b88bcd87c-vl24v\" (UID: \"3d6825f1-0e44-4318-b8b6-885c93a68a8f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88bcd87c-vl24v"
Jul 02 21:20:20.565231 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.565199    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkwsq\" (UniqueName: \"kubernetes.io/projected/3d6825f1-0e44-4318-b8b6-885c93a68a8f-kube-api-access-qkwsq\") pod \"klusterlet-addon-workmgr-5b88bcd87c-vl24v\" (UID: \"3d6825f1-0e44-4318-b8b6-885c93a68a8f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88bcd87c-vl24v"
Jul 02 21:20:20.655142 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.655047    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88bcd87c-vl24v"
Jul 02 21:20:20.655296 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.655199    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2ee94142-64e7-46b3-a91a-e362146cc451-installation-pull-secrets\") pod \"image-registry-5cc4899c7b-qpxzm\" (UID: \"2ee94142-64e7-46b3-a91a-e362146cc451\") " pod="openshift-image-registry/image-registry-5cc4899c7b-qpxzm"
Jul 02 21:20:20.655296 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.655242    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mw6p\" (UniqueName: \"kubernetes.io/projected/2ee94142-64e7-46b3-a91a-e362146cc451-kube-api-access-6mw6p\") pod \"image-registry-5cc4899c7b-qpxzm\" (UID: \"2ee94142-64e7-46b3-a91a-e362146cc451\") " pod="openshift-image-registry/image-registry-5cc4899c7b-qpxzm"
Jul 02 21:20:20.655296 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.655267    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2ee94142-64e7-46b3-a91a-e362146cc451-registry-certificates\") pod \"image-registry-5cc4899c7b-qpxzm\" (UID: \"2ee94142-64e7-46b3-a91a-e362146cc451\") " pod="openshift-image-registry/image-registry-5cc4899c7b-qpxzm"
Jul 02 21:20:20.655454 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.655305    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2ee94142-64e7-46b3-a91a-e362146cc451-trusted-ca\") pod \"image-registry-5cc4899c7b-qpxzm\" (UID: \"2ee94142-64e7-46b3-a91a-e362146cc451\") " pod="openshift-image-registry/image-registry-5cc4899c7b-qpxzm"
Jul 02 21:20:20.655454 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.655329    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2ee94142-64e7-46b3-a91a-e362146cc451-ca-trust-extracted\") pod \"image-registry-5cc4899c7b-qpxzm\" (UID: \"2ee94142-64e7-46b3-a91a-e362146cc451\") " pod="openshift-image-registry/image-registry-5cc4899c7b-qpxzm"
Jul 02 21:20:20.655454 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.655373    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ee94142-64e7-46b3-a91a-e362146cc451-bound-sa-token\") pod \"image-registry-5cc4899c7b-qpxzm\" (UID: \"2ee94142-64e7-46b3-a91a-e362146cc451\") " pod="openshift-image-registry/image-registry-5cc4899c7b-qpxzm"
Jul 02 21:20:20.655454 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.655397    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ee94142-64e7-46b3-a91a-e362146cc451-registry-tls\") pod \"image-registry-5cc4899c7b-qpxzm\" (UID: \"2ee94142-64e7-46b3-a91a-e362146cc451\") " pod="openshift-image-registry/image-registry-5cc4899c7b-qpxzm"
Jul 02 21:20:20.655674 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.655595    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2ee94142-64e7-46b3-a91a-e362146cc451-image-registry-private-configuration\") pod \"image-registry-5cc4899c7b-qpxzm\" (UID: \"2ee94142-64e7-46b3-a91a-e362146cc451\") " pod="openshift-image-registry/image-registry-5cc4899c7b-qpxzm"
Jul 02 21:20:20.655822 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.655799    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2ee94142-64e7-46b3-a91a-e362146cc451-ca-trust-extracted\") pod \"image-registry-5cc4899c7b-qpxzm\" (UID: \"2ee94142-64e7-46b3-a91a-e362146cc451\") " pod="openshift-image-registry/image-registry-5cc4899c7b-qpxzm"
Jul 02 21:20:20.656210 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.656190    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2ee94142-64e7-46b3-a91a-e362146cc451-registry-certificates\") pod \"image-registry-5cc4899c7b-qpxzm\" (UID: \"2ee94142-64e7-46b3-a91a-e362146cc451\") " pod="openshift-image-registry/image-registry-5cc4899c7b-qpxzm"
Jul 02 21:20:20.656566 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.656528    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2ee94142-64e7-46b3-a91a-e362146cc451-trusted-ca\") pod \"image-registry-5cc4899c7b-qpxzm\" (UID: \"2ee94142-64e7-46b3-a91a-e362146cc451\") " pod="openshift-image-registry/image-registry-5cc4899c7b-qpxzm"
Jul 02 21:20:20.658032 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.658010    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ee94142-64e7-46b3-a91a-e362146cc451-registry-tls\") pod \"image-registry-5cc4899c7b-qpxzm\" (UID: \"2ee94142-64e7-46b3-a91a-e362146cc451\") " pod="openshift-image-registry/image-registry-5cc4899c7b-qpxzm"
Jul 02 21:20:20.658128 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.658034    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2ee94142-64e7-46b3-a91a-e362146cc451-installation-pull-secrets\") pod \"image-registry-5cc4899c7b-qpxzm\" (UID: \"2ee94142-64e7-46b3-a91a-e362146cc451\") " pod="openshift-image-registry/image-registry-5cc4899c7b-qpxzm"
Jul 02 21:20:20.658177 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.658149    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2ee94142-64e7-46b3-a91a-e362146cc451-image-registry-private-configuration\") pod \"image-registry-5cc4899c7b-qpxzm\" (UID: \"2ee94142-64e7-46b3-a91a-e362146cc451\") " pod="openshift-image-registry/image-registry-5cc4899c7b-qpxzm"
Jul 02 21:20:20.672064 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.672044    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ee94142-64e7-46b3-a91a-e362146cc451-bound-sa-token\") pod \"image-registry-5cc4899c7b-qpxzm\" (UID: \"2ee94142-64e7-46b3-a91a-e362146cc451\") " pod="openshift-image-registry/image-registry-5cc4899c7b-qpxzm"
Jul 02 21:20:20.672310 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.672293    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mw6p\" (UniqueName: \"kubernetes.io/projected/2ee94142-64e7-46b3-a91a-e362146cc451-kube-api-access-6mw6p\") pod \"image-registry-5cc4899c7b-qpxzm\" (UID: \"2ee94142-64e7-46b3-a91a-e362146cc451\") " pod="openshift-image-registry/image-registry-5cc4899c7b-qpxzm"
Jul 02 21:20:20.758280 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.758251    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cc4899c7b-qpxzm"
Jul 02 21:20:20.804425 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:20:20.804394    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d6825f1_0e44_4318_b8b6_885c93a68a8f.slice/crio-82eec93db9b6b9a430907e39b162a2345927539a8d16df24daf054315933cf86 WatchSource:0}: Error finding container 82eec93db9b6b9a430907e39b162a2345927539a8d16df24daf054315933cf86: Status 404 returned error can't find the container with id 82eec93db9b6b9a430907e39b162a2345927539a8d16df24daf054315933cf86
Jul 02 21:20:20.804609 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.804548    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88bcd87c-vl24v"]
Jul 02 21:20:20.923595 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:20.923511    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cc4899c7b-qpxzm"]
Jul 02 21:20:20.926436 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:20:20.926408    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ee94142_64e7_46b3_a91a_e362146cc451.slice/crio-b19420aa50613ece6cf48063cf71c50abe12cc19d4f45662e5c11578e8c8e24f WatchSource:0}: Error finding container b19420aa50613ece6cf48063cf71c50abe12cc19d4f45662e5c11578e8c8e24f: Status 404 returned error can't find the container with id b19420aa50613ece6cf48063cf71c50abe12cc19d4f45662e5c11578e8c8e24f
Jul 02 21:20:21.380377 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:21.380294    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cc4899c7b-qpxzm" event={"ID":"2ee94142-64e7-46b3-a91a-e362146cc451","Type":"ContainerStarted","Data":"3106fa16943c46a099f093b0c90832df3805d2a3842a4cd098153d3c777ae5a2"}
Jul 02 21:20:21.380905 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:21.380545    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5cc4899c7b-qpxzm"
Jul 02 21:20:21.380905 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:21.380587    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cc4899c7b-qpxzm" event={"ID":"2ee94142-64e7-46b3-a91a-e362146cc451","Type":"ContainerStarted","Data":"b19420aa50613ece6cf48063cf71c50abe12cc19d4f45662e5c11578e8c8e24f"}
Jul 02 21:20:21.381894 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:21.381857    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88bcd87c-vl24v" event={"ID":"3d6825f1-0e44-4318-b8b6-885c93a68a8f","Type":"ContainerStarted","Data":"82eec93db9b6b9a430907e39b162a2345927539a8d16df24daf054315933cf86"}
Jul 02 21:20:21.444362 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:21.444296    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5cc4899c7b-qpxzm" podStartSLOduration=1.444281017 podStartE2EDuration="1.444281017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:20:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:21.44026873 +0000 UTC m=+93.883069153" watchObservedRunningTime="2026-07-02 21:20:21.444281017 +0000 UTC m=+93.887081436"
Jul 02 21:20:22.576685 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:22.576619    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b321922d-a68a-43a5-89bb-fd49e22371c3-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-cr85g\" (UID: \"b321922d-a68a-43a5-89bb-fd49e22371c3\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-cr85g"
Jul 02 21:20:22.579243 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:22.579218    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b321922d-a68a-43a5-89bb-fd49e22371c3-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-cr85g\" (UID: \"b321922d-a68a-43a5-89bb-fd49e22371c3\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-cr85g"
Jul 02 21:20:22.777863 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:22.777804    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-cr85g"
Jul 02 21:20:22.942599 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:22.942566    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-cr85g"]
Jul 02 21:20:24.030968 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:20:24.030924    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb321922d_a68a_43a5_89bb_fd49e22371c3.slice/crio-e0ebc93b2b34e324642cbca1b3b8f32f488370a0f432e3d0675e43f5f4282ad3 WatchSource:0}: Error finding container e0ebc93b2b34e324642cbca1b3b8f32f488370a0f432e3d0675e43f5f4282ad3: Status 404 returned error can't find the container with id e0ebc93b2b34e324642cbca1b3b8f32f488370a0f432e3d0675e43f5f4282ad3
Jul 02 21:20:24.394418 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:24.394325    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88bcd87c-vl24v" event={"ID":"3d6825f1-0e44-4318-b8b6-885c93a68a8f","Type":"ContainerStarted","Data":"1973bf29f19169ae1dd88735573486effaa389f05087a1401266102ce4f84874"}
Jul 02 21:20:24.394571 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:24.394539    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88bcd87c-vl24v"
Jul 02 21:20:24.395593 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:24.395567    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-cr85g" event={"ID":"b321922d-a68a-43a5-89bb-fd49e22371c3","Type":"ContainerStarted","Data":"e0ebc93b2b34e324642cbca1b3b8f32f488370a0f432e3d0675e43f5f4282ad3"}
Jul 02 21:20:24.396255 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:24.396234    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88bcd87c-vl24v"
Jul 02 21:20:24.438736 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:24.438681    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88bcd87c-vl24v" podStartSLOduration=1.149973865 podStartE2EDuration="4.438662627s" totalImagesPullingTime="3.288688762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:20:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:24.415960659 +0000 UTC m=+96.858761092" watchObservedRunningTime="2026-07-02 21:20:24.438662627 +0000 UTC m=+96.881463040"
Jul 02 21:20:25.400595 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:25.400547    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-cr85g" event={"ID":"b321922d-a68a-43a5-89bb-fd49e22371c3","Type":"ContainerStarted","Data":"15ef9cf2741997f7d95ecfc6e0fb07be4f28da083ded500d72c2c038ec3eccab"}
Jul 02 21:20:25.419397 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:25.419344    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-cr85g" podStartSLOduration=34.274744034 podStartE2EDuration="35.419328416s" totalImagesPullingTime="1.144584382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:19:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:25.418724854 +0000 UTC m=+97.861525278" watchObservedRunningTime="2026-07-02 21:20:25.419328416 +0000 UTC m=+97.862128838"
Jul 02 21:20:26.310322 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:26.310270    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2c34410-8c0a-4d13-8864-575ba506997d-metrics-tls\") pod \"dns-default-w5w24\" (UID: \"d2c34410-8c0a-4d13-8864-575ba506997d\") " pod="openshift-dns/dns-default-w5w24"
Jul 02 21:20:26.312615 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:26.312594    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2c34410-8c0a-4d13-8864-575ba506997d-metrics-tls\") pod \"dns-default-w5w24\" (UID: \"d2c34410-8c0a-4d13-8864-575ba506997d\") " pod="openshift-dns/dns-default-w5w24"
Jul 02 21:20:26.320143 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:26.320114    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sklqf\""
Jul 02 21:20:26.327336 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:26.327310    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w5w24"
Jul 02 21:20:26.473948 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:26.473914    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w5w24"]
Jul 02 21:20:26.477280 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:20:26.477244    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2c34410_8c0a_4d13_8864_575ba506997d.slice/crio-a85fc0e79373370ce6427e58650383effb3896d4e9d88c900bb87c743f844e52 WatchSource:0}: Error finding container a85fc0e79373370ce6427e58650383effb3896d4e9d88c900bb87c743f844e52: Status 404 returned error can't find the container with id a85fc0e79373370ce6427e58650383effb3896d4e9d88c900bb87c743f844e52
Jul 02 21:20:26.511673 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:26.511625    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9103d7e-3bea-4efc-ab49-8790e23e383a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dldh9\" (UID: \"d9103d7e-3bea-4efc-ab49-8790e23e383a\") " pod="openshift-insights/insights-runtime-extractor-dldh9"
Jul 02 21:20:26.514033 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:26.513997    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9103d7e-3bea-4efc-ab49-8790e23e383a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dldh9\" (UID: \"d9103d7e-3bea-4efc-ab49-8790e23e383a\") " pod="openshift-insights/insights-runtime-extractor-dldh9"
Jul 02 21:20:26.756828 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:26.756798    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dldh9"
Jul 02 21:20:26.887183 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:26.887161    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dldh9"]
Jul 02 21:20:26.889388 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:20:26.889362    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9103d7e_3bea_4efc_ab49_8790e23e383a.slice/crio-305d57d1803652c451d438c482a9782c68da47f3c750d9ed4534ed322eca6e95 WatchSource:0}: Error finding container 305d57d1803652c451d438c482a9782c68da47f3c750d9ed4534ed322eca6e95: Status 404 returned error can't find the container with id 305d57d1803652c451d438c482a9782c68da47f3c750d9ed4534ed322eca6e95
Jul 02 21:20:27.411218 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:27.411177    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dldh9" event={"ID":"d9103d7e-3bea-4efc-ab49-8790e23e383a","Type":"ContainerStarted","Data":"305d57d1803652c451d438c482a9782c68da47f3c750d9ed4534ed322eca6e95"}
Jul 02 21:20:27.412858 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:27.412828    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w5w24" event={"ID":"d2c34410-8c0a-4d13-8864-575ba506997d","Type":"ContainerStarted","Data":"a85fc0e79373370ce6427e58650383effb3896d4e9d88c900bb87c743f844e52"}
Jul 02 21:20:29.421467 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:29.421431    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w5w24" event={"ID":"d2c34410-8c0a-4d13-8864-575ba506997d","Type":"ContainerStarted","Data":"f74553c4b01f010021898142bdf6b3c2f315baf6403616efe60b242c932a8289"}
Jul 02 21:20:30.313429 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:30.313399    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-d495m"
Jul 02 21:20:30.354671 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:30.354619    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz"
Jul 02 21:20:31.433490 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:31.433450    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dldh9" event={"ID":"d9103d7e-3bea-4efc-ab49-8790e23e383a","Type":"ContainerStarted","Data":"a94b33fceb78bcb157e79a7e0b47a0ba73545298edb29dcaf0e190ffe7a8daca"}
Jul 02 21:20:31.435467 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:31.435428    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w5w24" event={"ID":"d2c34410-8c0a-4d13-8864-575ba506997d","Type":"ContainerStarted","Data":"20375c0e86e5ac6b22c467c8b8a8e9b525f202f06554acb44044c6b0d9861bbf"}
Jul 02 21:20:31.435642 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:31.435625    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-w5w24"
Jul 02 21:20:31.457761 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:31.457716    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-w5w24" podStartSLOduration=67.505043312 podStartE2EDuration="1m9.457703429s" totalImagesPullingTime="1.952660117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:19:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:31.455899296 +0000 UTC m=+103.898699730" watchObservedRunningTime="2026-07-02 21:20:31.457703429 +0000 UTC m=+103.900503841"
Jul 02 21:20:32.441456 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:32.441415    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dldh9" event={"ID":"d9103d7e-3bea-4efc-ab49-8790e23e383a","Type":"ContainerStarted","Data":"511d43717819560f0d0f27100d98a29ee82eba29f213eaf9d1e16c1e7767c278"}
Jul 02 21:20:34.786774 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.786696    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dbl59"]
Jul 02 21:20:34.789853 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.789832    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dbl59"
Jul 02 21:20:34.793288 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.793261    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 02 21:20:34.794814 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.794791    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-gfqn2\""
Jul 02 21:20:34.794937 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.794842    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 02 21:20:34.794937 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.794797    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 02 21:20:34.795051 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.795004    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 02 21:20:34.795106 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.795069    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 02 21:20:34.795106 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.795082    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 02 21:20:34.801011 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.800989    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hlq5v"]
Jul 02 21:20:34.804188 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.804162    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hlq5v"
Jul 02 21:20:34.804920 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.804898    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dbl59"]
Jul 02 21:20:34.808322 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.808299    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 02 21:20:34.808619 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.808358    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 02 21:20:34.808619 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.808381    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 02 21:20:34.809575 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.809501    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-np8j8\""
Jul 02 21:20:34.887246 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.887210    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/935457b1-cbad-4c83-99be-3cf040a6cfce-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-dbl59\" (UID: \"935457b1-cbad-4c83-99be-3cf040a6cfce\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dbl59"
Jul 02 21:20:34.887430 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.887254    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35f97721-70f8-4917-a619-e2f290712290-metrics-client-ca\") pod \"node-exporter-hlq5v\" (UID: \"35f97721-70f8-4917-a619-e2f290712290\") " pod="openshift-monitoring/node-exporter-hlq5v"
Jul 02 21:20:34.887430 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.887280    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/935457b1-cbad-4c83-99be-3cf040a6cfce-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-dbl59\" (UID: \"935457b1-cbad-4c83-99be-3cf040a6cfce\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dbl59"
Jul 02 21:20:34.887430 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.887319    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/35f97721-70f8-4917-a619-e2f290712290-root\") pod \"node-exporter-hlq5v\" (UID: \"35f97721-70f8-4917-a619-e2f290712290\") " pod="openshift-monitoring/node-exporter-hlq5v"
Jul 02 21:20:34.887430 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.887353    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlcwf\" (UniqueName: \"kubernetes.io/projected/935457b1-cbad-4c83-99be-3cf040a6cfce-kube-api-access-jlcwf\") pod \"kube-state-metrics-7c9dbb8b7d-dbl59\" (UID: \"935457b1-cbad-4c83-99be-3cf040a6cfce\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dbl59"
Jul 02 21:20:34.887430 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.887373    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/935457b1-cbad-4c83-99be-3cf040a6cfce-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-dbl59\" (UID: \"935457b1-cbad-4c83-99be-3cf040a6cfce\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dbl59"
Jul 02 21:20:34.887430 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.887414    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/935457b1-cbad-4c83-99be-3cf040a6cfce-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-dbl59\" (UID: \"935457b1-cbad-4c83-99be-3cf040a6cfce\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dbl59"
Jul 02 21:20:34.887769 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.887440    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35f97721-70f8-4917-a619-e2f290712290-sys\") pod \"node-exporter-hlq5v\" (UID: \"35f97721-70f8-4917-a619-e2f290712290\") " pod="openshift-monitoring/node-exporter-hlq5v"
Jul 02 21:20:34.887769 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.887472    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/935457b1-cbad-4c83-99be-3cf040a6cfce-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-dbl59\" (UID: \"935457b1-cbad-4c83-99be-3cf040a6cfce\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dbl59"
Jul 02 21:20:34.887769 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.887497    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/35f97721-70f8-4917-a619-e2f290712290-node-exporter-textfile\") pod \"node-exporter-hlq5v\" (UID: \"35f97721-70f8-4917-a619-e2f290712290\") " pod="openshift-monitoring/node-exporter-hlq5v"
Jul 02 21:20:34.887769 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.887519    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35f97721-70f8-4917-a619-e2f290712290-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hlq5v\" (UID: \"35f97721-70f8-4917-a619-e2f290712290\") " pod="openshift-monitoring/node-exporter-hlq5v"
Jul 02 21:20:34.887769 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.887549    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/35f97721-70f8-4917-a619-e2f290712290-node-exporter-accelerators-collector-config\") pod \"node-exporter-hlq5v\" (UID: \"35f97721-70f8-4917-a619-e2f290712290\") " pod="openshift-monitoring/node-exporter-hlq5v"
Jul 02 21:20:34.887769 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.887572    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s89gx\" (UniqueName: \"kubernetes.io/projected/35f97721-70f8-4917-a619-e2f290712290-kube-api-access-s89gx\") pod \"node-exporter-hlq5v\" (UID: \"35f97721-70f8-4917-a619-e2f290712290\") " pod="openshift-monitoring/node-exporter-hlq5v"
Jul 02 21:20:34.887769 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.887609    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/35f97721-70f8-4917-a619-e2f290712290-node-exporter-wtmp\") pod \"node-exporter-hlq5v\" (UID: \"35f97721-70f8-4917-a619-e2f290712290\") " pod="openshift-monitoring/node-exporter-hlq5v"
Jul 02 21:20:34.887769 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.887662    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/35f97721-70f8-4917-a619-e2f290712290-node-exporter-tls\") pod \"node-exporter-hlq5v\" (UID: \"35f97721-70f8-4917-a619-e2f290712290\") " pod="openshift-monitoring/node-exporter-hlq5v"
Jul 02 21:20:34.988665 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.988621    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/35f97721-70f8-4917-a619-e2f290712290-node-exporter-tls\") pod \"node-exporter-hlq5v\" (UID: \"35f97721-70f8-4917-a619-e2f290712290\") " pod="openshift-monitoring/node-exporter-hlq5v"
Jul 02 21:20:34.988854 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.988721    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/935457b1-cbad-4c83-99be-3cf040a6cfce-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-dbl59\" (UID: \"935457b1-cbad-4c83-99be-3cf040a6cfce\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dbl59"
Jul 02 21:20:34.988854 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.988747    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35f97721-70f8-4917-a619-e2f290712290-metrics-client-ca\") pod \"node-exporter-hlq5v\" (UID: \"35f97721-70f8-4917-a619-e2f290712290\") " pod="openshift-monitoring/node-exporter-hlq5v"
Jul 02 21:20:34.988854 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.988767    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/935457b1-cbad-4c83-99be-3cf040a6cfce-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-dbl59\" (UID: \"935457b1-cbad-4c83-99be-3cf040a6cfce\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dbl59"
Jul 02 21:20:34.988854 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.988785    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/35f97721-70f8-4917-a619-e2f290712290-root\") pod \"node-exporter-hlq5v\" (UID: \"35f97721-70f8-4917-a619-e2f290712290\") " pod="openshift-monitoring/node-exporter-hlq5v"
Jul 02 21:20:34.988854 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.988808    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlcwf\" (UniqueName: \"kubernetes.io/projected/935457b1-cbad-4c83-99be-3cf040a6cfce-kube-api-access-jlcwf\") pod \"kube-state-metrics-7c9dbb8b7d-dbl59\" (UID: \"935457b1-cbad-4c83-99be-3cf040a6cfce\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dbl59"
Jul 02 21:20:34.988854 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.988827    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/935457b1-cbad-4c83-99be-3cf040a6cfce-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-dbl59\" (UID: \"935457b1-cbad-4c83-99be-3cf040a6cfce\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dbl59"
Jul 02 21:20:34.988854 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.988852    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/935457b1-cbad-4c83-99be-3cf040a6cfce-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-dbl59\" (UID: \"935457b1-cbad-4c83-99be-3cf040a6cfce\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dbl59"
Jul 02 21:20:34.989208 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.988875    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35f97721-70f8-4917-a619-e2f290712290-sys\") pod \"node-exporter-hlq5v\" (UID: \"35f97721-70f8-4917-a619-e2f290712290\") " pod="openshift-monitoring/node-exporter-hlq5v"
Jul 02 21:20:34.989208 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.988903    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/935457b1-cbad-4c83-99be-3cf040a6cfce-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-dbl59\" (UID: \"935457b1-cbad-4c83-99be-3cf040a6cfce\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dbl59"
Jul 02 21:20:34.989208 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.988904    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/35f97721-70f8-4917-a619-e2f290712290-root\") pod \"node-exporter-hlq5v\" (UID: \"35f97721-70f8-4917-a619-e2f290712290\") " pod="openshift-monitoring/node-exporter-hlq5v"
Jul 02 21:20:34.989208 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.988927    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/35f97721-70f8-4917-a619-e2f290712290-node-exporter-textfile\") pod \"node-exporter-hlq5v\" (UID: \"35f97721-70f8-4917-a619-e2f290712290\") " pod="openshift-monitoring/node-exporter-hlq5v"
Jul 02 21:20:34.989208 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.988971    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35f97721-70f8-4917-a619-e2f290712290-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hlq5v\" (UID: \"35f97721-70f8-4917-a619-e2f290712290\") " pod="openshift-monitoring/node-exporter-hlq5v"
Jul 02 21:20:34.989208 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.989008    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/35f97721-70f8-4917-a619-e2f290712290-node-exporter-accelerators-collector-config\") pod \"node-exporter-hlq5v\" (UID: \"35f97721-70f8-4917-a619-e2f290712290\") " pod="openshift-monitoring/node-exporter-hlq5v"
Jul 02 21:20:34.989208 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.989038    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s89gx\" (UniqueName: \"kubernetes.io/projected/35f97721-70f8-4917-a619-e2f290712290-kube-api-access-s89gx\") pod \"node-exporter-hlq5v\" (UID: \"35f97721-70f8-4917-a619-e2f290712290\") " pod="openshift-monitoring/node-exporter-hlq5v"
Jul 02 21:20:34.989208 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.989082    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/35f97721-70f8-4917-a619-e2f290712290-node-exporter-wtmp\") pod \"node-exporter-hlq5v\" (UID: \"35f97721-70f8-4917-a619-e2f290712290\") " pod="openshift-monitoring/node-exporter-hlq5v"
Jul 02 21:20:34.989208 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.989180    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35f97721-70f8-4917-a619-e2f290712290-sys\") pod \"node-exporter-hlq5v\" (UID: \"35f97721-70f8-4917-a619-e2f290712290\") " pod="openshift-monitoring/node-exporter-hlq5v"
Jul 02 21:20:34.989208 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.989204    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/35f97721-70f8-4917-a619-e2f290712290-node-exporter-textfile\") pod \"node-exporter-hlq5v\" (UID: \"35f97721-70f8-4917-a619-e2f290712290\") " pod="openshift-monitoring/node-exporter-hlq5v"
Jul 02 21:20:34.989730 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.989213    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/935457b1-cbad-4c83-99be-3cf040a6cfce-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-dbl59\" (UID: \"935457b1-cbad-4c83-99be-3cf040a6cfce\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dbl59"
Jul 02 21:20:34.989730 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.989258    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/35f97721-70f8-4917-a619-e2f290712290-node-exporter-wtmp\") pod \"node-exporter-hlq5v\" (UID: \"35f97721-70f8-4917-a619-e2f290712290\") " pod="openshift-monitoring/node-exporter-hlq5v"
Jul 02 21:20:34.989843 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.989746    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/35f97721-70f8-4917-a619-e2f290712290-node-exporter-accelerators-collector-config\") pod \"node-exporter-hlq5v\" (UID: \"35f97721-70f8-4917-a619-e2f290712290\") " pod="openshift-monitoring/node-exporter-hlq5v"
Jul 02 21:20:34.989843 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.989804    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/935457b1-cbad-4c83-99be-3cf040a6cfce-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-dbl59\" (UID: \"935457b1-cbad-4c83-99be-3cf040a6cfce\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dbl59"
Jul 02 21:20:34.989954 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.989935    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/935457b1-cbad-4c83-99be-3cf040a6cfce-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-dbl59\" (UID: \"935457b1-cbad-4c83-99be-3cf040a6cfce\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dbl59"
Jul 02 21:20:34.990164 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.990124    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35f97721-70f8-4917-a619-e2f290712290-metrics-client-ca\") pod \"node-exporter-hlq5v\" (UID: \"35f97721-70f8-4917-a619-e2f290712290\") " pod="openshift-monitoring/node-exporter-hlq5v"
Jul 02 21:20:34.991418 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.991393    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/35f97721-70f8-4917-a619-e2f290712290-node-exporter-tls\") pod \"node-exporter-hlq5v\" (UID: \"35f97721-70f8-4917-a619-e2f290712290\") " pod="openshift-monitoring/node-exporter-hlq5v"
Jul 02 21:20:34.991728 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.991675    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/935457b1-cbad-4c83-99be-3cf040a6cfce-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-dbl59\" (UID: \"935457b1-cbad-4c83-99be-3cf040a6cfce\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dbl59"
Jul 02 21:20:34.991873 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.991851    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/935457b1-cbad-4c83-99be-3cf040a6cfce-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-dbl59\" (UID: \"935457b1-cbad-4c83-99be-3cf040a6cfce\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dbl59"
Jul 02 21:20:34.991965 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:34.991949    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35f97721-70f8-4917-a619-e2f290712290-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hlq5v\" (UID: \"35f97721-70f8-4917-a619-e2f290712290\") " pod="openshift-monitoring/node-exporter-hlq5v"
Jul 02 21:20:35.000464 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:35.000441    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s89gx\" (UniqueName: \"kubernetes.io/projected/35f97721-70f8-4917-a619-e2f290712290-kube-api-access-s89gx\") pod \"node-exporter-hlq5v\" (UID: \"35f97721-70f8-4917-a619-e2f290712290\") " pod="openshift-monitoring/node-exporter-hlq5v"
Jul 02 21:20:35.000572 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:35.000441    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlcwf\" (UniqueName: \"kubernetes.io/projected/935457b1-cbad-4c83-99be-3cf040a6cfce-kube-api-access-jlcwf\") pod \"kube-state-metrics-7c9dbb8b7d-dbl59\" (UID: \"935457b1-cbad-4c83-99be-3cf040a6cfce\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dbl59"
Jul 02 21:20:35.099936 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:35.099850    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dbl59"
Jul 02 21:20:35.114486 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:35.114456    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hlq5v"
Jul 02 21:20:35.125166 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:20:35.125133    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35f97721_70f8_4917_a619_e2f290712290.slice/crio-70fb09ea3483a47d9cc8892498faf6e00c752bcec6d1e69660a5ef9cc835cd24 WatchSource:0}: Error finding container 70fb09ea3483a47d9cc8892498faf6e00c752bcec6d1e69660a5ef9cc835cd24: Status 404 returned error can't find the container with id 70fb09ea3483a47d9cc8892498faf6e00c752bcec6d1e69660a5ef9cc835cd24
Jul 02 21:20:35.245726 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:35.245701    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dbl59"]
Jul 02 21:20:35.246345 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:20:35.246326    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod935457b1_cbad_4c83_99be_3cf040a6cfce.slice/crio-018b6839cf3321217d00e398aa06d1e4e6d65495fd8efb6963dcb31ab3f6abc6 WatchSource:0}: Error finding container 018b6839cf3321217d00e398aa06d1e4e6d65495fd8efb6963dcb31ab3f6abc6: Status 404 returned error can't find the container with id 018b6839cf3321217d00e398aa06d1e4e6d65495fd8efb6963dcb31ab3f6abc6
Jul 02 21:20:35.452625 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:35.452583    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hlq5v" event={"ID":"35f97721-70f8-4917-a619-e2f290712290","Type":"ContainerStarted","Data":"70fb09ea3483a47d9cc8892498faf6e00c752bcec6d1e69660a5ef9cc835cd24"}
Jul 02 21:20:35.454603 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:35.454578    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dldh9" event={"ID":"d9103d7e-3bea-4efc-ab49-8790e23e383a","Type":"ContainerStarted","Data":"0a44b5e80e481d1a1e598b32408816a2239e235eb9df8330e1b97a61a3606b2e"}
Jul 02 21:20:35.455769 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:35.455747    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dbl59" event={"ID":"935457b1-cbad-4c83-99be-3cf040a6cfce","Type":"ContainerStarted","Data":"018b6839cf3321217d00e398aa06d1e4e6d65495fd8efb6963dcb31ab3f6abc6"}
Jul 02 21:20:35.475392 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:35.475349    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dldh9" podStartSLOduration=34.059251125 podStartE2EDuration="41.475334377s" totalImagesPullingTime="7.416083252s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:19:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:35.474902119 +0000 UTC m=+107.917702541" watchObservedRunningTime="2026-07-02 21:20:35.475334377 +0000 UTC m=+107.918134816"
Jul 02 21:20:36.460750 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:36.460721    2586 generic.go:356] "Generic (PLEG): container finished" podID="35f97721-70f8-4917-a619-e2f290712290" containerID="fb56b231c1dbca7db791174b6778286335d1e8843dd1841c252f15ff346b2f67" exitCode=0
Jul 02 21:20:36.461146 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:36.460803    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hlq5v" event={"ID":"35f97721-70f8-4917-a619-e2f290712290","Type":"ContainerDied","Data":"fb56b231c1dbca7db791174b6778286335d1e8843dd1841c252f15ff346b2f67"}
Jul 02 21:20:37.466632 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:37.466596    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dbl59" event={"ID":"935457b1-cbad-4c83-99be-3cf040a6cfce","Type":"ContainerStarted","Data":"59142079f432a60d35934b9c0a51e756355732a60fe27984ac30e1de4333d2af"}
Jul 02 21:20:37.467074 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:37.466680    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dbl59" event={"ID":"935457b1-cbad-4c83-99be-3cf040a6cfce","Type":"ContainerStarted","Data":"b19c037ba0b9ce28f24d5b1698ffddb8e93cb8dcbf3f11189a60c8138c7096a5"}
Jul 02 21:20:37.467074 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:37.466711    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dbl59" event={"ID":"935457b1-cbad-4c83-99be-3cf040a6cfce","Type":"ContainerStarted","Data":"3f8e261d760ce765071728c367ef6f6c767faadf081b8233153436712f11d551"}
Jul 02 21:20:37.468768 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:37.468745    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hlq5v" event={"ID":"35f97721-70f8-4917-a619-e2f290712290","Type":"ContainerStarted","Data":"755f2e538c455dce1d7211e248617822a213ebc2fc860389ee634a16fd8cde98"}
Jul 02 21:20:37.468876 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:37.468790    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hlq5v" event={"ID":"35f97721-70f8-4917-a619-e2f290712290","Type":"ContainerStarted","Data":"0cee02e40179963952af914eb3297712fb04b8cbdf4860372be2da0493043184"}
Jul 02 21:20:37.491486 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:37.491441    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dbl59" podStartSLOduration=2.257196117 podStartE2EDuration="3.491427938s" totalImagesPullingTime="1.234231821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:37.489961464 +0000 UTC m=+109.932761886" watchObservedRunningTime="2026-07-02 21:20:37.491427938 +0000 UTC m=+109.934228359"
Jul 02 21:20:37.517488 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:37.517421    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hlq5v" podStartSLOduration=2.499805186 podStartE2EDuration="3.517403126s" totalImagesPullingTime="943.534462ms" totalInitContainerRuntime="74.063478ms" isStatefulPod=true podCreationTimestamp="2026-07-02 21:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:37.516418121 +0000 UTC m=+109.959218539" watchObservedRunningTime="2026-07-02 21:20:37.517403126 +0000 UTC m=+109.960203548"
Jul 02 21:20:38.444422 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:38.444393    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-w5w24"
Jul 02 21:20:39.100490 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.100450    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7b77b96467-7s78c"]
Jul 02 21:20:39.103529 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.103507    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7b77b96467-7s78c"
Jul 02 21:20:39.106681 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.106657    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 02 21:20:39.108109 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.108088    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 02 21:20:39.108253 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.108188    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-d3vcu9phi455g\""
Jul 02 21:20:39.108846 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.108823    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-768hv\""
Jul 02 21:20:39.108846 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.108840    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 02 21:20:39.109140 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.109119    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 02 21:20:39.115710 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.115684    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7b77b96467-7s78c"]
Jul 02 21:20:39.230396 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.230358    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/33aa0be6-449b-48aa-8f64-5f09346a08ad-metrics-server-audit-profiles\") pod \"metrics-server-7b77b96467-7s78c\" (UID: \"33aa0be6-449b-48aa-8f64-5f09346a08ad\") " pod="openshift-monitoring/metrics-server-7b77b96467-7s78c"
Jul 02 21:20:39.230580 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.230425    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/33aa0be6-449b-48aa-8f64-5f09346a08ad-client-ca-bundle\") pod \"metrics-server-7b77b96467-7s78c\" (UID: \"33aa0be6-449b-48aa-8f64-5f09346a08ad\") " pod="openshift-monitoring/metrics-server-7b77b96467-7s78c"
Jul 02 21:20:39.230580 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.230457    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/33aa0be6-449b-48aa-8f64-5f09346a08ad-secret-metrics-server-client-certs\") pod \"metrics-server-7b77b96467-7s78c\" (UID: \"33aa0be6-449b-48aa-8f64-5f09346a08ad\") " pod="openshift-monitoring/metrics-server-7b77b96467-7s78c"
Jul 02 21:20:39.230580 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.230492    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttlqk\" (UniqueName: \"kubernetes.io/projected/33aa0be6-449b-48aa-8f64-5f09346a08ad-kube-api-access-ttlqk\") pod \"metrics-server-7b77b96467-7s78c\" (UID: \"33aa0be6-449b-48aa-8f64-5f09346a08ad\") " pod="openshift-monitoring/metrics-server-7b77b96467-7s78c"
Jul 02 21:20:39.230580 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.230534    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33aa0be6-449b-48aa-8f64-5f09346a08ad-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b77b96467-7s78c\" (UID: \"33aa0be6-449b-48aa-8f64-5f09346a08ad\") " pod="openshift-monitoring/metrics-server-7b77b96467-7s78c"
Jul 02 21:20:39.230580 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.230552    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/33aa0be6-449b-48aa-8f64-5f09346a08ad-audit-log\") pod \"metrics-server-7b77b96467-7s78c\" (UID: \"33aa0be6-449b-48aa-8f64-5f09346a08ad\") " pod="openshift-monitoring/metrics-server-7b77b96467-7s78c"
Jul 02 21:20:39.230784 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.230691    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/33aa0be6-449b-48aa-8f64-5f09346a08ad-secret-metrics-server-tls\") pod \"metrics-server-7b77b96467-7s78c\" (UID: \"33aa0be6-449b-48aa-8f64-5f09346a08ad\") " pod="openshift-monitoring/metrics-server-7b77b96467-7s78c"
Jul 02 21:20:39.331892 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.331848    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33aa0be6-449b-48aa-8f64-5f09346a08ad-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b77b96467-7s78c\" (UID: \"33aa0be6-449b-48aa-8f64-5f09346a08ad\") " pod="openshift-monitoring/metrics-server-7b77b96467-7s78c"
Jul 02 21:20:39.331892 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.331890    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/33aa0be6-449b-48aa-8f64-5f09346a08ad-audit-log\") pod \"metrics-server-7b77b96467-7s78c\" (UID: \"33aa0be6-449b-48aa-8f64-5f09346a08ad\") " pod="openshift-monitoring/metrics-server-7b77b96467-7s78c"
Jul 02 21:20:39.332147 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.331949    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/33aa0be6-449b-48aa-8f64-5f09346a08ad-secret-metrics-server-tls\") pod \"metrics-server-7b77b96467-7s78c\" (UID: \"33aa0be6-449b-48aa-8f64-5f09346a08ad\") " pod="openshift-monitoring/metrics-server-7b77b96467-7s78c"
Jul 02 21:20:39.332147 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.331973    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/33aa0be6-449b-48aa-8f64-5f09346a08ad-metrics-server-audit-profiles\") pod \"metrics-server-7b77b96467-7s78c\" (UID: \"33aa0be6-449b-48aa-8f64-5f09346a08ad\") " pod="openshift-monitoring/metrics-server-7b77b96467-7s78c"
Jul 02 21:20:39.332147 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.332019    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/33aa0be6-449b-48aa-8f64-5f09346a08ad-client-ca-bundle\") pod \"metrics-server-7b77b96467-7s78c\" (UID: \"33aa0be6-449b-48aa-8f64-5f09346a08ad\") " pod="openshift-monitoring/metrics-server-7b77b96467-7s78c"
Jul 02 21:20:39.332147 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.332054    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/33aa0be6-449b-48aa-8f64-5f09346a08ad-secret-metrics-server-client-certs\") pod \"metrics-server-7b77b96467-7s78c\" (UID: \"33aa0be6-449b-48aa-8f64-5f09346a08ad\") " pod="openshift-monitoring/metrics-server-7b77b96467-7s78c"
Jul 02 21:20:39.332147 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.332107    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttlqk\" (UniqueName: \"kubernetes.io/projected/33aa0be6-449b-48aa-8f64-5f09346a08ad-kube-api-access-ttlqk\") pod \"metrics-server-7b77b96467-7s78c\" (UID: \"33aa0be6-449b-48aa-8f64-5f09346a08ad\") " pod="openshift-monitoring/metrics-server-7b77b96467-7s78c"
Jul 02 21:20:39.332517 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.332484    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/33aa0be6-449b-48aa-8f64-5f09346a08ad-audit-log\") pod \"metrics-server-7b77b96467-7s78c\" (UID: \"33aa0be6-449b-48aa-8f64-5f09346a08ad\") " pod="openshift-monitoring/metrics-server-7b77b96467-7s78c"
Jul 02 21:20:39.332759 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.332723    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33aa0be6-449b-48aa-8f64-5f09346a08ad-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b77b96467-7s78c\" (UID: \"33aa0be6-449b-48aa-8f64-5f09346a08ad\") " pod="openshift-monitoring/metrics-server-7b77b96467-7s78c"
Jul 02 21:20:39.333042 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.333021    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/33aa0be6-449b-48aa-8f64-5f09346a08ad-metrics-server-audit-profiles\") pod \"metrics-server-7b77b96467-7s78c\" (UID: \"33aa0be6-449b-48aa-8f64-5f09346a08ad\") " pod="openshift-monitoring/metrics-server-7b77b96467-7s78c"
Jul 02 21:20:39.334491 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.334465    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/33aa0be6-449b-48aa-8f64-5f09346a08ad-secret-metrics-server-client-certs\") pod \"metrics-server-7b77b96467-7s78c\" (UID: \"33aa0be6-449b-48aa-8f64-5f09346a08ad\") " pod="openshift-monitoring/metrics-server-7b77b96467-7s78c"
Jul 02 21:20:39.334634 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.334615    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/33aa0be6-449b-48aa-8f64-5f09346a08ad-secret-metrics-server-tls\") pod \"metrics-server-7b77b96467-7s78c\" (UID: \"33aa0be6-449b-48aa-8f64-5f09346a08ad\") " pod="openshift-monitoring/metrics-server-7b77b96467-7s78c"
Jul 02 21:20:39.334727 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.334712    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/33aa0be6-449b-48aa-8f64-5f09346a08ad-client-ca-bundle\") pod \"metrics-server-7b77b96467-7s78c\" (UID: \"33aa0be6-449b-48aa-8f64-5f09346a08ad\") " pod="openshift-monitoring/metrics-server-7b77b96467-7s78c"
Jul 02 21:20:39.343499 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.343476    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttlqk\" (UniqueName: \"kubernetes.io/projected/33aa0be6-449b-48aa-8f64-5f09346a08ad-kube-api-access-ttlqk\") pod \"metrics-server-7b77b96467-7s78c\" (UID: \"33aa0be6-449b-48aa-8f64-5f09346a08ad\") " pod="openshift-monitoring/metrics-server-7b77b96467-7s78c"
Jul 02 21:20:39.409168 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.409073    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-4qrvv"]
Jul 02 21:20:39.412677 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.412628    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7b77b96467-7s78c"
Jul 02 21:20:39.413332 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.413312    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4qrvv"
Jul 02 21:20:39.416259 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.416237    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-q5lrl\""
Jul 02 21:20:39.416425 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.416408    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 02 21:20:39.423736 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.423708    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-4qrvv"]
Jul 02 21:20:39.534243 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.534205    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8cph\" (UniqueName: \"kubernetes.io/projected/a39c9223-ec0e-4982-b586-0e55444a82ae-kube-api-access-r8cph\") pod \"monitoring-plugin-d5dcff7cf-4qrvv\" (UID: \"a39c9223-ec0e-4982-b586-0e55444a82ae\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4qrvv"
Jul 02 21:20:39.534391 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.534264    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a39c9223-ec0e-4982-b586-0e55444a82ae-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-4qrvv\" (UID: \"a39c9223-ec0e-4982-b586-0e55444a82ae\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4qrvv"
Jul 02 21:20:39.554261 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.554219    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7b77b96467-7s78c"]
Jul 02 21:20:39.557887 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:20:39.557857    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33aa0be6_449b_48aa_8f64_5f09346a08ad.slice/crio-0ae0d80a6b23fdaf18feea9a85b89ff2c3c8f404f7a96ad2dcbf84eac0f77359 WatchSource:0}: Error finding container 0ae0d80a6b23fdaf18feea9a85b89ff2c3c8f404f7a96ad2dcbf84eac0f77359: Status 404 returned error can't find the container with id 0ae0d80a6b23fdaf18feea9a85b89ff2c3c8f404f7a96ad2dcbf84eac0f77359
Jul 02 21:20:39.634877 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.634839    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a39c9223-ec0e-4982-b586-0e55444a82ae-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-4qrvv\" (UID: \"a39c9223-ec0e-4982-b586-0e55444a82ae\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4qrvv"
Jul 02 21:20:39.635048 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.634967    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8cph\" (UniqueName: \"kubernetes.io/projected/a39c9223-ec0e-4982-b586-0e55444a82ae-kube-api-access-r8cph\") pod \"monitoring-plugin-d5dcff7cf-4qrvv\" (UID: \"a39c9223-ec0e-4982-b586-0e55444a82ae\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4qrvv"
Jul 02 21:20:39.635048 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:20:39.635010    2586 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 02 21:20:39.635173 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:20:39.635094    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a39c9223-ec0e-4982-b586-0e55444a82ae-monitoring-plugin-cert podName:a39c9223-ec0e-4982-b586-0e55444a82ae nodeName:}" failed. No retries permitted until 2026-07-02 21:20:40.135072516 +0000 UTC m=+112.577872936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/a39c9223-ec0e-4982-b586-0e55444a82ae-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-4qrvv" (UID: "a39c9223-ec0e-4982-b586-0e55444a82ae") : secret "monitoring-plugin-cert" not found
Jul 02 21:20:39.651364 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:39.651333    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8cph\" (UniqueName: \"kubernetes.io/projected/a39c9223-ec0e-4982-b586-0e55444a82ae-kube-api-access-r8cph\") pod \"monitoring-plugin-d5dcff7cf-4qrvv\" (UID: \"a39c9223-ec0e-4982-b586-0e55444a82ae\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4qrvv"
Jul 02 21:20:40.140506 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:40.140455    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a39c9223-ec0e-4982-b586-0e55444a82ae-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-4qrvv\" (UID: \"a39c9223-ec0e-4982-b586-0e55444a82ae\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4qrvv"
Jul 02 21:20:40.143308 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:40.143282    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a39c9223-ec0e-4982-b586-0e55444a82ae-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-4qrvv\" (UID: \"a39c9223-ec0e-4982-b586-0e55444a82ae\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4qrvv"
Jul 02 21:20:40.335679 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:40.335625    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4qrvv"
Jul 02 21:20:40.485888 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:40.485632    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-4qrvv"]
Jul 02 21:20:40.486569 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:40.486524    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7b77b96467-7s78c" event={"ID":"33aa0be6-449b-48aa-8f64-5f09346a08ad","Type":"ContainerStarted","Data":"0ae0d80a6b23fdaf18feea9a85b89ff2c3c8f404f7a96ad2dcbf84eac0f77359"}
Jul 02 21:20:40.488338 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:20:40.488313    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda39c9223_ec0e_4982_b586_0e55444a82ae.slice/crio-65679c86f909a9c68b687cf42a2d646364c51c479d9b0b4884b844e24e480418 WatchSource:0}: Error finding container 65679c86f909a9c68b687cf42a2d646364c51c479d9b0b4884b844e24e480418: Status 404 returned error can't find the container with id 65679c86f909a9c68b687cf42a2d646364c51c479d9b0b4884b844e24e480418
Jul 02 21:20:41.492160 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:41.492120    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7b77b96467-7s78c" event={"ID":"33aa0be6-449b-48aa-8f64-5f09346a08ad","Type":"ContainerStarted","Data":"b2ec19d8eb888be7822d734938ec779d3aaa814186843bcf19ec04f3f7773991"}
Jul 02 21:20:41.493371 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:41.493340    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4qrvv" event={"ID":"a39c9223-ec0e-4982-b586-0e55444a82ae","Type":"ContainerStarted","Data":"65679c86f909a9c68b687cf42a2d646364c51c479d9b0b4884b844e24e480418"}
Jul 02 21:20:41.511520 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:41.511469    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7b77b96467-7s78c" podStartSLOduration=1.058888627 podStartE2EDuration="2.511452895s" totalImagesPullingTime="1.452564268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:20:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:41.510709575 +0000 UTC m=+113.953510105" watchObservedRunningTime="2026-07-02 21:20:41.511452895 +0000 UTC m=+113.954253316"
Jul 02 21:20:42.389738 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:42.389708    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5cc4899c7b-qpxzm"
Jul 02 21:20:42.498929 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:42.498884    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4qrvv" event={"ID":"a39c9223-ec0e-4982-b586-0e55444a82ae","Type":"ContainerStarted","Data":"d51aaa08b48b91da78b10b2a3afde4bcd387d6619a78f2831a166168cb94103b"}
Jul 02 21:20:42.520993 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:42.520939    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4qrvv" podStartSLOduration=2.150129378 podStartE2EDuration="3.520919825s" totalImagesPullingTime="1.370790447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:20:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:42.519263414 +0000 UTC m=+114.962063835" watchObservedRunningTime="2026-07-02 21:20:42.520919825 +0000 UTC m=+114.963720272"
Jul 02 21:20:43.505693 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:43.505628    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4qrvv"
Jul 02 21:20:43.509627 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:43.509604    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4qrvv"
Jul 02 21:20:45.368146 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:45.368109    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz" podUID="d30ee675-695d-42ad-9932-248d4adde3e3" containerName="registry" containerID="cri-o://d9d2870fa6b143206660eb46f26f3caf2a07a4872f79623cfeeb870e090e631e" gracePeriod=30
Jul 02 21:20:45.512951 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:45.512921    2586 generic.go:356] "Generic (PLEG): container finished" podID="d30ee675-695d-42ad-9932-248d4adde3e3" containerID="d9d2870fa6b143206660eb46f26f3caf2a07a4872f79623cfeeb870e090e631e" exitCode=0
Jul 02 21:20:45.513122 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:45.512994    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz" event={"ID":"d30ee675-695d-42ad-9932-248d4adde3e3","Type":"ContainerDied","Data":"d9d2870fa6b143206660eb46f26f3caf2a07a4872f79623cfeeb870e090e631e"}
Jul 02 21:20:45.603734 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:45.603709    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz"
Jul 02 21:20:45.792730 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:45.792692    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d30ee675-695d-42ad-9932-248d4adde3e3-registry-certificates\") pod \"d30ee675-695d-42ad-9932-248d4adde3e3\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") "
Jul 02 21:20:45.792730 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:45.792736    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d30ee675-695d-42ad-9932-248d4adde3e3-bound-sa-token\") pod \"d30ee675-695d-42ad-9932-248d4adde3e3\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") "
Jul 02 21:20:45.792926 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:45.792759    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d30ee675-695d-42ad-9932-248d4adde3e3-ca-trust-extracted\") pod \"d30ee675-695d-42ad-9932-248d4adde3e3\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") "
Jul 02 21:20:45.792926 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:45.792805    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d30ee675-695d-42ad-9932-248d4adde3e3-image-registry-private-configuration\") pod \"d30ee675-695d-42ad-9932-248d4adde3e3\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") "
Jul 02 21:20:45.792926 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:45.792835    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d30ee675-695d-42ad-9932-248d4adde3e3-trusted-ca\") pod \"d30ee675-695d-42ad-9932-248d4adde3e3\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") "
Jul 02 21:20:45.792926 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:45.792859    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d30ee675-695d-42ad-9932-248d4adde3e3-registry-tls\") pod \"d30ee675-695d-42ad-9932-248d4adde3e3\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") "
Jul 02 21:20:45.792926 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:45.792918    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mg4kn\" (UniqueName: \"kubernetes.io/projected/d30ee675-695d-42ad-9932-248d4adde3e3-kube-api-access-mg4kn\") pod \"d30ee675-695d-42ad-9932-248d4adde3e3\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") "
Jul 02 21:20:45.793183 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:45.792980    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d30ee675-695d-42ad-9932-248d4adde3e3-installation-pull-secrets\") pod \"d30ee675-695d-42ad-9932-248d4adde3e3\" (UID: \"d30ee675-695d-42ad-9932-248d4adde3e3\") "
Jul 02 21:20:45.793261 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:45.793226    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d30ee675-695d-42ad-9932-248d4adde3e3-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d30ee675-695d-42ad-9932-248d4adde3e3" (UID: "d30ee675-695d-42ad-9932-248d4adde3e3"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:20:45.793366 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:45.793339    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d30ee675-695d-42ad-9932-248d4adde3e3-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d30ee675-695d-42ad-9932-248d4adde3e3" (UID: "d30ee675-695d-42ad-9932-248d4adde3e3"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:20:45.795490 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:45.795460    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d30ee675-695d-42ad-9932-248d4adde3e3-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d30ee675-695d-42ad-9932-248d4adde3e3" (UID: "d30ee675-695d-42ad-9932-248d4adde3e3"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:20:45.795633 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:45.795451    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d30ee675-695d-42ad-9932-248d4adde3e3-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d30ee675-695d-42ad-9932-248d4adde3e3" (UID: "d30ee675-695d-42ad-9932-248d4adde3e3"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:20:45.795633 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:45.795554    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d30ee675-695d-42ad-9932-248d4adde3e3-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d30ee675-695d-42ad-9932-248d4adde3e3" (UID: "d30ee675-695d-42ad-9932-248d4adde3e3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:20:45.795776 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:45.795730    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d30ee675-695d-42ad-9932-248d4adde3e3-kube-api-access-mg4kn" (OuterVolumeSpecName: "kube-api-access-mg4kn") pod "d30ee675-695d-42ad-9932-248d4adde3e3" (UID: "d30ee675-695d-42ad-9932-248d4adde3e3"). InnerVolumeSpecName "kube-api-access-mg4kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:20:45.795776 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:45.795737    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d30ee675-695d-42ad-9932-248d4adde3e3-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d30ee675-695d-42ad-9932-248d4adde3e3" (UID: "d30ee675-695d-42ad-9932-248d4adde3e3"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:20:45.802243 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:45.802204    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30ee675-695d-42ad-9932-248d4adde3e3-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d30ee675-695d-42ad-9932-248d4adde3e3" (UID: "d30ee675-695d-42ad-9932-248d4adde3e3"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:20:45.894252 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:45.894211    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg4kn\" (UniqueName: \"kubernetes.io/projected/d30ee675-695d-42ad-9932-248d4adde3e3-kube-api-access-mg4kn\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:20:45.894252 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:45.894245    2586 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d30ee675-695d-42ad-9932-248d4adde3e3-installation-pull-secrets\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:20:45.894252 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:45.894258    2586 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d30ee675-695d-42ad-9932-248d4adde3e3-registry-certificates\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:20:45.894489 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:45.894271    2586 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d30ee675-695d-42ad-9932-248d4adde3e3-bound-sa-token\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:20:45.894489 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:45.894284    2586 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d30ee675-695d-42ad-9932-248d4adde3e3-ca-trust-extracted\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:20:45.894489 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:45.894297    2586 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d30ee675-695d-42ad-9932-248d4adde3e3-image-registry-private-configuration\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:20:45.894489 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:45.894310    2586 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d30ee675-695d-42ad-9932-248d4adde3e3-trusted-ca\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:20:45.894489 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:45.894322    2586 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d30ee675-695d-42ad-9932-248d4adde3e3-registry-tls\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:20:46.518303 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:46.518276    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz"
Jul 02 21:20:46.518743 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:46.518307    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69f78b9dc6-mvztz" event={"ID":"d30ee675-695d-42ad-9932-248d4adde3e3","Type":"ContainerDied","Data":"2ec2597f6ff4133fbadf4b3d3fce41d67b528dadadf57793243280df49061266"}
Jul 02 21:20:46.518743 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:46.518369    2586 scope.go:117] "RemoveContainer" containerID="d9d2870fa6b143206660eb46f26f3caf2a07a4872f79623cfeeb870e090e631e"
Jul 02 21:20:46.536779 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:46.536750    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69f78b9dc6-mvztz"]
Jul 02 21:20:46.540942 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:46.540922    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-69f78b9dc6-mvztz"]
Jul 02 21:20:48.021269 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:48.021236    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d30ee675-695d-42ad-9932-248d4adde3e3" path="/var/lib/kubelet/pods/d30ee675-695d-42ad-9932-248d4adde3e3/volumes"
Jul 02 21:20:48.294015 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:48.293982    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-8qcvz"]
Jul 02 21:20:48.294263 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:48.294250    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d30ee675-695d-42ad-9932-248d4adde3e3" containerName="registry"
Jul 02 21:20:48.294309 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:48.294264    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d30ee675-695d-42ad-9932-248d4adde3e3" containerName="registry"
Jul 02 21:20:48.294341 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:48.294320    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="d30ee675-695d-42ad-9932-248d4adde3e3" containerName="registry"
Jul 02 21:20:48.299096 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:48.299073    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-8qcvz"
Jul 02 21:20:48.302022 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:48.301987    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-xpssk\""
Jul 02 21:20:48.302022 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:48.301990    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 02 21:20:48.302199 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:48.301998    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 02 21:20:48.308338 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:48.308315    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-8qcvz"]
Jul 02 21:20:48.413998 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:48.413962    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4hfr\" (UniqueName: \"kubernetes.io/projected/9ad59fa2-6b4a-4eb6-8929-9a944438a7e5-kube-api-access-q4hfr\") pod \"downloads-685bfb5f64-8qcvz\" (UID: \"9ad59fa2-6b4a-4eb6-8929-9a944438a7e5\") " pod="openshift-console/downloads-685bfb5f64-8qcvz"
Jul 02 21:20:48.514543 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:48.514493    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4hfr\" (UniqueName: \"kubernetes.io/projected/9ad59fa2-6b4a-4eb6-8929-9a944438a7e5-kube-api-access-q4hfr\") pod \"downloads-685bfb5f64-8qcvz\" (UID: \"9ad59fa2-6b4a-4eb6-8929-9a944438a7e5\") " pod="openshift-console/downloads-685bfb5f64-8qcvz"
Jul 02 21:20:48.523804 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:48.523773    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4hfr\" (UniqueName: \"kubernetes.io/projected/9ad59fa2-6b4a-4eb6-8929-9a944438a7e5-kube-api-access-q4hfr\") pod \"downloads-685bfb5f64-8qcvz\" (UID: \"9ad59fa2-6b4a-4eb6-8929-9a944438a7e5\") " pod="openshift-console/downloads-685bfb5f64-8qcvz"
Jul 02 21:20:48.610281 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:48.610178    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-8qcvz"
Jul 02 21:20:48.737706 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:48.737673    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-8qcvz"]
Jul 02 21:20:48.740972 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:20:48.740938    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ad59fa2_6b4a_4eb6_8929_9a944438a7e5.slice/crio-ca0dadf4ed742f88d9dffae3ef309d153cc12e4100818325d9ac2e6cf81641d6 WatchSource:0}: Error finding container ca0dadf4ed742f88d9dffae3ef309d153cc12e4100818325d9ac2e6cf81641d6: Status 404 returned error can't find the container with id ca0dadf4ed742f88d9dffae3ef309d153cc12e4100818325d9ac2e6cf81641d6
Jul 02 21:20:49.532805 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:49.532762    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-8qcvz" event={"ID":"9ad59fa2-6b4a-4eb6-8929-9a944438a7e5","Type":"ContainerStarted","Data":"ca0dadf4ed742f88d9dffae3ef309d153cc12e4100818325d9ac2e6cf81641d6"}
Jul 02 21:20:57.789922 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:57.789883    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/006d61c8-eb34-4336-a3d3-82a7a4fc2069-metrics-certs\") pod \"network-metrics-daemon-s5xr4\" (UID: \"006d61c8-eb34-4336-a3d3-82a7a4fc2069\") " pod="openshift-multus/network-metrics-daemon-s5xr4"
Jul 02 21:20:57.792413 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:57.792386    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/006d61c8-eb34-4336-a3d3-82a7a4fc2069-metrics-certs\") pod \"network-metrics-daemon-s5xr4\" (UID: \"006d61c8-eb34-4336-a3d3-82a7a4fc2069\") " pod="openshift-multus/network-metrics-daemon-s5xr4"
Jul 02 21:20:57.844594 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:57.844562    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6k252\""
Jul 02 21:20:57.852869 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:57.852842    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5xr4"
Jul 02 21:20:58.760806 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:58.760774    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d69954986-b9kqc"]
Jul 02 21:20:58.764066 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:58.764039    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d69954986-b9kqc"
Jul 02 21:20:58.766772 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:58.766745    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 02 21:20:58.766772 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:58.766766    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 02 21:20:58.766957 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:58.766872    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 02 21:20:58.767878 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:58.767842    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ptcsr\""
Jul 02 21:20:58.768007 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:58.767972    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 02 21:20:58.768110 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:58.768045    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 02 21:20:58.776729 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:58.776709    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d69954986-b9kqc"]
Jul 02 21:20:58.798255 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:58.798200    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3624045-c72c-441d-81d2-199085401832-console-config\") pod \"console-6d69954986-b9kqc\" (UID: \"c3624045-c72c-441d-81d2-199085401832\") " pod="openshift-console/console-6d69954986-b9kqc"
Jul 02 21:20:58.798720 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:58.798275    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92gz2\" (UniqueName: \"kubernetes.io/projected/c3624045-c72c-441d-81d2-199085401832-kube-api-access-92gz2\") pod \"console-6d69954986-b9kqc\" (UID: \"c3624045-c72c-441d-81d2-199085401832\") " pod="openshift-console/console-6d69954986-b9kqc"
Jul 02 21:20:58.798720 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:58.798308    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3624045-c72c-441d-81d2-199085401832-console-serving-cert\") pod \"console-6d69954986-b9kqc\" (UID: \"c3624045-c72c-441d-81d2-199085401832\") " pod="openshift-console/console-6d69954986-b9kqc"
Jul 02 21:20:58.798720 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:58.798342    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3624045-c72c-441d-81d2-199085401832-service-ca\") pod \"console-6d69954986-b9kqc\" (UID: \"c3624045-c72c-441d-81d2-199085401832\") " pod="openshift-console/console-6d69954986-b9kqc"
Jul 02 21:20:58.798720 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:58.798473    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3624045-c72c-441d-81d2-199085401832-console-oauth-config\") pod \"console-6d69954986-b9kqc\" (UID: \"c3624045-c72c-441d-81d2-199085401832\") " pod="openshift-console/console-6d69954986-b9kqc"
Jul 02 21:20:58.798720 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:58.798575    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3624045-c72c-441d-81d2-199085401832-oauth-serving-cert\") pod \"console-6d69954986-b9kqc\" (UID: \"c3624045-c72c-441d-81d2-199085401832\") " pod="openshift-console/console-6d69954986-b9kqc"
Jul 02 21:20:58.899456 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:58.899420    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3624045-c72c-441d-81d2-199085401832-service-ca\") pod \"console-6d69954986-b9kqc\" (UID: \"c3624045-c72c-441d-81d2-199085401832\") " pod="openshift-console/console-6d69954986-b9kqc"
Jul 02 21:20:58.899688 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:58.899490    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3624045-c72c-441d-81d2-199085401832-console-oauth-config\") pod \"console-6d69954986-b9kqc\" (UID: \"c3624045-c72c-441d-81d2-199085401832\") " pod="openshift-console/console-6d69954986-b9kqc"
Jul 02 21:20:58.899688 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:58.899540    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3624045-c72c-441d-81d2-199085401832-oauth-serving-cert\") pod \"console-6d69954986-b9kqc\" (UID: \"c3624045-c72c-441d-81d2-199085401832\") " pod="openshift-console/console-6d69954986-b9kqc"
Jul 02 21:20:58.899688 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:58.899576    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3624045-c72c-441d-81d2-199085401832-console-config\") pod \"console-6d69954986-b9kqc\" (UID: \"c3624045-c72c-441d-81d2-199085401832\") " pod="openshift-console/console-6d69954986-b9kqc"
Jul 02 21:20:58.899688 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:58.899620    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92gz2\" (UniqueName: \"kubernetes.io/projected/c3624045-c72c-441d-81d2-199085401832-kube-api-access-92gz2\") pod \"console-6d69954986-b9kqc\" (UID: \"c3624045-c72c-441d-81d2-199085401832\") " pod="openshift-console/console-6d69954986-b9kqc"
Jul 02 21:20:58.899688 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:58.899665    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3624045-c72c-441d-81d2-199085401832-console-serving-cert\") pod \"console-6d69954986-b9kqc\" (UID: \"c3624045-c72c-441d-81d2-199085401832\") " pod="openshift-console/console-6d69954986-b9kqc"
Jul 02 21:20:58.900403 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:58.900369    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3624045-c72c-441d-81d2-199085401832-service-ca\") pod \"console-6d69954986-b9kqc\" (UID: \"c3624045-c72c-441d-81d2-199085401832\") " pod="openshift-console/console-6d69954986-b9kqc"
Jul 02 21:20:58.900511 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:58.900407    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3624045-c72c-441d-81d2-199085401832-oauth-serving-cert\") pod \"console-6d69954986-b9kqc\" (UID: \"c3624045-c72c-441d-81d2-199085401832\") " pod="openshift-console/console-6d69954986-b9kqc"
Jul 02 21:20:58.900511 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:58.900433    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3624045-c72c-441d-81d2-199085401832-console-config\") pod \"console-6d69954986-b9kqc\" (UID: \"c3624045-c72c-441d-81d2-199085401832\") " pod="openshift-console/console-6d69954986-b9kqc"
Jul 02 21:20:58.902549 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:58.902504    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3624045-c72c-441d-81d2-199085401832-console-oauth-config\") pod \"console-6d69954986-b9kqc\" (UID: \"c3624045-c72c-441d-81d2-199085401832\") " pod="openshift-console/console-6d69954986-b9kqc"
Jul 02 21:20:58.902683 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:58.902549    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3624045-c72c-441d-81d2-199085401832-console-serving-cert\") pod \"console-6d69954986-b9kqc\" (UID: \"c3624045-c72c-441d-81d2-199085401832\") " pod="openshift-console/console-6d69954986-b9kqc"
Jul 02 21:20:58.908168 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:58.908142    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92gz2\" (UniqueName: \"kubernetes.io/projected/c3624045-c72c-441d-81d2-199085401832-kube-api-access-92gz2\") pod \"console-6d69954986-b9kqc\" (UID: \"c3624045-c72c-441d-81d2-199085401832\") " pod="openshift-console/console-6d69954986-b9kqc"
Jul 02 21:20:59.075889 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:59.075801    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d69954986-b9kqc"
Jul 02 21:20:59.413784 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:59.413694    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7b77b96467-7s78c"
Jul 02 21:20:59.413784 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:20:59.413774    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7b77b96467-7s78c"
Jul 02 21:21:04.883460 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:04.883400    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d69954986-b9kqc"]
Jul 02 21:21:04.887394 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:21:04.887353    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3624045_c72c_441d_81d2_199085401832.slice/crio-6084ddb5adb20f9d21389462e1226df3a2fcd48425eec67cd6219dcd299195dd WatchSource:0}: Error finding container 6084ddb5adb20f9d21389462e1226df3a2fcd48425eec67cd6219dcd299195dd: Status 404 returned error can't find the container with id 6084ddb5adb20f9d21389462e1226df3a2fcd48425eec67cd6219dcd299195dd
Jul 02 21:21:04.902759 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:04.902735    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s5xr4"]
Jul 02 21:21:04.905344 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:21:04.905314    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod006d61c8_eb34_4336_a3d3_82a7a4fc2069.slice/crio-db87fb7af0f9ed4de3a53c2e9a5a2862d12c1309efc7e08354b179ad6bc40349 WatchSource:0}: Error finding container db87fb7af0f9ed4de3a53c2e9a5a2862d12c1309efc7e08354b179ad6bc40349: Status 404 returned error can't find the container with id db87fb7af0f9ed4de3a53c2e9a5a2862d12c1309efc7e08354b179ad6bc40349
Jul 02 21:21:05.602164 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:05.602049    2586 generic.go:356] "Generic (PLEG): container finished" podID="d2dfa800-1ffa-44d7-8dce-dd857531d58e" containerID="d78b7bbab21860ae93efa7c60ee2010a94d58924008850440e46b39be90df010" exitCode=0
Jul 02 21:21:05.602164 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:05.602119    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xzswr" event={"ID":"d2dfa800-1ffa-44d7-8dce-dd857531d58e","Type":"ContainerDied","Data":"d78b7bbab21860ae93efa7c60ee2010a94d58924008850440e46b39be90df010"}
Jul 02 21:21:05.602602 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:05.602551    2586 scope.go:117] "RemoveContainer" containerID="d78b7bbab21860ae93efa7c60ee2010a94d58924008850440e46b39be90df010"
Jul 02 21:21:05.604207 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:05.604176    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d69954986-b9kqc" event={"ID":"c3624045-c72c-441d-81d2-199085401832","Type":"ContainerStarted","Data":"6084ddb5adb20f9d21389462e1226df3a2fcd48425eec67cd6219dcd299195dd"}
Jul 02 21:21:05.607498 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:05.607474    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-8qcvz" event={"ID":"9ad59fa2-6b4a-4eb6-8929-9a944438a7e5","Type":"ContainerStarted","Data":"b44e96f60fed5ffdc85fbe9a61a3063cb047789ffb5ff7bd4696c5ebee70b4ec"}
Jul 02 21:21:05.608898 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:05.608879    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-8qcvz"
Jul 02 21:21:05.610538 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:05.610511    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s5xr4" event={"ID":"006d61c8-eb34-4336-a3d3-82a7a4fc2069","Type":"ContainerStarted","Data":"db87fb7af0f9ed4de3a53c2e9a5a2862d12c1309efc7e08354b179ad6bc40349"}
Jul 02 21:21:05.622274 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:05.622239    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-8qcvz"
Jul 02 21:21:05.663266 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:05.662689    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-8qcvz" podStartSLOduration=1.317117571 podStartE2EDuration="17.662665093s" totalImagesPullingTime="16.345547522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:20:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:21:05.64313882 +0000 UTC m=+138.085939268" watchObservedRunningTime="2026-07-02 21:21:05.662665093 +0000 UTC m=+138.105465624"
Jul 02 21:21:06.619087 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:06.618990    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xzswr" event={"ID":"d2dfa800-1ffa-44d7-8dce-dd857531d58e","Type":"ContainerStarted","Data":"907956efdac40fc0383e8d454fb7a360ac560b720401fb199e3db7dcdf0409ac"}
Jul 02 21:21:07.627453 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:07.627407    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s5xr4" event={"ID":"006d61c8-eb34-4336-a3d3-82a7a4fc2069","Type":"ContainerStarted","Data":"83d309fbd8a886ebb8329041c8c5593c1fc5fbc19d8cbb4517f599ad71daf01d"}
Jul 02 21:21:07.628078 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:07.627848    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s5xr4" event={"ID":"006d61c8-eb34-4336-a3d3-82a7a4fc2069","Type":"ContainerStarted","Data":"edcfbcccca51188a9ba7e320d8bb25fbd725203f79641860a45d9012bd975e52"}
Jul 02 21:21:07.651802 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:07.651736    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-s5xr4" podStartSLOduration=137.76653602 podStartE2EDuration="2m19.651713998s" totalImagesPullingTime="1.885177978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:18:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:21:07.651263059 +0000 UTC m=+140.094063482" watchObservedRunningTime="2026-07-02 21:21:07.651713998 +0000 UTC m=+140.094514420"
Jul 02 21:21:09.639274 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:09.639233    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d69954986-b9kqc" event={"ID":"c3624045-c72c-441d-81d2-199085401832","Type":"ContainerStarted","Data":"df10ddc518fa47668d6b3df53e343c4eeb0b701a829433ae9afb4e2aa622b279"}
Jul 02 21:21:09.658182 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:09.658138    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d69954986-b9kqc" podStartSLOduration=7.376590652 podStartE2EDuration="11.658121979s" totalImagesPullingTime="4.281531327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:20:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:21:09.657039725 +0000 UTC m=+142.099840170" watchObservedRunningTime="2026-07-02 21:21:09.658121979 +0000 UTC m=+142.100922401"
Jul 02 21:21:19.066714 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:19.066680    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d69954986-b9kqc"]
Jul 02 21:21:19.075893 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:19.075870    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d69954986-b9kqc"
Jul 02 21:21:19.419198 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:19.419166    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7b77b96467-7s78c"
Jul 02 21:21:19.423119 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:19.423095    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7b77b96467-7s78c"
Jul 02 21:21:19.680130 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:19.680103    2586 generic.go:356] "Generic (PLEG): container finished" podID="1b77428f-b316-4a33-9397-b3227211e3e3" containerID="93c44b9896ed68768f50681bbaa92cb302aec9bb40793e852eada7de7e6d2cb6" exitCode=0
Jul 02 21:21:19.680275 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:19.680178    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-c7tm5" event={"ID":"1b77428f-b316-4a33-9397-b3227211e3e3","Type":"ContainerDied","Data":"93c44b9896ed68768f50681bbaa92cb302aec9bb40793e852eada7de7e6d2cb6"}
Jul 02 21:21:19.680616 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:19.680589    2586 scope.go:117] "RemoveContainer" containerID="93c44b9896ed68768f50681bbaa92cb302aec9bb40793e852eada7de7e6d2cb6"
Jul 02 21:21:20.685531 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:20.685498    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-c7tm5" event={"ID":"1b77428f-b316-4a33-9397-b3227211e3e3","Type":"ContainerStarted","Data":"2c07117a48d269d52c8f891a9bba6850df7a13f0772f26397e4f5225c9a376ad"}
Jul 02 21:21:44.087033 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:44.086949    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6d69954986-b9kqc" podUID="c3624045-c72c-441d-81d2-199085401832" containerName="console" containerID="cri-o://df10ddc518fa47668d6b3df53e343c4eeb0b701a829433ae9afb4e2aa622b279" gracePeriod=15
Jul 02 21:21:44.347052 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:44.346988    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d69954986-b9kqc"
Jul 02 21:21:44.382097 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:44.382069    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92gz2\" (UniqueName: \"kubernetes.io/projected/c3624045-c72c-441d-81d2-199085401832-kube-api-access-92gz2\") pod \"c3624045-c72c-441d-81d2-199085401832\" (UID: \"c3624045-c72c-441d-81d2-199085401832\") "
Jul 02 21:21:44.382253 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:44.382103    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3624045-c72c-441d-81d2-199085401832-console-config\") pod \"c3624045-c72c-441d-81d2-199085401832\" (UID: \"c3624045-c72c-441d-81d2-199085401832\") "
Jul 02 21:21:44.382253 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:44.382159    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3624045-c72c-441d-81d2-199085401832-console-serving-cert\") pod \"c3624045-c72c-441d-81d2-199085401832\" (UID: \"c3624045-c72c-441d-81d2-199085401832\") "
Jul 02 21:21:44.382253 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:44.382202    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3624045-c72c-441d-81d2-199085401832-oauth-serving-cert\") pod \"c3624045-c72c-441d-81d2-199085401832\" (UID: \"c3624045-c72c-441d-81d2-199085401832\") "
Jul 02 21:21:44.382253 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:44.382244    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3624045-c72c-441d-81d2-199085401832-service-ca\") pod \"c3624045-c72c-441d-81d2-199085401832\" (UID: \"c3624045-c72c-441d-81d2-199085401832\") "
Jul 02 21:21:44.382472 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:44.382300    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3624045-c72c-441d-81d2-199085401832-console-oauth-config\") pod \"c3624045-c72c-441d-81d2-199085401832\" (UID: \"c3624045-c72c-441d-81d2-199085401832\") "
Jul 02 21:21:44.382920 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:44.382885    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3624045-c72c-441d-81d2-199085401832-service-ca" (OuterVolumeSpecName: "service-ca") pod "c3624045-c72c-441d-81d2-199085401832" (UID: "c3624045-c72c-441d-81d2-199085401832"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:21:44.383232 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:44.383207    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3624045-c72c-441d-81d2-199085401832-console-config" (OuterVolumeSpecName: "console-config") pod "c3624045-c72c-441d-81d2-199085401832" (UID: "c3624045-c72c-441d-81d2-199085401832"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:21:44.383540 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:44.383517    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3624045-c72c-441d-81d2-199085401832-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c3624045-c72c-441d-81d2-199085401832" (UID: "c3624045-c72c-441d-81d2-199085401832"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:21:44.384555 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:44.384530    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3624045-c72c-441d-81d2-199085401832-kube-api-access-92gz2" (OuterVolumeSpecName: "kube-api-access-92gz2") pod "c3624045-c72c-441d-81d2-199085401832" (UID: "c3624045-c72c-441d-81d2-199085401832"). InnerVolumeSpecName "kube-api-access-92gz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:21:44.384787 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:44.384762    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3624045-c72c-441d-81d2-199085401832-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c3624045-c72c-441d-81d2-199085401832" (UID: "c3624045-c72c-441d-81d2-199085401832"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:21:44.384910 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:44.384889    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3624045-c72c-441d-81d2-199085401832-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c3624045-c72c-441d-81d2-199085401832" (UID: "c3624045-c72c-441d-81d2-199085401832"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:21:44.483751 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:44.483722    2586 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3624045-c72c-441d-81d2-199085401832-service-ca\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:21:44.483751 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:44.483747    2586 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3624045-c72c-441d-81d2-199085401832-console-oauth-config\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:21:44.483751 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:44.483758    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92gz2\" (UniqueName: \"kubernetes.io/projected/c3624045-c72c-441d-81d2-199085401832-kube-api-access-92gz2\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:21:44.483981 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:44.483767    2586 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3624045-c72c-441d-81d2-199085401832-console-config\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:21:44.483981 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:44.483775    2586 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3624045-c72c-441d-81d2-199085401832-console-serving-cert\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:21:44.483981 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:44.483784    2586 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3624045-c72c-441d-81d2-199085401832-oauth-serving-cert\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:21:44.774428 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:44.774393    2586 generic.go:356] "Generic (PLEG): container finished" podID="c3624045-c72c-441d-81d2-199085401832" containerID="df10ddc518fa47668d6b3df53e343c4eeb0b701a829433ae9afb4e2aa622b279" exitCode=0
Jul 02 21:21:44.774596 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:44.774435    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d69954986-b9kqc"
Jul 02 21:21:44.774596 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:44.774480    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d69954986-b9kqc" event={"ID":"c3624045-c72c-441d-81d2-199085401832","Type":"ContainerDied","Data":"df10ddc518fa47668d6b3df53e343c4eeb0b701a829433ae9afb4e2aa622b279"}
Jul 02 21:21:44.774596 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:44.774527    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d69954986-b9kqc" event={"ID":"c3624045-c72c-441d-81d2-199085401832","Type":"ContainerDied","Data":"6084ddb5adb20f9d21389462e1226df3a2fcd48425eec67cd6219dcd299195dd"}
Jul 02 21:21:44.774596 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:44.774547    2586 scope.go:117] "RemoveContainer" containerID="df10ddc518fa47668d6b3df53e343c4eeb0b701a829433ae9afb4e2aa622b279"
Jul 02 21:21:44.783272 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:44.783250    2586 scope.go:117] "RemoveContainer" containerID="df10ddc518fa47668d6b3df53e343c4eeb0b701a829433ae9afb4e2aa622b279"
Jul 02 21:21:44.783555 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:21:44.783536    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df10ddc518fa47668d6b3df53e343c4eeb0b701a829433ae9afb4e2aa622b279\": container with ID starting with df10ddc518fa47668d6b3df53e343c4eeb0b701a829433ae9afb4e2aa622b279 not found: ID does not exist" containerID="df10ddc518fa47668d6b3df53e343c4eeb0b701a829433ae9afb4e2aa622b279"
Jul 02 21:21:44.783678 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:44.783563    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df10ddc518fa47668d6b3df53e343c4eeb0b701a829433ae9afb4e2aa622b279"} err="failed to get container status \"df10ddc518fa47668d6b3df53e343c4eeb0b701a829433ae9afb4e2aa622b279\": rpc error: code = NotFound desc = could not find container \"df10ddc518fa47668d6b3df53e343c4eeb0b701a829433ae9afb4e2aa622b279\": container with ID starting with df10ddc518fa47668d6b3df53e343c4eeb0b701a829433ae9afb4e2aa622b279 not found: ID does not exist"
Jul 02 21:21:44.794260 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:44.794236    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d69954986-b9kqc"]
Jul 02 21:21:44.798757 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:44.798738    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d69954986-b9kqc"]
Jul 02 21:21:46.021997 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:21:46.021965    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3624045-c72c-441d-81d2-199085401832" path="/var/lib/kubelet/pods/c3624045-c72c-441d-81d2-199085401832/volumes"
Jul 02 21:22:12.553066 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:12.553028    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7968ff4bdd-ck5gf"]
Jul 02 21:22:12.553549 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:12.553318    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3624045-c72c-441d-81d2-199085401832" containerName="console"
Jul 02 21:22:12.553549 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:12.553330    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3624045-c72c-441d-81d2-199085401832" containerName="console"
Jul 02 21:22:12.553549 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:12.553373    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3624045-c72c-441d-81d2-199085401832" containerName="console"
Jul 02 21:22:12.556231 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:12.556211    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7968ff4bdd-ck5gf"
Jul 02 21:22:12.560277 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:12.560243    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ptcsr\""
Jul 02 21:22:12.560404 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:12.560250    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 02 21:22:12.560404 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:12.560292    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 02 21:22:12.560404 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:12.560292    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 02 21:22:12.560404 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:12.560300    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 02 21:22:12.560818 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:12.560802    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 02 21:22:12.565083 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:12.565061    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 02 21:22:12.566259 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:12.566241    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7968ff4bdd-ck5gf"]
Jul 02 21:22:12.595641 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:12.595611    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlvxw\" (UniqueName: \"kubernetes.io/projected/75a3868b-74d3-4862-a653-fd9593f577c8-kube-api-access-rlvxw\") pod \"console-7968ff4bdd-ck5gf\" (UID: \"75a3868b-74d3-4862-a653-fd9593f577c8\") " pod="openshift-console/console-7968ff4bdd-ck5gf"
Jul 02 21:22:12.595833 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:12.595668    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75a3868b-74d3-4862-a653-fd9593f577c8-console-config\") pod \"console-7968ff4bdd-ck5gf\" (UID: \"75a3868b-74d3-4862-a653-fd9593f577c8\") " pod="openshift-console/console-7968ff4bdd-ck5gf"
Jul 02 21:22:12.595833 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:12.595697    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75a3868b-74d3-4862-a653-fd9593f577c8-console-oauth-config\") pod \"console-7968ff4bdd-ck5gf\" (UID: \"75a3868b-74d3-4862-a653-fd9593f577c8\") " pod="openshift-console/console-7968ff4bdd-ck5gf"
Jul 02 21:22:12.595833 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:12.595750    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75a3868b-74d3-4862-a653-fd9593f577c8-service-ca\") pod \"console-7968ff4bdd-ck5gf\" (UID: \"75a3868b-74d3-4862-a653-fd9593f577c8\") " pod="openshift-console/console-7968ff4bdd-ck5gf"
Jul 02 21:22:12.595996 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:12.595832    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75a3868b-74d3-4862-a653-fd9593f577c8-trusted-ca-bundle\") pod \"console-7968ff4bdd-ck5gf\" (UID: \"75a3868b-74d3-4862-a653-fd9593f577c8\") " pod="openshift-console/console-7968ff4bdd-ck5gf"
Jul 02 21:22:12.595996 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:12.595871    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75a3868b-74d3-4862-a653-fd9593f577c8-oauth-serving-cert\") pod \"console-7968ff4bdd-ck5gf\" (UID: \"75a3868b-74d3-4862-a653-fd9593f577c8\") " pod="openshift-console/console-7968ff4bdd-ck5gf"
Jul 02 21:22:12.595996 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:12.595945    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75a3868b-74d3-4862-a653-fd9593f577c8-console-serving-cert\") pod \"console-7968ff4bdd-ck5gf\" (UID: \"75a3868b-74d3-4862-a653-fd9593f577c8\") " pod="openshift-console/console-7968ff4bdd-ck5gf"
Jul 02 21:22:12.696451 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:12.696417    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75a3868b-74d3-4862-a653-fd9593f577c8-trusted-ca-bundle\") pod \"console-7968ff4bdd-ck5gf\" (UID: \"75a3868b-74d3-4862-a653-fd9593f577c8\") " pod="openshift-console/console-7968ff4bdd-ck5gf"
Jul 02 21:22:12.696451 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:12.696455    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75a3868b-74d3-4862-a653-fd9593f577c8-oauth-serving-cert\") pod \"console-7968ff4bdd-ck5gf\" (UID: \"75a3868b-74d3-4862-a653-fd9593f577c8\") " pod="openshift-console/console-7968ff4bdd-ck5gf"
Jul 02 21:22:12.696739 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:12.696477    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75a3868b-74d3-4862-a653-fd9593f577c8-console-serving-cert\") pod \"console-7968ff4bdd-ck5gf\" (UID: \"75a3868b-74d3-4862-a653-fd9593f577c8\") " pod="openshift-console/console-7968ff4bdd-ck5gf"
Jul 02 21:22:12.696739 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:12.696509    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlvxw\" (UniqueName: \"kubernetes.io/projected/75a3868b-74d3-4862-a653-fd9593f577c8-kube-api-access-rlvxw\") pod \"console-7968ff4bdd-ck5gf\" (UID: \"75a3868b-74d3-4862-a653-fd9593f577c8\") " pod="openshift-console/console-7968ff4bdd-ck5gf"
Jul 02 21:22:12.696739 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:12.696531    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75a3868b-74d3-4862-a653-fd9593f577c8-console-config\") pod \"console-7968ff4bdd-ck5gf\" (UID: \"75a3868b-74d3-4862-a653-fd9593f577c8\") " pod="openshift-console/console-7968ff4bdd-ck5gf"
Jul 02 21:22:12.696739 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:12.696558    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75a3868b-74d3-4862-a653-fd9593f577c8-console-oauth-config\") pod \"console-7968ff4bdd-ck5gf\" (UID: \"75a3868b-74d3-4862-a653-fd9593f577c8\") " pod="openshift-console/console-7968ff4bdd-ck5gf"
Jul 02 21:22:12.696739 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:12.696574    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75a3868b-74d3-4862-a653-fd9593f577c8-service-ca\") pod \"console-7968ff4bdd-ck5gf\" (UID: \"75a3868b-74d3-4862-a653-fd9593f577c8\") " pod="openshift-console/console-7968ff4bdd-ck5gf"
Jul 02 21:22:12.697340 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:12.697310    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75a3868b-74d3-4862-a653-fd9593f577c8-console-config\") pod \"console-7968ff4bdd-ck5gf\" (UID: \"75a3868b-74d3-4862-a653-fd9593f577c8\") " pod="openshift-console/console-7968ff4bdd-ck5gf"
Jul 02 21:22:12.697447 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:12.697343    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75a3868b-74d3-4862-a653-fd9593f577c8-service-ca\") pod \"console-7968ff4bdd-ck5gf\" (UID: \"75a3868b-74d3-4862-a653-fd9593f577c8\") " pod="openshift-console/console-7968ff4bdd-ck5gf"
Jul 02 21:22:12.697447 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:12.697359    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75a3868b-74d3-4862-a653-fd9593f577c8-oauth-serving-cert\") pod \"console-7968ff4bdd-ck5gf\" (UID: \"75a3868b-74d3-4862-a653-fd9593f577c8\") " pod="openshift-console/console-7968ff4bdd-ck5gf"
Jul 02 21:22:12.697686 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:12.697670    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75a3868b-74d3-4862-a653-fd9593f577c8-trusted-ca-bundle\") pod \"console-7968ff4bdd-ck5gf\" (UID: \"75a3868b-74d3-4862-a653-fd9593f577c8\") " pod="openshift-console/console-7968ff4bdd-ck5gf"
Jul 02 21:22:12.698992 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:12.698969    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75a3868b-74d3-4862-a653-fd9593f577c8-console-oauth-config\") pod \"console-7968ff4bdd-ck5gf\" (UID: \"75a3868b-74d3-4862-a653-fd9593f577c8\") " pod="openshift-console/console-7968ff4bdd-ck5gf"
Jul 02 21:22:12.699153 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:12.699136    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75a3868b-74d3-4862-a653-fd9593f577c8-console-serving-cert\") pod \"console-7968ff4bdd-ck5gf\" (UID: \"75a3868b-74d3-4862-a653-fd9593f577c8\") " pod="openshift-console/console-7968ff4bdd-ck5gf"
Jul 02 21:22:12.704632 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:12.704587    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlvxw\" (UniqueName: \"kubernetes.io/projected/75a3868b-74d3-4862-a653-fd9593f577c8-kube-api-access-rlvxw\") pod \"console-7968ff4bdd-ck5gf\" (UID: \"75a3868b-74d3-4862-a653-fd9593f577c8\") " pod="openshift-console/console-7968ff4bdd-ck5gf"
Jul 02 21:22:12.867542 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:12.867451    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7968ff4bdd-ck5gf"
Jul 02 21:22:12.996832 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:12.996798    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7968ff4bdd-ck5gf"]
Jul 02 21:22:13.000609 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:22:13.000574    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a3868b_74d3_4862_a653_fd9593f577c8.slice/crio-d6f2e8760d85e2457aeb0f9dc95cec242df78971daae1e9bbd298f877b06e2eb WatchSource:0}: Error finding container d6f2e8760d85e2457aeb0f9dc95cec242df78971daae1e9bbd298f877b06e2eb: Status 404 returned error can't find the container with id d6f2e8760d85e2457aeb0f9dc95cec242df78971daae1e9bbd298f877b06e2eb
Jul 02 21:22:13.894545 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:13.894504    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7968ff4bdd-ck5gf" event={"ID":"75a3868b-74d3-4862-a653-fd9593f577c8","Type":"ContainerStarted","Data":"c91b5ec792224430a44a8d3dca5223b7e2b7812d908bbe9dce2673dc11b7298f"}
Jul 02 21:22:13.894962 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:13.894564    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7968ff4bdd-ck5gf" event={"ID":"75a3868b-74d3-4862-a653-fd9593f577c8","Type":"ContainerStarted","Data":"d6f2e8760d85e2457aeb0f9dc95cec242df78971daae1e9bbd298f877b06e2eb"}
Jul 02 21:22:13.913933 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:13.913879    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7968ff4bdd-ck5gf" podStartSLOduration=1.913864032 podStartE2EDuration="1.913864032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:22:13.912601428 +0000 UTC m=+206.355401881" watchObservedRunningTime="2026-07-02 21:22:13.913864032 +0000 UTC m=+206.356664453"
Jul 02 21:22:22.868017 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:22.867956    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7968ff4bdd-ck5gf"
Jul 02 21:22:22.868017 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:22.868032    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7968ff4bdd-ck5gf"
Jul 02 21:22:22.872896 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:22.872871    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7968ff4bdd-ck5gf"
Jul 02 21:22:22.936736 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:22:22.936703    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7968ff4bdd-ck5gf"
Jul 02 21:23:47.319327 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:23:47.319290    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5895zk47"]
Jul 02 21:23:47.322758 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:23:47.322736    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5895zk47"
Jul 02 21:23:47.325997 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:23:47.325972    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 02 21:23:47.327169 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:23:47.327148    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 02 21:23:47.327271 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:23:47.327201    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rxgt5\""
Jul 02 21:23:47.332576 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:23:47.332555    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5895zk47"]
Jul 02 21:23:47.460002 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:23:47.459964    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp9w8\" (UniqueName: \"kubernetes.io/projected/21c99b83-b557-4c53-81fe-9393280e41c8-kube-api-access-lp9w8\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5895zk47\" (UID: \"21c99b83-b557-4c53-81fe-9393280e41c8\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5895zk47"
Jul 02 21:23:47.460207 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:23:47.460019    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21c99b83-b557-4c53-81fe-9393280e41c8-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5895zk47\" (UID: \"21c99b83-b557-4c53-81fe-9393280e41c8\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5895zk47"
Jul 02 21:23:47.460207 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:23:47.460110    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21c99b83-b557-4c53-81fe-9393280e41c8-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5895zk47\" (UID: \"21c99b83-b557-4c53-81fe-9393280e41c8\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5895zk47"
Jul 02 21:23:47.561532 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:23:47.561495    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21c99b83-b557-4c53-81fe-9393280e41c8-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5895zk47\" (UID: \"21c99b83-b557-4c53-81fe-9393280e41c8\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5895zk47"
Jul 02 21:23:47.561704 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:23:47.561554    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp9w8\" (UniqueName: \"kubernetes.io/projected/21c99b83-b557-4c53-81fe-9393280e41c8-kube-api-access-lp9w8\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5895zk47\" (UID: \"21c99b83-b557-4c53-81fe-9393280e41c8\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5895zk47"
Jul 02 21:23:47.561704 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:23:47.561589    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21c99b83-b557-4c53-81fe-9393280e41c8-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5895zk47\" (UID: \"21c99b83-b557-4c53-81fe-9393280e41c8\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5895zk47"
Jul 02 21:23:47.561891 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:23:47.561862    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21c99b83-b557-4c53-81fe-9393280e41c8-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5895zk47\" (UID: \"21c99b83-b557-4c53-81fe-9393280e41c8\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5895zk47"
Jul 02 21:23:47.561927 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:23:47.561902    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21c99b83-b557-4c53-81fe-9393280e41c8-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5895zk47\" (UID: \"21c99b83-b557-4c53-81fe-9393280e41c8\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5895zk47"
Jul 02 21:23:47.570263 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:23:47.570203    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp9w8\" (UniqueName: \"kubernetes.io/projected/21c99b83-b557-4c53-81fe-9393280e41c8-kube-api-access-lp9w8\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5895zk47\" (UID: \"21c99b83-b557-4c53-81fe-9393280e41c8\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5895zk47"
Jul 02 21:23:47.632101 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:23:47.632048    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5895zk47"
Jul 02 21:23:47.762625 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:23:47.762602    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5895zk47"]
Jul 02 21:23:47.765327 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:23:47.765297    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21c99b83_b557_4c53_81fe_9393280e41c8.slice/crio-a5c434338dd4c82e6a7eb99a9883cf5510f74450a7188557b47cd5c97fd8c434 WatchSource:0}: Error finding container a5c434338dd4c82e6a7eb99a9883cf5510f74450a7188557b47cd5c97fd8c434: Status 404 returned error can't find the container with id a5c434338dd4c82e6a7eb99a9883cf5510f74450a7188557b47cd5c97fd8c434
Jul 02 21:23:47.875889 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:23:47.875805    2586 kubelet.go:1648] "Image garbage collection succeeded"
Jul 02 21:23:48.232026 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:23:48.231978    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5895zk47" event={"ID":"21c99b83-b557-4c53-81fe-9393280e41c8","Type":"ContainerStarted","Data":"a5c434338dd4c82e6a7eb99a9883cf5510f74450a7188557b47cd5c97fd8c434"}
Jul 02 21:23:54.257386 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:23:54.257355    2586 generic.go:356] "Generic (PLEG): container finished" podID="21c99b83-b557-4c53-81fe-9393280e41c8" containerID="fa362bc3e53a231fcae158a85739c5e8f7b380bfa1279308f029469e06c4131a" exitCode=0
Jul 02 21:23:54.257749 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:23:54.257393    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5895zk47" event={"ID":"21c99b83-b557-4c53-81fe-9393280e41c8","Type":"ContainerDied","Data":"fa362bc3e53a231fcae158a85739c5e8f7b380bfa1279308f029469e06c4131a"}
Jul 02 21:23:54.258470 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:23:54.258453    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 21:23:56.268631 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:23:56.268590    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5895zk47" event={"ID":"21c99b83-b557-4c53-81fe-9393280e41c8","Type":"ContainerStarted","Data":"25f73481776f9b13a5bd5e9227d95798c04dd79cbeebd98751ed07678b5f393c"}
Jul 02 21:23:57.273609 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:23:57.273568    2586 generic.go:356] "Generic (PLEG): container finished" podID="21c99b83-b557-4c53-81fe-9393280e41c8" containerID="25f73481776f9b13a5bd5e9227d95798c04dd79cbeebd98751ed07678b5f393c" exitCode=0
Jul 02 21:23:57.274021 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:23:57.273616    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5895zk47" event={"ID":"21c99b83-b557-4c53-81fe-9393280e41c8","Type":"ContainerDied","Data":"25f73481776f9b13a5bd5e9227d95798c04dd79cbeebd98751ed07678b5f393c"}
Jul 02 21:24:05.307221 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:05.307181    2586 generic.go:356] "Generic (PLEG): container finished" podID="21c99b83-b557-4c53-81fe-9393280e41c8" containerID="4ada27b358568089984581d8fd867f2639fc02757b6a77844a8a38813d6c8eca" exitCode=0
Jul 02 21:24:05.307221 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:05.307222    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5895zk47" event={"ID":"21c99b83-b557-4c53-81fe-9393280e41c8","Type":"ContainerDied","Data":"4ada27b358568089984581d8fd867f2639fc02757b6a77844a8a38813d6c8eca"}
Jul 02 21:24:06.435677 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:06.435632    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5895zk47"
Jul 02 21:24:06.517332 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:06.517290    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lp9w8\" (UniqueName: \"kubernetes.io/projected/21c99b83-b557-4c53-81fe-9393280e41c8-kube-api-access-lp9w8\") pod \"21c99b83-b557-4c53-81fe-9393280e41c8\" (UID: \"21c99b83-b557-4c53-81fe-9393280e41c8\") "
Jul 02 21:24:06.517553 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:06.517371    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21c99b83-b557-4c53-81fe-9393280e41c8-util\") pod \"21c99b83-b557-4c53-81fe-9393280e41c8\" (UID: \"21c99b83-b557-4c53-81fe-9393280e41c8\") "
Jul 02 21:24:06.517553 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:06.517431    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21c99b83-b557-4c53-81fe-9393280e41c8-bundle\") pod \"21c99b83-b557-4c53-81fe-9393280e41c8\" (UID: \"21c99b83-b557-4c53-81fe-9393280e41c8\") "
Jul 02 21:24:06.518150 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:06.518122    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c99b83-b557-4c53-81fe-9393280e41c8-bundle" (OuterVolumeSpecName: "bundle") pod "21c99b83-b557-4c53-81fe-9393280e41c8" (UID: "21c99b83-b557-4c53-81fe-9393280e41c8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:24:06.519703 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:06.519681    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21c99b83-b557-4c53-81fe-9393280e41c8-kube-api-access-lp9w8" (OuterVolumeSpecName: "kube-api-access-lp9w8") pod "21c99b83-b557-4c53-81fe-9393280e41c8" (UID: "21c99b83-b557-4c53-81fe-9393280e41c8"). InnerVolumeSpecName "kube-api-access-lp9w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:24:06.521477 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:06.521448    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c99b83-b557-4c53-81fe-9393280e41c8-util" (OuterVolumeSpecName: "util") pod "21c99b83-b557-4c53-81fe-9393280e41c8" (UID: "21c99b83-b557-4c53-81fe-9393280e41c8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:24:06.618337 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:06.618232    2586 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21c99b83-b557-4c53-81fe-9393280e41c8-util\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:24:06.618337 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:06.618273    2586 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21c99b83-b557-4c53-81fe-9393280e41c8-bundle\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:24:06.618337 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:06.618287    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp9w8\" (UniqueName: \"kubernetes.io/projected/21c99b83-b557-4c53-81fe-9393280e41c8-kube-api-access-lp9w8\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:24:07.316106 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:07.316074    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5895zk47" event={"ID":"21c99b83-b557-4c53-81fe-9393280e41c8","Type":"ContainerDied","Data":"a5c434338dd4c82e6a7eb99a9883cf5510f74450a7188557b47cd5c97fd8c434"}
Jul 02 21:24:07.316292 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:07.316119    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5c434338dd4c82e6a7eb99a9883cf5510f74450a7188557b47cd5c97fd8c434"
Jul 02 21:24:07.316292 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:07.316124    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5895zk47"
Jul 02 21:24:09.955513 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:09.955477    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-j5w9q"]
Jul 02 21:24:09.955893 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:09.955799    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21c99b83-b557-4c53-81fe-9393280e41c8" containerName="util"
Jul 02 21:24:09.955893 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:09.955812    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="21c99b83-b557-4c53-81fe-9393280e41c8" containerName="util"
Jul 02 21:24:09.955893 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:09.955830    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21c99b83-b557-4c53-81fe-9393280e41c8" containerName="pull"
Jul 02 21:24:09.955893 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:09.955835    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="21c99b83-b557-4c53-81fe-9393280e41c8" containerName="pull"
Jul 02 21:24:09.955893 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:09.955842    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21c99b83-b557-4c53-81fe-9393280e41c8" containerName="extract"
Jul 02 21:24:09.955893 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:09.955847    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="21c99b83-b557-4c53-81fe-9393280e41c8" containerName="extract"
Jul 02 21:24:09.955893 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:09.955890    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="21c99b83-b557-4c53-81fe-9393280e41c8" containerName="extract"
Jul 02 21:24:09.958690 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:09.958672    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-j5w9q"
Jul 02 21:24:09.961772 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:09.961746    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 02 21:24:09.961905 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:09.961818    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 02 21:24:09.961990 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:09.961972    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-l282m\""
Jul 02 21:24:09.962294 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:09.962280    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 02 21:24:09.975764 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:09.975737    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-j5w9q"]
Jul 02 21:24:10.045807 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:10.045770    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqnzb\" (UniqueName: \"kubernetes.io/projected/f7f442f8-6b00-44a2-9b08-cba6cd6060c7-kube-api-access-vqnzb\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-j5w9q\" (UID: \"f7f442f8-6b00-44a2-9b08-cba6cd6060c7\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-j5w9q"
Jul 02 21:24:10.045807 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:10.045812    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/f7f442f8-6b00-44a2-9b08-cba6cd6060c7-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-j5w9q\" (UID: \"f7f442f8-6b00-44a2-9b08-cba6cd6060c7\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-j5w9q"
Jul 02 21:24:10.146420 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:10.146383    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqnzb\" (UniqueName: \"kubernetes.io/projected/f7f442f8-6b00-44a2-9b08-cba6cd6060c7-kube-api-access-vqnzb\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-j5w9q\" (UID: \"f7f442f8-6b00-44a2-9b08-cba6cd6060c7\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-j5w9q"
Jul 02 21:24:10.146560 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:10.146427    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/f7f442f8-6b00-44a2-9b08-cba6cd6060c7-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-j5w9q\" (UID: \"f7f442f8-6b00-44a2-9b08-cba6cd6060c7\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-j5w9q"
Jul 02 21:24:10.148860 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:10.148838    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/f7f442f8-6b00-44a2-9b08-cba6cd6060c7-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-j5w9q\" (UID: \"f7f442f8-6b00-44a2-9b08-cba6cd6060c7\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-j5w9q"
Jul 02 21:24:10.155961 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:10.155942    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqnzb\" (UniqueName: \"kubernetes.io/projected/f7f442f8-6b00-44a2-9b08-cba6cd6060c7-kube-api-access-vqnzb\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-j5w9q\" (UID: \"f7f442f8-6b00-44a2-9b08-cba6cd6060c7\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-j5w9q"
Jul 02 21:24:10.268859 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:10.268774    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-j5w9q"
Jul 02 21:24:10.405369 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:10.405315    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-j5w9q"]
Jul 02 21:24:10.407817 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:24:10.407790    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f442f8_6b00_44a2_9b08_cba6cd6060c7.slice/crio-8a4921dff70b28422ee7e03e36b754343aef9e35e77365feb2da841f1056a80a WatchSource:0}: Error finding container 8a4921dff70b28422ee7e03e36b754343aef9e35e77365feb2da841f1056a80a: Status 404 returned error can't find the container with id 8a4921dff70b28422ee7e03e36b754343aef9e35e77365feb2da841f1056a80a
Jul 02 21:24:11.337382 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:11.337341    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-j5w9q" event={"ID":"f7f442f8-6b00-44a2-9b08-cba6cd6060c7","Type":"ContainerStarted","Data":"8a4921dff70b28422ee7e03e36b754343aef9e35e77365feb2da841f1056a80a"}
Jul 02 21:24:14.351846 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:14.351810    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-j5w9q" event={"ID":"f7f442f8-6b00-44a2-9b08-cba6cd6060c7","Type":"ContainerStarted","Data":"1616ead0b4014dd875ad01cb9f35d788dd08079d7bd73a089f2ebafd5ab5d625"}
Jul 02 21:24:14.352248 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:14.351903    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-j5w9q"
Jul 02 21:24:14.374801 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:14.374751    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-j5w9q" podStartSLOduration=1.9166941720000001 podStartE2EDuration="5.374737049s" totalImagesPullingTime="3.458042877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:24:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:24:14.372178236 +0000 UTC m=+326.814978703" watchObservedRunningTime="2026-07-02 21:24:14.374737049 +0000 UTC m=+326.817537471"
Jul 02 21:24:35.358168 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:24:35.358138    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-j5w9q"
Jul 02 21:25:23.726285 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:23.726251    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-56668bddcf-x646t"]
Jul 02 21:25:23.729363 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:23.729337    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-56668bddcf-x646t"
Jul 02 21:25:23.730116 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:23.730087    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-5h4d2"]
Jul 02 21:25:23.733210 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:23.733188    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67d95fdc48-5h4d2"
Jul 02 21:25:23.735638 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:23.735615    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 02 21:25:23.735865 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:23.735847    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 02 21:25:23.736952 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:23.736933    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 02 21:25:23.737071 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:23.736959    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 02 21:25:23.737071 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:23.736996    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-br88d\""
Jul 02 21:25:23.737071 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:23.737034    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-wb2ht\""
Jul 02 21:25:23.748728 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:23.748681    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-56668bddcf-x646t"]
Jul 02 21:25:23.750515 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:23.750494    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-5h4d2"]
Jul 02 21:25:23.847401 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:23.847362    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15f58830-8ba4-4736-bcc3-d7d8fb46e2d9-cert\") pod \"kserve-controller-manager-56668bddcf-x646t\" (UID: \"15f58830-8ba4-4736-bcc3-d7d8fb46e2d9\") " pod="kserve/kserve-controller-manager-56668bddcf-x646t"
Jul 02 21:25:23.847401 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:23.847400    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqcbr\" (UniqueName: \"kubernetes.io/projected/15f58830-8ba4-4736-bcc3-d7d8fb46e2d9-kube-api-access-hqcbr\") pod \"kserve-controller-manager-56668bddcf-x646t\" (UID: \"15f58830-8ba4-4736-bcc3-d7d8fb46e2d9\") " pod="kserve/kserve-controller-manager-56668bddcf-x646t"
Jul 02 21:25:23.847679 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:23.847430    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35cb1ddb-bfb7-46bb-89eb-7472e0c33b7e-cert\") pod \"llmisvc-controller-manager-67d95fdc48-5h4d2\" (UID: \"35cb1ddb-bfb7-46bb-89eb-7472e0c33b7e\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-5h4d2"
Jul 02 21:25:23.847679 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:23.847521    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqfw8\" (UniqueName: \"kubernetes.io/projected/35cb1ddb-bfb7-46bb-89eb-7472e0c33b7e-kube-api-access-dqfw8\") pod \"llmisvc-controller-manager-67d95fdc48-5h4d2\" (UID: \"35cb1ddb-bfb7-46bb-89eb-7472e0c33b7e\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-5h4d2"
Jul 02 21:25:23.948685 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:23.948630    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35cb1ddb-bfb7-46bb-89eb-7472e0c33b7e-cert\") pod \"llmisvc-controller-manager-67d95fdc48-5h4d2\" (UID: \"35cb1ddb-bfb7-46bb-89eb-7472e0c33b7e\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-5h4d2"
Jul 02 21:25:23.948685 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:23.948690    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqfw8\" (UniqueName: \"kubernetes.io/projected/35cb1ddb-bfb7-46bb-89eb-7472e0c33b7e-kube-api-access-dqfw8\") pod \"llmisvc-controller-manager-67d95fdc48-5h4d2\" (UID: \"35cb1ddb-bfb7-46bb-89eb-7472e0c33b7e\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-5h4d2"
Jul 02 21:25:23.948928 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:23.948743    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15f58830-8ba4-4736-bcc3-d7d8fb46e2d9-cert\") pod \"kserve-controller-manager-56668bddcf-x646t\" (UID: \"15f58830-8ba4-4736-bcc3-d7d8fb46e2d9\") " pod="kserve/kserve-controller-manager-56668bddcf-x646t"
Jul 02 21:25:23.948928 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:23.948764    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqcbr\" (UniqueName: \"kubernetes.io/projected/15f58830-8ba4-4736-bcc3-d7d8fb46e2d9-kube-api-access-hqcbr\") pod \"kserve-controller-manager-56668bddcf-x646t\" (UID: \"15f58830-8ba4-4736-bcc3-d7d8fb46e2d9\") " pod="kserve/kserve-controller-manager-56668bddcf-x646t"
Jul 02 21:25:23.948928 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:25:23.948811    2586 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 02 21:25:23.948928 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:25:23.948884    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35cb1ddb-bfb7-46bb-89eb-7472e0c33b7e-cert podName:35cb1ddb-bfb7-46bb-89eb-7472e0c33b7e nodeName:}" failed. No retries permitted until 2026-07-02 21:25:24.448864042 +0000 UTC m=+396.891664442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/35cb1ddb-bfb7-46bb-89eb-7472e0c33b7e-cert") pod "llmisvc-controller-manager-67d95fdc48-5h4d2" (UID: "35cb1ddb-bfb7-46bb-89eb-7472e0c33b7e") : secret "llmisvc-webhook-server-cert" not found
Jul 02 21:25:23.951184 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:23.951153    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15f58830-8ba4-4736-bcc3-d7d8fb46e2d9-cert\") pod \"kserve-controller-manager-56668bddcf-x646t\" (UID: \"15f58830-8ba4-4736-bcc3-d7d8fb46e2d9\") " pod="kserve/kserve-controller-manager-56668bddcf-x646t"
Jul 02 21:25:23.959736 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:23.959705    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqfw8\" (UniqueName: \"kubernetes.io/projected/35cb1ddb-bfb7-46bb-89eb-7472e0c33b7e-kube-api-access-dqfw8\") pod \"llmisvc-controller-manager-67d95fdc48-5h4d2\" (UID: \"35cb1ddb-bfb7-46bb-89eb-7472e0c33b7e\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-5h4d2"
Jul 02 21:25:23.960047 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:23.960027    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqcbr\" (UniqueName: \"kubernetes.io/projected/15f58830-8ba4-4736-bcc3-d7d8fb46e2d9-kube-api-access-hqcbr\") pod \"kserve-controller-manager-56668bddcf-x646t\" (UID: \"15f58830-8ba4-4736-bcc3-d7d8fb46e2d9\") " pod="kserve/kserve-controller-manager-56668bddcf-x646t"
Jul 02 21:25:24.040753 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:24.040665    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-56668bddcf-x646t"
Jul 02 21:25:24.170394 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:24.170272    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-56668bddcf-x646t"]
Jul 02 21:25:24.172386 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:25:24.172364    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15f58830_8ba4_4736_bcc3_d7d8fb46e2d9.slice/crio-94bd9c9f33a3cf5291aab013d6a122d79de83da483a72b7d2b9c98a42c1bc1b1 WatchSource:0}: Error finding container 94bd9c9f33a3cf5291aab013d6a122d79de83da483a72b7d2b9c98a42c1bc1b1: Status 404 returned error can't find the container with id 94bd9c9f33a3cf5291aab013d6a122d79de83da483a72b7d2b9c98a42c1bc1b1
Jul 02 21:25:24.454059 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:24.454015    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35cb1ddb-bfb7-46bb-89eb-7472e0c33b7e-cert\") pod \"llmisvc-controller-manager-67d95fdc48-5h4d2\" (UID: \"35cb1ddb-bfb7-46bb-89eb-7472e0c33b7e\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-5h4d2"
Jul 02 21:25:24.456327 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:24.456298    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35cb1ddb-bfb7-46bb-89eb-7472e0c33b7e-cert\") pod \"llmisvc-controller-manager-67d95fdc48-5h4d2\" (UID: \"35cb1ddb-bfb7-46bb-89eb-7472e0c33b7e\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-5h4d2"
Jul 02 21:25:24.646263 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:24.646213    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67d95fdc48-5h4d2"
Jul 02 21:25:24.647526 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:24.647491    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-56668bddcf-x646t" event={"ID":"15f58830-8ba4-4736-bcc3-d7d8fb46e2d9","Type":"ContainerStarted","Data":"94bd9c9f33a3cf5291aab013d6a122d79de83da483a72b7d2b9c98a42c1bc1b1"}
Jul 02 21:25:24.804078 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:24.804040    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-5h4d2"]
Jul 02 21:25:24.807183 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:25:24.807134    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod35cb1ddb_bfb7_46bb_89eb_7472e0c33b7e.slice/crio-056d35b98d108359a966b53159f15915197b117767374a39531242b81f84d2b5 WatchSource:0}: Error finding container 056d35b98d108359a966b53159f15915197b117767374a39531242b81f84d2b5: Status 404 returned error can't find the container with id 056d35b98d108359a966b53159f15915197b117767374a39531242b81f84d2b5
Jul 02 21:25:25.654258 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:25.654217    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67d95fdc48-5h4d2" event={"ID":"35cb1ddb-bfb7-46bb-89eb-7472e0c33b7e","Type":"ContainerStarted","Data":"056d35b98d108359a966b53159f15915197b117767374a39531242b81f84d2b5"}
Jul 02 21:25:28.670580 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:28.670529    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67d95fdc48-5h4d2" event={"ID":"35cb1ddb-bfb7-46bb-89eb-7472e0c33b7e","Type":"ContainerStarted","Data":"a50dbf7efdf99349a701e800e2f934e140e301f158741ec14252b2314ef5105d"}
Jul 02 21:25:28.671031 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:28.670594    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-67d95fdc48-5h4d2"
Jul 02 21:25:28.672021 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:28.671996    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-56668bddcf-x646t" event={"ID":"15f58830-8ba4-4736-bcc3-d7d8fb46e2d9","Type":"ContainerStarted","Data":"598666acca6a2c1dd438c7ebab7d4d12bdeeba237bf6c1463e3173d628516ff0"}
Jul 02 21:25:28.672162 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:28.672059    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-56668bddcf-x646t"
Jul 02 21:25:28.689351 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:28.689305    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-67d95fdc48-5h4d2" podStartSLOduration=2.525915547 podStartE2EDuration="5.689292185s" totalImagesPullingTime="3.163376638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:25:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:25:28.687183757 +0000 UTC m=+401.129984180" watchObservedRunningTime="2026-07-02 21:25:28.689292185 +0000 UTC m=+401.132092607"
Jul 02 21:25:28.705376 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:28.705322    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-56668bddcf-x646t" podStartSLOduration=2.794406976 podStartE2EDuration="5.705304185s" totalImagesPullingTime="2.910897209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:25:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:25:28.704404851 +0000 UTC m=+401.147205300" watchObservedRunningTime="2026-07-02 21:25:28.705304185 +0000 UTC m=+401.148104608"
Jul 02 21:25:59.678240 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:59.678208    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-67d95fdc48-5h4d2"
Jul 02 21:25:59.681279 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:25:59.681258    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-56668bddcf-x646t"
Jul 02 21:26:01.255499 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:01.255465    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-56668bddcf-x646t"]
Jul 02 21:26:01.255922 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:01.255673    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/kserve-controller-manager-56668bddcf-x646t" podUID="15f58830-8ba4-4736-bcc3-d7d8fb46e2d9" containerName="manager" containerID="cri-o://598666acca6a2c1dd438c7ebab7d4d12bdeeba237bf6c1463e3173d628516ff0" gracePeriod=10
Jul 02 21:26:01.281664 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:01.281621    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-56668bddcf-9nl57"]
Jul 02 21:26:01.284809 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:01.284793    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-56668bddcf-9nl57"
Jul 02 21:26:01.292699 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:01.292672    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-56668bddcf-9nl57"]
Jul 02 21:26:01.368071 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:01.368031    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/df26b232-65c5-427e-af02-bebdbc530620-cert\") pod \"kserve-controller-manager-56668bddcf-9nl57\" (UID: \"df26b232-65c5-427e-af02-bebdbc530620\") " pod="kserve/kserve-controller-manager-56668bddcf-9nl57"
Jul 02 21:26:01.368237 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:01.368147    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zwj6\" (UniqueName: \"kubernetes.io/projected/df26b232-65c5-427e-af02-bebdbc530620-kube-api-access-6zwj6\") pod \"kserve-controller-manager-56668bddcf-9nl57\" (UID: \"df26b232-65c5-427e-af02-bebdbc530620\") " pod="kserve/kserve-controller-manager-56668bddcf-9nl57"
Jul 02 21:26:01.468993 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:01.468960    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zwj6\" (UniqueName: \"kubernetes.io/projected/df26b232-65c5-427e-af02-bebdbc530620-kube-api-access-6zwj6\") pod \"kserve-controller-manager-56668bddcf-9nl57\" (UID: \"df26b232-65c5-427e-af02-bebdbc530620\") " pod="kserve/kserve-controller-manager-56668bddcf-9nl57"
Jul 02 21:26:01.469153 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:01.469017    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/df26b232-65c5-427e-af02-bebdbc530620-cert\") pod \"kserve-controller-manager-56668bddcf-9nl57\" (UID: \"df26b232-65c5-427e-af02-bebdbc530620\") " pod="kserve/kserve-controller-manager-56668bddcf-9nl57"
Jul 02 21:26:01.471216 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:01.471189    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/df26b232-65c5-427e-af02-bebdbc530620-cert\") pod \"kserve-controller-manager-56668bddcf-9nl57\" (UID: \"df26b232-65c5-427e-af02-bebdbc530620\") " pod="kserve/kserve-controller-manager-56668bddcf-9nl57"
Jul 02 21:26:01.479178 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:01.479153    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zwj6\" (UniqueName: \"kubernetes.io/projected/df26b232-65c5-427e-af02-bebdbc530620-kube-api-access-6zwj6\") pod \"kserve-controller-manager-56668bddcf-9nl57\" (UID: \"df26b232-65c5-427e-af02-bebdbc530620\") " pod="kserve/kserve-controller-manager-56668bddcf-9nl57"
Jul 02 21:26:01.496415 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:01.496393    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-56668bddcf-x646t"
Jul 02 21:26:01.569583 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:01.569495    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqcbr\" (UniqueName: \"kubernetes.io/projected/15f58830-8ba4-4736-bcc3-d7d8fb46e2d9-kube-api-access-hqcbr\") pod \"15f58830-8ba4-4736-bcc3-d7d8fb46e2d9\" (UID: \"15f58830-8ba4-4736-bcc3-d7d8fb46e2d9\") "
Jul 02 21:26:01.569583 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:01.569530    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15f58830-8ba4-4736-bcc3-d7d8fb46e2d9-cert\") pod \"15f58830-8ba4-4736-bcc3-d7d8fb46e2d9\" (UID: \"15f58830-8ba4-4736-bcc3-d7d8fb46e2d9\") "
Jul 02 21:26:01.571626 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:01.571600    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15f58830-8ba4-4736-bcc3-d7d8fb46e2d9-cert" (OuterVolumeSpecName: "cert") pod "15f58830-8ba4-4736-bcc3-d7d8fb46e2d9" (UID: "15f58830-8ba4-4736-bcc3-d7d8fb46e2d9"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:26:01.571765 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:01.571623    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f58830-8ba4-4736-bcc3-d7d8fb46e2d9-kube-api-access-hqcbr" (OuterVolumeSpecName: "kube-api-access-hqcbr") pod "15f58830-8ba4-4736-bcc3-d7d8fb46e2d9" (UID: "15f58830-8ba4-4736-bcc3-d7d8fb46e2d9"). InnerVolumeSpecName "kube-api-access-hqcbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:26:01.634606 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:01.634559    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-56668bddcf-9nl57"
Jul 02 21:26:01.670881 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:01.670838    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqcbr\" (UniqueName: \"kubernetes.io/projected/15f58830-8ba4-4736-bcc3-d7d8fb46e2d9-kube-api-access-hqcbr\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:26:01.670881 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:01.670867    2586 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15f58830-8ba4-4736-bcc3-d7d8fb46e2d9-cert\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:26:01.762563 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:01.762537    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-56668bddcf-9nl57"]
Jul 02 21:26:01.764997 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:26:01.764970    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf26b232_65c5_427e_af02_bebdbc530620.slice/crio-a4a6e9054112f0408c88c5667dd0d0a2877c41aef85fade07bbdf31a200bbbe0 WatchSource:0}: Error finding container a4a6e9054112f0408c88c5667dd0d0a2877c41aef85fade07bbdf31a200bbbe0: Status 404 returned error can't find the container with id a4a6e9054112f0408c88c5667dd0d0a2877c41aef85fade07bbdf31a200bbbe0
Jul 02 21:26:01.811460 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:01.811422    2586 generic.go:356] "Generic (PLEG): container finished" podID="15f58830-8ba4-4736-bcc3-d7d8fb46e2d9" containerID="598666acca6a2c1dd438c7ebab7d4d12bdeeba237bf6c1463e3173d628516ff0" exitCode=0
Jul 02 21:26:01.811670 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:01.811464    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-56668bddcf-x646t"
Jul 02 21:26:01.811670 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:01.811511    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-56668bddcf-x646t" event={"ID":"15f58830-8ba4-4736-bcc3-d7d8fb46e2d9","Type":"ContainerDied","Data":"598666acca6a2c1dd438c7ebab7d4d12bdeeba237bf6c1463e3173d628516ff0"}
Jul 02 21:26:01.811670 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:01.811561    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-56668bddcf-x646t" event={"ID":"15f58830-8ba4-4736-bcc3-d7d8fb46e2d9","Type":"ContainerDied","Data":"94bd9c9f33a3cf5291aab013d6a122d79de83da483a72b7d2b9c98a42c1bc1b1"}
Jul 02 21:26:01.811670 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:01.811587    2586 scope.go:117] "RemoveContainer" containerID="598666acca6a2c1dd438c7ebab7d4d12bdeeba237bf6c1463e3173d628516ff0"
Jul 02 21:26:01.812762 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:01.812739    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-56668bddcf-9nl57" event={"ID":"df26b232-65c5-427e-af02-bebdbc530620","Type":"ContainerStarted","Data":"a4a6e9054112f0408c88c5667dd0d0a2877c41aef85fade07bbdf31a200bbbe0"}
Jul 02 21:26:01.820363 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:01.820332    2586 scope.go:117] "RemoveContainer" containerID="598666acca6a2c1dd438c7ebab7d4d12bdeeba237bf6c1463e3173d628516ff0"
Jul 02 21:26:01.820613 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:26:01.820594    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"598666acca6a2c1dd438c7ebab7d4d12bdeeba237bf6c1463e3173d628516ff0\": container with ID starting with 598666acca6a2c1dd438c7ebab7d4d12bdeeba237bf6c1463e3173d628516ff0 not found: ID does not exist" containerID="598666acca6a2c1dd438c7ebab7d4d12bdeeba237bf6c1463e3173d628516ff0"
Jul 02 21:26:01.820680 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:01.820623    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"598666acca6a2c1dd438c7ebab7d4d12bdeeba237bf6c1463e3173d628516ff0"} err="failed to get container status \"598666acca6a2c1dd438c7ebab7d4d12bdeeba237bf6c1463e3173d628516ff0\": rpc error: code = NotFound desc = could not find container \"598666acca6a2c1dd438c7ebab7d4d12bdeeba237bf6c1463e3173d628516ff0\": container with ID starting with 598666acca6a2c1dd438c7ebab7d4d12bdeeba237bf6c1463e3173d628516ff0 not found: ID does not exist"
Jul 02 21:26:01.834160 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:01.834132    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-56668bddcf-x646t"]
Jul 02 21:26:01.841123 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:01.841100    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-56668bddcf-x646t"]
Jul 02 21:26:02.022661 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:02.022618    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15f58830-8ba4-4736-bcc3-d7d8fb46e2d9" path="/var/lib/kubelet/pods/15f58830-8ba4-4736-bcc3-d7d8fb46e2d9/volumes"
Jul 02 21:26:02.818602 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:02.818565    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-56668bddcf-9nl57" event={"ID":"df26b232-65c5-427e-af02-bebdbc530620","Type":"ContainerStarted","Data":"f648051e3f66ff3eb65846a5df896745e212b65641dbc9292f91842018cf786e"}
Jul 02 21:26:02.819092 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:02.818690    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-56668bddcf-9nl57"
Jul 02 21:26:02.841875 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:02.838891    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-56668bddcf-9nl57" podStartSLOduration=1.512728653 podStartE2EDuration="1.838867238s" totalImagesPullingTime="326.138585ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:26:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:26:02.83587923 +0000 UTC m=+435.278679665" watchObservedRunningTime="2026-07-02 21:26:02.838867238 +0000 UTC m=+435.281667661"
Jul 02 21:26:15.274922 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:15.274851    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7968ff4bdd-ck5gf"]
Jul 02 21:26:33.830199 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:33.830166    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-56668bddcf-9nl57"
Jul 02 21:26:34.989357 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:34.989325    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-rmwb4"]
Jul 02 21:26:34.989746 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:34.989638    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15f58830-8ba4-4736-bcc3-d7d8fb46e2d9" containerName="manager"
Jul 02 21:26:34.989746 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:34.989663    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="15f58830-8ba4-4736-bcc3-d7d8fb46e2d9" containerName="manager"
Jul 02 21:26:34.989746 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:34.989713    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="15f58830-8ba4-4736-bcc3-d7d8fb46e2d9" containerName="manager"
Jul 02 21:26:34.992959 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:34.992937    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-rmwb4"
Jul 02 21:26:34.997108 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:34.997082    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-w94v4\""
Jul 02 21:26:34.997244 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:34.997114    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 02 21:26:35.003056 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:35.003029    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-rmwb4"]
Jul 02 21:26:35.047435 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:35.047389    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aaef0172-0d63-40b6-8293-de7f631bbdd5-cert\") pod \"odh-model-controller-84b54b7f97-rmwb4\" (UID: \"aaef0172-0d63-40b6-8293-de7f631bbdd5\") " pod="kserve/odh-model-controller-84b54b7f97-rmwb4"
Jul 02 21:26:35.047435 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:35.047437    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vs8f\" (UniqueName: \"kubernetes.io/projected/aaef0172-0d63-40b6-8293-de7f631bbdd5-kube-api-access-8vs8f\") pod \"odh-model-controller-84b54b7f97-rmwb4\" (UID: \"aaef0172-0d63-40b6-8293-de7f631bbdd5\") " pod="kserve/odh-model-controller-84b54b7f97-rmwb4"
Jul 02 21:26:35.148323 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:35.148286    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aaef0172-0d63-40b6-8293-de7f631bbdd5-cert\") pod \"odh-model-controller-84b54b7f97-rmwb4\" (UID: \"aaef0172-0d63-40b6-8293-de7f631bbdd5\") " pod="kserve/odh-model-controller-84b54b7f97-rmwb4"
Jul 02 21:26:35.148323 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:35.148326    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vs8f\" (UniqueName: \"kubernetes.io/projected/aaef0172-0d63-40b6-8293-de7f631bbdd5-kube-api-access-8vs8f\") pod \"odh-model-controller-84b54b7f97-rmwb4\" (UID: \"aaef0172-0d63-40b6-8293-de7f631bbdd5\") " pod="kserve/odh-model-controller-84b54b7f97-rmwb4"
Jul 02 21:26:35.148584 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:26:35.148448    2586 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 02 21:26:35.148584 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:26:35.148537    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaef0172-0d63-40b6-8293-de7f631bbdd5-cert podName:aaef0172-0d63-40b6-8293-de7f631bbdd5 nodeName:}" failed. No retries permitted until 2026-07-02 21:26:35.648513939 +0000 UTC m=+468.091314352 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/aaef0172-0d63-40b6-8293-de7f631bbdd5-cert") pod "odh-model-controller-84b54b7f97-rmwb4" (UID: "aaef0172-0d63-40b6-8293-de7f631bbdd5") : secret "odh-model-controller-webhook-cert" not found
Jul 02 21:26:35.160310 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:35.160280    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vs8f\" (UniqueName: \"kubernetes.io/projected/aaef0172-0d63-40b6-8293-de7f631bbdd5-kube-api-access-8vs8f\") pod \"odh-model-controller-84b54b7f97-rmwb4\" (UID: \"aaef0172-0d63-40b6-8293-de7f631bbdd5\") " pod="kserve/odh-model-controller-84b54b7f97-rmwb4"
Jul 02 21:26:35.652151 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:35.652111    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aaef0172-0d63-40b6-8293-de7f631bbdd5-cert\") pod \"odh-model-controller-84b54b7f97-rmwb4\" (UID: \"aaef0172-0d63-40b6-8293-de7f631bbdd5\") " pod="kserve/odh-model-controller-84b54b7f97-rmwb4"
Jul 02 21:26:35.654565 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:35.654545    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aaef0172-0d63-40b6-8293-de7f631bbdd5-cert\") pod \"odh-model-controller-84b54b7f97-rmwb4\" (UID: \"aaef0172-0d63-40b6-8293-de7f631bbdd5\") " pod="kserve/odh-model-controller-84b54b7f97-rmwb4"
Jul 02 21:26:35.904947 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:35.904840    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-rmwb4"
Jul 02 21:26:36.038223 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:36.038191    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-rmwb4"]
Jul 02 21:26:36.042290 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:26:36.042260    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaef0172_0d63_40b6_8293_de7f631bbdd5.slice/crio-7dabc4d083895801844be88b877b69d4030dc25f4a9e12df083a9a5b374bb25d WatchSource:0}: Error finding container 7dabc4d083895801844be88b877b69d4030dc25f4a9e12df083a9a5b374bb25d: Status 404 returned error can't find the container with id 7dabc4d083895801844be88b877b69d4030dc25f4a9e12df083a9a5b374bb25d
Jul 02 21:26:36.966984 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:36.966942    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-rmwb4" event={"ID":"aaef0172-0d63-40b6-8293-de7f631bbdd5","Type":"ContainerStarted","Data":"7dabc4d083895801844be88b877b69d4030dc25f4a9e12df083a9a5b374bb25d"}
Jul 02 21:26:38.977889 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:38.977848    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-rmwb4" event={"ID":"aaef0172-0d63-40b6-8293-de7f631bbdd5","Type":"ContainerStarted","Data":"8c54cb137c99a363cfa3fd5def6d68a5f75567fd1bc5f8792f0cf2be2423cf4a"}
Jul 02 21:26:38.978307 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:38.978016    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-rmwb4"
Jul 02 21:26:38.996960 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:38.996904    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-rmwb4" podStartSLOduration=2.241452248 podStartE2EDuration="4.99688736s" totalImagesPullingTime="2.755435112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:26:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:26:38.995625177 +0000 UTC m=+471.438425619" watchObservedRunningTime="2026-07-02 21:26:38.99688736 +0000 UTC m=+471.439687782"
Jul 02 21:26:40.295055 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:40.295006    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7968ff4bdd-ck5gf" podUID="75a3868b-74d3-4862-a653-fd9593f577c8" containerName="console" containerID="cri-o://c91b5ec792224430a44a8d3dca5223b7e2b7812d908bbe9dce2673dc11b7298f" gracePeriod=15
Jul 02 21:26:40.537562 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:40.537533    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7968ff4bdd-ck5gf"
Jul 02 21:26:40.587000 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:40.586931    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75a3868b-74d3-4862-a653-fd9593f577c8-oauth-serving-cert\") pod \"75a3868b-74d3-4862-a653-fd9593f577c8\" (UID: \"75a3868b-74d3-4862-a653-fd9593f577c8\") "
Jul 02 21:26:40.587000 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:40.586963    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75a3868b-74d3-4862-a653-fd9593f577c8-console-serving-cert\") pod \"75a3868b-74d3-4862-a653-fd9593f577c8\" (UID: \"75a3868b-74d3-4862-a653-fd9593f577c8\") "
Jul 02 21:26:40.587000 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:40.586996    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75a3868b-74d3-4862-a653-fd9593f577c8-console-oauth-config\") pod \"75a3868b-74d3-4862-a653-fd9593f577c8\" (UID: \"75a3868b-74d3-4862-a653-fd9593f577c8\") "
Jul 02 21:26:40.587255 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:40.587023    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75a3868b-74d3-4862-a653-fd9593f577c8-console-config\") pod \"75a3868b-74d3-4862-a653-fd9593f577c8\" (UID: \"75a3868b-74d3-4862-a653-fd9593f577c8\") "
Jul 02 21:26:40.587255 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:40.587054    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75a3868b-74d3-4862-a653-fd9593f577c8-service-ca\") pod \"75a3868b-74d3-4862-a653-fd9593f577c8\" (UID: \"75a3868b-74d3-4862-a653-fd9593f577c8\") "
Jul 02 21:26:40.587255 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:40.587082    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlvxw\" (UniqueName: \"kubernetes.io/projected/75a3868b-74d3-4862-a653-fd9593f577c8-kube-api-access-rlvxw\") pod \"75a3868b-74d3-4862-a653-fd9593f577c8\" (UID: \"75a3868b-74d3-4862-a653-fd9593f577c8\") "
Jul 02 21:26:40.587255 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:40.587120    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75a3868b-74d3-4862-a653-fd9593f577c8-trusted-ca-bundle\") pod \"75a3868b-74d3-4862-a653-fd9593f577c8\" (UID: \"75a3868b-74d3-4862-a653-fd9593f577c8\") "
Jul 02 21:26:40.587450 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:40.587376    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75a3868b-74d3-4862-a653-fd9593f577c8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "75a3868b-74d3-4862-a653-fd9593f577c8" (UID: "75a3868b-74d3-4862-a653-fd9593f577c8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:26:40.587506 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:40.587443    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75a3868b-74d3-4862-a653-fd9593f577c8-console-config" (OuterVolumeSpecName: "console-config") pod "75a3868b-74d3-4862-a653-fd9593f577c8" (UID: "75a3868b-74d3-4862-a653-fd9593f577c8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:26:40.587595 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:40.587473    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75a3868b-74d3-4862-a653-fd9593f577c8-service-ca" (OuterVolumeSpecName: "service-ca") pod "75a3868b-74d3-4862-a653-fd9593f577c8" (UID: "75a3868b-74d3-4862-a653-fd9593f577c8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:26:40.587746 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:40.587644    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75a3868b-74d3-4862-a653-fd9593f577c8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "75a3868b-74d3-4862-a653-fd9593f577c8" (UID: "75a3868b-74d3-4862-a653-fd9593f577c8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:26:40.589265 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:40.589237    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75a3868b-74d3-4862-a653-fd9593f577c8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "75a3868b-74d3-4862-a653-fd9593f577c8" (UID: "75a3868b-74d3-4862-a653-fd9593f577c8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:26:40.589265 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:40.589246    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75a3868b-74d3-4862-a653-fd9593f577c8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "75a3868b-74d3-4862-a653-fd9593f577c8" (UID: "75a3868b-74d3-4862-a653-fd9593f577c8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:26:40.589469 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:40.589446    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a3868b-74d3-4862-a653-fd9593f577c8-kube-api-access-rlvxw" (OuterVolumeSpecName: "kube-api-access-rlvxw") pod "75a3868b-74d3-4862-a653-fd9593f577c8" (UID: "75a3868b-74d3-4862-a653-fd9593f577c8"). InnerVolumeSpecName "kube-api-access-rlvxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:26:40.687876 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:40.687847    2586 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75a3868b-74d3-4862-a653-fd9593f577c8-oauth-serving-cert\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:26:40.687876 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:40.687873    2586 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75a3868b-74d3-4862-a653-fd9593f577c8-console-serving-cert\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:26:40.687876 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:40.687883    2586 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75a3868b-74d3-4862-a653-fd9593f577c8-console-oauth-config\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:26:40.688090 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:40.687893    2586 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75a3868b-74d3-4862-a653-fd9593f577c8-console-config\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:26:40.688090 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:40.687901    2586 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75a3868b-74d3-4862-a653-fd9593f577c8-service-ca\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:26:40.688090 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:40.687909    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlvxw\" (UniqueName: \"kubernetes.io/projected/75a3868b-74d3-4862-a653-fd9593f577c8-kube-api-access-rlvxw\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:26:40.688090 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:40.687917    2586 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75a3868b-74d3-4862-a653-fd9593f577c8-trusted-ca-bundle\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:26:40.988483 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:40.988443    2586 generic.go:356] "Generic (PLEG): container finished" podID="75a3868b-74d3-4862-a653-fd9593f577c8" containerID="c91b5ec792224430a44a8d3dca5223b7e2b7812d908bbe9dce2673dc11b7298f" exitCode=0
Jul 02 21:26:40.988637 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:40.988493    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7968ff4bdd-ck5gf"
Jul 02 21:26:40.988637 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:40.988526    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7968ff4bdd-ck5gf" event={"ID":"75a3868b-74d3-4862-a653-fd9593f577c8","Type":"ContainerDied","Data":"c91b5ec792224430a44a8d3dca5223b7e2b7812d908bbe9dce2673dc11b7298f"}
Jul 02 21:26:40.988637 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:40.988582    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7968ff4bdd-ck5gf" event={"ID":"75a3868b-74d3-4862-a653-fd9593f577c8","Type":"ContainerDied","Data":"d6f2e8760d85e2457aeb0f9dc95cec242df78971daae1e9bbd298f877b06e2eb"}
Jul 02 21:26:40.988637 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:40.988605    2586 scope.go:117] "RemoveContainer" containerID="c91b5ec792224430a44a8d3dca5223b7e2b7812d908bbe9dce2673dc11b7298f"
Jul 02 21:26:40.997574 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:40.997555    2586 scope.go:117] "RemoveContainer" containerID="c91b5ec792224430a44a8d3dca5223b7e2b7812d908bbe9dce2673dc11b7298f"
Jul 02 21:26:40.997880 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:26:40.997860    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c91b5ec792224430a44a8d3dca5223b7e2b7812d908bbe9dce2673dc11b7298f\": container with ID starting with c91b5ec792224430a44a8d3dca5223b7e2b7812d908bbe9dce2673dc11b7298f not found: ID does not exist" containerID="c91b5ec792224430a44a8d3dca5223b7e2b7812d908bbe9dce2673dc11b7298f"
Jul 02 21:26:40.997956 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:40.997889    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c91b5ec792224430a44a8d3dca5223b7e2b7812d908bbe9dce2673dc11b7298f"} err="failed to get container status \"c91b5ec792224430a44a8d3dca5223b7e2b7812d908bbe9dce2673dc11b7298f\": rpc error: code = NotFound desc = could not find container \"c91b5ec792224430a44a8d3dca5223b7e2b7812d908bbe9dce2673dc11b7298f\": container with ID starting with c91b5ec792224430a44a8d3dca5223b7e2b7812d908bbe9dce2673dc11b7298f not found: ID does not exist"
Jul 02 21:26:41.009736 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:41.009705    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7968ff4bdd-ck5gf"]
Jul 02 21:26:41.014927 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:41.014891    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7968ff4bdd-ck5gf"]
Jul 02 21:26:42.021842 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:42.021804    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75a3868b-74d3-4862-a653-fd9593f577c8" path="/var/lib/kubelet/pods/75a3868b-74d3-4862-a653-fd9593f577c8/volumes"
Jul 02 21:26:49.985147 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:49.985115    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-rmwb4"
Jul 02 21:26:50.810072 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:50.810034    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-v8dmh"]
Jul 02 21:26:50.810349 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:50.810337    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75a3868b-74d3-4862-a653-fd9593f577c8" containerName="console"
Jul 02 21:26:50.810425 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:50.810350    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="75a3868b-74d3-4862-a653-fd9593f577c8" containerName="console"
Jul 02 21:26:50.810425 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:50.810408    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="75a3868b-74d3-4862-a653-fd9593f577c8" containerName="console"
Jul 02 21:26:50.814900 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:50.814882    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-v8dmh"
Jul 02 21:26:50.817429 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:50.817403    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-sx7n8\""
Jul 02 21:26:50.817555 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:50.817403    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 02 21:26:50.820619 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:50.820593    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-v8dmh"]
Jul 02 21:26:50.869890 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:50.869862    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/491466ac-ca40-4477-9967-f1fa459c3fa1-model-cache\") pod \"s3-init-v8dmh\" (UID: \"491466ac-ca40-4477-9967-f1fa459c3fa1\") " pod="kserve/s3-init-v8dmh"
Jul 02 21:26:50.869890 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:50.869895    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfnvq\" (UniqueName: \"kubernetes.io/projected/491466ac-ca40-4477-9967-f1fa459c3fa1-kube-api-access-jfnvq\") pod \"s3-init-v8dmh\" (UID: \"491466ac-ca40-4477-9967-f1fa459c3fa1\") " pod="kserve/s3-init-v8dmh"
Jul 02 21:26:50.970642 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:50.970605    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/491466ac-ca40-4477-9967-f1fa459c3fa1-model-cache\") pod \"s3-init-v8dmh\" (UID: \"491466ac-ca40-4477-9967-f1fa459c3fa1\") " pod="kserve/s3-init-v8dmh"
Jul 02 21:26:50.970758 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:50.970671    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfnvq\" (UniqueName: \"kubernetes.io/projected/491466ac-ca40-4477-9967-f1fa459c3fa1-kube-api-access-jfnvq\") pod \"s3-init-v8dmh\" (UID: \"491466ac-ca40-4477-9967-f1fa459c3fa1\") " pod="kserve/s3-init-v8dmh"
Jul 02 21:26:50.971045 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:50.971023    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/491466ac-ca40-4477-9967-f1fa459c3fa1-model-cache\") pod \"s3-init-v8dmh\" (UID: \"491466ac-ca40-4477-9967-f1fa459c3fa1\") " pod="kserve/s3-init-v8dmh"
Jul 02 21:26:50.978662 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:50.978624    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfnvq\" (UniqueName: \"kubernetes.io/projected/491466ac-ca40-4477-9967-f1fa459c3fa1-kube-api-access-jfnvq\") pod \"s3-init-v8dmh\" (UID: \"491466ac-ca40-4477-9967-f1fa459c3fa1\") " pod="kserve/s3-init-v8dmh"
Jul 02 21:26:51.137149 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:51.137049    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-v8dmh"
Jul 02 21:26:51.278983 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:51.278951    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-v8dmh"]
Jul 02 21:26:51.283137 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:26:51.283109    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod491466ac_ca40_4477_9967_f1fa459c3fa1.slice/crio-ec936648fc3cedf6d12147d318bf4fb3e5a02c9bf2afef4f9687a294ebbda24c WatchSource:0}: Error finding container ec936648fc3cedf6d12147d318bf4fb3e5a02c9bf2afef4f9687a294ebbda24c: Status 404 returned error can't find the container with id ec936648fc3cedf6d12147d318bf4fb3e5a02c9bf2afef4f9687a294ebbda24c
Jul 02 21:26:52.040579 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:52.040540    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-v8dmh" event={"ID":"491466ac-ca40-4477-9967-f1fa459c3fa1","Type":"ContainerStarted","Data":"ec936648fc3cedf6d12147d318bf4fb3e5a02c9bf2afef4f9687a294ebbda24c"}
Jul 02 21:26:56.060500 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:56.060463    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-v8dmh" event={"ID":"491466ac-ca40-4477-9967-f1fa459c3fa1","Type":"ContainerStarted","Data":"ec04e85fdad16c1f1c1908460d4c677c1bea41e7a82591375c8dcac19175a37d"}
Jul 02 21:26:57.065937 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:57.065898    2586 generic.go:356] "Generic (PLEG): container finished" podID="491466ac-ca40-4477-9967-f1fa459c3fa1" containerID="ec04e85fdad16c1f1c1908460d4c677c1bea41e7a82591375c8dcac19175a37d" exitCode=0
Jul 02 21:26:57.066289 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:26:57.065985    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-v8dmh" event={"ID":"491466ac-ca40-4477-9967-f1fa459c3fa1","Type":"ContainerDied","Data":"ec04e85fdad16c1f1c1908460d4c677c1bea41e7a82591375c8dcac19175a37d"}
Jul 02 21:27:02.092400 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:27:02.092314    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-v8dmh" event={"ID":"491466ac-ca40-4477-9967-f1fa459c3fa1","Type":"ContainerStarted","Data":"9f8b68ee156672d246c7c4df235857739ace049bfd1d5614614f25b9111076ea"}
Jul 02 21:27:02.111495 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:27:02.111444    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-v8dmh" podStartSLOduration=1.7610842820000001 podStartE2EDuration="12.111427644s" totalImagesPullingTime="8.774000158s" totalInitContainerRuntime="1.576343204s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:26:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:27:02.111292887 +0000 UTC m=+494.554093345" watchObservedRunningTime="2026-07-02 21:27:02.111427644 +0000 UTC m=+494.554228066"
Jul 02 21:28:07.016715 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:07.016632    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/s3-init-v8dmh" podUID="491466ac-ca40-4477-9967-f1fa459c3fa1" containerName="s3-init" containerID="cri-o://9f8b68ee156672d246c7c4df235857739ace049bfd1d5614614f25b9111076ea" gracePeriod=30
Jul 02 21:28:07.361513 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:07.361423    2586 generic.go:356] "Generic (PLEG): container finished" podID="491466ac-ca40-4477-9967-f1fa459c3fa1" containerID="9f8b68ee156672d246c7c4df235857739ace049bfd1d5614614f25b9111076ea" exitCode=0
Jul 02 21:28:07.361513 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:07.361494    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-v8dmh" event={"ID":"491466ac-ca40-4477-9967-f1fa459c3fa1","Type":"ContainerDied","Data":"9f8b68ee156672d246c7c4df235857739ace049bfd1d5614614f25b9111076ea"}
Jul 02 21:28:07.361711 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:07.361548    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-v8dmh" event={"ID":"491466ac-ca40-4477-9967-f1fa459c3fa1","Type":"ContainerDied","Data":"ec936648fc3cedf6d12147d318bf4fb3e5a02c9bf2afef4f9687a294ebbda24c"}
Jul 02 21:28:07.361711 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:07.361561    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec936648fc3cedf6d12147d318bf4fb3e5a02c9bf2afef4f9687a294ebbda24c"
Jul 02 21:28:07.361711 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:07.361618    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-v8dmh"
Jul 02 21:28:08.365759 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:08.365729    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-v8dmh"
Jul 02 21:28:08.373614 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:08.373596    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-v8dmh"
Jul 02 21:28:08.487858 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:08.487827    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/491466ac-ca40-4477-9967-f1fa459c3fa1-model-cache\") pod \"491466ac-ca40-4477-9967-f1fa459c3fa1\" (UID: \"491466ac-ca40-4477-9967-f1fa459c3fa1\") "
Jul 02 21:28:08.488012 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:08.487875    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfnvq\" (UniqueName: \"kubernetes.io/projected/491466ac-ca40-4477-9967-f1fa459c3fa1-kube-api-access-jfnvq\") pod \"491466ac-ca40-4477-9967-f1fa459c3fa1\" (UID: \"491466ac-ca40-4477-9967-f1fa459c3fa1\") "
Jul 02 21:28:08.489982 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:08.489948    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/491466ac-ca40-4477-9967-f1fa459c3fa1-kube-api-access-jfnvq" (OuterVolumeSpecName: "kube-api-access-jfnvq") pod "491466ac-ca40-4477-9967-f1fa459c3fa1" (UID: "491466ac-ca40-4477-9967-f1fa459c3fa1"). InnerVolumeSpecName "kube-api-access-jfnvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:28:08.506791 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:08.506752    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/491466ac-ca40-4477-9967-f1fa459c3fa1-model-cache" (OuterVolumeSpecName: "model-cache") pod "491466ac-ca40-4477-9967-f1fa459c3fa1" (UID: "491466ac-ca40-4477-9967-f1fa459c3fa1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:28:08.588532 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:08.588498    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/491466ac-ca40-4477-9967-f1fa459c3fa1-model-cache\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:28:08.588532 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:08.588529    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfnvq\" (UniqueName: \"kubernetes.io/projected/491466ac-ca40-4477-9967-f1fa459c3fa1-kube-api-access-jfnvq\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:28:09.369090 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:09.369056    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-v8dmh"
Jul 02 21:28:11.170969 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:11.170936    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-2mxq4"]
Jul 02 21:28:11.171323 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:11.171244    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="491466ac-ca40-4477-9967-f1fa459c3fa1" containerName="s3-init"
Jul 02 21:28:11.171323 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:11.171255    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="491466ac-ca40-4477-9967-f1fa459c3fa1" containerName="s3-init"
Jul 02 21:28:11.171323 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:11.171264    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="491466ac-ca40-4477-9967-f1fa459c3fa1" containerName="download-hf-model"
Jul 02 21:28:11.171323 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:11.171270    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="491466ac-ca40-4477-9967-f1fa459c3fa1" containerName="download-hf-model"
Jul 02 21:28:11.171323 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:11.171316    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="491466ac-ca40-4477-9967-f1fa459c3fa1" containerName="s3-init"
Jul 02 21:28:11.174109 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:11.174092    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2mxq4"
Jul 02 21:28:11.179113 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:11.179091    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 02 21:28:11.179241 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:11.179214    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-sx7n8\""
Jul 02 21:28:11.193629 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:11.193598    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-2mxq4"]
Jul 02 21:28:11.209156 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:11.209113    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54g4z\" (UniqueName: \"kubernetes.io/projected/54222686-9dbf-4df8-91a1-05b1ca92c86f-kube-api-access-54g4z\") pod \"seaweedfs-tls-custom-6644bf5d6c-2mxq4\" (UID: \"54222686-9dbf-4df8-91a1-05b1ca92c86f\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2mxq4"
Jul 02 21:28:11.209347 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:11.209196    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/54222686-9dbf-4df8-91a1-05b1ca92c86f-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-2mxq4\" (UID: \"54222686-9dbf-4df8-91a1-05b1ca92c86f\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2mxq4"
Jul 02 21:28:11.309788 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:11.309740    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54g4z\" (UniqueName: \"kubernetes.io/projected/54222686-9dbf-4df8-91a1-05b1ca92c86f-kube-api-access-54g4z\") pod \"seaweedfs-tls-custom-6644bf5d6c-2mxq4\" (UID: \"54222686-9dbf-4df8-91a1-05b1ca92c86f\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2mxq4"
Jul 02 21:28:11.309958 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:11.309800    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/54222686-9dbf-4df8-91a1-05b1ca92c86f-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-2mxq4\" (UID: \"54222686-9dbf-4df8-91a1-05b1ca92c86f\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2mxq4"
Jul 02 21:28:11.310149 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:11.310131    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/54222686-9dbf-4df8-91a1-05b1ca92c86f-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-2mxq4\" (UID: \"54222686-9dbf-4df8-91a1-05b1ca92c86f\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2mxq4"
Jul 02 21:28:11.319440 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:11.319403    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54g4z\" (UniqueName: \"kubernetes.io/projected/54222686-9dbf-4df8-91a1-05b1ca92c86f-kube-api-access-54g4z\") pod \"seaweedfs-tls-custom-6644bf5d6c-2mxq4\" (UID: \"54222686-9dbf-4df8-91a1-05b1ca92c86f\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2mxq4"
Jul 02 21:28:11.483374 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:11.483329    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2mxq4"
Jul 02 21:28:11.616674 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:11.616631    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-2mxq4"]
Jul 02 21:28:11.619515 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:28:11.619483    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54222686_9dbf_4df8_91a1_05b1ca92c86f.slice/crio-93e0f230193ff0aa56e50ad63b5ee3d64892f3d8dffb721f0f8a8c5fae5f2835 WatchSource:0}: Error finding container 93e0f230193ff0aa56e50ad63b5ee3d64892f3d8dffb721f0f8a8c5fae5f2835: Status 404 returned error can't find the container with id 93e0f230193ff0aa56e50ad63b5ee3d64892f3d8dffb721f0f8a8c5fae5f2835
Jul 02 21:28:12.384975 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:12.384932    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2mxq4" event={"ID":"54222686-9dbf-4df8-91a1-05b1ca92c86f","Type":"ContainerStarted","Data":"93e0f230193ff0aa56e50ad63b5ee3d64892f3d8dffb721f0f8a8c5fae5f2835"}
Jul 02 21:28:15.400209 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:15.400172    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2mxq4" event={"ID":"54222686-9dbf-4df8-91a1-05b1ca92c86f","Type":"ContainerStarted","Data":"208c219d90489b4611cb150129bd546345d8c8e63fb80ed9684899719fa8eb08"}
Jul 02 21:28:15.416433 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:15.416368    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2mxq4" podStartSLOduration=1.422106752 podStartE2EDuration="4.416351856s" totalImagesPullingTime="2.994245104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:28:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:28:15.415352637 +0000 UTC m=+567.858153081" watchObservedRunningTime="2026-07-02 21:28:15.416351856 +0000 UTC m=+567.859152279"
Jul 02 21:28:17.246496 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:17.246456    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-2mxq4"]
Jul 02 21:28:17.409284 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:17.409242    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2mxq4" podUID="54222686-9dbf-4df8-91a1-05b1ca92c86f" containerName="seaweedfs-tls-custom" containerID="cri-o://208c219d90489b4611cb150129bd546345d8c8e63fb80ed9684899719fa8eb08" gracePeriod=30
Jul 02 21:28:18.652555 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:18.652531    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2mxq4"
Jul 02 21:28:18.776870 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:18.776832    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/54222686-9dbf-4df8-91a1-05b1ca92c86f-data\") pod \"54222686-9dbf-4df8-91a1-05b1ca92c86f\" (UID: \"54222686-9dbf-4df8-91a1-05b1ca92c86f\") "
Jul 02 21:28:18.777039 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:18.776910    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54g4z\" (UniqueName: \"kubernetes.io/projected/54222686-9dbf-4df8-91a1-05b1ca92c86f-kube-api-access-54g4z\") pod \"54222686-9dbf-4df8-91a1-05b1ca92c86f\" (UID: \"54222686-9dbf-4df8-91a1-05b1ca92c86f\") "
Jul 02 21:28:18.778157 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:18.778127    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54222686-9dbf-4df8-91a1-05b1ca92c86f-data" (OuterVolumeSpecName: "data") pod "54222686-9dbf-4df8-91a1-05b1ca92c86f" (UID: "54222686-9dbf-4df8-91a1-05b1ca92c86f"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:28:18.779019 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:18.778993    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54222686-9dbf-4df8-91a1-05b1ca92c86f-kube-api-access-54g4z" (OuterVolumeSpecName: "kube-api-access-54g4z") pod "54222686-9dbf-4df8-91a1-05b1ca92c86f" (UID: "54222686-9dbf-4df8-91a1-05b1ca92c86f"). InnerVolumeSpecName "kube-api-access-54g4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:28:18.877419 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:18.877375    2586 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/54222686-9dbf-4df8-91a1-05b1ca92c86f-data\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:28:18.877419 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:18.877414    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54g4z\" (UniqueName: \"kubernetes.io/projected/54222686-9dbf-4df8-91a1-05b1ca92c86f-kube-api-access-54g4z\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:28:19.419485 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:19.419447    2586 generic.go:356] "Generic (PLEG): container finished" podID="54222686-9dbf-4df8-91a1-05b1ca92c86f" containerID="208c219d90489b4611cb150129bd546345d8c8e63fb80ed9684899719fa8eb08" exitCode=0
Jul 02 21:28:19.419485 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:19.419479    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2mxq4" event={"ID":"54222686-9dbf-4df8-91a1-05b1ca92c86f","Type":"ContainerDied","Data":"208c219d90489b4611cb150129bd546345d8c8e63fb80ed9684899719fa8eb08"}
Jul 02 21:28:19.419794 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:19.419503    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2mxq4"
Jul 02 21:28:19.419794 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:19.419530    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2mxq4" event={"ID":"54222686-9dbf-4df8-91a1-05b1ca92c86f","Type":"ContainerDied","Data":"93e0f230193ff0aa56e50ad63b5ee3d64892f3d8dffb721f0f8a8c5fae5f2835"}
Jul 02 21:28:19.419794 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:19.419551    2586 scope.go:117] "RemoveContainer" containerID="208c219d90489b4611cb150129bd546345d8c8e63fb80ed9684899719fa8eb08"
Jul 02 21:28:19.429259 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:19.429241    2586 scope.go:117] "RemoveContainer" containerID="208c219d90489b4611cb150129bd546345d8c8e63fb80ed9684899719fa8eb08"
Jul 02 21:28:19.429572 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:28:19.429551    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"208c219d90489b4611cb150129bd546345d8c8e63fb80ed9684899719fa8eb08\": container with ID starting with 208c219d90489b4611cb150129bd546345d8c8e63fb80ed9684899719fa8eb08 not found: ID does not exist" containerID="208c219d90489b4611cb150129bd546345d8c8e63fb80ed9684899719fa8eb08"
Jul 02 21:28:19.429667 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:19.429581    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"208c219d90489b4611cb150129bd546345d8c8e63fb80ed9684899719fa8eb08"} err="failed to get container status \"208c219d90489b4611cb150129bd546345d8c8e63fb80ed9684899719fa8eb08\": rpc error: code = NotFound desc = could not find container \"208c219d90489b4611cb150129bd546345d8c8e63fb80ed9684899719fa8eb08\": container with ID starting with 208c219d90489b4611cb150129bd546345d8c8e63fb80ed9684899719fa8eb08 not found: ID does not exist"
Jul 02 21:28:19.441806 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:19.441779    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-2mxq4"]
Jul 02 21:28:19.446382 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:19.446355    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-2mxq4"]
Jul 02 21:28:20.022026 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:20.021992    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54222686-9dbf-4df8-91a1-05b1ca92c86f" path="/var/lib/kubelet/pods/54222686-9dbf-4df8-91a1-05b1ca92c86f/volumes"
Jul 02 21:28:21.070363 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:21.070327    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-qc5tr"]
Jul 02 21:28:21.070737 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:21.070627    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54222686-9dbf-4df8-91a1-05b1ca92c86f" containerName="seaweedfs-tls-custom"
Jul 02 21:28:21.070737 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:21.070639    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="54222686-9dbf-4df8-91a1-05b1ca92c86f" containerName="seaweedfs-tls-custom"
Jul 02 21:28:21.070821 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:21.070744    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="54222686-9dbf-4df8-91a1-05b1ca92c86f" containerName="seaweedfs-tls-custom"
Jul 02 21:28:21.075165 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:21.075145    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-qc5tr"
Jul 02 21:28:21.077715 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:21.077694    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 02 21:28:21.077800 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:21.077776    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-sx7n8\""
Jul 02 21:28:21.081963 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:21.081935    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-qc5tr"]
Jul 02 21:28:21.194070 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:21.194035    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bf4e53c-efa8-4886-8404-83229c8a37e7-model-cache\") pod \"s3-tls-init-custom-qc5tr\" (UID: \"6bf4e53c-efa8-4886-8404-83229c8a37e7\") " pod="kserve/s3-tls-init-custom-qc5tr"
Jul 02 21:28:21.194070 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:21.194075    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjwh8\" (UniqueName: \"kubernetes.io/projected/6bf4e53c-efa8-4886-8404-83229c8a37e7-kube-api-access-hjwh8\") pod \"s3-tls-init-custom-qc5tr\" (UID: \"6bf4e53c-efa8-4886-8404-83229c8a37e7\") " pod="kserve/s3-tls-init-custom-qc5tr"
Jul 02 21:28:21.295245 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:21.295204    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bf4e53c-efa8-4886-8404-83229c8a37e7-model-cache\") pod \"s3-tls-init-custom-qc5tr\" (UID: \"6bf4e53c-efa8-4886-8404-83229c8a37e7\") " pod="kserve/s3-tls-init-custom-qc5tr"
Jul 02 21:28:21.295245 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:21.295248    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjwh8\" (UniqueName: \"kubernetes.io/projected/6bf4e53c-efa8-4886-8404-83229c8a37e7-kube-api-access-hjwh8\") pod \"s3-tls-init-custom-qc5tr\" (UID: \"6bf4e53c-efa8-4886-8404-83229c8a37e7\") " pod="kserve/s3-tls-init-custom-qc5tr"
Jul 02 21:28:21.295614 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:21.295588    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bf4e53c-efa8-4886-8404-83229c8a37e7-model-cache\") pod \"s3-tls-init-custom-qc5tr\" (UID: \"6bf4e53c-efa8-4886-8404-83229c8a37e7\") " pod="kserve/s3-tls-init-custom-qc5tr"
Jul 02 21:28:21.303733 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:21.303698    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjwh8\" (UniqueName: \"kubernetes.io/projected/6bf4e53c-efa8-4886-8404-83229c8a37e7-kube-api-access-hjwh8\") pod \"s3-tls-init-custom-qc5tr\" (UID: \"6bf4e53c-efa8-4886-8404-83229c8a37e7\") " pod="kserve/s3-tls-init-custom-qc5tr"
Jul 02 21:28:21.398597 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:21.398505    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-qc5tr"
Jul 02 21:28:21.528727 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:21.528674    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-qc5tr"]
Jul 02 21:28:21.531555 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:28:21.531526    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6bf4e53c_efa8_4886_8404_83229c8a37e7.slice/crio-a0c214bdee62b187f8c555d265b0629349d9dbda1a750d45e7e44bf5fae18b56 WatchSource:0}: Error finding container a0c214bdee62b187f8c555d265b0629349d9dbda1a750d45e7e44bf5fae18b56: Status 404 returned error can't find the container with id a0c214bdee62b187f8c555d265b0629349d9dbda1a750d45e7e44bf5fae18b56
Jul 02 21:28:22.435592 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:22.435550    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-qc5tr" event={"ID":"6bf4e53c-efa8-4886-8404-83229c8a37e7","Type":"ContainerStarted","Data":"bdcb07ba2fc9c2cab54139732940adb550ded155878445145cbbf34578c0a1a1"}
Jul 02 21:28:22.435592 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:22.435599    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-qc5tr" event={"ID":"6bf4e53c-efa8-4886-8404-83229c8a37e7","Type":"ContainerStarted","Data":"a0c214bdee62b187f8c555d265b0629349d9dbda1a750d45e7e44bf5fae18b56"}
Jul 02 21:28:25.451631 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:25.451596    2586 generic.go:356] "Generic (PLEG): container finished" podID="6bf4e53c-efa8-4886-8404-83229c8a37e7" containerID="bdcb07ba2fc9c2cab54139732940adb550ded155878445145cbbf34578c0a1a1" exitCode=0
Jul 02 21:28:25.452028 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:25.451675    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-qc5tr" event={"ID":"6bf4e53c-efa8-4886-8404-83229c8a37e7","Type":"ContainerDied","Data":"bdcb07ba2fc9c2cab54139732940adb550ded155878445145cbbf34578c0a1a1"}
Jul 02 21:28:26.456902 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:26.456871    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-qc5tr" event={"ID":"6bf4e53c-efa8-4886-8404-83229c8a37e7","Type":"ContainerStarted","Data":"3c30bc49eaed7864f2b9aa50ca163e60dd01f7d1012b4da458d29c621d5466d5"}
Jul 02 21:28:26.474836 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:26.474789    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-qc5tr" podStartSLOduration=2.5403435500000002 podStartE2EDuration="5.474772026s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.934428476s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:28:26.473090852 +0000 UTC m=+578.915891278" watchObservedRunningTime="2026-07-02 21:28:26.474772026 +0000 UTC m=+578.917572449"
Jul 02 21:28:33.486582 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:33.486544    2586 generic.go:356] "Generic (PLEG): container finished" podID="6bf4e53c-efa8-4886-8404-83229c8a37e7" containerID="3c30bc49eaed7864f2b9aa50ca163e60dd01f7d1012b4da458d29c621d5466d5" exitCode=0
Jul 02 21:28:33.486942 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:33.486610    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-qc5tr" event={"ID":"6bf4e53c-efa8-4886-8404-83229c8a37e7","Type":"ContainerDied","Data":"3c30bc49eaed7864f2b9aa50ca163e60dd01f7d1012b4da458d29c621d5466d5"}
Jul 02 21:28:34.645025 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:34.644999    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-qc5tr"
Jul 02 21:28:34.714669 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:34.714619    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjwh8\" (UniqueName: \"kubernetes.io/projected/6bf4e53c-efa8-4886-8404-83229c8a37e7-kube-api-access-hjwh8\") pod \"6bf4e53c-efa8-4886-8404-83229c8a37e7\" (UID: \"6bf4e53c-efa8-4886-8404-83229c8a37e7\") "
Jul 02 21:28:34.714847 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:34.714716    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bf4e53c-efa8-4886-8404-83229c8a37e7-model-cache\") pod \"6bf4e53c-efa8-4886-8404-83229c8a37e7\" (UID: \"6bf4e53c-efa8-4886-8404-83229c8a37e7\") "
Jul 02 21:28:34.716749 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:34.716719    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bf4e53c-efa8-4886-8404-83229c8a37e7-kube-api-access-hjwh8" (OuterVolumeSpecName: "kube-api-access-hjwh8") pod "6bf4e53c-efa8-4886-8404-83229c8a37e7" (UID: "6bf4e53c-efa8-4886-8404-83229c8a37e7"). InnerVolumeSpecName "kube-api-access-hjwh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:28:34.734565 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:34.734524    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bf4e53c-efa8-4886-8404-83229c8a37e7-model-cache" (OuterVolumeSpecName: "model-cache") pod "6bf4e53c-efa8-4886-8404-83229c8a37e7" (UID: "6bf4e53c-efa8-4886-8404-83229c8a37e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:28:34.815482 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:34.815391    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjwh8\" (UniqueName: \"kubernetes.io/projected/6bf4e53c-efa8-4886-8404-83229c8a37e7-kube-api-access-hjwh8\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:28:34.815482 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:34.815422    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bf4e53c-efa8-4886-8404-83229c8a37e7-model-cache\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:28:35.496301 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:35.496260    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-qc5tr" event={"ID":"6bf4e53c-efa8-4886-8404-83229c8a37e7","Type":"ContainerDied","Data":"a0c214bdee62b187f8c555d265b0629349d9dbda1a750d45e7e44bf5fae18b56"}
Jul 02 21:28:35.496301 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:35.496305    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0c214bdee62b187f8c555d265b0629349d9dbda1a750d45e7e44bf5fae18b56"
Jul 02 21:28:35.496565 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:35.496271    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-qc5tr"
Jul 02 21:28:39.396568 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:39.396531    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-cd4vg"]
Jul 02 21:28:39.397065 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:39.396872    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bf4e53c-efa8-4886-8404-83229c8a37e7" containerName="download-hf-model"
Jul 02 21:28:39.397065 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:39.396885    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bf4e53c-efa8-4886-8404-83229c8a37e7" containerName="download-hf-model"
Jul 02 21:28:39.397065 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:39.396901    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bf4e53c-efa8-4886-8404-83229c8a37e7" containerName="s3-tls-init-custom"
Jul 02 21:28:39.397065 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:39.396906    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bf4e53c-efa8-4886-8404-83229c8a37e7" containerName="s3-tls-init-custom"
Jul 02 21:28:39.397065 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:39.396963    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bf4e53c-efa8-4886-8404-83229c8a37e7" containerName="s3-tls-init-custom"
Jul 02 21:28:39.403450 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:39.403427    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-cd4vg"
Jul 02 21:28:39.406379 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:39.406354    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 02 21:28:39.406604 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:39.406399    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-sx7n8\""
Jul 02 21:28:39.409755 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:39.409731    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-cd4vg"]
Jul 02 21:28:39.556840 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:39.556809    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc5fx\" (UniqueName: \"kubernetes.io/projected/1903668d-3ca5-4c5f-9243-92adb5d752f6-kube-api-access-kc5fx\") pod \"s3-tls-init-serving-cd4vg\" (UID: \"1903668d-3ca5-4c5f-9243-92adb5d752f6\") " pod="kserve/s3-tls-init-serving-cd4vg"
Jul 02 21:28:39.557024 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:39.556861    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1903668d-3ca5-4c5f-9243-92adb5d752f6-model-cache\") pod \"s3-tls-init-serving-cd4vg\" (UID: \"1903668d-3ca5-4c5f-9243-92adb5d752f6\") " pod="kserve/s3-tls-init-serving-cd4vg"
Jul 02 21:28:39.658202 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:39.658111    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc5fx\" (UniqueName: \"kubernetes.io/projected/1903668d-3ca5-4c5f-9243-92adb5d752f6-kube-api-access-kc5fx\") pod \"s3-tls-init-serving-cd4vg\" (UID: \"1903668d-3ca5-4c5f-9243-92adb5d752f6\") " pod="kserve/s3-tls-init-serving-cd4vg"
Jul 02 21:28:39.658202 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:39.658166    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1903668d-3ca5-4c5f-9243-92adb5d752f6-model-cache\") pod \"s3-tls-init-serving-cd4vg\" (UID: \"1903668d-3ca5-4c5f-9243-92adb5d752f6\") " pod="kserve/s3-tls-init-serving-cd4vg"
Jul 02 21:28:39.658497 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:39.658482    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1903668d-3ca5-4c5f-9243-92adb5d752f6-model-cache\") pod \"s3-tls-init-serving-cd4vg\" (UID: \"1903668d-3ca5-4c5f-9243-92adb5d752f6\") " pod="kserve/s3-tls-init-serving-cd4vg"
Jul 02 21:28:39.666611 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:39.666574    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc5fx\" (UniqueName: \"kubernetes.io/projected/1903668d-3ca5-4c5f-9243-92adb5d752f6-kube-api-access-kc5fx\") pod \"s3-tls-init-serving-cd4vg\" (UID: \"1903668d-3ca5-4c5f-9243-92adb5d752f6\") " pod="kserve/s3-tls-init-serving-cd4vg"
Jul 02 21:28:39.727077 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:39.727037    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-cd4vg"
Jul 02 21:28:39.862248 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:39.862219    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-cd4vg"]
Jul 02 21:28:39.865552 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:28:39.865510    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1903668d_3ca5_4c5f_9243_92adb5d752f6.slice/crio-0cc00c44f38ad7619c5406d49091fb21ad665961ffb9d6ad8d3b0d65ddb47f7a WatchSource:0}: Error finding container 0cc00c44f38ad7619c5406d49091fb21ad665961ffb9d6ad8d3b0d65ddb47f7a: Status 404 returned error can't find the container with id 0cc00c44f38ad7619c5406d49091fb21ad665961ffb9d6ad8d3b0d65ddb47f7a
Jul 02 21:28:40.518247 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:40.518205    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-cd4vg" event={"ID":"1903668d-3ca5-4c5f-9243-92adb5d752f6","Type":"ContainerStarted","Data":"3117ec4195a26412ac0d816edd1cfb78af26b230fcbd1920bde50d714dc81818"}
Jul 02 21:28:40.518247 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:40.518254    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-cd4vg" event={"ID":"1903668d-3ca5-4c5f-9243-92adb5d752f6","Type":"ContainerStarted","Data":"0cc00c44f38ad7619c5406d49091fb21ad665961ffb9d6ad8d3b0d65ddb47f7a"}
Jul 02 21:28:42.530732 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:42.530693    2586 generic.go:356] "Generic (PLEG): container finished" podID="1903668d-3ca5-4c5f-9243-92adb5d752f6" containerID="3117ec4195a26412ac0d816edd1cfb78af26b230fcbd1920bde50d714dc81818" exitCode=0
Jul 02 21:28:42.531390 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:42.530765    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-cd4vg" event={"ID":"1903668d-3ca5-4c5f-9243-92adb5d752f6","Type":"ContainerDied","Data":"3117ec4195a26412ac0d816edd1cfb78af26b230fcbd1920bde50d714dc81818"}
Jul 02 21:28:43.537115 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:43.537082    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-cd4vg" event={"ID":"1903668d-3ca5-4c5f-9243-92adb5d752f6","Type":"ContainerStarted","Data":"309f6ee354ddd935c3b170fb3ee1185fdcc787549c74ff51705c64cc95285fcc"}
Jul 02 21:28:43.554135 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:43.554083    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-cd4vg" podStartSLOduration=2.949066255 podStartE2EDuration="4.554072174s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.605005919s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:28:43.552760926 +0000 UTC m=+595.995561349" watchObservedRunningTime="2026-07-02 21:28:43.554072174 +0000 UTC m=+595.996872596"
Jul 02 21:28:49.564226 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:49.564191    2586 generic.go:356] "Generic (PLEG): container finished" podID="1903668d-3ca5-4c5f-9243-92adb5d752f6" containerID="309f6ee354ddd935c3b170fb3ee1185fdcc787549c74ff51705c64cc95285fcc" exitCode=0
Jul 02 21:28:49.564625 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:49.564231    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-cd4vg" event={"ID":"1903668d-3ca5-4c5f-9243-92adb5d752f6","Type":"ContainerDied","Data":"309f6ee354ddd935c3b170fb3ee1185fdcc787549c74ff51705c64cc95285fcc"}
Jul 02 21:28:50.710558 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:50.710525    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-cd4vg"
Jul 02 21:28:50.754861 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:50.754833    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1903668d-3ca5-4c5f-9243-92adb5d752f6-model-cache\") pod \"1903668d-3ca5-4c5f-9243-92adb5d752f6\" (UID: \"1903668d-3ca5-4c5f-9243-92adb5d752f6\") "
Jul 02 21:28:50.755055 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:50.754881    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kc5fx\" (UniqueName: \"kubernetes.io/projected/1903668d-3ca5-4c5f-9243-92adb5d752f6-kube-api-access-kc5fx\") pod \"1903668d-3ca5-4c5f-9243-92adb5d752f6\" (UID: \"1903668d-3ca5-4c5f-9243-92adb5d752f6\") "
Jul 02 21:28:50.757791 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:50.757760    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1903668d-3ca5-4c5f-9243-92adb5d752f6-kube-api-access-kc5fx" (OuterVolumeSpecName: "kube-api-access-kc5fx") pod "1903668d-3ca5-4c5f-9243-92adb5d752f6" (UID: "1903668d-3ca5-4c5f-9243-92adb5d752f6"). InnerVolumeSpecName "kube-api-access-kc5fx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:28:50.781704 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:50.781673    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1903668d-3ca5-4c5f-9243-92adb5d752f6-model-cache" (OuterVolumeSpecName: "model-cache") pod "1903668d-3ca5-4c5f-9243-92adb5d752f6" (UID: "1903668d-3ca5-4c5f-9243-92adb5d752f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:28:50.856215 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:50.856126    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1903668d-3ca5-4c5f-9243-92adb5d752f6-model-cache\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:28:50.856215 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:50.856158    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc5fx\" (UniqueName: \"kubernetes.io/projected/1903668d-3ca5-4c5f-9243-92adb5d752f6-kube-api-access-kc5fx\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:28:51.575179 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:51.575152    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-cd4vg"
Jul 02 21:28:51.575337 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:51.575152    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-cd4vg" event={"ID":"1903668d-3ca5-4c5f-9243-92adb5d752f6","Type":"ContainerDied","Data":"0cc00c44f38ad7619c5406d49091fb21ad665961ffb9d6ad8d3b0d65ddb47f7a"}
Jul 02 21:28:51.575337 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:28:51.575256    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cc00c44f38ad7619c5406d49091fb21ad665961ffb9d6ad8d3b0d65ddb47f7a"
Jul 02 21:29:03.373112 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:03.373074    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4"]
Jul 02 21:29:03.373485 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:03.373421    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1903668d-3ca5-4c5f-9243-92adb5d752f6" containerName="download-hf-model"
Jul 02 21:29:03.373485 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:03.373433    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="1903668d-3ca5-4c5f-9243-92adb5d752f6" containerName="download-hf-model"
Jul 02 21:29:03.373485 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:03.373442    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1903668d-3ca5-4c5f-9243-92adb5d752f6" containerName="s3-tls-init-serving"
Jul 02 21:29:03.373485 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:03.373448    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="1903668d-3ca5-4c5f-9243-92adb5d752f6" containerName="s3-tls-init-serving"
Jul 02 21:29:03.373623 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:03.373500    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="1903668d-3ca5-4c5f-9243-92adb5d752f6" containerName="s3-tls-init-serving"
Jul 02 21:29:03.377698 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:03.377679    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4"
Jul 02 21:29:03.380385 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:03.380361    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 21:29:03.380503 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:03.380362    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 02 21:29:03.380503 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:03.380363    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 21:29:03.381532 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:03.381517    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pbfjv\""
Jul 02 21:29:03.381867 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:03.381850    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 21:29:03.381959 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:03.381877    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 02 21:29:03.381959 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:03.381894    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 21:29:03.392244 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:03.392221    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4"]
Jul 02 21:29:03.452168 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:03.452128    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3b7deba-0535-4464-adde-8deb987c6539-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-956c77b89-x84t4\" (UID: \"d3b7deba-0535-4464-adde-8deb987c6539\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4"
Jul 02 21:29:03.452318 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:03.452178    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3b7deba-0535-4464-adde-8deb987c6539-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-956c77b89-x84t4\" (UID: \"d3b7deba-0535-4464-adde-8deb987c6539\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4"
Jul 02 21:29:03.452318 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:03.452256    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3b7deba-0535-4464-adde-8deb987c6539-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-956c77b89-x84t4\" (UID: \"d3b7deba-0535-4464-adde-8deb987c6539\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4"
Jul 02 21:29:03.452426 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:03.452312    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3b7deba-0535-4464-adde-8deb987c6539-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-956c77b89-x84t4\" (UID: \"d3b7deba-0535-4464-adde-8deb987c6539\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4"
Jul 02 21:29:03.452426 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:03.452354    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdd7j\" (UniqueName: \"kubernetes.io/projected/d3b7deba-0535-4464-adde-8deb987c6539-kube-api-access-jdd7j\") pod \"isvc-sklearn-batcher-predictor-956c77b89-x84t4\" (UID: \"d3b7deba-0535-4464-adde-8deb987c6539\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4"
Jul 02 21:29:03.553665 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:03.553617    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3b7deba-0535-4464-adde-8deb987c6539-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-956c77b89-x84t4\" (UID: \"d3b7deba-0535-4464-adde-8deb987c6539\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4"
Jul 02 21:29:03.553845 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:03.553697    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3b7deba-0535-4464-adde-8deb987c6539-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-956c77b89-x84t4\" (UID: \"d3b7deba-0535-4464-adde-8deb987c6539\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4"
Jul 02 21:29:03.553845 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:03.553730    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3b7deba-0535-4464-adde-8deb987c6539-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-956c77b89-x84t4\" (UID: \"d3b7deba-0535-4464-adde-8deb987c6539\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4"
Jul 02 21:29:03.553845 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:03.553755    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3b7deba-0535-4464-adde-8deb987c6539-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-956c77b89-x84t4\" (UID: \"d3b7deba-0535-4464-adde-8deb987c6539\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4"
Jul 02 21:29:03.553845 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:03.553777    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdd7j\" (UniqueName: \"kubernetes.io/projected/d3b7deba-0535-4464-adde-8deb987c6539-kube-api-access-jdd7j\") pod \"isvc-sklearn-batcher-predictor-956c77b89-x84t4\" (UID: \"d3b7deba-0535-4464-adde-8deb987c6539\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4"
Jul 02 21:29:03.554094 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:03.554069    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3b7deba-0535-4464-adde-8deb987c6539-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-956c77b89-x84t4\" (UID: \"d3b7deba-0535-4464-adde-8deb987c6539\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4"
Jul 02 21:29:03.554351 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:03.554330    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3b7deba-0535-4464-adde-8deb987c6539-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-956c77b89-x84t4\" (UID: \"d3b7deba-0535-4464-adde-8deb987c6539\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4"
Jul 02 21:29:03.554571 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:03.554548    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3b7deba-0535-4464-adde-8deb987c6539-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-956c77b89-x84t4\" (UID: \"d3b7deba-0535-4464-adde-8deb987c6539\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4"
Jul 02 21:29:03.556263 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:03.556241    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3b7deba-0535-4464-adde-8deb987c6539-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-956c77b89-x84t4\" (UID: \"d3b7deba-0535-4464-adde-8deb987c6539\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4"
Jul 02 21:29:03.562020 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:03.562000    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdd7j\" (UniqueName: \"kubernetes.io/projected/d3b7deba-0535-4464-adde-8deb987c6539-kube-api-access-jdd7j\") pod \"isvc-sklearn-batcher-predictor-956c77b89-x84t4\" (UID: \"d3b7deba-0535-4464-adde-8deb987c6539\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4"
Jul 02 21:29:03.688768 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:03.688740    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4"
Jul 02 21:29:03.825266 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:03.825237    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4"]
Jul 02 21:29:03.827596 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:29:03.827560    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3b7deba_0535_4464_adde_8deb987c6539.slice/crio-46dff043f9c16d116151ac5682eec18d5c94b6d99be295df02f172baee612c65 WatchSource:0}: Error finding container 46dff043f9c16d116151ac5682eec18d5c94b6d99be295df02f172baee612c65: Status 404 returned error can't find the container with id 46dff043f9c16d116151ac5682eec18d5c94b6d99be295df02f172baee612c65
Jul 02 21:29:04.632554 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:04.632518    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" event={"ID":"d3b7deba-0535-4464-adde-8deb987c6539","Type":"ContainerStarted","Data":"f489d2bb0bcafc71adfb5a28c1b47cf142cd0b13d230945312e3c792ca6a1b6d"}
Jul 02 21:29:04.632958 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:04.632584    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" event={"ID":"d3b7deba-0535-4464-adde-8deb987c6539","Type":"ContainerStarted","Data":"46dff043f9c16d116151ac5682eec18d5c94b6d99be295df02f172baee612c65"}
Jul 02 21:29:08.652727 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:08.652694    2586 generic.go:356] "Generic (PLEG): container finished" podID="d3b7deba-0535-4464-adde-8deb987c6539" containerID="f489d2bb0bcafc71adfb5a28c1b47cf142cd0b13d230945312e3c792ca6a1b6d" exitCode=0
Jul 02 21:29:08.653128 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:08.652757    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" event={"ID":"d3b7deba-0535-4464-adde-8deb987c6539","Type":"ContainerDied","Data":"f489d2bb0bcafc71adfb5a28c1b47cf142cd0b13d230945312e3c792ca6a1b6d"}
Jul 02 21:29:08.653878 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:08.653841    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 21:29:22.744950 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:22.744851    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" event={"ID":"d3b7deba-0535-4464-adde-8deb987c6539","Type":"ContainerStarted","Data":"fdad76d84ec1459de4be52a2b873708f0449e4f1adf567370c5dd3de9a879747"}
Jul 02 21:29:24.756793 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:24.756760    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" event={"ID":"d3b7deba-0535-4464-adde-8deb987c6539","Type":"ContainerStarted","Data":"0bd504da6c19f3ef1b9bd78b333e7faf6fb1e65b6c8c14ffe1adc7d14c77e3e3"}
Jul 02 21:29:26.774993 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:26.774946    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" event={"ID":"d3b7deba-0535-4464-adde-8deb987c6539","Type":"ContainerStarted","Data":"8a1ec84c9a2f1369ee51e82a9dfa9f5b715c975a2d3f3852f77d6c51031e6716"}
Jul 02 21:29:26.775392 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:26.775086    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4"
Jul 02 21:29:26.775392 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:26.775140    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4"
Jul 02 21:29:26.776437 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:26.776405    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 21:29:26.798410 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:26.798363    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" podStartSLOduration=2.25197875 podStartE2EDuration="23.798343165s" totalImagesPullingTime="17.77256551s" totalInitContainerRuntime="3.773798905s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:29:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:29:26.797300998 +0000 UTC m=+639.240101420" watchObservedRunningTime="2026-07-02 21:29:26.798343165 +0000 UTC m=+639.241143587"
Jul 02 21:29:27.779696 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:27.779661    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4"
Jul 02 21:29:27.780157 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:27.779819    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 21:29:27.780981 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:27.780950    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:29:28.784455 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:28.784414    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 21:29:28.784865 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:28.784778    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:29:33.788826 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:33.788801    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4"
Jul 02 21:29:33.789350 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:33.789319    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 21:29:33.789615 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:33.789585    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:29:43.789861 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:43.789817    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 21:29:43.790339 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:43.790294    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:29:53.790056 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:53.790012    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 21:29:53.790579 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:29:53.790512    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:30:03.789354 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:03.789311    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 21:30:03.789942 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:03.789785    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:30:13.789421 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:13.789363    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 21:30:13.789937 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:13.789824    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:30:23.789505 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:23.789446    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 21:30:23.790004 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:23.789838    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:30:33.790246 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:33.790196    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 21:30:33.790752 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:33.790725    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:30:43.790707 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:43.790597    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4"
Jul 02 21:30:43.791068 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:43.790717    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4"
Jul 02 21:30:58.571147 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:58.571112    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f"]
Jul 02 21:30:58.575437 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:58.575414    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f"
Jul 02 21:30:58.578116 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:58.578096    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 02 21:30:58.578512 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:58.578492    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 02 21:30:58.594169 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:58.594143    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f"]
Jul 02 21:30:58.607103 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:58.607065    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f\" (UID: \"cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f"
Jul 02 21:30:58.607311 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:58.607103    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mmsq\" (UniqueName: \"kubernetes.io/projected/cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925-kube-api-access-5mmsq\") pod \"isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f\" (UID: \"cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f"
Jul 02 21:30:58.607311 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:58.607217    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f\" (UID: \"cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f"
Jul 02 21:30:58.607311 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:58.607269    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f\" (UID: \"cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f"
Jul 02 21:30:58.607562 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:58.607308    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f\" (UID: \"cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f"
Jul 02 21:30:58.610137 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:58.610112    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4"]
Jul 02 21:30:58.610415 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:58.610362    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="kserve-container" containerID="cri-o://fdad76d84ec1459de4be52a2b873708f0449e4f1adf567370c5dd3de9a879747" gracePeriod=30
Jul 02 21:30:58.610415 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:58.610400    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="agent" containerID="cri-o://8a1ec84c9a2f1369ee51e82a9dfa9f5b715c975a2d3f3852f77d6c51031e6716" gracePeriod=30
Jul 02 21:30:58.610578 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:58.610405    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="kube-rbac-proxy" containerID="cri-o://0bd504da6c19f3ef1b9bd78b333e7faf6fb1e65b6c8c14ffe1adc7d14c77e3e3" gracePeriod=30
Jul 02 21:30:58.708386 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:58.708350    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f\" (UID: \"cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f"
Jul 02 21:30:58.708386 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:58.708391    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mmsq\" (UniqueName: \"kubernetes.io/projected/cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925-kube-api-access-5mmsq\") pod \"isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f\" (UID: \"cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f"
Jul 02 21:30:58.708684 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:58.708420    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f\" (UID: \"cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f"
Jul 02 21:30:58.708684 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:58.708455    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f\" (UID: \"cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f"
Jul 02 21:30:58.708684 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:58.708485    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f\" (UID: \"cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f"
Jul 02 21:30:58.708910 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:58.708885    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f\" (UID: \"cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f"
Jul 02 21:30:58.709285 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:58.709262    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f\" (UID: \"cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f"
Jul 02 21:30:58.709365 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:58.709264    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f\" (UID: \"cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f"
Jul 02 21:30:58.711052 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:58.711032    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f\" (UID: \"cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f"
Jul 02 21:30:58.717583 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:58.717554    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mmsq\" (UniqueName: \"kubernetes.io/projected/cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925-kube-api-access-5mmsq\") pod \"isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f\" (UID: \"cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f"
Jul 02 21:30:58.785309 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:58.785262    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 02 21:30:58.887473 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:58.887377    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f"
Jul 02 21:30:59.022371 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:59.022343    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f"]
Jul 02 21:30:59.024693 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:30:59.024667    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf3e2e4f_c62f_49c6_a622_c4d9c7d3d925.slice/crio-18ed06b9b32c62134753d2013f16ec68a42395c7dfb01a9c4795a94158f0a46e WatchSource:0}: Error finding container 18ed06b9b32c62134753d2013f16ec68a42395c7dfb01a9c4795a94158f0a46e: Status 404 returned error can't find the container with id 18ed06b9b32c62134753d2013f16ec68a42395c7dfb01a9c4795a94158f0a46e
Jul 02 21:30:59.192884 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:59.192854    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" event={"ID":"cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925","Type":"ContainerStarted","Data":"c6a3d8964048221ed2c74192912fcef75074a93cbd9857c6b93a5d8c470ffe13"}
Jul 02 21:30:59.193081 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:59.192923    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" event={"ID":"cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925","Type":"ContainerStarted","Data":"18ed06b9b32c62134753d2013f16ec68a42395c7dfb01a9c4795a94158f0a46e"}
Jul 02 21:30:59.195557 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:59.195529    2586 generic.go:356] "Generic (PLEG): container finished" podID="d3b7deba-0535-4464-adde-8deb987c6539" containerID="0bd504da6c19f3ef1b9bd78b333e7faf6fb1e65b6c8c14ffe1adc7d14c77e3e3" exitCode=2
Jul 02 21:30:59.195701 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:30:59.195576    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" event={"ID":"d3b7deba-0535-4464-adde-8deb987c6539","Type":"ContainerDied","Data":"0bd504da6c19f3ef1b9bd78b333e7faf6fb1e65b6c8c14ffe1adc7d14c77e3e3"}
Jul 02 21:31:03.215916 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:03.215877    2586 generic.go:356] "Generic (PLEG): container finished" podID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerID="c6a3d8964048221ed2c74192912fcef75074a93cbd9857c6b93a5d8c470ffe13" exitCode=0
Jul 02 21:31:03.216328 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:03.215958    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" event={"ID":"cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925","Type":"ContainerDied","Data":"c6a3d8964048221ed2c74192912fcef75074a93cbd9857c6b93a5d8c470ffe13"}
Jul 02 21:31:03.785029 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:03.784940    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 02 21:31:03.789439 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:03.789406    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 21:31:03.789782 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:03.789755    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:31:04.223328 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:04.223296    2586 generic.go:356] "Generic (PLEG): container finished" podID="d3b7deba-0535-4464-adde-8deb987c6539" containerID="fdad76d84ec1459de4be52a2b873708f0449e4f1adf567370c5dd3de9a879747" exitCode=0
Jul 02 21:31:04.223812 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:04.223374    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" event={"ID":"d3b7deba-0535-4464-adde-8deb987c6539","Type":"ContainerDied","Data":"fdad76d84ec1459de4be52a2b873708f0449e4f1adf567370c5dd3de9a879747"}
Jul 02 21:31:04.225752 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:04.225725    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" event={"ID":"cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925","Type":"ContainerStarted","Data":"b84468ecb4b1485fed07fb6c3e0c48dd62be9c995324f2fa531da8c3e2b9b9b3"}
Jul 02 21:31:04.225890 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:04.225784    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" event={"ID":"cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925","Type":"ContainerStarted","Data":"921762d3c45bf596dcb2f20f69cbe9837406d6f95cd52244b4cbc1213df6b75c"}
Jul 02 21:31:04.225890 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:04.225808    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" event={"ID":"cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925","Type":"ContainerStarted","Data":"7813e031f11dbf45c6e3517efafe5412df8e3afe4a2b5581a53bd4d15e341f4c"}
Jul 02 21:31:04.226134 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:04.226114    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f"
Jul 02 21:31:04.226227 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:04.226170    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f"
Jul 02 21:31:04.226227 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:04.226222    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f"
Jul 02 21:31:04.227771 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:04.227744    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:5000: connect: connection refused"
Jul 02 21:31:04.228394 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:04.228373    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:31:04.248826 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:04.248777    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" podStartSLOduration=2.721916642 podStartE2EDuration="6.248746313s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.526829671s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:30:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:31:04.246970249 +0000 UTC m=+736.689770692" watchObservedRunningTime="2026-07-02 21:31:04.248746313 +0000 UTC m=+736.691546734"
Jul 02 21:31:05.231929 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:05.231881    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:5000: connect: connection refused"
Jul 02 21:31:05.232489 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:05.232447    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:31:08.785127 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:08.785083    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 02 21:31:08.785513 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:08.785216    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4"
Jul 02 21:31:10.235968 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:10.235925    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f"
Jul 02 21:31:10.236596 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:10.236568    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:5000: connect: connection refused"
Jul 02 21:31:10.236931 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:10.236907    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:31:13.785127 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:13.785077    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 02 21:31:13.789456 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:13.789424    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 21:31:13.789776 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:13.789751    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:31:18.785297 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:18.785258    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 02 21:31:20.237059 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:20.237015    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:5000: connect: connection refused"
Jul 02 21:31:20.237536 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:20.237510    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:31:23.785510 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:23.785469    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 02 21:31:23.789888 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:23.789842    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 21:31:23.790030 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:23.790004    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4"
Jul 02 21:31:23.790172 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:23.790151    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:31:23.790269 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:23.790255    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4"
Jul 02 21:31:28.784962 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:28.784919    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 02 21:31:29.263406 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:29.263378    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4"
Jul 02 21:31:29.346843 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:29.346805    2586 generic.go:356] "Generic (PLEG): container finished" podID="d3b7deba-0535-4464-adde-8deb987c6539" containerID="8a1ec84c9a2f1369ee51e82a9dfa9f5b715c975a2d3f3852f77d6c51031e6716" exitCode=0
Jul 02 21:31:29.347084 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:29.346859    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4"
Jul 02 21:31:29.347084 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:29.346883    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" event={"ID":"d3b7deba-0535-4464-adde-8deb987c6539","Type":"ContainerDied","Data":"8a1ec84c9a2f1369ee51e82a9dfa9f5b715c975a2d3f3852f77d6c51031e6716"}
Jul 02 21:31:29.347084 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:29.346949    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4" event={"ID":"d3b7deba-0535-4464-adde-8deb987c6539","Type":"ContainerDied","Data":"46dff043f9c16d116151ac5682eec18d5c94b6d99be295df02f172baee612c65"}
Jul 02 21:31:29.347084 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:29.346982    2586 scope.go:117] "RemoveContainer" containerID="8a1ec84c9a2f1369ee51e82a9dfa9f5b715c975a2d3f3852f77d6c51031e6716"
Jul 02 21:31:29.355220 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:29.355201    2586 scope.go:117] "RemoveContainer" containerID="0bd504da6c19f3ef1b9bd78b333e7faf6fb1e65b6c8c14ffe1adc7d14c77e3e3"
Jul 02 21:31:29.362311 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:29.362293    2586 scope.go:117] "RemoveContainer" containerID="fdad76d84ec1459de4be52a2b873708f0449e4f1adf567370c5dd3de9a879747"
Jul 02 21:31:29.369257 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:29.369238    2586 scope.go:117] "RemoveContainer" containerID="f489d2bb0bcafc71adfb5a28c1b47cf142cd0b13d230945312e3c792ca6a1b6d"
Jul 02 21:31:29.371337 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:29.371318    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3b7deba-0535-4464-adde-8deb987c6539-kserve-provision-location\") pod \"d3b7deba-0535-4464-adde-8deb987c6539\" (UID: \"d3b7deba-0535-4464-adde-8deb987c6539\") "
Jul 02 21:31:29.371457 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:29.371363    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdd7j\" (UniqueName: \"kubernetes.io/projected/d3b7deba-0535-4464-adde-8deb987c6539-kube-api-access-jdd7j\") pod \"d3b7deba-0535-4464-adde-8deb987c6539\" (UID: \"d3b7deba-0535-4464-adde-8deb987c6539\") "
Jul 02 21:31:29.371457 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:29.371416    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3b7deba-0535-4464-adde-8deb987c6539-kube-rbac-proxy-sar-config\") pod \"d3b7deba-0535-4464-adde-8deb987c6539\" (UID: \"d3b7deba-0535-4464-adde-8deb987c6539\") "
Jul 02 21:31:29.371457 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:29.371445    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3b7deba-0535-4464-adde-8deb987c6539-proxy-tls\") pod \"d3b7deba-0535-4464-adde-8deb987c6539\" (UID: \"d3b7deba-0535-4464-adde-8deb987c6539\") "
Jul 02 21:31:29.371634 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:29.371608    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3b7deba-0535-4464-adde-8deb987c6539-cabundle-cert\") pod \"d3b7deba-0535-4464-adde-8deb987c6539\" (UID: \"d3b7deba-0535-4464-adde-8deb987c6539\") "
Jul 02 21:31:29.371716 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:29.371635    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b7deba-0535-4464-adde-8deb987c6539-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d3b7deba-0535-4464-adde-8deb987c6539" (UID: "d3b7deba-0535-4464-adde-8deb987c6539"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:31:29.371885 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:29.371862    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3b7deba-0535-4464-adde-8deb987c6539-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d3b7deba-0535-4464-adde-8deb987c6539" (UID: "d3b7deba-0535-4464-adde-8deb987c6539"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:31:29.371976 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:29.371951    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3b7deba-0535-4464-adde-8deb987c6539-kserve-provision-location\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:31:29.372035 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:29.371999    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3b7deba-0535-4464-adde-8deb987c6539-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d3b7deba-0535-4464-adde-8deb987c6539" (UID: "d3b7deba-0535-4464-adde-8deb987c6539"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:31:29.373662 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:29.373617    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3b7deba-0535-4464-adde-8deb987c6539-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d3b7deba-0535-4464-adde-8deb987c6539" (UID: "d3b7deba-0535-4464-adde-8deb987c6539"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:31:29.373762 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:29.373752    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3b7deba-0535-4464-adde-8deb987c6539-kube-api-access-jdd7j" (OuterVolumeSpecName: "kube-api-access-jdd7j") pod "d3b7deba-0535-4464-adde-8deb987c6539" (UID: "d3b7deba-0535-4464-adde-8deb987c6539"). InnerVolumeSpecName "kube-api-access-jdd7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:31:29.376791 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:29.376776    2586 scope.go:117] "RemoveContainer" containerID="8a1ec84c9a2f1369ee51e82a9dfa9f5b715c975a2d3f3852f77d6c51031e6716"
Jul 02 21:31:29.377055 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:31:29.377034    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a1ec84c9a2f1369ee51e82a9dfa9f5b715c975a2d3f3852f77d6c51031e6716\": container with ID starting with 8a1ec84c9a2f1369ee51e82a9dfa9f5b715c975a2d3f3852f77d6c51031e6716 not found: ID does not exist" containerID="8a1ec84c9a2f1369ee51e82a9dfa9f5b715c975a2d3f3852f77d6c51031e6716"
Jul 02 21:31:29.377123 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:29.377067    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a1ec84c9a2f1369ee51e82a9dfa9f5b715c975a2d3f3852f77d6c51031e6716"} err="failed to get container status \"8a1ec84c9a2f1369ee51e82a9dfa9f5b715c975a2d3f3852f77d6c51031e6716\": rpc error: code = NotFound desc = could not find container \"8a1ec84c9a2f1369ee51e82a9dfa9f5b715c975a2d3f3852f77d6c51031e6716\": container with ID starting with 8a1ec84c9a2f1369ee51e82a9dfa9f5b715c975a2d3f3852f77d6c51031e6716 not found: ID does not exist"
Jul 02 21:31:29.377123 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:29.377103    2586 scope.go:117] "RemoveContainer" containerID="0bd504da6c19f3ef1b9bd78b333e7faf6fb1e65b6c8c14ffe1adc7d14c77e3e3"
Jul 02 21:31:29.377358 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:31:29.377340    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bd504da6c19f3ef1b9bd78b333e7faf6fb1e65b6c8c14ffe1adc7d14c77e3e3\": container with ID starting with 0bd504da6c19f3ef1b9bd78b333e7faf6fb1e65b6c8c14ffe1adc7d14c77e3e3 not found: ID does not exist" containerID="0bd504da6c19f3ef1b9bd78b333e7faf6fb1e65b6c8c14ffe1adc7d14c77e3e3"
Jul 02 21:31:29.377395 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:29.377364    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bd504da6c19f3ef1b9bd78b333e7faf6fb1e65b6c8c14ffe1adc7d14c77e3e3"} err="failed to get container status \"0bd504da6c19f3ef1b9bd78b333e7faf6fb1e65b6c8c14ffe1adc7d14c77e3e3\": rpc error: code = NotFound desc = could not find container \"0bd504da6c19f3ef1b9bd78b333e7faf6fb1e65b6c8c14ffe1adc7d14c77e3e3\": container with ID starting with 0bd504da6c19f3ef1b9bd78b333e7faf6fb1e65b6c8c14ffe1adc7d14c77e3e3 not found: ID does not exist"
Jul 02 21:31:29.377395 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:29.377379    2586 scope.go:117] "RemoveContainer" containerID="fdad76d84ec1459de4be52a2b873708f0449e4f1adf567370c5dd3de9a879747"
Jul 02 21:31:29.377666 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:31:29.377608    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdad76d84ec1459de4be52a2b873708f0449e4f1adf567370c5dd3de9a879747\": container with ID starting with fdad76d84ec1459de4be52a2b873708f0449e4f1adf567370c5dd3de9a879747 not found: ID does not exist" containerID="fdad76d84ec1459de4be52a2b873708f0449e4f1adf567370c5dd3de9a879747"
Jul 02 21:31:29.377666 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:29.377628    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdad76d84ec1459de4be52a2b873708f0449e4f1adf567370c5dd3de9a879747"} err="failed to get container status \"fdad76d84ec1459de4be52a2b873708f0449e4f1adf567370c5dd3de9a879747\": rpc error: code = NotFound desc = could not find container \"fdad76d84ec1459de4be52a2b873708f0449e4f1adf567370c5dd3de9a879747\": container with ID starting with fdad76d84ec1459de4be52a2b873708f0449e4f1adf567370c5dd3de9a879747 not found: ID does not exist"
Jul 02 21:31:29.377666 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:29.377642    2586 scope.go:117] "RemoveContainer" containerID="f489d2bb0bcafc71adfb5a28c1b47cf142cd0b13d230945312e3c792ca6a1b6d"
Jul 02 21:31:29.377946 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:31:29.377926    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f489d2bb0bcafc71adfb5a28c1b47cf142cd0b13d230945312e3c792ca6a1b6d\": container with ID starting with f489d2bb0bcafc71adfb5a28c1b47cf142cd0b13d230945312e3c792ca6a1b6d not found: ID does not exist" containerID="f489d2bb0bcafc71adfb5a28c1b47cf142cd0b13d230945312e3c792ca6a1b6d"
Jul 02 21:31:29.377990 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:29.377951    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f489d2bb0bcafc71adfb5a28c1b47cf142cd0b13d230945312e3c792ca6a1b6d"} err="failed to get container status \"f489d2bb0bcafc71adfb5a28c1b47cf142cd0b13d230945312e3c792ca6a1b6d\": rpc error: code = NotFound desc = could not find container \"f489d2bb0bcafc71adfb5a28c1b47cf142cd0b13d230945312e3c792ca6a1b6d\": container with ID starting with f489d2bb0bcafc71adfb5a28c1b47cf142cd0b13d230945312e3c792ca6a1b6d not found: ID does not exist"
Jul 02 21:31:29.472492 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:29.472459    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3b7deba-0535-4464-adde-8deb987c6539-cabundle-cert\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:31:29.472492 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:29.472483    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdd7j\" (UniqueName: \"kubernetes.io/projected/d3b7deba-0535-4464-adde-8deb987c6539-kube-api-access-jdd7j\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:31:29.472492 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:29.472492    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3b7deba-0535-4464-adde-8deb987c6539-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:31:29.472755 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:29.472507    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3b7deba-0535-4464-adde-8deb987c6539-proxy-tls\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:31:29.670138 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:29.670109    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4"]
Jul 02 21:31:29.674092 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:29.674068    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-x84t4"]
Jul 02 21:31:30.022706 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:30.022674    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3b7deba-0535-4464-adde-8deb987c6539" path="/var/lib/kubelet/pods/d3b7deba-0535-4464-adde-8deb987c6539/volumes"
Jul 02 21:31:30.236820 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:30.236781    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:5000: connect: connection refused"
Jul 02 21:31:30.237310 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:30.237282    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:31:40.237164 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:40.237114    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:5000: connect: connection refused"
Jul 02 21:31:40.237668 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:40.237630    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:31:50.236894 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:50.236843    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:5000: connect: connection refused"
Jul 02 21:31:50.237382 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:31:50.237334    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:32:00.237148 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:00.237102    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:5000: connect: connection refused"
Jul 02 21:32:00.237664 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:00.237622    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:32:10.236498 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:10.236397    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:5000: connect: connection refused"
Jul 02 21:32:10.236931 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:10.236911    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:32:20.237810 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:20.237779    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f"
Jul 02 21:32:20.238334 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:20.238022    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f"
Jul 02 21:32:33.710892 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:33.710859    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-bt87v"]
Jul 02 21:32:33.711250 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:33.711182    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="agent"
Jul 02 21:32:33.711250 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:33.711192    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="agent"
Jul 02 21:32:33.711250 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:33.711207    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="kserve-container"
Jul 02 21:32:33.711250 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:33.711213    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="kserve-container"
Jul 02 21:32:33.711250 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:33.711223    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="kube-rbac-proxy"
Jul 02 21:32:33.711250 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:33.711228    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="kube-rbac-proxy"
Jul 02 21:32:33.711250 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:33.711238    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="storage-initializer"
Jul 02 21:32:33.711250 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:33.711244    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="storage-initializer"
Jul 02 21:32:33.711493 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:33.711292    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="kserve-container"
Jul 02 21:32:33.711493 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:33.711300    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="kube-rbac-proxy"
Jul 02 21:32:33.711493 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:33.711310    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3b7deba-0535-4464-adde-8deb987c6539" containerName="agent"
Jul 02 21:32:33.714479 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:33.714460    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-bt87v"
Jul 02 21:32:33.718731 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:33.718703    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Jul 02 21:32:33.719054 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:33.719028    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Jul 02 21:32:33.727679 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:33.727636    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-bt87v"]
Jul 02 21:32:33.755350 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:33.755316    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f"]
Jul 02 21:32:33.755619 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:33.755583    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="kserve-container" containerID="cri-o://7813e031f11dbf45c6e3517efafe5412df8e3afe4a2b5581a53bd4d15e341f4c" gracePeriod=30
Jul 02 21:32:33.755619 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:33.755602    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="kube-rbac-proxy" containerID="cri-o://921762d3c45bf596dcb2f20f69cbe9837406d6f95cd52244b4cbc1213df6b75c" gracePeriod=30
Jul 02 21:32:33.755806 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:33.755604    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="agent" containerID="cri-o://b84468ecb4b1485fed07fb6c3e0c48dd62be9c995324f2fa531da8c3e2b9b9b3" gracePeriod=30
Jul 02 21:32:33.820751 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:33.820713    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv4z4\" (UniqueName: \"kubernetes.io/projected/37135ac3-b594-414c-8135-3a462e132e1f-kube-api-access-gv4z4\") pod \"message-dumper-predictor-6575bcfdfd-bt87v\" (UID: \"37135ac3-b594-414c-8135-3a462e132e1f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-bt87v"
Jul 02 21:32:33.821034 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:33.820770    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37135ac3-b594-414c-8135-3a462e132e1f-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-bt87v\" (UID: \"37135ac3-b594-414c-8135-3a462e132e1f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-bt87v"
Jul 02 21:32:33.821034 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:33.820859    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37135ac3-b594-414c-8135-3a462e132e1f-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-bt87v\" (UID: \"37135ac3-b594-414c-8135-3a462e132e1f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-bt87v"
Jul 02 21:32:33.921358 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:33.921319    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv4z4\" (UniqueName: \"kubernetes.io/projected/37135ac3-b594-414c-8135-3a462e132e1f-kube-api-access-gv4z4\") pod \"message-dumper-predictor-6575bcfdfd-bt87v\" (UID: \"37135ac3-b594-414c-8135-3a462e132e1f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-bt87v"
Jul 02 21:32:33.921358 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:33.921364    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37135ac3-b594-414c-8135-3a462e132e1f-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-bt87v\" (UID: \"37135ac3-b594-414c-8135-3a462e132e1f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-bt87v"
Jul 02 21:32:33.921628 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:33.921600    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37135ac3-b594-414c-8135-3a462e132e1f-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-bt87v\" (UID: \"37135ac3-b594-414c-8135-3a462e132e1f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-bt87v"
Jul 02 21:32:33.922023 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:33.922004    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37135ac3-b594-414c-8135-3a462e132e1f-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-bt87v\" (UID: \"37135ac3-b594-414c-8135-3a462e132e1f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-bt87v"
Jul 02 21:32:33.924014 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:33.923985    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37135ac3-b594-414c-8135-3a462e132e1f-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-bt87v\" (UID: \"37135ac3-b594-414c-8135-3a462e132e1f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-bt87v"
Jul 02 21:32:33.929193 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:33.929169    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv4z4\" (UniqueName: \"kubernetes.io/projected/37135ac3-b594-414c-8135-3a462e132e1f-kube-api-access-gv4z4\") pod \"message-dumper-predictor-6575bcfdfd-bt87v\" (UID: \"37135ac3-b594-414c-8135-3a462e132e1f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-bt87v"
Jul 02 21:32:34.026049 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:34.025965    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-bt87v"
Jul 02 21:32:34.159318 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:34.159294    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-bt87v"]
Jul 02 21:32:34.161722 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:32:34.161689    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37135ac3_b594_414c_8135_3a462e132e1f.slice/crio-ff8d28e2173f0ba238f53a0b892af30f4a04e4ef1a5f7fe3950c8bca504ac99d WatchSource:0}: Error finding container ff8d28e2173f0ba238f53a0b892af30f4a04e4ef1a5f7fe3950c8bca504ac99d: Status 404 returned error can't find the container with id ff8d28e2173f0ba238f53a0b892af30f4a04e4ef1a5f7fe3950c8bca504ac99d
Jul 02 21:32:34.643756 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:34.643714    2586 generic.go:356] "Generic (PLEG): container finished" podID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerID="921762d3c45bf596dcb2f20f69cbe9837406d6f95cd52244b4cbc1213df6b75c" exitCode=2
Jul 02 21:32:34.643960 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:34.643791    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" event={"ID":"cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925","Type":"ContainerDied","Data":"921762d3c45bf596dcb2f20f69cbe9837406d6f95cd52244b4cbc1213df6b75c"}
Jul 02 21:32:34.644900 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:34.644861    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-bt87v" event={"ID":"37135ac3-b594-414c-8135-3a462e132e1f","Type":"ContainerStarted","Data":"ff8d28e2173f0ba238f53a0b892af30f4a04e4ef1a5f7fe3950c8bca504ac99d"}
Jul 02 21:32:35.232324 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:35.232275    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 02 21:32:35.651295 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:35.651267    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-bt87v" event={"ID":"37135ac3-b594-414c-8135-3a462e132e1f","Type":"ContainerStarted","Data":"9a57663d0010d3f01933ca427b4a80cd002d668dac34f4dd3a70cc996b0e2e0f"}
Jul 02 21:32:36.656919 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:36.656885    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-bt87v" event={"ID":"37135ac3-b594-414c-8135-3a462e132e1f","Type":"ContainerStarted","Data":"8bd93c35e193cf767a2ee5eebbf7ea06c08cec7e08467649ceccf885ca78dea0"}
Jul 02 21:32:36.657363 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:36.656947    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-bt87v"
Jul 02 21:32:36.657363 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:36.656995    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-bt87v"
Jul 02 21:32:36.658569 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:36.658547    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-bt87v"
Jul 02 21:32:36.673969 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:36.673926    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-bt87v" podStartSLOduration=2.288463342 podStartE2EDuration="3.673912177s" totalImagesPullingTime="1.385448835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:32:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:32:36.672911529 +0000 UTC m=+829.115711976" watchObservedRunningTime="2026-07-02 21:32:36.673912177 +0000 UTC m=+829.116712600"
Jul 02 21:32:38.667308 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:38.667274    2586 generic.go:356] "Generic (PLEG): container finished" podID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerID="7813e031f11dbf45c6e3517efafe5412df8e3afe4a2b5581a53bd4d15e341f4c" exitCode=0
Jul 02 21:32:38.667773 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:38.667342    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" event={"ID":"cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925","Type":"ContainerDied","Data":"7813e031f11dbf45c6e3517efafe5412df8e3afe4a2b5581a53bd4d15e341f4c"}
Jul 02 21:32:40.232854 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:40.232801    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 02 21:32:40.237135 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:40.237092    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:5000: connect: connection refused"
Jul 02 21:32:40.237469 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:40.237443    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:32:43.673978 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:43.673948    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-bt87v"
Jul 02 21:32:45.232786 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:45.232747    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 02 21:32:45.233171 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:45.232874    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f"
Jul 02 21:32:50.232722 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:50.232681    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 02 21:32:50.236974 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:50.236935    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:5000: connect: connection refused"
Jul 02 21:32:50.237422 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:50.237397    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:32:53.770541 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:53.770504    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h"]
Jul 02 21:32:53.774912 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:53.774890    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h"
Jul 02 21:32:53.777520 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:53.777498    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-757779fc66-gmt6h\" (UID: \"d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h"
Jul 02 21:32:53.777641 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:53.777542    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd-proxy-tls\") pod \"isvc-logger-predictor-757779fc66-gmt6h\" (UID: \"d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h"
Jul 02 21:32:53.777641 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:53.777595    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htpgt\" (UniqueName: \"kubernetes.io/projected/d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd-kube-api-access-htpgt\") pod \"isvc-logger-predictor-757779fc66-gmt6h\" (UID: \"d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h"
Jul 02 21:32:53.777641 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:53.777614    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 02 21:32:53.777641 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:53.777619    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd-cabundle-cert\") pod \"isvc-logger-predictor-757779fc66-gmt6h\" (UID: \"d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h"
Jul 02 21:32:53.777900 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:53.777673    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 02 21:32:53.777900 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:53.777706    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd-kserve-provision-location\") pod \"isvc-logger-predictor-757779fc66-gmt6h\" (UID: \"d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h"
Jul 02 21:32:53.786241 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:53.786208    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h"]
Jul 02 21:32:53.878182 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:53.878150    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-757779fc66-gmt6h\" (UID: \"d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h"
Jul 02 21:32:53.878182 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:53.878184    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd-proxy-tls\") pod \"isvc-logger-predictor-757779fc66-gmt6h\" (UID: \"d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h"
Jul 02 21:32:53.878410 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:53.878327    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htpgt\" (UniqueName: \"kubernetes.io/projected/d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd-kube-api-access-htpgt\") pod \"isvc-logger-predictor-757779fc66-gmt6h\" (UID: \"d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h"
Jul 02 21:32:53.878410 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:53.878370    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd-cabundle-cert\") pod \"isvc-logger-predictor-757779fc66-gmt6h\" (UID: \"d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h"
Jul 02 21:32:53.878508 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:53.878420    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd-kserve-provision-location\") pod \"isvc-logger-predictor-757779fc66-gmt6h\" (UID: \"d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h"
Jul 02 21:32:53.878831 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:53.878802    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd-kserve-provision-location\") pod \"isvc-logger-predictor-757779fc66-gmt6h\" (UID: \"d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h"
Jul 02 21:32:53.878966 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:53.878914    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-757779fc66-gmt6h\" (UID: \"d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h"
Jul 02 21:32:53.879009 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:53.878971    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd-cabundle-cert\") pod \"isvc-logger-predictor-757779fc66-gmt6h\" (UID: \"d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h"
Jul 02 21:32:53.880625 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:53.880606    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd-proxy-tls\") pod \"isvc-logger-predictor-757779fc66-gmt6h\" (UID: \"d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h"
Jul 02 21:32:53.886247 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:53.886222    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htpgt\" (UniqueName: \"kubernetes.io/projected/d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd-kube-api-access-htpgt\") pod \"isvc-logger-predictor-757779fc66-gmt6h\" (UID: \"d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h"
Jul 02 21:32:54.086418 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:54.086333    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h"
Jul 02 21:32:54.225882 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:54.225850    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h"]
Jul 02 21:32:54.228049 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:32:54.228013    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd00dbfb7_153a_4f77_bdd2_0fba5dbdadbd.slice/crio-fa0628b34cbebbedeff96456c02bba9cf4b59b2df9de70ace724decbbe1d943c WatchSource:0}: Error finding container fa0628b34cbebbedeff96456c02bba9cf4b59b2df9de70ace724decbbe1d943c: Status 404 returned error can't find the container with id fa0628b34cbebbedeff96456c02bba9cf4b59b2df9de70ace724decbbe1d943c
Jul 02 21:32:54.741532 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:54.741497    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" event={"ID":"d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd","Type":"ContainerStarted","Data":"3a36424578468c272b666e6ad6d55bb9a54b81bbeaff9ac532aa42d8457413ca"}
Jul 02 21:32:54.741695 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:54.741561    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" event={"ID":"d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd","Type":"ContainerStarted","Data":"fa0628b34cbebbedeff96456c02bba9cf4b59b2df9de70ace724decbbe1d943c"}
Jul 02 21:32:55.232169 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:55.232123    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 02 21:32:58.762532 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:58.762498    2586 generic.go:356] "Generic (PLEG): container finished" podID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerID="3a36424578468c272b666e6ad6d55bb9a54b81bbeaff9ac532aa42d8457413ca" exitCode=0
Jul 02 21:32:58.762971 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:58.762578    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" event={"ID":"d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd","Type":"ContainerDied","Data":"3a36424578468c272b666e6ad6d55bb9a54b81bbeaff9ac532aa42d8457413ca"}
Jul 02 21:32:59.769826 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:59.769790    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" event={"ID":"d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd","Type":"ContainerStarted","Data":"c5a9f797683fbb6dec0cdbc84ed4f95b71f05aeeba37642d223e57ff90cc00ec"}
Jul 02 21:32:59.770177 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:59.769861    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" event={"ID":"d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd","Type":"ContainerStarted","Data":"303d6748b47f429b9354f2d83b567cf1529c6ef2b8e782e1c3e176681c9b9ff0"}
Jul 02 21:32:59.770177 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:59.769899    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" event={"ID":"d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd","Type":"ContainerStarted","Data":"fe58ad59fd645abc19a02c91d2eb2a0316d18698077ea85d433501b200f36e9a"}
Jul 02 21:32:59.770261 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:59.770203    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h"
Jul 02 21:32:59.770261 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:59.770258    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h"
Jul 02 21:32:59.770323 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:59.770292    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h"
Jul 02 21:32:59.771674 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:59.771625    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 02 21:32:59.772283 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:59.772263    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:32:59.792148 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:32:59.792105    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" podStartSLOduration=2.887904659 podStartE2EDuration="6.792090803s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.904186144s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:32:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:32:59.790066772 +0000 UTC m=+852.232879379" watchObservedRunningTime="2026-07-02 21:32:59.792090803 +0000 UTC m=+852.234891225"
Jul 02 21:33:00.232154 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:00.232109    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 02 21:33:00.236858 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:00.236816    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:5000: connect: connection refused"
Jul 02 21:33:00.237012 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:00.236958    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f"
Jul 02 21:33:00.237419 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:00.237394    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:33:00.237515 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:00.237505    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f"
Jul 02 21:33:00.777737 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:00.777698    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 02 21:33:00.778128 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:00.778104    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:33:03.794618 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:03.794584    2586 generic.go:356] "Generic (PLEG): container finished" podID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerID="b84468ecb4b1485fed07fb6c3e0c48dd62be9c995324f2fa531da8c3e2b9b9b3" exitCode=0
Jul 02 21:33:03.794967 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:03.794675    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" event={"ID":"cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925","Type":"ContainerDied","Data":"b84468ecb4b1485fed07fb6c3e0c48dd62be9c995324f2fa531da8c3e2b9b9b3"}
Jul 02 21:33:03.911952 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:03.911919    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f"
Jul 02 21:33:03.954307 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:03.954273    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mmsq\" (UniqueName: \"kubernetes.io/projected/cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925-kube-api-access-5mmsq\") pod \"cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925\" (UID: \"cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925\") "
Jul 02 21:33:03.954307 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:03.954314    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925-cabundle-cert\") pod \"cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925\" (UID: \"cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925\") "
Jul 02 21:33:03.954526 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:03.954371    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925-proxy-tls\") pod \"cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925\" (UID: \"cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925\") "
Jul 02 21:33:03.954526 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:03.954391    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925-kserve-provision-location\") pod \"cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925\" (UID: \"cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925\") "
Jul 02 21:33:03.954526 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:03.954414    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925-kube-rbac-proxy-sar-config\") pod \"cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925\" (UID: \"cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925\") "
Jul 02 21:33:03.954751 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:03.954722    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" (UID: "cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:33:03.954823 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:03.954749    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" (UID: "cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:33:03.954904 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:03.954877    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" (UID: "cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:33:03.956483 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:03.956454    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925-kube-api-access-5mmsq" (OuterVolumeSpecName: "kube-api-access-5mmsq") pod "cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" (UID: "cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925"). InnerVolumeSpecName "kube-api-access-5mmsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:33:03.956563 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:03.956460    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" (UID: "cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:33:04.055300 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:04.055271    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925-proxy-tls\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:33:04.055300 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:04.055298    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925-kserve-provision-location\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:33:04.055512 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:04.055308    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:33:04.055512 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:04.055317    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mmsq\" (UniqueName: \"kubernetes.io/projected/cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925-kube-api-access-5mmsq\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:33:04.055512 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:04.055326    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925-cabundle-cert\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:33:04.802169 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:04.802136    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f" event={"ID":"cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925","Type":"ContainerDied","Data":"18ed06b9b32c62134753d2013f16ec68a42395c7dfb01a9c4795a94158f0a46e"}
Jul 02 21:33:04.802567 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:04.802178    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f"
Jul 02 21:33:04.802567 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:04.802217    2586 scope.go:117] "RemoveContainer" containerID="b84468ecb4b1485fed07fb6c3e0c48dd62be9c995324f2fa531da8c3e2b9b9b3"
Jul 02 21:33:04.810392 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:04.810371    2586 scope.go:117] "RemoveContainer" containerID="921762d3c45bf596dcb2f20f69cbe9837406d6f95cd52244b4cbc1213df6b75c"
Jul 02 21:33:04.818090 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:04.818070    2586 scope.go:117] "RemoveContainer" containerID="7813e031f11dbf45c6e3517efafe5412df8e3afe4a2b5581a53bd4d15e341f4c"
Jul 02 21:33:04.822462 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:04.822440    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f"]
Jul 02 21:33:04.826762 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:04.826735    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-ghs4f"]
Jul 02 21:33:04.827737 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:04.827717    2586 scope.go:117] "RemoveContainer" containerID="c6a3d8964048221ed2c74192912fcef75074a93cbd9857c6b93a5d8c470ffe13"
Jul 02 21:33:05.782223 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:05.782194    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h"
Jul 02 21:33:05.782899 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:05.782862    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 02 21:33:05.783165 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:05.783137    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:33:06.022246 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:06.022210    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" path="/var/lib/kubelet/pods/cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925/volumes"
Jul 02 21:33:15.783202 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:15.783149    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 02 21:33:15.783715 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:15.783687    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:33:25.783065 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:25.783016    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 02 21:33:25.783553 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:25.783361    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:33:35.783499 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:35.783458    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 02 21:33:35.783997 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:35.783971    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:33:45.783306 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:45.783250    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 02 21:33:45.783761 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:45.783734    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:33:55.782831 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:55.782781    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 02 21:33:55.783335 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:33:55.783308    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:34:05.782910 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:05.782874    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 02 21:34:05.783398 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:05.783090    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:34:10.018133 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:10.018075    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 02 21:34:10.018599 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:10.018512    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:34:20.022835 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:20.022805    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h"
Jul 02 21:34:20.023217 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:20.022907    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h"
Jul 02 21:34:28.790443 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:28.790410    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-bt87v_37135ac3-b594-414c-8135-3a462e132e1f/kserve-container/0.log"
Jul 02 21:34:29.070454 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.070365    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h"]
Jul 02 21:34:29.070738 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.070686    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="kserve-container" containerID="cri-o://fe58ad59fd645abc19a02c91d2eb2a0316d18698077ea85d433501b200f36e9a" gracePeriod=30
Jul 02 21:34:29.070880 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.070741    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="kube-rbac-proxy" containerID="cri-o://303d6748b47f429b9354f2d83b567cf1529c6ef2b8e782e1c3e176681c9b9ff0" gracePeriod=30
Jul 02 21:34:29.070880 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.070712    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="agent" containerID="cri-o://c5a9f797683fbb6dec0cdbc84ed4f95b71f05aeeba37642d223e57ff90cc00ec" gracePeriod=30
Jul 02 21:34:29.098439 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.098387    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp"]
Jul 02 21:34:29.098858 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.098836    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="kserve-container"
Jul 02 21:34:29.098940 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.098863    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="kserve-container"
Jul 02 21:34:29.098940 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.098879    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="storage-initializer"
Jul 02 21:34:29.098940 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.098889    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="storage-initializer"
Jul 02 21:34:29.098940 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.098906    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="agent"
Jul 02 21:34:29.098940 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.098914    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="agent"
Jul 02 21:34:29.098940 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.098927    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="kube-rbac-proxy"
Jul 02 21:34:29.098940 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.098936    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="kube-rbac-proxy"
Jul 02 21:34:29.099269 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.099017    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="agent"
Jul 02 21:34:29.099269 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.099027    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="kserve-container"
Jul 02 21:34:29.099269 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.099035    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf3e2e4f-c62f-49c6-a622-c4d9c7d3d925" containerName="kube-rbac-proxy"
Jul 02 21:34:29.102556 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.102478    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp"
Jul 02 21:34:29.106343 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.105495    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Jul 02 21:34:29.106343 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.105852    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Jul 02 21:34:29.116359 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.116204    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp"]
Jul 02 21:34:29.131433 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.131390    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-bt87v"]
Jul 02 21:34:29.131746 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.131718    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-bt87v" podUID="37135ac3-b594-414c-8135-3a462e132e1f" containerName="kserve-container" containerID="cri-o://9a57663d0010d3f01933ca427b4a80cd002d668dac34f4dd3a70cc996b0e2e0f" gracePeriod=30
Jul 02 21:34:29.131875 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.131719    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-bt87v" podUID="37135ac3-b594-414c-8135-3a462e132e1f" containerName="kube-rbac-proxy" containerID="cri-o://8bd93c35e193cf767a2ee5eebbf7ea06c08cec7e08467649ceccf885ca78dea0" gracePeriod=30
Jul 02 21:34:29.136559 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.136528    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n8n5\" (UniqueName: \"kubernetes.io/projected/ff252377-bb33-498f-b199-fbc1221ea67e-kube-api-access-9n8n5\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp\" (UID: \"ff252377-bb33-498f-b199-fbc1221ea67e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp"
Jul 02 21:34:29.136709 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.136633    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff252377-bb33-498f-b199-fbc1221ea67e-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp\" (UID: \"ff252377-bb33-498f-b199-fbc1221ea67e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp"
Jul 02 21:34:29.136784 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.136708    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff252377-bb33-498f-b199-fbc1221ea67e-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp\" (UID: \"ff252377-bb33-498f-b199-fbc1221ea67e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp"
Jul 02 21:34:29.136784 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.136748    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff252377-bb33-498f-b199-fbc1221ea67e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp\" (UID: \"ff252377-bb33-498f-b199-fbc1221ea67e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp"
Jul 02 21:34:29.137137 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.136798    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff252377-bb33-498f-b199-fbc1221ea67e-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp\" (UID: \"ff252377-bb33-498f-b199-fbc1221ea67e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp"
Jul 02 21:34:29.185517 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.185486    2586 generic.go:356] "Generic (PLEG): container finished" podID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerID="303d6748b47f429b9354f2d83b567cf1529c6ef2b8e782e1c3e176681c9b9ff0" exitCode=2
Jul 02 21:34:29.185622 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.185548    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" event={"ID":"d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd","Type":"ContainerDied","Data":"303d6748b47f429b9354f2d83b567cf1529c6ef2b8e782e1c3e176681c9b9ff0"}
Jul 02 21:34:29.237998 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.237957    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff252377-bb33-498f-b199-fbc1221ea67e-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp\" (UID: \"ff252377-bb33-498f-b199-fbc1221ea67e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp"
Jul 02 21:34:29.237998 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.238002    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff252377-bb33-498f-b199-fbc1221ea67e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp\" (UID: \"ff252377-bb33-498f-b199-fbc1221ea67e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp"
Jul 02 21:34:29.238215 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.238027    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff252377-bb33-498f-b199-fbc1221ea67e-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp\" (UID: \"ff252377-bb33-498f-b199-fbc1221ea67e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp"
Jul 02 21:34:29.238215 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.238071    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n8n5\" (UniqueName: \"kubernetes.io/projected/ff252377-bb33-498f-b199-fbc1221ea67e-kube-api-access-9n8n5\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp\" (UID: \"ff252377-bb33-498f-b199-fbc1221ea67e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp"
Jul 02 21:34:29.238215 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.238179    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff252377-bb33-498f-b199-fbc1221ea67e-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp\" (UID: \"ff252377-bb33-498f-b199-fbc1221ea67e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp"
Jul 02 21:34:29.238450 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.238426    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff252377-bb33-498f-b199-fbc1221ea67e-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp\" (UID: \"ff252377-bb33-498f-b199-fbc1221ea67e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp"
Jul 02 21:34:29.238883 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.238852    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff252377-bb33-498f-b199-fbc1221ea67e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp\" (UID: \"ff252377-bb33-498f-b199-fbc1221ea67e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp"
Jul 02 21:34:29.239151 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.239129    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff252377-bb33-498f-b199-fbc1221ea67e-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp\" (UID: \"ff252377-bb33-498f-b199-fbc1221ea67e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp"
Jul 02 21:34:29.240758 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.240730    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff252377-bb33-498f-b199-fbc1221ea67e-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp\" (UID: \"ff252377-bb33-498f-b199-fbc1221ea67e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp"
Jul 02 21:34:29.246621 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.246597    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n8n5\" (UniqueName: \"kubernetes.io/projected/ff252377-bb33-498f-b199-fbc1221ea67e-kube-api-access-9n8n5\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp\" (UID: \"ff252377-bb33-498f-b199-fbc1221ea67e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp"
Jul 02 21:34:29.371120 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.371097    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-bt87v"
Jul 02 21:34:29.420935 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.420903    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp"
Jul 02 21:34:29.439999 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.439962    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37135ac3-b594-414c-8135-3a462e132e1f-proxy-tls\") pod \"37135ac3-b594-414c-8135-3a462e132e1f\" (UID: \"37135ac3-b594-414c-8135-3a462e132e1f\") "
Jul 02 21:34:29.440178 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.440013    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gv4z4\" (UniqueName: \"kubernetes.io/projected/37135ac3-b594-414c-8135-3a462e132e1f-kube-api-access-gv4z4\") pod \"37135ac3-b594-414c-8135-3a462e132e1f\" (UID: \"37135ac3-b594-414c-8135-3a462e132e1f\") "
Jul 02 21:34:29.440178 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.440127    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37135ac3-b594-414c-8135-3a462e132e1f-kube-rbac-proxy-sar-config\") pod \"37135ac3-b594-414c-8135-3a462e132e1f\" (UID: \"37135ac3-b594-414c-8135-3a462e132e1f\") "
Jul 02 21:34:29.440537 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.440505    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37135ac3-b594-414c-8135-3a462e132e1f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "37135ac3-b594-414c-8135-3a462e132e1f" (UID: "37135ac3-b594-414c-8135-3a462e132e1f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:34:29.442376 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.442335    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37135ac3-b594-414c-8135-3a462e132e1f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "37135ac3-b594-414c-8135-3a462e132e1f" (UID: "37135ac3-b594-414c-8135-3a462e132e1f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:34:29.442376 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.442346    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37135ac3-b594-414c-8135-3a462e132e1f-kube-api-access-gv4z4" (OuterVolumeSpecName: "kube-api-access-gv4z4") pod "37135ac3-b594-414c-8135-3a462e132e1f" (UID: "37135ac3-b594-414c-8135-3a462e132e1f"). InnerVolumeSpecName "kube-api-access-gv4z4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:34:29.541429 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.541392    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37135ac3-b594-414c-8135-3a462e132e1f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:34:29.541429 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.541422    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37135ac3-b594-414c-8135-3a462e132e1f-proxy-tls\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:34:29.541429 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.541431    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv4z4\" (UniqueName: \"kubernetes.io/projected/37135ac3-b594-414c-8135-3a462e132e1f-kube-api-access-gv4z4\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:34:29.553196 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:29.553149    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp"]
Jul 02 21:34:29.555854 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:34:29.555830    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff252377_bb33_498f_b199_fbc1221ea67e.slice/crio-7953396bd775471ff59791b6b2624d47fdb4b34dc5396662f10310a8c6b650dd WatchSource:0}: Error finding container 7953396bd775471ff59791b6b2624d47fdb4b34dc5396662f10310a8c6b650dd: Status 404 returned error can't find the container with id 7953396bd775471ff59791b6b2624d47fdb4b34dc5396662f10310a8c6b650dd
Jul 02 21:34:30.018709 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:30.018663    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 02 21:34:30.019226 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:30.019058    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:34:30.191163 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:30.191128    2586 generic.go:356] "Generic (PLEG): container finished" podID="37135ac3-b594-414c-8135-3a462e132e1f" containerID="8bd93c35e193cf767a2ee5eebbf7ea06c08cec7e08467649ceccf885ca78dea0" exitCode=2
Jul 02 21:34:30.191163 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:30.191153    2586 generic.go:356] "Generic (PLEG): container finished" podID="37135ac3-b594-414c-8135-3a462e132e1f" containerID="9a57663d0010d3f01933ca427b4a80cd002d668dac34f4dd3a70cc996b0e2e0f" exitCode=2
Jul 02 21:34:30.191380 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:30.191181    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-bt87v"
Jul 02 21:34:30.191380 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:30.191219    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-bt87v" event={"ID":"37135ac3-b594-414c-8135-3a462e132e1f","Type":"ContainerDied","Data":"8bd93c35e193cf767a2ee5eebbf7ea06c08cec7e08467649ceccf885ca78dea0"}
Jul 02 21:34:30.191380 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:30.191284    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-bt87v" event={"ID":"37135ac3-b594-414c-8135-3a462e132e1f","Type":"ContainerDied","Data":"9a57663d0010d3f01933ca427b4a80cd002d668dac34f4dd3a70cc996b0e2e0f"}
Jul 02 21:34:30.191380 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:30.191321    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-bt87v" event={"ID":"37135ac3-b594-414c-8135-3a462e132e1f","Type":"ContainerDied","Data":"ff8d28e2173f0ba238f53a0b892af30f4a04e4ef1a5f7fe3950c8bca504ac99d"}
Jul 02 21:34:30.191380 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:30.191359    2586 scope.go:117] "RemoveContainer" containerID="8bd93c35e193cf767a2ee5eebbf7ea06c08cec7e08467649ceccf885ca78dea0"
Jul 02 21:34:30.193083 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:30.193048    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp" event={"ID":"ff252377-bb33-498f-b199-fbc1221ea67e","Type":"ContainerStarted","Data":"54511ac8fab2c379372fb936118cc4be9d575a8b02c4341001efb459ac33d686"}
Jul 02 21:34:30.193219 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:30.193109    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp" event={"ID":"ff252377-bb33-498f-b199-fbc1221ea67e","Type":"ContainerStarted","Data":"7953396bd775471ff59791b6b2624d47fdb4b34dc5396662f10310a8c6b650dd"}
Jul 02 21:34:30.199718 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:30.199698    2586 scope.go:117] "RemoveContainer" containerID="9a57663d0010d3f01933ca427b4a80cd002d668dac34f4dd3a70cc996b0e2e0f"
Jul 02 21:34:30.208286 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:30.208268    2586 scope.go:117] "RemoveContainer" containerID="8bd93c35e193cf767a2ee5eebbf7ea06c08cec7e08467649ceccf885ca78dea0"
Jul 02 21:34:30.208579 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:34:30.208561    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bd93c35e193cf767a2ee5eebbf7ea06c08cec7e08467649ceccf885ca78dea0\": container with ID starting with 8bd93c35e193cf767a2ee5eebbf7ea06c08cec7e08467649ceccf885ca78dea0 not found: ID does not exist" containerID="8bd93c35e193cf767a2ee5eebbf7ea06c08cec7e08467649ceccf885ca78dea0"
Jul 02 21:34:30.208686 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:30.208592    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bd93c35e193cf767a2ee5eebbf7ea06c08cec7e08467649ceccf885ca78dea0"} err="failed to get container status \"8bd93c35e193cf767a2ee5eebbf7ea06c08cec7e08467649ceccf885ca78dea0\": rpc error: code = NotFound desc = could not find container \"8bd93c35e193cf767a2ee5eebbf7ea06c08cec7e08467649ceccf885ca78dea0\": container with ID starting with 8bd93c35e193cf767a2ee5eebbf7ea06c08cec7e08467649ceccf885ca78dea0 not found: ID does not exist"
Jul 02 21:34:30.208686 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:30.208617    2586 scope.go:117] "RemoveContainer" containerID="9a57663d0010d3f01933ca427b4a80cd002d668dac34f4dd3a70cc996b0e2e0f"
Jul 02 21:34:30.208880 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:34:30.208859    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a57663d0010d3f01933ca427b4a80cd002d668dac34f4dd3a70cc996b0e2e0f\": container with ID starting with 9a57663d0010d3f01933ca427b4a80cd002d668dac34f4dd3a70cc996b0e2e0f not found: ID does not exist" containerID="9a57663d0010d3f01933ca427b4a80cd002d668dac34f4dd3a70cc996b0e2e0f"
Jul 02 21:34:30.208922 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:30.208886    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a57663d0010d3f01933ca427b4a80cd002d668dac34f4dd3a70cc996b0e2e0f"} err="failed to get container status \"9a57663d0010d3f01933ca427b4a80cd002d668dac34f4dd3a70cc996b0e2e0f\": rpc error: code = NotFound desc = could not find container \"9a57663d0010d3f01933ca427b4a80cd002d668dac34f4dd3a70cc996b0e2e0f\": container with ID starting with 9a57663d0010d3f01933ca427b4a80cd002d668dac34f4dd3a70cc996b0e2e0f not found: ID does not exist"
Jul 02 21:34:30.208922 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:30.208902    2586 scope.go:117] "RemoveContainer" containerID="8bd93c35e193cf767a2ee5eebbf7ea06c08cec7e08467649ceccf885ca78dea0"
Jul 02 21:34:30.209106 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:30.209087    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bd93c35e193cf767a2ee5eebbf7ea06c08cec7e08467649ceccf885ca78dea0"} err="failed to get container status \"8bd93c35e193cf767a2ee5eebbf7ea06c08cec7e08467649ceccf885ca78dea0\": rpc error: code = NotFound desc = could not find container \"8bd93c35e193cf767a2ee5eebbf7ea06c08cec7e08467649ceccf885ca78dea0\": container with ID starting with 8bd93c35e193cf767a2ee5eebbf7ea06c08cec7e08467649ceccf885ca78dea0 not found: ID does not exist"
Jul 02 21:34:30.209155 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:30.209108    2586 scope.go:117] "RemoveContainer" containerID="9a57663d0010d3f01933ca427b4a80cd002d668dac34f4dd3a70cc996b0e2e0f"
Jul 02 21:34:30.209343 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:30.209328    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a57663d0010d3f01933ca427b4a80cd002d668dac34f4dd3a70cc996b0e2e0f"} err="failed to get container status \"9a57663d0010d3f01933ca427b4a80cd002d668dac34f4dd3a70cc996b0e2e0f\": rpc error: code = NotFound desc = could not find container \"9a57663d0010d3f01933ca427b4a80cd002d668dac34f4dd3a70cc996b0e2e0f\": container with ID starting with 9a57663d0010d3f01933ca427b4a80cd002d668dac34f4dd3a70cc996b0e2e0f not found: ID does not exist"
Jul 02 21:34:30.237699 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:30.237660    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-bt87v"]
Jul 02 21:34:30.240014 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:30.239988    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-bt87v"]
Jul 02 21:34:30.778569 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:30.778528    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Jul 02 21:34:32.022214 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:32.022177    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37135ac3-b594-414c-8135-3a462e132e1f" path="/var/lib/kubelet/pods/37135ac3-b594-414c-8135-3a462e132e1f/volumes"
Jul 02 21:34:34.215494 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:34.215463    2586 generic.go:356] "Generic (PLEG): container finished" podID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerID="fe58ad59fd645abc19a02c91d2eb2a0316d18698077ea85d433501b200f36e9a" exitCode=0
Jul 02 21:34:34.215894 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:34.215534    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" event={"ID":"d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd","Type":"ContainerDied","Data":"fe58ad59fd645abc19a02c91d2eb2a0316d18698077ea85d433501b200f36e9a"}
Jul 02 21:34:35.778671 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:35.778618    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Jul 02 21:34:40.018569 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:40.018524    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 02 21:34:40.019169 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:40.018928    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:34:40.778419 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:40.778380    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Jul 02 21:34:40.778605 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:40.778556    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h"
Jul 02 21:34:45.778375 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:45.778333    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Jul 02 21:34:50.018570 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:50.018509    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 02 21:34:50.019118 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:50.018831    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:34:50.022006 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:50.021985    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h"
Jul 02 21:34:50.022132 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:50.022059    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h"
Jul 02 21:34:50.778672 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:50.778616    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Jul 02 21:34:55.778702 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:55.778639    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Jul 02 21:34:59.229568 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.229528    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h"
Jul 02 21:34:59.332433 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.332344    2586 generic.go:356] "Generic (PLEG): container finished" podID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerID="c5a9f797683fbb6dec0cdbc84ed4f95b71f05aeeba37642d223e57ff90cc00ec" exitCode=0
Jul 02 21:34:59.332433 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.332412    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h"
Jul 02 21:34:59.332696 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.332436    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" event={"ID":"d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd","Type":"ContainerDied","Data":"c5a9f797683fbb6dec0cdbc84ed4f95b71f05aeeba37642d223e57ff90cc00ec"}
Jul 02 21:34:59.332696 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.332520    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h" event={"ID":"d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd","Type":"ContainerDied","Data":"fa0628b34cbebbedeff96456c02bba9cf4b59b2df9de70ace724decbbe1d943c"}
Jul 02 21:34:59.332696 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.332563    2586 scope.go:117] "RemoveContainer" containerID="c5a9f797683fbb6dec0cdbc84ed4f95b71f05aeeba37642d223e57ff90cc00ec"
Jul 02 21:34:59.334056 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.334027    2586 generic.go:356] "Generic (PLEG): container finished" podID="ff252377-bb33-498f-b199-fbc1221ea67e" containerID="54511ac8fab2c379372fb936118cc4be9d575a8b02c4341001efb459ac33d686" exitCode=0
Jul 02 21:34:59.334218 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.334070    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp" event={"ID":"ff252377-bb33-498f-b199-fbc1221ea67e","Type":"ContainerDied","Data":"54511ac8fab2c379372fb936118cc4be9d575a8b02c4341001efb459ac33d686"}
Jul 02 21:34:59.335418 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.335400    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 21:34:59.341138 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.341121    2586 scope.go:117] "RemoveContainer" containerID="303d6748b47f429b9354f2d83b567cf1529c6ef2b8e782e1c3e176681c9b9ff0"
Jul 02 21:34:59.348710 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.348696    2586 scope.go:117] "RemoveContainer" containerID="fe58ad59fd645abc19a02c91d2eb2a0316d18698077ea85d433501b200f36e9a"
Jul 02 21:34:59.356233 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.356215    2586 scope.go:117] "RemoveContainer" containerID="3a36424578468c272b666e6ad6d55bb9a54b81bbeaff9ac532aa42d8457413ca"
Jul 02 21:34:59.363442 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.363424    2586 scope.go:117] "RemoveContainer" containerID="c5a9f797683fbb6dec0cdbc84ed4f95b71f05aeeba37642d223e57ff90cc00ec"
Jul 02 21:34:59.363723 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:34:59.363699    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5a9f797683fbb6dec0cdbc84ed4f95b71f05aeeba37642d223e57ff90cc00ec\": container with ID starting with c5a9f797683fbb6dec0cdbc84ed4f95b71f05aeeba37642d223e57ff90cc00ec not found: ID does not exist" containerID="c5a9f797683fbb6dec0cdbc84ed4f95b71f05aeeba37642d223e57ff90cc00ec"
Jul 02 21:34:59.363788 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.363747    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5a9f797683fbb6dec0cdbc84ed4f95b71f05aeeba37642d223e57ff90cc00ec"} err="failed to get container status \"c5a9f797683fbb6dec0cdbc84ed4f95b71f05aeeba37642d223e57ff90cc00ec\": rpc error: code = NotFound desc = could not find container \"c5a9f797683fbb6dec0cdbc84ed4f95b71f05aeeba37642d223e57ff90cc00ec\": container with ID starting with c5a9f797683fbb6dec0cdbc84ed4f95b71f05aeeba37642d223e57ff90cc00ec not found: ID does not exist"
Jul 02 21:34:59.363788 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.363786    2586 scope.go:117] "RemoveContainer" containerID="303d6748b47f429b9354f2d83b567cf1529c6ef2b8e782e1c3e176681c9b9ff0"
Jul 02 21:34:59.364039 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:34:59.364023    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"303d6748b47f429b9354f2d83b567cf1529c6ef2b8e782e1c3e176681c9b9ff0\": container with ID starting with 303d6748b47f429b9354f2d83b567cf1529c6ef2b8e782e1c3e176681c9b9ff0 not found: ID does not exist" containerID="303d6748b47f429b9354f2d83b567cf1529c6ef2b8e782e1c3e176681c9b9ff0"
Jul 02 21:34:59.364095 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.364047    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"303d6748b47f429b9354f2d83b567cf1529c6ef2b8e782e1c3e176681c9b9ff0"} err="failed to get container status \"303d6748b47f429b9354f2d83b567cf1529c6ef2b8e782e1c3e176681c9b9ff0\": rpc error: code = NotFound desc = could not find container \"303d6748b47f429b9354f2d83b567cf1529c6ef2b8e782e1c3e176681c9b9ff0\": container with ID starting with 303d6748b47f429b9354f2d83b567cf1529c6ef2b8e782e1c3e176681c9b9ff0 not found: ID does not exist"
Jul 02 21:34:59.364095 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.364061    2586 scope.go:117] "RemoveContainer" containerID="fe58ad59fd645abc19a02c91d2eb2a0316d18698077ea85d433501b200f36e9a"
Jul 02 21:34:59.364293 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:34:59.364276    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe58ad59fd645abc19a02c91d2eb2a0316d18698077ea85d433501b200f36e9a\": container with ID starting with fe58ad59fd645abc19a02c91d2eb2a0316d18698077ea85d433501b200f36e9a not found: ID does not exist" containerID="fe58ad59fd645abc19a02c91d2eb2a0316d18698077ea85d433501b200f36e9a"
Jul 02 21:34:59.364334 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.364299    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe58ad59fd645abc19a02c91d2eb2a0316d18698077ea85d433501b200f36e9a"} err="failed to get container status \"fe58ad59fd645abc19a02c91d2eb2a0316d18698077ea85d433501b200f36e9a\": rpc error: code = NotFound desc = could not find container \"fe58ad59fd645abc19a02c91d2eb2a0316d18698077ea85d433501b200f36e9a\": container with ID starting with fe58ad59fd645abc19a02c91d2eb2a0316d18698077ea85d433501b200f36e9a not found: ID does not exist"
Jul 02 21:34:59.364334 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.364312    2586 scope.go:117] "RemoveContainer" containerID="3a36424578468c272b666e6ad6d55bb9a54b81bbeaff9ac532aa42d8457413ca"
Jul 02 21:34:59.364548 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:34:59.364532    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a36424578468c272b666e6ad6d55bb9a54b81bbeaff9ac532aa42d8457413ca\": container with ID starting with 3a36424578468c272b666e6ad6d55bb9a54b81bbeaff9ac532aa42d8457413ca not found: ID does not exist" containerID="3a36424578468c272b666e6ad6d55bb9a54b81bbeaff9ac532aa42d8457413ca"
Jul 02 21:34:59.364588 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.364552    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a36424578468c272b666e6ad6d55bb9a54b81bbeaff9ac532aa42d8457413ca"} err="failed to get container status \"3a36424578468c272b666e6ad6d55bb9a54b81bbeaff9ac532aa42d8457413ca\": rpc error: code = NotFound desc = could not find container \"3a36424578468c272b666e6ad6d55bb9a54b81bbeaff9ac532aa42d8457413ca\": container with ID starting with 3a36424578468c272b666e6ad6d55bb9a54b81bbeaff9ac532aa42d8457413ca not found: ID does not exist"
Jul 02 21:34:59.398913 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.398877    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd-kube-rbac-proxy-sar-config\") pod \"d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd\" (UID: \"d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd\") "
Jul 02 21:34:59.398913 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.398919    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htpgt\" (UniqueName: \"kubernetes.io/projected/d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd-kube-api-access-htpgt\") pod \"d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd\" (UID: \"d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd\") "
Jul 02 21:34:59.399104 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.398984    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd-cabundle-cert\") pod \"d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd\" (UID: \"d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd\") "
Jul 02 21:34:59.399104 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.399052    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd-kserve-provision-location\") pod \"d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd\" (UID: \"d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd\") "
Jul 02 21:34:59.399104 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.399082    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd-proxy-tls\") pod \"d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd\" (UID: \"d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd\") "
Jul 02 21:34:59.399372 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.399349    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" (UID: "d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:34:59.399483 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.399459    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" (UID: "d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:34:59.399835 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.399803    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" (UID: "d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:34:59.401188 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.401157    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd-kube-api-access-htpgt" (OuterVolumeSpecName: "kube-api-access-htpgt") pod "d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" (UID: "d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd"). InnerVolumeSpecName "kube-api-access-htpgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:34:59.401256 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.401240    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" (UID: "d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:34:59.500642 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.500606    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:34:59.500642 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.500638    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htpgt\" (UniqueName: \"kubernetes.io/projected/d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd-kube-api-access-htpgt\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:34:59.500642 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.500677    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd-cabundle-cert\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:34:59.500941 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.500688    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd-kserve-provision-location\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:34:59.500941 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.500698    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd-proxy-tls\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:34:59.656686 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.656642    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h"]
Jul 02 21:34:59.660209 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:34:59.660181    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-gmt6h"]
Jul 02 21:35:00.025094 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:35:00.025062    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" path="/var/lib/kubelet/pods/d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd/volumes"
Jul 02 21:37:12.995921 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:12.995876    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp" event={"ID":"ff252377-bb33-498f-b199-fbc1221ea67e","Type":"ContainerStarted","Data":"2de06eb6145c4a3513dcc039e2cb6a0d50d82eaafa6cd257d763d9d911f09888"}
Jul 02 21:37:12.996462 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:12.995960    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp" event={"ID":"ff252377-bb33-498f-b199-fbc1221ea67e","Type":"ContainerStarted","Data":"d0604b73e69886d8de8c0413a10f704c7088ac9aa86d8e95c0f6fe10dfe4ae1a"}
Jul 02 21:37:12.996462 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:12.995996    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp"
Jul 02 21:37:13.026098 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:13.025918    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp" podStartSLOduration=2.401509686 podStartE2EDuration="2m44.025894459s" totalImagesPullingTime="2m12.441191262s" totalInitContainerRuntime="29.183193511s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:34:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:37:13.02512125 +0000 UTC m=+1105.467921693" watchObservedRunningTime="2026-07-02 21:37:13.025894459 +0000 UTC m=+1105.468694882"
Jul 02 21:37:14.001194 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:14.001156    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp"
Jul 02 21:37:14.002320 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:14.002292    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp" podUID="ff252377-bb33-498f-b199-fbc1221ea67e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 02 21:37:15.006298 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:15.006251    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp" podUID="ff252377-bb33-498f-b199-fbc1221ea67e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 02 21:37:20.011815 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:20.011779    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp"
Jul 02 21:37:20.012419 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:20.012285    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp" podUID="ff252377-bb33-498f-b199-fbc1221ea67e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 02 21:37:30.012818 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:30.012784    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp"
Jul 02 21:37:39.226488 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:39.226452    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp"]
Jul 02 21:37:39.226932 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:39.226768    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp" podUID="ff252377-bb33-498f-b199-fbc1221ea67e" containerName="kserve-container" containerID="cri-o://d0604b73e69886d8de8c0413a10f704c7088ac9aa86d8e95c0f6fe10dfe4ae1a" gracePeriod=30
Jul 02 21:37:39.226932 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:39.226791    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp" podUID="ff252377-bb33-498f-b199-fbc1221ea67e" containerName="kube-rbac-proxy" containerID="cri-o://2de06eb6145c4a3513dcc039e2cb6a0d50d82eaafa6cd257d763d9d911f09888" gracePeriod=30
Jul 02 21:37:40.007332 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:40.007281    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp" podUID="ff252377-bb33-498f-b199-fbc1221ea67e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jul 02 21:37:40.012639 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:40.012598    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp" podUID="ff252377-bb33-498f-b199-fbc1221ea67e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 02 21:37:40.126020 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:40.125985    2586 generic.go:356] "Generic (PLEG): container finished" podID="ff252377-bb33-498f-b199-fbc1221ea67e" containerID="2de06eb6145c4a3513dcc039e2cb6a0d50d82eaafa6cd257d763d9d911f09888" exitCode=2
Jul 02 21:37:40.126020 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:40.126009    2586 generic.go:356] "Generic (PLEG): container finished" podID="ff252377-bb33-498f-b199-fbc1221ea67e" containerID="d0604b73e69886d8de8c0413a10f704c7088ac9aa86d8e95c0f6fe10dfe4ae1a" exitCode=0
Jul 02 21:37:40.126208 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:40.126062    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp" event={"ID":"ff252377-bb33-498f-b199-fbc1221ea67e","Type":"ContainerDied","Data":"2de06eb6145c4a3513dcc039e2cb6a0d50d82eaafa6cd257d763d9d911f09888"}
Jul 02 21:37:40.126208 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:40.126143    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp" event={"ID":"ff252377-bb33-498f-b199-fbc1221ea67e","Type":"ContainerDied","Data":"d0604b73e69886d8de8c0413a10f704c7088ac9aa86d8e95c0f6fe10dfe4ae1a"}
Jul 02 21:37:40.203485 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:40.203446    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp"
Jul 02 21:37:40.272260 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:40.272185    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9n8n5\" (UniqueName: \"kubernetes.io/projected/ff252377-bb33-498f-b199-fbc1221ea67e-kube-api-access-9n8n5\") pod \"ff252377-bb33-498f-b199-fbc1221ea67e\" (UID: \"ff252377-bb33-498f-b199-fbc1221ea67e\") "
Jul 02 21:37:40.272260 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:40.272217    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff252377-bb33-498f-b199-fbc1221ea67e-cabundle-cert\") pod \"ff252377-bb33-498f-b199-fbc1221ea67e\" (UID: \"ff252377-bb33-498f-b199-fbc1221ea67e\") "
Jul 02 21:37:40.272260 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:40.272251    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff252377-bb33-498f-b199-fbc1221ea67e-kube-rbac-proxy-sar-config\") pod \"ff252377-bb33-498f-b199-fbc1221ea67e\" (UID: \"ff252377-bb33-498f-b199-fbc1221ea67e\") "
Jul 02 21:37:40.272725 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:40.272304    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff252377-bb33-498f-b199-fbc1221ea67e-kserve-provision-location\") pod \"ff252377-bb33-498f-b199-fbc1221ea67e\" (UID: \"ff252377-bb33-498f-b199-fbc1221ea67e\") "
Jul 02 21:37:40.272725 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:40.272337    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff252377-bb33-498f-b199-fbc1221ea67e-proxy-tls\") pod \"ff252377-bb33-498f-b199-fbc1221ea67e\" (UID: \"ff252377-bb33-498f-b199-fbc1221ea67e\") "
Jul 02 21:37:40.272725 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:40.272550    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff252377-bb33-498f-b199-fbc1221ea67e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ff252377-bb33-498f-b199-fbc1221ea67e" (UID: "ff252377-bb33-498f-b199-fbc1221ea67e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:37:40.272725 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:40.272672    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff252377-bb33-498f-b199-fbc1221ea67e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ff252377-bb33-498f-b199-fbc1221ea67e" (UID: "ff252377-bb33-498f-b199-fbc1221ea67e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:37:40.274328 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:40.274307    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff252377-bb33-498f-b199-fbc1221ea67e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ff252377-bb33-498f-b199-fbc1221ea67e" (UID: "ff252377-bb33-498f-b199-fbc1221ea67e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:37:40.274446 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:40.274428    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff252377-bb33-498f-b199-fbc1221ea67e-kube-api-access-9n8n5" (OuterVolumeSpecName: "kube-api-access-9n8n5") pod "ff252377-bb33-498f-b199-fbc1221ea67e" (UID: "ff252377-bb33-498f-b199-fbc1221ea67e"). InnerVolumeSpecName "kube-api-access-9n8n5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:37:40.275069 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:40.275046    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff252377-bb33-498f-b199-fbc1221ea67e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ff252377-bb33-498f-b199-fbc1221ea67e" (UID: "ff252377-bb33-498f-b199-fbc1221ea67e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:37:40.373271 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:40.373227    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n8n5\" (UniqueName: \"kubernetes.io/projected/ff252377-bb33-498f-b199-fbc1221ea67e-kube-api-access-9n8n5\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:37:40.373271 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:40.373269    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff252377-bb33-498f-b199-fbc1221ea67e-cabundle-cert\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:37:40.373271 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:40.373280    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff252377-bb33-498f-b199-fbc1221ea67e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:37:40.373271 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:40.373290    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff252377-bb33-498f-b199-fbc1221ea67e-kserve-provision-location\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:37:40.373549 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:40.373300    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff252377-bb33-498f-b199-fbc1221ea67e-proxy-tls\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:37:41.134306 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:41.134269    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp" event={"ID":"ff252377-bb33-498f-b199-fbc1221ea67e","Type":"ContainerDied","Data":"7953396bd775471ff59791b6b2624d47fdb4b34dc5396662f10310a8c6b650dd"}
Jul 02 21:37:41.134558 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:41.134329    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp"
Jul 02 21:37:41.134558 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:41.134365    2586 scope.go:117] "RemoveContainer" containerID="2de06eb6145c4a3513dcc039e2cb6a0d50d82eaafa6cd257d763d9d911f09888"
Jul 02 21:37:41.143786 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:41.143761    2586 scope.go:117] "RemoveContainer" containerID="d0604b73e69886d8de8c0413a10f704c7088ac9aa86d8e95c0f6fe10dfe4ae1a"
Jul 02 21:37:41.151050 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:41.151030    2586 scope.go:117] "RemoveContainer" containerID="54511ac8fab2c379372fb936118cc4be9d575a8b02c4341001efb459ac33d686"
Jul 02 21:37:41.157275 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:41.157253    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp"]
Jul 02 21:37:41.161276 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:41.161258    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-9rzcp"]
Jul 02 21:37:42.022678 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:42.022618    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff252377-bb33-498f-b199-fbc1221ea67e" path="/var/lib/kubelet/pods/ff252377-bb33-498f-b199-fbc1221ea67e/volumes"
Jul 02 21:37:44.551720 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.551687    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr"]
Jul 02 21:37:44.552108 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.551983    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37135ac3-b594-414c-8135-3a462e132e1f" containerName="kserve-container"
Jul 02 21:37:44.552108 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.551995    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="37135ac3-b594-414c-8135-3a462e132e1f" containerName="kserve-container"
Jul 02 21:37:44.552108 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.552004    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37135ac3-b594-414c-8135-3a462e132e1f" containerName="kube-rbac-proxy"
Jul 02 21:37:44.552108 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.552010    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="37135ac3-b594-414c-8135-3a462e132e1f" containerName="kube-rbac-proxy"
Jul 02 21:37:44.552108 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.552019    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="kube-rbac-proxy"
Jul 02 21:37:44.552108 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.552025    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="kube-rbac-proxy"
Jul 02 21:37:44.552108 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.552034    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff252377-bb33-498f-b199-fbc1221ea67e" containerName="kserve-container"
Jul 02 21:37:44.552108 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.552038    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff252377-bb33-498f-b199-fbc1221ea67e" containerName="kserve-container"
Jul 02 21:37:44.552108 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.552047    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff252377-bb33-498f-b199-fbc1221ea67e" containerName="storage-initializer"
Jul 02 21:37:44.552108 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.552053    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff252377-bb33-498f-b199-fbc1221ea67e" containerName="storage-initializer"
Jul 02 21:37:44.552108 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.552062    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="storage-initializer"
Jul 02 21:37:44.552108 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.552067    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="storage-initializer"
Jul 02 21:37:44.552108 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.552075    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="kserve-container"
Jul 02 21:37:44.552108 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.552080    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="kserve-container"
Jul 02 21:37:44.552108 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.552089    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="agent"
Jul 02 21:37:44.552108 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.552094    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="agent"
Jul 02 21:37:44.552108 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.552101    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff252377-bb33-498f-b199-fbc1221ea67e" containerName="kube-rbac-proxy"
Jul 02 21:37:44.552108 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.552107    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff252377-bb33-498f-b199-fbc1221ea67e" containerName="kube-rbac-proxy"
Jul 02 21:37:44.552761 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.552158    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="kserve-container"
Jul 02 21:37:44.552761 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.552168    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="37135ac3-b594-414c-8135-3a462e132e1f" containerName="kube-rbac-proxy"
Jul 02 21:37:44.552761 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.552173    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="kube-rbac-proxy"
Jul 02 21:37:44.552761 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.552178    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="d00dbfb7-153a-4f77-bdd2-0fba5dbdadbd" containerName="agent"
Jul 02 21:37:44.552761 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.552183    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff252377-bb33-498f-b199-fbc1221ea67e" containerName="kube-rbac-proxy"
Jul 02 21:37:44.552761 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.552190    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff252377-bb33-498f-b199-fbc1221ea67e" containerName="kserve-container"
Jul 02 21:37:44.552761 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.552195    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="37135ac3-b594-414c-8135-3a462e132e1f" containerName="kserve-container"
Jul 02 21:37:44.582902 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.582876    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr"]
Jul 02 21:37:44.583024 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.582990    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr"
Jul 02 21:37:44.585536 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.585517    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Jul 02 21:37:44.585855 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.585837    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 21:37:44.585964 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.585947    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Jul 02 21:37:44.586028 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.586013    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 21:37:44.586838 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.586819    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 21:37:44.586940 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.586839    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pbfjv\""
Jul 02 21:37:44.586940 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.586844    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 21:37:44.711621 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.711581    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8867abea-f006-493a-9925-b75b73ffca4d-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-bmxbr\" (UID: \"8867abea-f006-493a-9925-b75b73ffca4d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr"
Jul 02 21:37:44.711821 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.711626    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8867abea-f006-493a-9925-b75b73ffca4d-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-bmxbr\" (UID: \"8867abea-f006-493a-9925-b75b73ffca4d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr"
Jul 02 21:37:44.711821 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.711730    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl9z4\" (UniqueName: \"kubernetes.io/projected/8867abea-f006-493a-9925-b75b73ffca4d-kube-api-access-cl9z4\") pod \"isvc-autogluon-v2-predictor-67b849d456-bmxbr\" (UID: \"8867abea-f006-493a-9925-b75b73ffca4d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr"
Jul 02 21:37:44.711821 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.711759    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8867abea-f006-493a-9925-b75b73ffca4d-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-bmxbr\" (UID: \"8867abea-f006-493a-9925-b75b73ffca4d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr"
Jul 02 21:37:44.711821 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.711798    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8867abea-f006-493a-9925-b75b73ffca4d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-bmxbr\" (UID: \"8867abea-f006-493a-9925-b75b73ffca4d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr"
Jul 02 21:37:44.812778 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.812688    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl9z4\" (UniqueName: \"kubernetes.io/projected/8867abea-f006-493a-9925-b75b73ffca4d-kube-api-access-cl9z4\") pod \"isvc-autogluon-v2-predictor-67b849d456-bmxbr\" (UID: \"8867abea-f006-493a-9925-b75b73ffca4d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr"
Jul 02 21:37:44.812778 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.812729    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8867abea-f006-493a-9925-b75b73ffca4d-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-bmxbr\" (UID: \"8867abea-f006-493a-9925-b75b73ffca4d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr"
Jul 02 21:37:44.812979 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.812901    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8867abea-f006-493a-9925-b75b73ffca4d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-bmxbr\" (UID: \"8867abea-f006-493a-9925-b75b73ffca4d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr"
Jul 02 21:37:44.812979 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.812952    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8867abea-f006-493a-9925-b75b73ffca4d-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-bmxbr\" (UID: \"8867abea-f006-493a-9925-b75b73ffca4d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr"
Jul 02 21:37:44.813066 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.812984    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8867abea-f006-493a-9925-b75b73ffca4d-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-bmxbr\" (UID: \"8867abea-f006-493a-9925-b75b73ffca4d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr"
Jul 02 21:37:44.813415 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.813393    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8867abea-f006-493a-9925-b75b73ffca4d-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-bmxbr\" (UID: \"8867abea-f006-493a-9925-b75b73ffca4d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr"
Jul 02 21:37:44.813575 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.813553    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8867abea-f006-493a-9925-b75b73ffca4d-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-bmxbr\" (UID: \"8867abea-f006-493a-9925-b75b73ffca4d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr"
Jul 02 21:37:44.813612 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.813558    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8867abea-f006-493a-9925-b75b73ffca4d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-bmxbr\" (UID: \"8867abea-f006-493a-9925-b75b73ffca4d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr"
Jul 02 21:37:44.815611 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.815587    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8867abea-f006-493a-9925-b75b73ffca4d-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-bmxbr\" (UID: \"8867abea-f006-493a-9925-b75b73ffca4d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr"
Jul 02 21:37:44.822332 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.822301    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl9z4\" (UniqueName: \"kubernetes.io/projected/8867abea-f006-493a-9925-b75b73ffca4d-kube-api-access-cl9z4\") pod \"isvc-autogluon-v2-predictor-67b849d456-bmxbr\" (UID: \"8867abea-f006-493a-9925-b75b73ffca4d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr"
Jul 02 21:37:44.894285 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:44.894240    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr"
Jul 02 21:37:45.046128 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:45.046103    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr"]
Jul 02 21:37:45.048244 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:37:45.048211    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8867abea_f006_493a_9925_b75b73ffca4d.slice/crio-f0d170a3d0f2d32b816046e3627eeb6448cc15194a077547e66d75dc052be563 WatchSource:0}: Error finding container f0d170a3d0f2d32b816046e3627eeb6448cc15194a077547e66d75dc052be563: Status 404 returned error can't find the container with id f0d170a3d0f2d32b816046e3627eeb6448cc15194a077547e66d75dc052be563
Jul 02 21:37:45.155019 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:45.154991    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr" event={"ID":"8867abea-f006-493a-9925-b75b73ffca4d","Type":"ContainerStarted","Data":"450cd52d84c31fd7a912aaf246a5be18f72b04ed414e1f0f98bc9d9ae23b7fbb"}
Jul 02 21:37:45.155129 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:37:45.155057    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr" event={"ID":"8867abea-f006-493a-9925-b75b73ffca4d","Type":"ContainerStarted","Data":"f0d170a3d0f2d32b816046e3627eeb6448cc15194a077547e66d75dc052be563"}
Jul 02 21:38:14.280985 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:38:14.280895    2586 generic.go:356] "Generic (PLEG): container finished" podID="8867abea-f006-493a-9925-b75b73ffca4d" containerID="450cd52d84c31fd7a912aaf246a5be18f72b04ed414e1f0f98bc9d9ae23b7fbb" exitCode=0
Jul 02 21:38:14.280985 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:38:14.280968    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr" event={"ID":"8867abea-f006-493a-9925-b75b73ffca4d","Type":"ContainerDied","Data":"450cd52d84c31fd7a912aaf246a5be18f72b04ed414e1f0f98bc9d9ae23b7fbb"}
Jul 02 21:38:15.287745 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:38:15.287704    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr" event={"ID":"8867abea-f006-493a-9925-b75b73ffca4d","Type":"ContainerStarted","Data":"dd03b41064823318f97286422e5ef4c27946bcad02e59edf63b7573d42aa3134"}
Jul 02 21:38:15.288127 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:38:15.287775    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr" event={"ID":"8867abea-f006-493a-9925-b75b73ffca4d","Type":"ContainerStarted","Data":"704aa9776f37272181dafc7aeaa381c951075dfa0c13976fc594796a9aa0531d"}
Jul 02 21:38:15.288127 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:38:15.287978    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr"
Jul 02 21:38:15.309261 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:38:15.309176    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr" podStartSLOduration=2.443583668 podStartE2EDuration="31.309159261s" totalImagesPullingTime="0s" totalInitContainerRuntime="28.865575593s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:37:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:38:15.307402825 +0000 UTC m=+1167.750203281" watchObservedRunningTime="2026-07-02 21:38:15.309159261 +0000 UTC m=+1167.751959754"
Jul 02 21:38:16.293074 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:38:16.293037    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr"
Jul 02 21:38:22.303528 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:38:22.303495    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr"
Jul 02 21:39:53.022777 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:53.022740    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr"
Jul 02 21:39:54.711298 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:54.711263    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr"]
Jul 02 21:39:54.711770 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:54.711501    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr" podUID="8867abea-f006-493a-9925-b75b73ffca4d" containerName="kserve-container" containerID="cri-o://704aa9776f37272181dafc7aeaa381c951075dfa0c13976fc594796a9aa0531d" gracePeriod=30
Jul 02 21:39:54.711770 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:54.711560    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr" podUID="8867abea-f006-493a-9925-b75b73ffca4d" containerName="kube-rbac-proxy" containerID="cri-o://dd03b41064823318f97286422e5ef4c27946bcad02e59edf63b7573d42aa3134" gracePeriod=30
Jul 02 21:39:55.566077 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.566053    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr"
Jul 02 21:39:55.703360 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.703329    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8867abea-f006-493a-9925-b75b73ffca4d-kserve-provision-location\") pod \"8867abea-f006-493a-9925-b75b73ffca4d\" (UID: \"8867abea-f006-493a-9925-b75b73ffca4d\") "
Jul 02 21:39:55.703527 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.703372    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8867abea-f006-493a-9925-b75b73ffca4d-proxy-tls\") pod \"8867abea-f006-493a-9925-b75b73ffca4d\" (UID: \"8867abea-f006-493a-9925-b75b73ffca4d\") "
Jul 02 21:39:55.703527 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.703414    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8867abea-f006-493a-9925-b75b73ffca4d-cabundle-cert\") pod \"8867abea-f006-493a-9925-b75b73ffca4d\" (UID: \"8867abea-f006-493a-9925-b75b73ffca4d\") "
Jul 02 21:39:55.703527 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.703477    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cl9z4\" (UniqueName: \"kubernetes.io/projected/8867abea-f006-493a-9925-b75b73ffca4d-kube-api-access-cl9z4\") pod \"8867abea-f006-493a-9925-b75b73ffca4d\" (UID: \"8867abea-f006-493a-9925-b75b73ffca4d\") "
Jul 02 21:39:55.703527 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.703497    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8867abea-f006-493a-9925-b75b73ffca4d-kube-rbac-proxy-sar-config\") pod \"8867abea-f006-493a-9925-b75b73ffca4d\" (UID: \"8867abea-f006-493a-9925-b75b73ffca4d\") "
Jul 02 21:39:55.704095 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.703908    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8867abea-f006-493a-9925-b75b73ffca4d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8867abea-f006-493a-9925-b75b73ffca4d" (UID: "8867abea-f006-493a-9925-b75b73ffca4d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:39:55.704095 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.703961    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8867abea-f006-493a-9925-b75b73ffca4d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8867abea-f006-493a-9925-b75b73ffca4d" (UID: "8867abea-f006-493a-9925-b75b73ffca4d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:39:55.705839 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.705808    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8867abea-f006-493a-9925-b75b73ffca4d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8867abea-f006-493a-9925-b75b73ffca4d" (UID: "8867abea-f006-493a-9925-b75b73ffca4d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:39:55.705839 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.705817    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8867abea-f006-493a-9925-b75b73ffca4d-kube-api-access-cl9z4" (OuterVolumeSpecName: "kube-api-access-cl9z4") pod "8867abea-f006-493a-9925-b75b73ffca4d" (UID: "8867abea-f006-493a-9925-b75b73ffca4d"). InnerVolumeSpecName "kube-api-access-cl9z4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:39:55.708581 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.708552    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8867abea-f006-493a-9925-b75b73ffca4d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8867abea-f006-493a-9925-b75b73ffca4d" (UID: "8867abea-f006-493a-9925-b75b73ffca4d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:39:55.741563 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.741531    2586 generic.go:356] "Generic (PLEG): container finished" podID="8867abea-f006-493a-9925-b75b73ffca4d" containerID="dd03b41064823318f97286422e5ef4c27946bcad02e59edf63b7573d42aa3134" exitCode=2
Jul 02 21:39:55.741563 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.741555    2586 generic.go:356] "Generic (PLEG): container finished" podID="8867abea-f006-493a-9925-b75b73ffca4d" containerID="704aa9776f37272181dafc7aeaa381c951075dfa0c13976fc594796a9aa0531d" exitCode=0
Jul 02 21:39:55.741997 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.741592    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr"
Jul 02 21:39:55.741997 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.741605    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr" event={"ID":"8867abea-f006-493a-9925-b75b73ffca4d","Type":"ContainerDied","Data":"dd03b41064823318f97286422e5ef4c27946bcad02e59edf63b7573d42aa3134"}
Jul 02 21:39:55.741997 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.741675    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr" event={"ID":"8867abea-f006-493a-9925-b75b73ffca4d","Type":"ContainerDied","Data":"704aa9776f37272181dafc7aeaa381c951075dfa0c13976fc594796a9aa0531d"}
Jul 02 21:39:55.741997 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.741696    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr" event={"ID":"8867abea-f006-493a-9925-b75b73ffca4d","Type":"ContainerDied","Data":"f0d170a3d0f2d32b816046e3627eeb6448cc15194a077547e66d75dc052be563"}
Jul 02 21:39:55.741997 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.741718    2586 scope.go:117] "RemoveContainer" containerID="dd03b41064823318f97286422e5ef4c27946bcad02e59edf63b7573d42aa3134"
Jul 02 21:39:55.751673 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.751609    2586 scope.go:117] "RemoveContainer" containerID="704aa9776f37272181dafc7aeaa381c951075dfa0c13976fc594796a9aa0531d"
Jul 02 21:39:55.760137 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.760120    2586 scope.go:117] "RemoveContainer" containerID="450cd52d84c31fd7a912aaf246a5be18f72b04ed414e1f0f98bc9d9ae23b7fbb"
Jul 02 21:39:55.767426 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.767410    2586 scope.go:117] "RemoveContainer" containerID="dd03b41064823318f97286422e5ef4c27946bcad02e59edf63b7573d42aa3134"
Jul 02 21:39:55.767759 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:39:55.767731    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd03b41064823318f97286422e5ef4c27946bcad02e59edf63b7573d42aa3134\": container with ID starting with dd03b41064823318f97286422e5ef4c27946bcad02e59edf63b7573d42aa3134 not found: ID does not exist" containerID="dd03b41064823318f97286422e5ef4c27946bcad02e59edf63b7573d42aa3134"
Jul 02 21:39:55.767859 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.767770    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd03b41064823318f97286422e5ef4c27946bcad02e59edf63b7573d42aa3134"} err="failed to get container status \"dd03b41064823318f97286422e5ef4c27946bcad02e59edf63b7573d42aa3134\": rpc error: code = NotFound desc = could not find container \"dd03b41064823318f97286422e5ef4c27946bcad02e59edf63b7573d42aa3134\": container with ID starting with dd03b41064823318f97286422e5ef4c27946bcad02e59edf63b7573d42aa3134 not found: ID does not exist"
Jul 02 21:39:55.767859 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.767793    2586 scope.go:117] "RemoveContainer" containerID="704aa9776f37272181dafc7aeaa381c951075dfa0c13976fc594796a9aa0531d"
Jul 02 21:39:55.768084 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:39:55.768063    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"704aa9776f37272181dafc7aeaa381c951075dfa0c13976fc594796a9aa0531d\": container with ID starting with 704aa9776f37272181dafc7aeaa381c951075dfa0c13976fc594796a9aa0531d not found: ID does not exist" containerID="704aa9776f37272181dafc7aeaa381c951075dfa0c13976fc594796a9aa0531d"
Jul 02 21:39:55.768180 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.768088    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"704aa9776f37272181dafc7aeaa381c951075dfa0c13976fc594796a9aa0531d"} err="failed to get container status \"704aa9776f37272181dafc7aeaa381c951075dfa0c13976fc594796a9aa0531d\": rpc error: code = NotFound desc = could not find container \"704aa9776f37272181dafc7aeaa381c951075dfa0c13976fc594796a9aa0531d\": container with ID starting with 704aa9776f37272181dafc7aeaa381c951075dfa0c13976fc594796a9aa0531d not found: ID does not exist"
Jul 02 21:39:55.768180 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.768103    2586 scope.go:117] "RemoveContainer" containerID="450cd52d84c31fd7a912aaf246a5be18f72b04ed414e1f0f98bc9d9ae23b7fbb"
Jul 02 21:39:55.768330 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:39:55.768315    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"450cd52d84c31fd7a912aaf246a5be18f72b04ed414e1f0f98bc9d9ae23b7fbb\": container with ID starting with 450cd52d84c31fd7a912aaf246a5be18f72b04ed414e1f0f98bc9d9ae23b7fbb not found: ID does not exist" containerID="450cd52d84c31fd7a912aaf246a5be18f72b04ed414e1f0f98bc9d9ae23b7fbb"
Jul 02 21:39:55.768367 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.768334    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"450cd52d84c31fd7a912aaf246a5be18f72b04ed414e1f0f98bc9d9ae23b7fbb"} err="failed to get container status \"450cd52d84c31fd7a912aaf246a5be18f72b04ed414e1f0f98bc9d9ae23b7fbb\": rpc error: code = NotFound desc = could not find container \"450cd52d84c31fd7a912aaf246a5be18f72b04ed414e1f0f98bc9d9ae23b7fbb\": container with ID starting with 450cd52d84c31fd7a912aaf246a5be18f72b04ed414e1f0f98bc9d9ae23b7fbb not found: ID does not exist"
Jul 02 21:39:55.768367 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.768347    2586 scope.go:117] "RemoveContainer" containerID="dd03b41064823318f97286422e5ef4c27946bcad02e59edf63b7573d42aa3134"
Jul 02 21:39:55.768519 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.768499    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr"]
Jul 02 21:39:55.768561 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.768546    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd03b41064823318f97286422e5ef4c27946bcad02e59edf63b7573d42aa3134"} err="failed to get container status \"dd03b41064823318f97286422e5ef4c27946bcad02e59edf63b7573d42aa3134\": rpc error: code = NotFound desc = could not find container \"dd03b41064823318f97286422e5ef4c27946bcad02e59edf63b7573d42aa3134\": container with ID starting with dd03b41064823318f97286422e5ef4c27946bcad02e59edf63b7573d42aa3134 not found: ID does not exist"
Jul 02 21:39:55.768599 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.768564    2586 scope.go:117] "RemoveContainer" containerID="704aa9776f37272181dafc7aeaa381c951075dfa0c13976fc594796a9aa0531d"
Jul 02 21:39:55.768796 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.768775    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"704aa9776f37272181dafc7aeaa381c951075dfa0c13976fc594796a9aa0531d"} err="failed to get container status \"704aa9776f37272181dafc7aeaa381c951075dfa0c13976fc594796a9aa0531d\": rpc error: code = NotFound desc = could not find container \"704aa9776f37272181dafc7aeaa381c951075dfa0c13976fc594796a9aa0531d\": container with ID starting with 704aa9776f37272181dafc7aeaa381c951075dfa0c13976fc594796a9aa0531d not found: ID does not exist"
Jul 02 21:39:55.768796 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.768796    2586 scope.go:117] "RemoveContainer" containerID="450cd52d84c31fd7a912aaf246a5be18f72b04ed414e1f0f98bc9d9ae23b7fbb"
Jul 02 21:39:55.768983 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.768963    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"450cd52d84c31fd7a912aaf246a5be18f72b04ed414e1f0f98bc9d9ae23b7fbb"} err="failed to get container status \"450cd52d84c31fd7a912aaf246a5be18f72b04ed414e1f0f98bc9d9ae23b7fbb\": rpc error: code = NotFound desc = could not find container \"450cd52d84c31fd7a912aaf246a5be18f72b04ed414e1f0f98bc9d9ae23b7fbb\": container with ID starting with 450cd52d84c31fd7a912aaf246a5be18f72b04ed414e1f0f98bc9d9ae23b7fbb not found: ID does not exist"
Jul 02 21:39:55.772683 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.772640    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bmxbr"]
Jul 02 21:39:55.804823 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.804775    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl9z4\" (UniqueName: \"kubernetes.io/projected/8867abea-f006-493a-9925-b75b73ffca4d-kube-api-access-cl9z4\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:39:55.804823 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.804810    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8867abea-f006-493a-9925-b75b73ffca4d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:39:55.804823 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.804820    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8867abea-f006-493a-9925-b75b73ffca4d-kserve-provision-location\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:39:55.804823 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.804831    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8867abea-f006-493a-9925-b75b73ffca4d-proxy-tls\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:39:55.805102 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:55.804841    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8867abea-f006-493a-9925-b75b73ffca4d-cabundle-cert\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:39:56.022984 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:56.022888    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8867abea-f006-493a-9925-b75b73ffca4d" path="/var/lib/kubelet/pods/8867abea-f006-493a-9925-b75b73ffca4d/volumes"
Jul 02 21:39:59.891174 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:59.891144    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7"]
Jul 02 21:39:59.891667 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:59.891510    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8867abea-f006-493a-9925-b75b73ffca4d" containerName="storage-initializer"
Jul 02 21:39:59.891667 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:59.891529    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="8867abea-f006-493a-9925-b75b73ffca4d" containerName="storage-initializer"
Jul 02 21:39:59.891667 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:59.891541    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8867abea-f006-493a-9925-b75b73ffca4d" containerName="kserve-container"
Jul 02 21:39:59.891667 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:59.891547    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="8867abea-f006-493a-9925-b75b73ffca4d" containerName="kserve-container"
Jul 02 21:39:59.891667 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:59.891553    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8867abea-f006-493a-9925-b75b73ffca4d" containerName="kube-rbac-proxy"
Jul 02 21:39:59.891667 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:59.891559    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="8867abea-f006-493a-9925-b75b73ffca4d" containerName="kube-rbac-proxy"
Jul 02 21:39:59.891667 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:59.891636    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="8867abea-f006-493a-9925-b75b73ffca4d" containerName="kserve-container"
Jul 02 21:39:59.891667 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:59.891664    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="8867abea-f006-493a-9925-b75b73ffca4d" containerName="kube-rbac-proxy"
Jul 02 21:39:59.896768 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:59.896743    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7"
Jul 02 21:39:59.899776 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:59.899748    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 21:39:59.899981 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:59.899781    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 21:39:59.899981 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:59.899844    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Jul 02 21:39:59.899981 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:59.899850    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Jul 02 21:39:59.899981 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:59.899863    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 21:39:59.900877 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:59.900856    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 21:39:59.900980 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:59.900885    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pbfjv\""
Jul 02 21:39:59.906134 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:39:59.906113    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7"]
Jul 02 21:40:00.039614 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:40:00.039577    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7\" (UID: \"7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7"
Jul 02 21:40:00.039827 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:40:00.039633    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7\" (UID: \"7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7"
Jul 02 21:40:00.039827 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:40:00.039685    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7\" (UID: \"7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7"
Jul 02 21:40:00.039827 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:40:00.039711    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-647rs\" (UniqueName: \"kubernetes.io/projected/7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde-kube-api-access-647rs\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7\" (UID: \"7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7"
Jul 02 21:40:00.039827 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:40:00.039747    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7\" (UID: \"7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7"
Jul 02 21:40:00.140592 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:40:00.140553    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7\" (UID: \"7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7"
Jul 02 21:40:00.140799 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:40:00.140601    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7\" (UID: \"7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7"
Jul 02 21:40:00.140799 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:40:00.140642    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7\" (UID: \"7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7"
Jul 02 21:40:00.140799 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:40:00.140696    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7\" (UID: \"7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7"
Jul 02 21:40:00.140799 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:40:00.140716    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-647rs\" (UniqueName: \"kubernetes.io/projected/7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde-kube-api-access-647rs\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7\" (UID: \"7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7"
Jul 02 21:40:00.141091 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:40:00.141067    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7\" (UID: \"7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7"
Jul 02 21:40:00.141401 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:40:00.141359    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7\" (UID: \"7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7"
Jul 02 21:40:00.141401 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:40:00.141386    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7\" (UID: \"7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7"
Jul 02 21:40:00.143072 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:40:00.143056    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7\" (UID: \"7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7"
Jul 02 21:40:00.149035 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:40:00.149014    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-647rs\" (UniqueName: \"kubernetes.io/projected/7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde-kube-api-access-647rs\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7\" (UID: \"7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7"
Jul 02 21:40:00.208362 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:40:00.208322    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7"
Jul 02 21:40:00.338319 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:40:00.338279    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7"]
Jul 02 21:40:00.341273 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:40:00.341240    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fb6ff88_e1d5_43ba_a91a_5ea0171ffdde.slice/crio-66517ccd06e2529f33707e44c43a776f99dcf49f805ad8b8e74c2e77d8010da4 WatchSource:0}: Error finding container 66517ccd06e2529f33707e44c43a776f99dcf49f805ad8b8e74c2e77d8010da4: Status 404 returned error can't find the container with id 66517ccd06e2529f33707e44c43a776f99dcf49f805ad8b8e74c2e77d8010da4
Jul 02 21:40:00.769901 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:40:00.769863    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7" event={"ID":"7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde","Type":"ContainerStarted","Data":"36f8ed22fb1e10f6ba860e60cd1cc1550ccc38bf344a19f57dc87dd9a8194712"}
Jul 02 21:40:00.770062 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:40:00.769934    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7" event={"ID":"7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde","Type":"ContainerStarted","Data":"66517ccd06e2529f33707e44c43a776f99dcf49f805ad8b8e74c2e77d8010da4"}
Jul 02 21:40:20.859195 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:40:20.859102    2586 generic.go:356] "Generic (PLEG): container finished" podID="7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde" containerID="36f8ed22fb1e10f6ba860e60cd1cc1550ccc38bf344a19f57dc87dd9a8194712" exitCode=0
Jul 02 21:40:20.859195 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:40:20.859180    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7" event={"ID":"7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde","Type":"ContainerDied","Data":"36f8ed22fb1e10f6ba860e60cd1cc1550ccc38bf344a19f57dc87dd9a8194712"}
Jul 02 21:40:21.866423 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:40:21.866386    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7" event={"ID":"7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde","Type":"ContainerStarted","Data":"4e9111bcb450d4dda6bc88f047cc606871298fb075e06dc5f7191075d850d9d1"}
Jul 02 21:40:21.866795 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:40:21.866474    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7" event={"ID":"7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde","Type":"ContainerStarted","Data":"dd4482c10c14bf452e142531f38ab291977065c762512368755cd25063e88a0e"}
Jul 02 21:40:21.866795 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:40:21.866703    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7"
Jul 02 21:40:21.866893 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:40:21.866875    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7"
Jul 02 21:40:21.888591 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:40:21.888544    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7" podStartSLOduration=2.755669325 podStartE2EDuration="22.888530855s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.13286153s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:40:21.886207518 +0000 UTC m=+1294.329007939" watchObservedRunningTime="2026-07-02 21:40:21.888530855 +0000 UTC m=+1294.331331276"
Jul 02 21:40:27.876346 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:40:27.876309    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7"
Jul 02 21:41:56.022519 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:41:56.022488    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7"
Jul 02 21:42:00.026363 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:00.026327    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7"]
Jul 02 21:42:00.026813 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:00.026594    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7" podUID="7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde" containerName="kserve-container" containerID="cri-o://dd4482c10c14bf452e142531f38ab291977065c762512368755cd25063e88a0e" gracePeriod=30
Jul 02 21:42:00.026813 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:00.026632    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7" podUID="7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde" containerName="kube-rbac-proxy" containerID="cri-o://4e9111bcb450d4dda6bc88f047cc606871298fb075e06dc5f7191075d850d9d1" gracePeriod=30
Jul 02 21:42:00.314090 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:00.313980    2586 generic.go:356] "Generic (PLEG): container finished" podID="7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde" containerID="4e9111bcb450d4dda6bc88f047cc606871298fb075e06dc5f7191075d850d9d1" exitCode=2
Jul 02 21:42:00.314090 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:00.314059    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7" event={"ID":"7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde","Type":"ContainerDied","Data":"4e9111bcb450d4dda6bc88f047cc606871298fb075e06dc5f7191075d850d9d1"}
Jul 02 21:42:00.877919 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:00.877892    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7"
Jul 02 21:42:00.957970 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:00.957936    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde-kube-rbac-proxy-sar-config\") pod \"7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde\" (UID: \"7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde\") "
Jul 02 21:42:00.958170 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:00.957986    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde-kserve-provision-location\") pod \"7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde\" (UID: \"7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde\") "
Jul 02 21:42:00.958170 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:00.958043    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde-cabundle-cert\") pod \"7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde\" (UID: \"7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde\") "
Jul 02 21:42:00.958170 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:00.958069    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-647rs\" (UniqueName: \"kubernetes.io/projected/7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde-kube-api-access-647rs\") pod \"7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde\" (UID: \"7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde\") "
Jul 02 21:42:00.958170 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:00.958115    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde-proxy-tls\") pod \"7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde\" (UID: \"7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde\") "
Jul 02 21:42:00.958382 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:00.958347    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde" (UID: "7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:42:00.958433 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:00.958399    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde" (UID: "7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:42:00.960276 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:00.960238    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde" (UID: "7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:42:00.960276 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:00.960244    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde-kube-api-access-647rs" (OuterVolumeSpecName: "kube-api-access-647rs") pod "7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde" (UID: "7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde"). InnerVolumeSpecName "kube-api-access-647rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:42:00.962739 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:00.962711    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde" (UID: "7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:42:01.058917 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:01.058876    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:42:01.058917 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:01.058913    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde-kserve-provision-location\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:42:01.059292 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:01.058929    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde-cabundle-cert\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:42:01.059292 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:01.058944    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-647rs\" (UniqueName: \"kubernetes.io/projected/7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde-kube-api-access-647rs\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:42:01.059292 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:01.058959    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde-proxy-tls\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:42:01.320642 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:01.320553    2586 generic.go:356] "Generic (PLEG): container finished" podID="7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde" containerID="dd4482c10c14bf452e142531f38ab291977065c762512368755cd25063e88a0e" exitCode=0
Jul 02 21:42:01.320642 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:01.320621    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7"
Jul 02 21:42:01.320852 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:01.320660    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7" event={"ID":"7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde","Type":"ContainerDied","Data":"dd4482c10c14bf452e142531f38ab291977065c762512368755cd25063e88a0e"}
Jul 02 21:42:01.320852 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:01.320731    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7" event={"ID":"7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde","Type":"ContainerDied","Data":"66517ccd06e2529f33707e44c43a776f99dcf49f805ad8b8e74c2e77d8010da4"}
Jul 02 21:42:01.320852 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:01.320760    2586 scope.go:117] "RemoveContainer" containerID="4e9111bcb450d4dda6bc88f047cc606871298fb075e06dc5f7191075d850d9d1"
Jul 02 21:42:01.330143 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:01.330120    2586 scope.go:117] "RemoveContainer" containerID="dd4482c10c14bf452e142531f38ab291977065c762512368755cd25063e88a0e"
Jul 02 21:42:01.338217 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:01.338195    2586 scope.go:117] "RemoveContainer" containerID="36f8ed22fb1e10f6ba860e60cd1cc1550ccc38bf344a19f57dc87dd9a8194712"
Jul 02 21:42:01.343359 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:01.343335    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7"]
Jul 02 21:42:01.347051 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:01.347019    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-xkpx7"]
Jul 02 21:42:01.347119 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:01.347101    2586 scope.go:117] "RemoveContainer" containerID="4e9111bcb450d4dda6bc88f047cc606871298fb075e06dc5f7191075d850d9d1"
Jul 02 21:42:01.347391 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:42:01.347373    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e9111bcb450d4dda6bc88f047cc606871298fb075e06dc5f7191075d850d9d1\": container with ID starting with 4e9111bcb450d4dda6bc88f047cc606871298fb075e06dc5f7191075d850d9d1 not found: ID does not exist" containerID="4e9111bcb450d4dda6bc88f047cc606871298fb075e06dc5f7191075d850d9d1"
Jul 02 21:42:01.347434 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:01.347400    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e9111bcb450d4dda6bc88f047cc606871298fb075e06dc5f7191075d850d9d1"} err="failed to get container status \"4e9111bcb450d4dda6bc88f047cc606871298fb075e06dc5f7191075d850d9d1\": rpc error: code = NotFound desc = could not find container \"4e9111bcb450d4dda6bc88f047cc606871298fb075e06dc5f7191075d850d9d1\": container with ID starting with 4e9111bcb450d4dda6bc88f047cc606871298fb075e06dc5f7191075d850d9d1 not found: ID does not exist"
Jul 02 21:42:01.347434 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:01.347425    2586 scope.go:117] "RemoveContainer" containerID="dd4482c10c14bf452e142531f38ab291977065c762512368755cd25063e88a0e"
Jul 02 21:42:01.347678 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:42:01.347639    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd4482c10c14bf452e142531f38ab291977065c762512368755cd25063e88a0e\": container with ID starting with dd4482c10c14bf452e142531f38ab291977065c762512368755cd25063e88a0e not found: ID does not exist" containerID="dd4482c10c14bf452e142531f38ab291977065c762512368755cd25063e88a0e"
Jul 02 21:42:01.347718 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:01.347686    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd4482c10c14bf452e142531f38ab291977065c762512368755cd25063e88a0e"} err="failed to get container status \"dd4482c10c14bf452e142531f38ab291977065c762512368755cd25063e88a0e\": rpc error: code = NotFound desc = could not find container \"dd4482c10c14bf452e142531f38ab291977065c762512368755cd25063e88a0e\": container with ID starting with dd4482c10c14bf452e142531f38ab291977065c762512368755cd25063e88a0e not found: ID does not exist"
Jul 02 21:42:01.347718 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:01.347702    2586 scope.go:117] "RemoveContainer" containerID="36f8ed22fb1e10f6ba860e60cd1cc1550ccc38bf344a19f57dc87dd9a8194712"
Jul 02 21:42:01.347920 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:42:01.347897    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36f8ed22fb1e10f6ba860e60cd1cc1550ccc38bf344a19f57dc87dd9a8194712\": container with ID starting with 36f8ed22fb1e10f6ba860e60cd1cc1550ccc38bf344a19f57dc87dd9a8194712 not found: ID does not exist" containerID="36f8ed22fb1e10f6ba860e60cd1cc1550ccc38bf344a19f57dc87dd9a8194712"
Jul 02 21:42:01.348006 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:01.347927    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36f8ed22fb1e10f6ba860e60cd1cc1550ccc38bf344a19f57dc87dd9a8194712"} err="failed to get container status \"36f8ed22fb1e10f6ba860e60cd1cc1550ccc38bf344a19f57dc87dd9a8194712\": rpc error: code = NotFound desc = could not find container \"36f8ed22fb1e10f6ba860e60cd1cc1550ccc38bf344a19f57dc87dd9a8194712\": container with ID starting with 36f8ed22fb1e10f6ba860e60cd1cc1550ccc38bf344a19f57dc87dd9a8194712 not found: ID does not exist"
Jul 02 21:42:02.022232 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:02.022196    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde" path="/var/lib/kubelet/pods/7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde/volumes"
Jul 02 21:42:05.175129 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:05.175097    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc"]
Jul 02 21:42:05.175496 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:05.175411    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde" containerName="kserve-container"
Jul 02 21:42:05.175496 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:05.175423    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde" containerName="kserve-container"
Jul 02 21:42:05.175496 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:05.175433    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde" containerName="storage-initializer"
Jul 02 21:42:05.175496 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:05.175439    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde" containerName="storage-initializer"
Jul 02 21:42:05.175496 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:05.175458    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde" containerName="kube-rbac-proxy"
Jul 02 21:42:05.175496 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:05.175464    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde" containerName="kube-rbac-proxy"
Jul 02 21:42:05.175769 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:05.175509    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde" containerName="kserve-container"
Jul 02 21:42:05.175769 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:05.175515    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fb6ff88-e1d5-43ba-a91a-5ea0171ffdde" containerName="kube-rbac-proxy"
Jul 02 21:42:05.178993 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:05.178975    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc"
Jul 02 21:42:05.181797 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:05.181775    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 21:42:05.181905 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:05.181794    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Jul 02 21:42:05.182147 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:05.182132    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pbfjv\""
Jul 02 21:42:05.182205 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:05.182159    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 21:42:05.182268 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:05.182249    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Jul 02 21:42:05.182327 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:05.182288    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 21:42:05.182384 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:05.182375    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 21:42:05.187258 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:05.187233    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0738e969-6412-41b7-99bb-1d6f152d0cbf-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc\" (UID: \"0738e969-6412-41b7-99bb-1d6f152d0cbf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc"
Jul 02 21:42:05.187453 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:05.187433    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0738e969-6412-41b7-99bb-1d6f152d0cbf-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc\" (UID: \"0738e969-6412-41b7-99bb-1d6f152d0cbf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc"
Jul 02 21:42:05.187596 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:05.187582    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0738e969-6412-41b7-99bb-1d6f152d0cbf-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc\" (UID: \"0738e969-6412-41b7-99bb-1d6f152d0cbf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc"
Jul 02 21:42:05.187736 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:05.187720    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0738e969-6412-41b7-99bb-1d6f152d0cbf-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc\" (UID: \"0738e969-6412-41b7-99bb-1d6f152d0cbf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc"
Jul 02 21:42:05.187854 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:05.187841    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ldm6\" (UniqueName: \"kubernetes.io/projected/0738e969-6412-41b7-99bb-1d6f152d0cbf-kube-api-access-9ldm6\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc\" (UID: \"0738e969-6412-41b7-99bb-1d6f152d0cbf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc"
Jul 02 21:42:05.195248 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:05.195189    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc"]
Jul 02 21:42:05.288464 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:05.288427    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0738e969-6412-41b7-99bb-1d6f152d0cbf-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc\" (UID: \"0738e969-6412-41b7-99bb-1d6f152d0cbf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc"
Jul 02 21:42:05.288464 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:05.288470    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0738e969-6412-41b7-99bb-1d6f152d0cbf-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc\" (UID: \"0738e969-6412-41b7-99bb-1d6f152d0cbf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc"
Jul 02 21:42:05.288729 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:05.288594    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0738e969-6412-41b7-99bb-1d6f152d0cbf-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc\" (UID: \"0738e969-6412-41b7-99bb-1d6f152d0cbf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc"
Jul 02 21:42:05.288729 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:05.288638    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0738e969-6412-41b7-99bb-1d6f152d0cbf-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc\" (UID: \"0738e969-6412-41b7-99bb-1d6f152d0cbf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc"
Jul 02 21:42:05.288729 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:05.288696    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ldm6\" (UniqueName: \"kubernetes.io/projected/0738e969-6412-41b7-99bb-1d6f152d0cbf-kube-api-access-9ldm6\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc\" (UID: \"0738e969-6412-41b7-99bb-1d6f152d0cbf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc"
Jul 02 21:42:05.288855 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:42:05.288777    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-serving-cert: secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Jul 02 21:42:05.288855 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:42:05.288843    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0738e969-6412-41b7-99bb-1d6f152d0cbf-proxy-tls podName:0738e969-6412-41b7-99bb-1d6f152d0cbf nodeName:}" failed. No retries permitted until 2026-07-02 21:42:05.788823627 +0000 UTC m=+1398.231624030 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0738e969-6412-41b7-99bb-1d6f152d0cbf-proxy-tls") pod "isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc" (UID: "0738e969-6412-41b7-99bb-1d6f152d0cbf") : secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Jul 02 21:42:05.289024 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:05.288998    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0738e969-6412-41b7-99bb-1d6f152d0cbf-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc\" (UID: \"0738e969-6412-41b7-99bb-1d6f152d0cbf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc"
Jul 02 21:42:05.289213 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:05.289190    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0738e969-6412-41b7-99bb-1d6f152d0cbf-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc\" (UID: \"0738e969-6412-41b7-99bb-1d6f152d0cbf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc"
Jul 02 21:42:05.289320 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:05.289217    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0738e969-6412-41b7-99bb-1d6f152d0cbf-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc\" (UID: \"0738e969-6412-41b7-99bb-1d6f152d0cbf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc"
Jul 02 21:42:05.297441 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:05.297412    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ldm6\" (UniqueName: \"kubernetes.io/projected/0738e969-6412-41b7-99bb-1d6f152d0cbf-kube-api-access-9ldm6\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc\" (UID: \"0738e969-6412-41b7-99bb-1d6f152d0cbf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc"
Jul 02 21:42:05.794143 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:05.794100    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0738e969-6412-41b7-99bb-1d6f152d0cbf-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc\" (UID: \"0738e969-6412-41b7-99bb-1d6f152d0cbf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc"
Jul 02 21:42:05.796721 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:05.796695    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0738e969-6412-41b7-99bb-1d6f152d0cbf-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc\" (UID: \"0738e969-6412-41b7-99bb-1d6f152d0cbf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc"
Jul 02 21:42:06.096964 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:06.096850    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc"
Jul 02 21:42:06.225548 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:06.225522    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc"]
Jul 02 21:42:06.227531 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:42:06.227501    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0738e969_6412_41b7_99bb_1d6f152d0cbf.slice/crio-2fac0d4e64eb8766e70a2f7fe21a21399813d0fc00c61e23cbddbd5a09b08b28 WatchSource:0}: Error finding container 2fac0d4e64eb8766e70a2f7fe21a21399813d0fc00c61e23cbddbd5a09b08b28: Status 404 returned error can't find the container with id 2fac0d4e64eb8766e70a2f7fe21a21399813d0fc00c61e23cbddbd5a09b08b28
Jul 02 21:42:06.346723 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:06.346670    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc" event={"ID":"0738e969-6412-41b7-99bb-1d6f152d0cbf","Type":"ContainerStarted","Data":"127b5ca312d2ef2979ed5ffb22cc0b93c190cfc49337291984f43c8066cb224f"}
Jul 02 21:42:06.346914 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:06.346756    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc" event={"ID":"0738e969-6412-41b7-99bb-1d6f152d0cbf","Type":"ContainerStarted","Data":"2fac0d4e64eb8766e70a2f7fe21a21399813d0fc00c61e23cbddbd5a09b08b28"}
Jul 02 21:42:27.442163 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:27.442125    2586 generic.go:356] "Generic (PLEG): container finished" podID="0738e969-6412-41b7-99bb-1d6f152d0cbf" containerID="127b5ca312d2ef2979ed5ffb22cc0b93c190cfc49337291984f43c8066cb224f" exitCode=0
Jul 02 21:42:27.442569 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:27.442197    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc" event={"ID":"0738e969-6412-41b7-99bb-1d6f152d0cbf","Type":"ContainerDied","Data":"127b5ca312d2ef2979ed5ffb22cc0b93c190cfc49337291984f43c8066cb224f"}
Jul 02 21:42:28.449214 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:28.449177    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc" event={"ID":"0738e969-6412-41b7-99bb-1d6f152d0cbf","Type":"ContainerStarted","Data":"538df0e351dcc1978ae343a4c134dae5e445d6395b08cd4f8ca8b9628ea63f5e"}
Jul 02 21:42:28.449724 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:28.449279    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc" event={"ID":"0738e969-6412-41b7-99bb-1d6f152d0cbf","Type":"ContainerStarted","Data":"ffdeeb938b7359e689fb887d1f4b2a26a10848103e70b23108de0031e441e55c"}
Jul 02 21:42:28.449724 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:28.449514    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc"
Jul 02 21:42:28.468155 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:28.468100    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc" podStartSLOduration=3.167900827 podStartE2EDuration="23.468081638s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.300180811s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:42:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:42:28.466814336 +0000 UTC m=+1420.909614796" watchObservedRunningTime="2026-07-02 21:42:28.468081638 +0000 UTC m=+1420.910882060"
Jul 02 21:42:29.454231 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:29.454204    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc"
Jul 02 21:42:35.464083 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:42:35.464051    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc"
Jul 02 21:44:04.023224 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:04.023191    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc"
Jul 02 21:44:05.340106 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:05.340068    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc"]
Jul 02 21:44:05.340527 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:05.340326    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc" podUID="0738e969-6412-41b7-99bb-1d6f152d0cbf" containerName="kserve-container" containerID="cri-o://ffdeeb938b7359e689fb887d1f4b2a26a10848103e70b23108de0031e441e55c" gracePeriod=30
Jul 02 21:44:05.340527 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:05.340336    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc" podUID="0738e969-6412-41b7-99bb-1d6f152d0cbf" containerName="kube-rbac-proxy" containerID="cri-o://538df0e351dcc1978ae343a4c134dae5e445d6395b08cd4f8ca8b9628ea63f5e" gracePeriod=30
Jul 02 21:44:05.458704 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:05.458668    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc" podUID="0738e969-6412-41b7-99bb-1d6f152d0cbf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Jul 02 21:44:05.884823 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:05.884788    2586 generic.go:356] "Generic (PLEG): container finished" podID="0738e969-6412-41b7-99bb-1d6f152d0cbf" containerID="538df0e351dcc1978ae343a4c134dae5e445d6395b08cd4f8ca8b9628ea63f5e" exitCode=2
Jul 02 21:44:05.884994 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:05.884871    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc" event={"ID":"0738e969-6412-41b7-99bb-1d6f152d0cbf","Type":"ContainerDied","Data":"538df0e351dcc1978ae343a4c134dae5e445d6395b08cd4f8ca8b9628ea63f5e"}
Jul 02 21:44:06.293111 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:06.293077    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc"
Jul 02 21:44:06.312516 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:06.312482    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0738e969-6412-41b7-99bb-1d6f152d0cbf-proxy-tls\") pod \"0738e969-6412-41b7-99bb-1d6f152d0cbf\" (UID: \"0738e969-6412-41b7-99bb-1d6f152d0cbf\") "
Jul 02 21:44:06.312721 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:06.312526    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0738e969-6412-41b7-99bb-1d6f152d0cbf-cabundle-cert\") pod \"0738e969-6412-41b7-99bb-1d6f152d0cbf\" (UID: \"0738e969-6412-41b7-99bb-1d6f152d0cbf\") "
Jul 02 21:44:06.312721 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:06.312565    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0738e969-6412-41b7-99bb-1d6f152d0cbf-kserve-provision-location\") pod \"0738e969-6412-41b7-99bb-1d6f152d0cbf\" (UID: \"0738e969-6412-41b7-99bb-1d6f152d0cbf\") "
Jul 02 21:44:06.312721 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:06.312598    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0738e969-6412-41b7-99bb-1d6f152d0cbf-kube-rbac-proxy-sar-config\") pod \"0738e969-6412-41b7-99bb-1d6f152d0cbf\" (UID: \"0738e969-6412-41b7-99bb-1d6f152d0cbf\") "
Jul 02 21:44:06.312721 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:06.312638    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ldm6\" (UniqueName: \"kubernetes.io/projected/0738e969-6412-41b7-99bb-1d6f152d0cbf-kube-api-access-9ldm6\") pod \"0738e969-6412-41b7-99bb-1d6f152d0cbf\" (UID: \"0738e969-6412-41b7-99bb-1d6f152d0cbf\") "
Jul 02 21:44:06.313002 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:06.312976    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0738e969-6412-41b7-99bb-1d6f152d0cbf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0738e969-6412-41b7-99bb-1d6f152d0cbf" (UID: "0738e969-6412-41b7-99bb-1d6f152d0cbf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:44:06.313064 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:06.313025    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0738e969-6412-41b7-99bb-1d6f152d0cbf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0738e969-6412-41b7-99bb-1d6f152d0cbf" (UID: "0738e969-6412-41b7-99bb-1d6f152d0cbf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:44:06.315203 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:06.315174    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0738e969-6412-41b7-99bb-1d6f152d0cbf-kube-api-access-9ldm6" (OuterVolumeSpecName: "kube-api-access-9ldm6") pod "0738e969-6412-41b7-99bb-1d6f152d0cbf" (UID: "0738e969-6412-41b7-99bb-1d6f152d0cbf"). InnerVolumeSpecName "kube-api-access-9ldm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:44:06.316088 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:06.316057    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0738e969-6412-41b7-99bb-1d6f152d0cbf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0738e969-6412-41b7-99bb-1d6f152d0cbf" (UID: "0738e969-6412-41b7-99bb-1d6f152d0cbf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:44:06.318432 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:06.318404    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0738e969-6412-41b7-99bb-1d6f152d0cbf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0738e969-6412-41b7-99bb-1d6f152d0cbf" (UID: "0738e969-6412-41b7-99bb-1d6f152d0cbf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:44:06.414052 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:06.413953    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ldm6\" (UniqueName: \"kubernetes.io/projected/0738e969-6412-41b7-99bb-1d6f152d0cbf-kube-api-access-9ldm6\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:44:06.414052 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:06.413988    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0738e969-6412-41b7-99bb-1d6f152d0cbf-proxy-tls\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:44:06.414052 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:06.414000    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0738e969-6412-41b7-99bb-1d6f152d0cbf-cabundle-cert\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:44:06.414052 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:06.414008    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0738e969-6412-41b7-99bb-1d6f152d0cbf-kserve-provision-location\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:44:06.414052 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:06.414016    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0738e969-6412-41b7-99bb-1d6f152d0cbf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:44:06.891392 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:06.891351    2586 generic.go:356] "Generic (PLEG): container finished" podID="0738e969-6412-41b7-99bb-1d6f152d0cbf" containerID="ffdeeb938b7359e689fb887d1f4b2a26a10848103e70b23108de0031e441e55c" exitCode=0
Jul 02 21:44:06.891597 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:06.891405    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc" event={"ID":"0738e969-6412-41b7-99bb-1d6f152d0cbf","Type":"ContainerDied","Data":"ffdeeb938b7359e689fb887d1f4b2a26a10848103e70b23108de0031e441e55c"}
Jul 02 21:44:06.891597 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:06.891413    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc"
Jul 02 21:44:06.891597 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:06.891500    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc" event={"ID":"0738e969-6412-41b7-99bb-1d6f152d0cbf","Type":"ContainerDied","Data":"2fac0d4e64eb8766e70a2f7fe21a21399813d0fc00c61e23cbddbd5a09b08b28"}
Jul 02 21:44:06.891597 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:06.891540    2586 scope.go:117] "RemoveContainer" containerID="538df0e351dcc1978ae343a4c134dae5e445d6395b08cd4f8ca8b9628ea63f5e"
Jul 02 21:44:06.900756 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:06.900739    2586 scope.go:117] "RemoveContainer" containerID="ffdeeb938b7359e689fb887d1f4b2a26a10848103e70b23108de0031e441e55c"
Jul 02 21:44:06.908411 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:06.908389    2586 scope.go:117] "RemoveContainer" containerID="127b5ca312d2ef2979ed5ffb22cc0b93c190cfc49337291984f43c8066cb224f"
Jul 02 21:44:06.914990 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:06.914962    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc"]
Jul 02 21:44:06.917418 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:06.917393    2586 scope.go:117] "RemoveContainer" containerID="538df0e351dcc1978ae343a4c134dae5e445d6395b08cd4f8ca8b9628ea63f5e"
Jul 02 21:44:06.917797 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:44:06.917775    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"538df0e351dcc1978ae343a4c134dae5e445d6395b08cd4f8ca8b9628ea63f5e\": container with ID starting with 538df0e351dcc1978ae343a4c134dae5e445d6395b08cd4f8ca8b9628ea63f5e not found: ID does not exist" containerID="538df0e351dcc1978ae343a4c134dae5e445d6395b08cd4f8ca8b9628ea63f5e"
Jul 02 21:44:06.917858 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:06.917807    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"538df0e351dcc1978ae343a4c134dae5e445d6395b08cd4f8ca8b9628ea63f5e"} err="failed to get container status \"538df0e351dcc1978ae343a4c134dae5e445d6395b08cd4f8ca8b9628ea63f5e\": rpc error: code = NotFound desc = could not find container \"538df0e351dcc1978ae343a4c134dae5e445d6395b08cd4f8ca8b9628ea63f5e\": container with ID starting with 538df0e351dcc1978ae343a4c134dae5e445d6395b08cd4f8ca8b9628ea63f5e not found: ID does not exist"
Jul 02 21:44:06.917858 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:06.917831    2586 scope.go:117] "RemoveContainer" containerID="ffdeeb938b7359e689fb887d1f4b2a26a10848103e70b23108de0031e441e55c"
Jul 02 21:44:06.918088 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:44:06.918063    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffdeeb938b7359e689fb887d1f4b2a26a10848103e70b23108de0031e441e55c\": container with ID starting with ffdeeb938b7359e689fb887d1f4b2a26a10848103e70b23108de0031e441e55c not found: ID does not exist" containerID="ffdeeb938b7359e689fb887d1f4b2a26a10848103e70b23108de0031e441e55c"
Jul 02 21:44:06.918172 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:06.918094    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffdeeb938b7359e689fb887d1f4b2a26a10848103e70b23108de0031e441e55c"} err="failed to get container status \"ffdeeb938b7359e689fb887d1f4b2a26a10848103e70b23108de0031e441e55c\": rpc error: code = NotFound desc = could not find container \"ffdeeb938b7359e689fb887d1f4b2a26a10848103e70b23108de0031e441e55c\": container with ID starting with ffdeeb938b7359e689fb887d1f4b2a26a10848103e70b23108de0031e441e55c not found: ID does not exist"
Jul 02 21:44:06.918172 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:06.918112    2586 scope.go:117] "RemoveContainer" containerID="127b5ca312d2ef2979ed5ffb22cc0b93c190cfc49337291984f43c8066cb224f"
Jul 02 21:44:06.918172 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:06.918143    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-92nmc"]
Jul 02 21:44:06.918382 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:44:06.918364    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"127b5ca312d2ef2979ed5ffb22cc0b93c190cfc49337291984f43c8066cb224f\": container with ID starting with 127b5ca312d2ef2979ed5ffb22cc0b93c190cfc49337291984f43c8066cb224f not found: ID does not exist" containerID="127b5ca312d2ef2979ed5ffb22cc0b93c190cfc49337291984f43c8066cb224f"
Jul 02 21:44:06.918419 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:06.918390    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"127b5ca312d2ef2979ed5ffb22cc0b93c190cfc49337291984f43c8066cb224f"} err="failed to get container status \"127b5ca312d2ef2979ed5ffb22cc0b93c190cfc49337291984f43c8066cb224f\": rpc error: code = NotFound desc = could not find container \"127b5ca312d2ef2979ed5ffb22cc0b93c190cfc49337291984f43c8066cb224f\": container with ID starting with 127b5ca312d2ef2979ed5ffb22cc0b93c190cfc49337291984f43c8066cb224f not found: ID does not exist"
Jul 02 21:44:08.022745 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:08.022708    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0738e969-6412-41b7-99bb-1d6f152d0cbf" path="/var/lib/kubelet/pods/0738e969-6412-41b7-99bb-1d6f152d0cbf/volumes"
Jul 02 21:44:10.485129 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:10.485045    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv"]
Jul 02 21:44:10.485526 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:10.485346    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0738e969-6412-41b7-99bb-1d6f152d0cbf" containerName="storage-initializer"
Jul 02 21:44:10.485526 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:10.485357    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0738e969-6412-41b7-99bb-1d6f152d0cbf" containerName="storage-initializer"
Jul 02 21:44:10.485526 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:10.485369    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0738e969-6412-41b7-99bb-1d6f152d0cbf" containerName="kserve-container"
Jul 02 21:44:10.485526 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:10.485376    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0738e969-6412-41b7-99bb-1d6f152d0cbf" containerName="kserve-container"
Jul 02 21:44:10.485526 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:10.485387    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0738e969-6412-41b7-99bb-1d6f152d0cbf" containerName="kube-rbac-proxy"
Jul 02 21:44:10.485526 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:10.485395    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0738e969-6412-41b7-99bb-1d6f152d0cbf" containerName="kube-rbac-proxy"
Jul 02 21:44:10.485526 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:10.485449    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="0738e969-6412-41b7-99bb-1d6f152d0cbf" containerName="kserve-container"
Jul 02 21:44:10.485526 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:10.485458    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="0738e969-6412-41b7-99bb-1d6f152d0cbf" containerName="kube-rbac-proxy"
Jul 02 21:44:10.490035 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:10.490018    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv"
Jul 02 21:44:10.493357 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:10.493323    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 21:44:10.493357 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:10.493338    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Jul 02 21:44:10.493633 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:10.493430    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 21:44:10.493633 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:10.493448    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 21:44:10.493797 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:10.493774    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 21:44:10.494422 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:10.494404    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pbfjv\""
Jul 02 21:44:10.494528 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:10.494407    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Jul 02 21:44:10.499755 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:10.499733    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv"]
Jul 02 21:44:10.544835 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:10.544794    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca50158d-ce0c-4ebd-bddc-c87df7753b08-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv\" (UID: \"ca50158d-ce0c-4ebd-bddc-c87df7753b08\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv"
Jul 02 21:44:10.545003 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:10.544840    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbw7h\" (UniqueName: \"kubernetes.io/projected/ca50158d-ce0c-4ebd-bddc-c87df7753b08-kube-api-access-dbw7h\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv\" (UID: \"ca50158d-ce0c-4ebd-bddc-c87df7753b08\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv"
Jul 02 21:44:10.545003 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:10.544868    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca50158d-ce0c-4ebd-bddc-c87df7753b08-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv\" (UID: \"ca50158d-ce0c-4ebd-bddc-c87df7753b08\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv"
Jul 02 21:44:10.545003 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:10.544929    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca50158d-ce0c-4ebd-bddc-c87df7753b08-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv\" (UID: \"ca50158d-ce0c-4ebd-bddc-c87df7753b08\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv"
Jul 02 21:44:10.545003 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:10.544973    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca50158d-ce0c-4ebd-bddc-c87df7753b08-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv\" (UID: \"ca50158d-ce0c-4ebd-bddc-c87df7753b08\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv"
Jul 02 21:44:10.646135 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:10.646093    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbw7h\" (UniqueName: \"kubernetes.io/projected/ca50158d-ce0c-4ebd-bddc-c87df7753b08-kube-api-access-dbw7h\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv\" (UID: \"ca50158d-ce0c-4ebd-bddc-c87df7753b08\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv"
Jul 02 21:44:10.646135 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:10.646136    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca50158d-ce0c-4ebd-bddc-c87df7753b08-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv\" (UID: \"ca50158d-ce0c-4ebd-bddc-c87df7753b08\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv"
Jul 02 21:44:10.646397 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:10.646170    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca50158d-ce0c-4ebd-bddc-c87df7753b08-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv\" (UID: \"ca50158d-ce0c-4ebd-bddc-c87df7753b08\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv"
Jul 02 21:44:10.646397 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:10.646206    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca50158d-ce0c-4ebd-bddc-c87df7753b08-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv\" (UID: \"ca50158d-ce0c-4ebd-bddc-c87df7753b08\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv"
Jul 02 21:44:10.646397 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:10.646289    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca50158d-ce0c-4ebd-bddc-c87df7753b08-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv\" (UID: \"ca50158d-ce0c-4ebd-bddc-c87df7753b08\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv"
Jul 02 21:44:10.646397 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:44:10.646395    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-serving-cert: secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Jul 02 21:44:10.646636 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:44:10.646456    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca50158d-ce0c-4ebd-bddc-c87df7753b08-proxy-tls podName:ca50158d-ce0c-4ebd-bddc-c87df7753b08 nodeName:}" failed. No retries permitted until 2026-07-02 21:44:11.146434241 +0000 UTC m=+1523.589234659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ca50158d-ce0c-4ebd-bddc-c87df7753b08-proxy-tls") pod "isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv" (UID: "ca50158d-ce0c-4ebd-bddc-c87df7753b08") : secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Jul 02 21:44:10.646636 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:10.646574    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca50158d-ce0c-4ebd-bddc-c87df7753b08-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv\" (UID: \"ca50158d-ce0c-4ebd-bddc-c87df7753b08\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv"
Jul 02 21:44:10.646899 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:10.646881    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca50158d-ce0c-4ebd-bddc-c87df7753b08-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv\" (UID: \"ca50158d-ce0c-4ebd-bddc-c87df7753b08\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv"
Jul 02 21:44:10.646937 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:10.646915    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca50158d-ce0c-4ebd-bddc-c87df7753b08-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv\" (UID: \"ca50158d-ce0c-4ebd-bddc-c87df7753b08\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv"
Jul 02 21:44:10.656816 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:10.656795    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbw7h\" (UniqueName: \"kubernetes.io/projected/ca50158d-ce0c-4ebd-bddc-c87df7753b08-kube-api-access-dbw7h\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv\" (UID: \"ca50158d-ce0c-4ebd-bddc-c87df7753b08\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv"
Jul 02 21:44:11.151015 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:11.150968    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca50158d-ce0c-4ebd-bddc-c87df7753b08-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv\" (UID: \"ca50158d-ce0c-4ebd-bddc-c87df7753b08\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv"
Jul 02 21:44:11.153433 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:11.153407    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca50158d-ce0c-4ebd-bddc-c87df7753b08-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv\" (UID: \"ca50158d-ce0c-4ebd-bddc-c87df7753b08\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv"
Jul 02 21:44:11.401710 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:11.401583    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv"
Jul 02 21:44:11.536850 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:11.536824    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv"]
Jul 02 21:44:11.538596 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:44:11.538563    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca50158d_ce0c_4ebd_bddc_c87df7753b08.slice/crio-cc1ce46a9cce3fc55ad9d0375896c6e16a584c30a1af38836a86e6380efe730f WatchSource:0}: Error finding container cc1ce46a9cce3fc55ad9d0375896c6e16a584c30a1af38836a86e6380efe730f: Status 404 returned error can't find the container with id cc1ce46a9cce3fc55ad9d0375896c6e16a584c30a1af38836a86e6380efe730f
Jul 02 21:44:11.922571 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:11.922525    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv" event={"ID":"ca50158d-ce0c-4ebd-bddc-c87df7753b08","Type":"ContainerStarted","Data":"78ec1cc6830391f3e54d00b95034ec4484e3c9e4f0a8856c5990bba0468f55da"}
Jul 02 21:44:11.922738 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:11.922615    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv" event={"ID":"ca50158d-ce0c-4ebd-bddc-c87df7753b08","Type":"ContainerStarted","Data":"cc1ce46a9cce3fc55ad9d0375896c6e16a584c30a1af38836a86e6380efe730f"}
Jul 02 21:44:19.960906 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:19.960865    2586 generic.go:356] "Generic (PLEG): container finished" podID="ca50158d-ce0c-4ebd-bddc-c87df7753b08" containerID="78ec1cc6830391f3e54d00b95034ec4484e3c9e4f0a8856c5990bba0468f55da" exitCode=0
Jul 02 21:44:19.960906 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:19.960900    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv" event={"ID":"ca50158d-ce0c-4ebd-bddc-c87df7753b08","Type":"ContainerDied","Data":"78ec1cc6830391f3e54d00b95034ec4484e3c9e4f0a8856c5990bba0468f55da"}
Jul 02 21:44:20.967362 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:20.967330    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv" event={"ID":"ca50158d-ce0c-4ebd-bddc-c87df7753b08","Type":"ContainerStarted","Data":"233fd589f21dd000243aee83969e2cd65ae473623959b4883741c19d7518848a"}
Jul 02 21:44:20.967763 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:20.967395    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv" event={"ID":"ca50158d-ce0c-4ebd-bddc-c87df7753b08","Type":"ContainerStarted","Data":"a3663b3c0c90460d3f93428bb8d97a154732c29ffa7cd4fe6ab10e9cd6960304"}
Jul 02 21:44:20.967763 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:20.967611    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv"
Jul 02 21:44:20.990298 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:20.990241    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv" podStartSLOduration=3.352179137 podStartE2EDuration="10.990224083s" totalImagesPullingTime="0s" totalInitContainerRuntime="7.638044946s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:44:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:44:20.99019293 +0000 UTC m=+1533.432993389" watchObservedRunningTime="2026-07-02 21:44:20.990224083 +0000 UTC m=+1533.433024507"
Jul 02 21:44:21.971509 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:21.971471    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv"
Jul 02 21:44:21.973061 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:21.973027    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv" podUID="ca50158d-ce0c-4ebd-bddc-c87df7753b08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 02 21:44:22.975738 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:22.975701    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv" podUID="ca50158d-ce0c-4ebd-bddc-c87df7753b08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 02 21:44:27.981870 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:27.981840    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv"
Jul 02 21:44:27.982316 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:27.982291    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv" podUID="ca50158d-ce0c-4ebd-bddc-c87df7753b08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 02 21:44:37.982812 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:37.982784    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv"
Jul 02 21:44:40.539498 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:40.539459    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv"]
Jul 02 21:44:40.539992 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:40.539793    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv" podUID="ca50158d-ce0c-4ebd-bddc-c87df7753b08" containerName="kserve-container" containerID="cri-o://a3663b3c0c90460d3f93428bb8d97a154732c29ffa7cd4fe6ab10e9cd6960304" gracePeriod=30
Jul 02 21:44:40.539992 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:40.539970    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv" podUID="ca50158d-ce0c-4ebd-bddc-c87df7753b08" containerName="kube-rbac-proxy" containerID="cri-o://233fd589f21dd000243aee83969e2cd65ae473623959b4883741c19d7518848a" gracePeriod=30
Jul 02 21:44:41.062126 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:41.062086    2586 generic.go:356] "Generic (PLEG): container finished" podID="ca50158d-ce0c-4ebd-bddc-c87df7753b08" containerID="233fd589f21dd000243aee83969e2cd65ae473623959b4883741c19d7518848a" exitCode=2
Jul 02 21:44:41.062320 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:41.062172    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv" event={"ID":"ca50158d-ce0c-4ebd-bddc-c87df7753b08","Type":"ContainerDied","Data":"233fd589f21dd000243aee83969e2cd65ae473623959b4883741c19d7518848a"}
Jul 02 21:44:41.489403 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:41.489377    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv"
Jul 02 21:44:41.614171 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:41.614053    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbw7h\" (UniqueName: \"kubernetes.io/projected/ca50158d-ce0c-4ebd-bddc-c87df7753b08-kube-api-access-dbw7h\") pod \"ca50158d-ce0c-4ebd-bddc-c87df7753b08\" (UID: \"ca50158d-ce0c-4ebd-bddc-c87df7753b08\") "
Jul 02 21:44:41.614171 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:41.614139    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca50158d-ce0c-4ebd-bddc-c87df7753b08-kserve-provision-location\") pod \"ca50158d-ce0c-4ebd-bddc-c87df7753b08\" (UID: \"ca50158d-ce0c-4ebd-bddc-c87df7753b08\") "
Jul 02 21:44:41.614755 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:41.614187    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca50158d-ce0c-4ebd-bddc-c87df7753b08-proxy-tls\") pod \"ca50158d-ce0c-4ebd-bddc-c87df7753b08\" (UID: \"ca50158d-ce0c-4ebd-bddc-c87df7753b08\") "
Jul 02 21:44:41.614755 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:41.614230    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca50158d-ce0c-4ebd-bddc-c87df7753b08-kube-rbac-proxy-sar-config\") pod \"ca50158d-ce0c-4ebd-bddc-c87df7753b08\" (UID: \"ca50158d-ce0c-4ebd-bddc-c87df7753b08\") "
Jul 02 21:44:41.614755 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:41.614250    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca50158d-ce0c-4ebd-bddc-c87df7753b08-cabundle-cert\") pod \"ca50158d-ce0c-4ebd-bddc-c87df7753b08\" (UID: \"ca50158d-ce0c-4ebd-bddc-c87df7753b08\") "
Jul 02 21:44:41.614755 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:41.614693    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca50158d-ce0c-4ebd-bddc-c87df7753b08-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ca50158d-ce0c-4ebd-bddc-c87df7753b08" (UID: "ca50158d-ce0c-4ebd-bddc-c87df7753b08"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:44:41.614755 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:41.614702    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca50158d-ce0c-4ebd-bddc-c87df7753b08-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ca50158d-ce0c-4ebd-bddc-c87df7753b08" (UID: "ca50158d-ce0c-4ebd-bddc-c87df7753b08"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:44:41.615173 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:41.615146    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca50158d-ce0c-4ebd-bddc-c87df7753b08-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ca50158d-ce0c-4ebd-bddc-c87df7753b08" (UID: "ca50158d-ce0c-4ebd-bddc-c87df7753b08"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:44:41.616466 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:41.616441    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca50158d-ce0c-4ebd-bddc-c87df7753b08-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ca50158d-ce0c-4ebd-bddc-c87df7753b08" (UID: "ca50158d-ce0c-4ebd-bddc-c87df7753b08"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:44:41.616466 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:41.616444    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca50158d-ce0c-4ebd-bddc-c87df7753b08-kube-api-access-dbw7h" (OuterVolumeSpecName: "kube-api-access-dbw7h") pod "ca50158d-ce0c-4ebd-bddc-c87df7753b08" (UID: "ca50158d-ce0c-4ebd-bddc-c87df7753b08"). InnerVolumeSpecName "kube-api-access-dbw7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:44:41.715277 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:41.715234    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbw7h\" (UniqueName: \"kubernetes.io/projected/ca50158d-ce0c-4ebd-bddc-c87df7753b08-kube-api-access-dbw7h\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:44:41.715277 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:41.715268    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca50158d-ce0c-4ebd-bddc-c87df7753b08-kserve-provision-location\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:44:41.715277 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:41.715280    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca50158d-ce0c-4ebd-bddc-c87df7753b08-proxy-tls\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:44:41.715277 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:41.715289    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca50158d-ce0c-4ebd-bddc-c87df7753b08-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:44:41.715553 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:41.715298    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca50158d-ce0c-4ebd-bddc-c87df7753b08-cabundle-cert\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\""
Jul 02 21:44:42.068356 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:42.068319    2586 generic.go:356] "Generic (PLEG): container finished" podID="ca50158d-ce0c-4ebd-bddc-c87df7753b08" containerID="a3663b3c0c90460d3f93428bb8d97a154732c29ffa7cd4fe6ab10e9cd6960304" exitCode=0
Jul 02 21:44:42.068507 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:42.068383    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv"
Jul 02 21:44:42.068507 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:42.068389    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv" event={"ID":"ca50158d-ce0c-4ebd-bddc-c87df7753b08","Type":"ContainerDied","Data":"a3663b3c0c90460d3f93428bb8d97a154732c29ffa7cd4fe6ab10e9cd6960304"}
Jul 02 21:44:42.068507 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:42.068454    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv" event={"ID":"ca50158d-ce0c-4ebd-bddc-c87df7753b08","Type":"ContainerDied","Data":"cc1ce46a9cce3fc55ad9d0375896c6e16a584c30a1af38836a86e6380efe730f"}
Jul 02 21:44:42.068507 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:42.068494    2586 scope.go:117] "RemoveContainer" containerID="233fd589f21dd000243aee83969e2cd65ae473623959b4883741c19d7518848a"
Jul 02 21:44:42.076619 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:42.076599    2586 scope.go:117] "RemoveContainer" containerID="a3663b3c0c90460d3f93428bb8d97a154732c29ffa7cd4fe6ab10e9cd6960304"
Jul 02 21:44:42.083715 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:42.083694    2586 scope.go:117] "RemoveContainer" containerID="78ec1cc6830391f3e54d00b95034ec4484e3c9e4f0a8856c5990bba0468f55da"
Jul 02 21:44:42.089413 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:42.089387    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv"]
Jul 02 21:44:42.091853 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:42.091731    2586 scope.go:117] "RemoveContainer" containerID="233fd589f21dd000243aee83969e2cd65ae473623959b4883741c19d7518848a"
Jul 02 21:44:42.092286 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:44:42.092259    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"233fd589f21dd000243aee83969e2cd65ae473623959b4883741c19d7518848a\": container with ID starting with 233fd589f21dd000243aee83969e2cd65ae473623959b4883741c19d7518848a not found: ID does not exist" containerID="233fd589f21dd000243aee83969e2cd65ae473623959b4883741c19d7518848a"
Jul 02 21:44:42.092396 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:42.092296    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"233fd589f21dd000243aee83969e2cd65ae473623959b4883741c19d7518848a"} err="failed to get container status \"233fd589f21dd000243aee83969e2cd65ae473623959b4883741c19d7518848a\": rpc error: code = NotFound desc = could not find container \"233fd589f21dd000243aee83969e2cd65ae473623959b4883741c19d7518848a\": container with ID starting with 233fd589f21dd000243aee83969e2cd65ae473623959b4883741c19d7518848a not found: ID does not exist"
Jul 02 21:44:42.092396 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:42.092314    2586 scope.go:117] "RemoveContainer" containerID="a3663b3c0c90460d3f93428bb8d97a154732c29ffa7cd4fe6ab10e9cd6960304"
Jul 02 21:44:42.092592 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:44:42.092576    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3663b3c0c90460d3f93428bb8d97a154732c29ffa7cd4fe6ab10e9cd6960304\": container with ID starting with a3663b3c0c90460d3f93428bb8d97a154732c29ffa7cd4fe6ab10e9cd6960304 not found: ID does not exist" containerID="a3663b3c0c90460d3f93428bb8d97a154732c29ffa7cd4fe6ab10e9cd6960304"
Jul 02 21:44:42.092634 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:42.092596    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3663b3c0c90460d3f93428bb8d97a154732c29ffa7cd4fe6ab10e9cd6960304"} err="failed to get container status \"a3663b3c0c90460d3f93428bb8d97a154732c29ffa7cd4fe6ab10e9cd6960304\": rpc error: code = NotFound desc = could not find container \"a3663b3c0c90460d3f93428bb8d97a154732c29ffa7cd4fe6ab10e9cd6960304\": container with ID starting with a3663b3c0c90460d3f93428bb8d97a154732c29ffa7cd4fe6ab10e9cd6960304 not found: ID does not exist"
Jul 02 21:44:42.092634 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:42.092609    2586 scope.go:117] "RemoveContainer" containerID="78ec1cc6830391f3e54d00b95034ec4484e3c9e4f0a8856c5990bba0468f55da"
Jul 02 21:44:42.092906 ip-10-0-134-124 kubenswrapper[2586]: E0702 21:44:42.092888    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78ec1cc6830391f3e54d00b95034ec4484e3c9e4f0a8856c5990bba0468f55da\": container with ID starting with 78ec1cc6830391f3e54d00b95034ec4484e3c9e4f0a8856c5990bba0468f55da not found: ID does not exist" containerID="78ec1cc6830391f3e54d00b95034ec4484e3c9e4f0a8856c5990bba0468f55da"
Jul 02 21:44:42.092960 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:42.092910    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78ec1cc6830391f3e54d00b95034ec4484e3c9e4f0a8856c5990bba0468f55da"} err="failed to get container status \"78ec1cc6830391f3e54d00b95034ec4484e3c9e4f0a8856c5990bba0468f55da\": rpc error: code = NotFound desc = could not find container \"78ec1cc6830391f3e54d00b95034ec4484e3c9e4f0a8856c5990bba0468f55da\": container with ID starting with 78ec1cc6830391f3e54d00b95034ec4484e3c9e4f0a8856c5990bba0468f55da not found: ID does not exist"
Jul 02 21:44:42.094219 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:42.094201    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rfhkv"]
Jul 02 21:44:44.022924 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:44:44.022888    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca50158d-ce0c-4ebd-bddc-c87df7753b08" path="/var/lib/kubelet/pods/ca50158d-ce0c-4ebd-bddc-c87df7753b08/volumes"
Jul 02 21:45:13.729035 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:13.728994    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-p2hh8/must-gather-tnbc4"]
Jul 02 21:45:13.729568 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:13.729302    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca50158d-ce0c-4ebd-bddc-c87df7753b08" containerName="storage-initializer"
Jul 02 21:45:13.729568 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:13.729313    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca50158d-ce0c-4ebd-bddc-c87df7753b08" containerName="storage-initializer"
Jul 02 21:45:13.729568 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:13.729324    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca50158d-ce0c-4ebd-bddc-c87df7753b08" containerName="kserve-container"
Jul 02 21:45:13.729568 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:13.729331    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca50158d-ce0c-4ebd-bddc-c87df7753b08" containerName="kserve-container"
Jul 02 21:45:13.729568 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:13.729339    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca50158d-ce0c-4ebd-bddc-c87df7753b08" containerName="kube-rbac-proxy"
Jul 02 21:45:13.729568 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:13.729344    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca50158d-ce0c-4ebd-bddc-c87df7753b08" containerName="kube-rbac-proxy"
Jul 02 21:45:13.729568 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:13.729398    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca50158d-ce0c-4ebd-bddc-c87df7753b08" containerName="kserve-container"
Jul 02 21:45:13.729568 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:13.729410    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca50158d-ce0c-4ebd-bddc-c87df7753b08" containerName="kube-rbac-proxy"
Jul 02 21:45:13.732485 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:13.732467    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p2hh8/must-gather-tnbc4"
Jul 02 21:45:13.735268 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:13.735244    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p2hh8\"/\"openshift-service-ca.crt\""
Jul 02 21:45:13.736740 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:13.736714    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-p2hh8\"/\"default-dockercfg-fjmss\""
Jul 02 21:45:13.736870 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:13.736737    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p2hh8\"/\"kube-root-ca.crt\""
Jul 02 21:45:13.741722 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:13.741701    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p2hh8/must-gather-tnbc4"]
Jul 02 21:45:13.776473 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:13.776450    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6467d156-a866-4b09-becd-85b81756913b-must-gather-output\") pod \"must-gather-tnbc4\" (UID: \"6467d156-a866-4b09-becd-85b81756913b\") " pod="openshift-must-gather-p2hh8/must-gather-tnbc4"
Jul 02 21:45:13.776578 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:13.776491    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr92f\" (UniqueName: \"kubernetes.io/projected/6467d156-a866-4b09-becd-85b81756913b-kube-api-access-dr92f\") pod \"must-gather-tnbc4\" (UID: \"6467d156-a866-4b09-becd-85b81756913b\") " pod="openshift-must-gather-p2hh8/must-gather-tnbc4"
Jul 02 21:45:13.877477 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:13.877446    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6467d156-a866-4b09-becd-85b81756913b-must-gather-output\") pod \"must-gather-tnbc4\" (UID: \"6467d156-a866-4b09-becd-85b81756913b\") " pod="openshift-must-gather-p2hh8/must-gather-tnbc4"
Jul 02 21:45:13.877672 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:13.877492    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr92f\" (UniqueName: \"kubernetes.io/projected/6467d156-a866-4b09-becd-85b81756913b-kube-api-access-dr92f\") pod \"must-gather-tnbc4\" (UID: \"6467d156-a866-4b09-becd-85b81756913b\") " pod="openshift-must-gather-p2hh8/must-gather-tnbc4"
Jul 02 21:45:13.877810 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:13.877789    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6467d156-a866-4b09-becd-85b81756913b-must-gather-output\") pod \"must-gather-tnbc4\" (UID: \"6467d156-a866-4b09-becd-85b81756913b\") " pod="openshift-must-gather-p2hh8/must-gather-tnbc4"
Jul 02 21:45:13.887730 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:13.887704    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr92f\" (UniqueName: \"kubernetes.io/projected/6467d156-a866-4b09-becd-85b81756913b-kube-api-access-dr92f\") pod \"must-gather-tnbc4\" (UID: \"6467d156-a866-4b09-becd-85b81756913b\") " pod="openshift-must-gather-p2hh8/must-gather-tnbc4"
Jul 02 21:45:14.053817 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:14.053718    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p2hh8/must-gather-tnbc4"
Jul 02 21:45:14.184985 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:14.184954    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p2hh8/must-gather-tnbc4"]
Jul 02 21:45:14.187376 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:45:14.187351    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6467d156_a866_4b09_becd_85b81756913b.slice/crio-e72a400f9f49e3d67d2fa2a1a5858dc38b23a5991e7ff8f6fe741fe37ca96110 WatchSource:0}: Error finding container e72a400f9f49e3d67d2fa2a1a5858dc38b23a5991e7ff8f6fe741fe37ca96110: Status 404 returned error can't find the container with id e72a400f9f49e3d67d2fa2a1a5858dc38b23a5991e7ff8f6fe741fe37ca96110
Jul 02 21:45:14.189368 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:14.189348    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 21:45:14.205721 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:14.205684    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p2hh8/must-gather-tnbc4" event={"ID":"6467d156-a866-4b09-becd-85b81756913b","Type":"ContainerStarted","Data":"e72a400f9f49e3d67d2fa2a1a5858dc38b23a5991e7ff8f6fe741fe37ca96110"}
Jul 02 21:45:15.220026 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:15.219974    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p2hh8/must-gather-tnbc4" event={"ID":"6467d156-a866-4b09-becd-85b81756913b","Type":"ContainerStarted","Data":"9982f5a92024d50ae386f90bd2e2cdbfaa7e33b520d4d182120ef3a7db72e11c"}
Jul 02 21:45:15.220485 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:15.220094    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p2hh8/must-gather-tnbc4" event={"ID":"6467d156-a866-4b09-becd-85b81756913b","Type":"ContainerStarted","Data":"1a562cc92c6a89714e33649badf4e521976ed6d37a4feaeffdd231b0f2eba74f"}
Jul 02 21:45:15.239483 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:15.239421    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-p2hh8/must-gather-tnbc4" podStartSLOduration=1.523103718 podStartE2EDuration="2.239400541s" totalImagesPullingTime="716.296823ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:45:15.237528212 +0000 UTC m=+1587.680328635" watchObservedRunningTime="2026-07-02 21:45:15.239400541 +0000 UTC m=+1587.682200963"
Jul 02 21:45:16.492948 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:16.492903    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5gfl7_9c214277-f44b-4f90-be58-8c8078403fc2/global-pull-secret-syncer/0.log"
Jul 02 21:45:16.665373 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:16.665346    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xn9ss_ea0b0920-86bd-418f-8daf-ec2078dc58d5/konnectivity-agent/0.log"
Jul 02 21:45:16.686729 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:16.686689    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-124.ec2.internal_25c1e22383a6ef722c235e04d1247f6b/haproxy/0.log"
Jul 02 21:45:20.552551 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:20.552513    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-dbl59_935457b1-cbad-4c83-99be-3cf040a6cfce/kube-state-metrics/0.log"
Jul 02 21:45:20.579558 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:20.579523    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-dbl59_935457b1-cbad-4c83-99be-3cf040a6cfce/kube-rbac-proxy-main/0.log"
Jul 02 21:45:20.612742 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:20.612679    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-dbl59_935457b1-cbad-4c83-99be-3cf040a6cfce/kube-rbac-proxy-self/0.log"
Jul 02 21:45:20.670513 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:20.670472    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7b77b96467-7s78c_33aa0be6-449b-48aa-8f64-5f09346a08ad/metrics-server/0.log"
Jul 02 21:45:20.700393 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:20.700353    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-4qrvv_a39c9223-ec0e-4982-b586-0e55444a82ae/monitoring-plugin/0.log"
Jul 02 21:45:20.848235 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:20.848142    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hlq5v_35f97721-70f8-4917-a619-e2f290712290/node-exporter/0.log"
Jul 02 21:45:20.878134 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:20.878113    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hlq5v_35f97721-70f8-4917-a619-e2f290712290/kube-rbac-proxy/0.log"
Jul 02 21:45:20.908892 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:20.908860    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hlq5v_35f97721-70f8-4917-a619-e2f290712290/init-textfile/0.log"
Jul 02 21:45:23.088872 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:23.088838    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-cr85g_b321922d-a68a-43a5-89bb-fd49e22371c3/networking-console-plugin/0.log"
Jul 02 21:45:24.157305 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:24.157263    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-8qcvz_9ad59fa2-6b4a-4eb6-8929-9a944438a7e5/download-server/0.log"
Jul 02 21:45:25.641856 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:25.641820    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-p2hh8/perf-node-gather-daemonset-ggphs"]
Jul 02 21:45:25.647516 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:25.647491    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p2hh8/perf-node-gather-daemonset-ggphs"
Jul 02 21:45:25.653892 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:25.653872    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p2hh8/perf-node-gather-daemonset-ggphs"]
Jul 02 21:45:25.701790 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:25.701760    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w5w24_d2c34410-8c0a-4d13-8864-575ba506997d/dns/0.log"
Jul 02 21:45:25.725005 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:25.724976    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w5w24_d2c34410-8c0a-4d13-8864-575ba506997d/kube-rbac-proxy/0.log"
Jul 02 21:45:25.776809 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:25.776781    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h77tf_83d2714f-a9bc-4830-b379-871a817b6f32/dns-node-resolver/0.log"
Jul 02 21:45:25.796757 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:25.796724    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8b155954-b62b-4074-a44e-e23fc98f5218-podres\") pod \"perf-node-gather-daemonset-ggphs\" (UID: \"8b155954-b62b-4074-a44e-e23fc98f5218\") " pod="openshift-must-gather-p2hh8/perf-node-gather-daemonset-ggphs"
Jul 02 21:45:25.796924 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:25.796776    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8b155954-b62b-4074-a44e-e23fc98f5218-proc\") pod \"perf-node-gather-daemonset-ggphs\" (UID: \"8b155954-b62b-4074-a44e-e23fc98f5218\") " pod="openshift-must-gather-p2hh8/perf-node-gather-daemonset-ggphs"
Jul 02 21:45:25.796924 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:25.796835    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b155954-b62b-4074-a44e-e23fc98f5218-sys\") pod \"perf-node-gather-daemonset-ggphs\" (UID: \"8b155954-b62b-4074-a44e-e23fc98f5218\") " pod="openshift-must-gather-p2hh8/perf-node-gather-daemonset-ggphs"
Jul 02 21:45:25.796924 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:25.796876    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hjv6\" (UniqueName: \"kubernetes.io/projected/8b155954-b62b-4074-a44e-e23fc98f5218-kube-api-access-7hjv6\") pod \"perf-node-gather-daemonset-ggphs\" (UID: \"8b155954-b62b-4074-a44e-e23fc98f5218\") " pod="openshift-must-gather-p2hh8/perf-node-gather-daemonset-ggphs"
Jul 02 21:45:25.797036 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:25.796973    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b155954-b62b-4074-a44e-e23fc98f5218-lib-modules\") pod \"perf-node-gather-daemonset-ggphs\" (UID: \"8b155954-b62b-4074-a44e-e23fc98f5218\") " pod="openshift-must-gather-p2hh8/perf-node-gather-daemonset-ggphs"
Jul 02 21:45:25.897926 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:25.897819    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b155954-b62b-4074-a44e-e23fc98f5218-lib-modules\") pod \"perf-node-gather-daemonset-ggphs\" (UID: \"8b155954-b62b-4074-a44e-e23fc98f5218\") " pod="openshift-must-gather-p2hh8/perf-node-gather-daemonset-ggphs"
Jul 02 21:45:25.898192 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:25.898154    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b155954-b62b-4074-a44e-e23fc98f5218-lib-modules\") pod \"perf-node-gather-daemonset-ggphs\" (UID: \"8b155954-b62b-4074-a44e-e23fc98f5218\") " pod="openshift-must-gather-p2hh8/perf-node-gather-daemonset-ggphs"
Jul 02 21:45:25.898336 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:25.898165    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8b155954-b62b-4074-a44e-e23fc98f5218-podres\") pod \"perf-node-gather-daemonset-ggphs\" (UID: \"8b155954-b62b-4074-a44e-e23fc98f5218\") " pod="openshift-must-gather-p2hh8/perf-node-gather-daemonset-ggphs"
Jul 02 21:45:25.898336 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:25.898268    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8b155954-b62b-4074-a44e-e23fc98f5218-proc\") pod \"perf-node-gather-daemonset-ggphs\" (UID: \"8b155954-b62b-4074-a44e-e23fc98f5218\") " pod="openshift-must-gather-p2hh8/perf-node-gather-daemonset-ggphs"
Jul 02 21:45:25.898336 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:25.898284    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8b155954-b62b-4074-a44e-e23fc98f5218-podres\") pod \"perf-node-gather-daemonset-ggphs\" (UID: \"8b155954-b62b-4074-a44e-e23fc98f5218\") " pod="openshift-must-gather-p2hh8/perf-node-gather-daemonset-ggphs"
Jul 02 21:45:25.898336 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:25.898299    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b155954-b62b-4074-a44e-e23fc98f5218-sys\") pod \"perf-node-gather-daemonset-ggphs\" (UID: \"8b155954-b62b-4074-a44e-e23fc98f5218\") " pod="openshift-must-gather-p2hh8/perf-node-gather-daemonset-ggphs"
Jul 02 21:45:25.898336 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:25.898330    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hjv6\" (UniqueName: \"kubernetes.io/projected/8b155954-b62b-4074-a44e-e23fc98f5218-kube-api-access-7hjv6\") pod \"perf-node-gather-daemonset-ggphs\" (UID: \"8b155954-b62b-4074-a44e-e23fc98f5218\") " pod="openshift-must-gather-p2hh8/perf-node-gather-daemonset-ggphs"
Jul 02 21:45:25.898583 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:25.898349    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8b155954-b62b-4074-a44e-e23fc98f5218-proc\") pod \"perf-node-gather-daemonset-ggphs\" (UID: \"8b155954-b62b-4074-a44e-e23fc98f5218\") " pod="openshift-must-gather-p2hh8/perf-node-gather-daemonset-ggphs"
Jul 02 21:45:25.898583 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:25.898375    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b155954-b62b-4074-a44e-e23fc98f5218-sys\") pod \"perf-node-gather-daemonset-ggphs\" (UID: \"8b155954-b62b-4074-a44e-e23fc98f5218\") " pod="openshift-must-gather-p2hh8/perf-node-gather-daemonset-ggphs"
Jul 02 21:45:25.907403 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:25.907365    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hjv6\" (UniqueName: \"kubernetes.io/projected/8b155954-b62b-4074-a44e-e23fc98f5218-kube-api-access-7hjv6\") pod \"perf-node-gather-daemonset-ggphs\" (UID: \"8b155954-b62b-4074-a44e-e23fc98f5218\") " pod="openshift-must-gather-p2hh8/perf-node-gather-daemonset-ggphs"
Jul 02 21:45:25.960287 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:25.960250    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p2hh8/perf-node-gather-daemonset-ggphs"
Jul 02 21:45:26.122158 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:26.120279    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p2hh8/perf-node-gather-daemonset-ggphs"]
Jul 02 21:45:26.126512 ip-10-0-134-124 kubenswrapper[2586]: W0702 21:45:26.126473    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8b155954_b62b_4074_a44e_e23fc98f5218.slice/crio-117e67996b9daf67eb09ddace569fb655c4dff062c82c18225397973dd95b6e5 WatchSource:0}: Error finding container 117e67996b9daf67eb09ddace569fb655c4dff062c82c18225397973dd95b6e5: Status 404 returned error can't find the container with id 117e67996b9daf67eb09ddace569fb655c4dff062c82c18225397973dd95b6e5
Jul 02 21:45:26.286014 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:26.285977    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p2hh8/perf-node-gather-daemonset-ggphs" event={"ID":"8b155954-b62b-4074-a44e-e23fc98f5218","Type":"ContainerStarted","Data":"669d9fe5e1606e440e1ad9f7243a473f8c5ba3bbb596ca6603d74984d130cffa"}
Jul 02 21:45:26.286151 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:26.286051    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-p2hh8/perf-node-gather-daemonset-ggphs"
Jul 02 21:45:26.286151 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:26.286071    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p2hh8/perf-node-gather-daemonset-ggphs" event={"ID":"8b155954-b62b-4074-a44e-e23fc98f5218","Type":"ContainerStarted","Data":"117e67996b9daf67eb09ddace569fb655c4dff062c82c18225397973dd95b6e5"}
Jul 02 21:45:26.318124 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:26.318068    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-p2hh8/perf-node-gather-daemonset-ggphs" podStartSLOduration=1.318049332 podStartE2EDuration="1.318049332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:45:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:45:26.317364335 +0000 UTC m=+1598.760164758" watchObservedRunningTime="2026-07-02 21:45:26.318049332 +0000 UTC m=+1598.760849754"
Jul 02 21:45:26.330738 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:26.330706    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5cc4899c7b-qpxzm_2ee94142-64e7-46b3-a91a-e362146cc451/registry/0.log"
Jul 02 21:45:26.397542 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:26.397513    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6fpnv_75f700f7-60d3-4bf0-aef0-2052610d6cba/node-ca/0.log"
Jul 02 21:45:27.710356 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:27.710319    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mvrrr_2fc74b27-fbce-4841-b23e-100f5fffdb1b/serve-healthcheck-canary/0.log"
Jul 02 21:45:28.232574 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:28.232540    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dldh9_d9103d7e-3bea-4efc-ab49-8790e23e383a/kube-rbac-proxy/0.log"
Jul 02 21:45:28.254112 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:28.254083    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dldh9_d9103d7e-3bea-4efc-ab49-8790e23e383a/exporter/0.log"
Jul 02 21:45:28.275611 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:28.275592    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dldh9_d9103d7e-3bea-4efc-ab49-8790e23e383a/extractor/0.log"
Jul 02 21:45:30.434630 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:30.434597    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-56668bddcf-9nl57_df26b232-65c5-427e-af02-bebdbc530620/manager/0.log"
Jul 02 21:45:30.455414 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:30.455388    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-67d95fdc48-5h4d2_35cb1ddb-bfb7-46bb-89eb-7472e0c33b7e/manager/0.log"
Jul 02 21:45:30.550789 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:30.550755    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-rmwb4_aaef0172-0d63-40b6-8293-de7f631bbdd5/manager/0.log"
Jul 02 21:45:30.584397 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:30.584374    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-v8dmh_491466ac-ca40-4477-9967-f1fa459c3fa1/s3-init/0.log"
Jul 02 21:45:30.601549 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:30.601522    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-v8dmh_491466ac-ca40-4477-9967-f1fa459c3fa1/download-hf-model/0.log"
Jul 02 21:45:30.631884 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:30.631847    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-qc5tr_6bf4e53c-efa8-4886-8404-83229c8a37e7/s3-tls-init-custom/0.log"
Jul 02 21:45:30.649575 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:30.649549    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-qc5tr_6bf4e53c-efa8-4886-8404-83229c8a37e7/download-hf-model/0.log"
Jul 02 21:45:30.677397 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:30.677375    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-cd4vg_1903668d-3ca5-4c5f-9243-92adb5d752f6/s3-tls-init-serving/0.log"
Jul 02 21:45:30.723156 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:30.723071    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-cd4vg_1903668d-3ca5-4c5f-9243-92adb5d752f6/download-hf-model/0.log"
Jul 02 21:45:32.304046 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:32.303991    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-p2hh8/perf-node-gather-daemonset-ggphs"
Jul 02 21:45:34.892874 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:34.892848    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-pxlmv_01ce737f-793f-48cd-be14-aa4431880a61/migrator/0.log"
Jul 02 21:45:34.914847 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:34.914823    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-pxlmv_01ce737f-793f-48cd-be14-aa4431880a61/graceful-termination/0.log"
Jul 02 21:45:35.226859 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:35.226819    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-c7tm5_1b77428f-b316-4a33-9397-b3227211e3e3/kube-storage-version-migrator-operator/1.log"
Jul 02 21:45:35.228193 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:35.228169    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-c7tm5_1b77428f-b316-4a33-9397-b3227211e3e3/kube-storage-version-migrator-operator/0.log"
Jul 02 21:45:36.195699 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:36.195671    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8tkxq_ce3cb284-71b0-4202-b141-7ad511895978/kube-multus-additional-cni-plugins/0.log"
Jul 02 21:45:36.218978 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:36.218952    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8tkxq_ce3cb284-71b0-4202-b141-7ad511895978/egress-router-binary-copy/0.log"
Jul 02 21:45:36.240346 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:36.240315    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8tkxq_ce3cb284-71b0-4202-b141-7ad511895978/cni-plugins/0.log"
Jul 02 21:45:36.261322 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:36.261297    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8tkxq_ce3cb284-71b0-4202-b141-7ad511895978/bond-cni-plugin/0.log"
Jul 02 21:45:36.282476 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:36.282453    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8tkxq_ce3cb284-71b0-4202-b141-7ad511895978/routeoverride-cni/0.log"
Jul 02 21:45:36.304211 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:36.304176    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8tkxq_ce3cb284-71b0-4202-b141-7ad511895978/whereabouts-cni-bincopy/0.log"
Jul 02 21:45:36.326504 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:36.326478    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8tkxq_ce3cb284-71b0-4202-b141-7ad511895978/whereabouts-cni/0.log"
Jul 02 21:45:36.700311 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:36.700279    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-c56wq_e26f8e6a-05cd-4164-bc55-6f44d1c8bf8a/kube-multus/0.log"
Jul 02 21:45:36.825660 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:36.825617    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-s5xr4_006d61c8-eb34-4336-a3d3-82a7a4fc2069/network-metrics-daemon/0.log"
Jul 02 21:45:36.846698 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:36.846676    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-s5xr4_006d61c8-eb34-4336-a3d3-82a7a4fc2069/kube-rbac-proxy/0.log"
Jul 02 21:45:38.107936 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:38.107908    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mvqc5_da774dac-c1e6-4f51-b69e-95a97f5bf0f1/ovn-controller/0.log"
Jul 02 21:45:38.126744 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:38.126717    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mvqc5_da774dac-c1e6-4f51-b69e-95a97f5bf0f1/ovn-acl-logging/0.log"
Jul 02 21:45:38.156166 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:38.156140    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mvqc5_da774dac-c1e6-4f51-b69e-95a97f5bf0f1/kube-rbac-proxy-node/0.log"
Jul 02 21:45:38.178355 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:38.178332    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mvqc5_da774dac-c1e6-4f51-b69e-95a97f5bf0f1/kube-rbac-proxy-ovn-metrics/0.log"
Jul 02 21:45:38.198424 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:38.198396    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mvqc5_da774dac-c1e6-4f51-b69e-95a97f5bf0f1/northd/0.log"
Jul 02 21:45:38.221263 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:38.221239    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mvqc5_da774dac-c1e6-4f51-b69e-95a97f5bf0f1/nbdb/0.log"
Jul 02 21:45:38.245521 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:38.245491    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mvqc5_da774dac-c1e6-4f51-b69e-95a97f5bf0f1/sbdb/0.log"
Jul 02 21:45:38.293594 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:38.293565    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mvqc5_da774dac-c1e6-4f51-b69e-95a97f5bf0f1/ovnkube-controller/0.log"
Jul 02 21:45:39.270271 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:39.270243    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-d495m_45cb060e-3948-4b0e-b407-86ea9b6af872/network-check-target-container/0.log"
Jul 02 21:45:40.285309 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:40.285285    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-gmfs8_732f8ea2-dc3e-426a-96a2-d8a7ba62d935/iptables-alerter/0.log"
Jul 02 21:45:40.916710 ip-10-0-134-124 kubenswrapper[2586]: I0702 21:45:40.916678    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-8j5v7_1c79d16b-84a9-4c32-af87-3e104b416604/tuned/0.log"