Jul 29 10:26:54.572617 ip-10-0-137-111 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 29 10:26:54.572630 ip-10-0-137-111 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 29 10:26:54.572641 ip-10-0-137-111 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 29 10:26:54.573114 ip-10-0-137-111 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 29 10:27:03.878267 ip-10-0-137-111 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Jul 29 10:27:03.878286 ip-10-0-137-111 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 585554afa639480386c9a90b22272738 --
Jul 29 10:28:43.147354 ip-10-0-137-111 systemd[1]: Starting Kubernetes Kubelet...
Jul 29 10:28:43.597990 ip-10-0-137-111 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 10:28:43.597990 ip-10-0-137-111 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 29 10:28:43.597990 ip-10-0-137-111 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 10:28:43.597990 ip-10-0-137-111 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 29 10:28:43.597990 ip-10-0-137-111 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 10:28:43.599084 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.598980    2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 29 10:28:43.603904 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.603889    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 10:28:43.603904 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.603904    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 10:28:43.603966 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.603909    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 10:28:43.603966 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.603912    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 10:28:43.603966 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.603917    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 10:28:43.603966 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.603920    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 10:28:43.603966 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.603923    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 10:28:43.603966 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.603926    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 10:28:43.603966 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.603929    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 10:28:43.603966 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.603932    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 10:28:43.603966 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.603935    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 10:28:43.603966 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.603938    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 10:28:43.603966 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.603940    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 10:28:43.603966 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.603946    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 10:28:43.603966 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.603949    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 10:28:43.603966 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.603952    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 10:28:43.603966 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.603955    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 10:28:43.603966 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.603958    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 10:28:43.603966 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.603960    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 10:28:43.603966 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.603963    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 10:28:43.603966 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.603965    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 10:28:43.603966 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.603968    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 10:28:43.604478 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.603971    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 10:28:43.604478 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.603974    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 10:28:43.604478 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.603977    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 10:28:43.604478 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.603980    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 10:28:43.604478 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.603983    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 10:28:43.604478 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.603986    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 10:28:43.604478 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.603988    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 10:28:43.604478 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.603991    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 10:28:43.604478 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.603994    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 10:28:43.604478 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.603996    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 10:28:43.604478 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.603999    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 10:28:43.604478 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604001    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 10:28:43.604478 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604004    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 10:28:43.604478 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604006    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 10:28:43.604478 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604009    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 10:28:43.604478 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604011    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 10:28:43.604478 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604014    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 10:28:43.604478 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604016    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 10:28:43.604478 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604019    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 10:28:43.604478 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604021    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 10:28:43.604978 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604024    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 10:28:43.604978 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604027    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 10:28:43.604978 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604029    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 10:28:43.604978 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604032    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 10:28:43.604978 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604035    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 10:28:43.604978 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604037    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 10:28:43.604978 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604040    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 10:28:43.604978 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604042    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 10:28:43.604978 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604044    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 10:28:43.604978 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604047    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 10:28:43.604978 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604049    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 10:28:43.604978 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604052    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 10:28:43.604978 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604055    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 10:28:43.604978 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604058    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 10:28:43.604978 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604061    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 10:28:43.604978 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604063    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 10:28:43.604978 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604066    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 10:28:43.604978 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604081    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 10:28:43.604978 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604084    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 10:28:43.604978 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604086    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 10:28:43.605490 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604089    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 10:28:43.605490 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604092    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 10:28:43.605490 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604095    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 10:28:43.605490 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604097    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 10:28:43.605490 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604100    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 10:28:43.605490 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604103    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 10:28:43.605490 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604105    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 10:28:43.605490 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604108    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 10:28:43.605490 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604112    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 10:28:43.605490 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604114    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 10:28:43.605490 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604116    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 10:28:43.605490 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604119    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 10:28:43.605490 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604122    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 10:28:43.605490 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604126    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 10:28:43.605490 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604130    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 10:28:43.605490 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604133    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 10:28:43.605490 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604139    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 10:28:43.605490 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604142    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 10:28:43.605490 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604144    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 10:28:43.605963 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604147    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 10:28:43.605963 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604149    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 10:28:43.605963 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604153    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 10:28:43.605963 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604157    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 10:28:43.605963 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604160    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 10:28:43.605963 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604163    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 10:28:43.605963 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604166    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 10:28:43.605963 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604169    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 10:28:43.605963 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604172    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 10:28:43.605963 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604175    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 10:28:43.605963 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604178    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 10:28:43.605963 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604180    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 10:28:43.605963 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604185    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 10:28:43.605963 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604188    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 10:28:43.605963 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604191    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 10:28:43.605963 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604193    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 10:28:43.605963 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604196    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 10:28:43.605963 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604199    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 10:28:43.605963 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604917    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 10:28:43.606425 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604923    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 10:28:43.606425 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604926    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 10:28:43.606425 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604929    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 10:28:43.606425 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604931    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 10:28:43.606425 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604934    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 10:28:43.606425 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604937    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 10:28:43.606425 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604940    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 10:28:43.606425 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604942    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 10:28:43.606425 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604945    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 10:28:43.606425 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604947    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 10:28:43.606425 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604951    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 10:28:43.606425 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604956    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 10:28:43.606425 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604959    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 10:28:43.606425 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604962    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 10:28:43.606425 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604964    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 10:28:43.606425 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604968    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 10:28:43.606425 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604972    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 10:28:43.606425 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604975    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 10:28:43.606425 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604979    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 10:28:43.606892 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604983    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 10:28:43.606892 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604986    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 10:28:43.606892 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604989    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 10:28:43.606892 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604991    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 10:28:43.606892 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604994    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 10:28:43.606892 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.604997    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 10:28:43.606892 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605000    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 10:28:43.606892 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605002    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 10:28:43.606892 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605005    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 10:28:43.606892 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605008    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 10:28:43.606892 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605010    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 10:28:43.606892 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605013    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 10:28:43.606892 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605015    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 10:28:43.606892 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605018    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 10:28:43.606892 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605020    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 10:28:43.606892 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605024    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 10:28:43.606892 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605027    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 10:28:43.606892 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605030    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 10:28:43.606892 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605033    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 10:28:43.607370 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605035    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 10:28:43.607370 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605038    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 10:28:43.607370 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605040    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 10:28:43.607370 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605043    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 10:28:43.607370 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605045    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 10:28:43.607370 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605047    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 10:28:43.607370 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605050    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 10:28:43.607370 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605053    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 10:28:43.607370 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605055    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 10:28:43.607370 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605058    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 10:28:43.607370 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605060    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 10:28:43.607370 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605063    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 10:28:43.607370 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605067    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 10:28:43.607370 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605084    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 10:28:43.607370 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605087    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 10:28:43.607370 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605090    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 10:28:43.607370 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605093    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 10:28:43.607370 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605095    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 10:28:43.607370 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605098    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 10:28:43.607838 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605101    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 10:28:43.607838 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605103    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 10:28:43.607838 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605107    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 10:28:43.607838 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605109    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 10:28:43.607838 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605112    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 10:28:43.607838 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605114    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 10:28:43.607838 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605117    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 10:28:43.607838 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605120    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 10:28:43.607838 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605122    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 10:28:43.607838 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605126    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 10:28:43.607838 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605129    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 10:28:43.607838 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605132    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 10:28:43.607838 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605134    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 10:28:43.607838 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605137    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 10:28:43.607838 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605139    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 10:28:43.607838 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605142    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 10:28:43.607838 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605145    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 10:28:43.607838 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605147    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 10:28:43.607838 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605150    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 10:28:43.607838 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605152    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 10:28:43.608455 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605155    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 10:28:43.608455 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605158    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 10:28:43.608455 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605160    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 10:28:43.608455 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605163    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 10:28:43.608455 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605166    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 10:28:43.608455 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605169    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 10:28:43.608455 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605172    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 10:28:43.608455 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605174    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 10:28:43.608455 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605177    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 10:28:43.608455 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605179    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 10:28:43.608455 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605182    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 10:28:43.608455 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605184    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 10:28:43.608455 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605187    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 10:28:43.608455 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605189    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 10:28:43.608455 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605192    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 10:28:43.608455 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605194    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 10:28:43.608455 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605197    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 10:28:43.608455 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605199    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 10:28:43.608455 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605202    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 10:28:43.608455 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605204    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 10:28:43.608953 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605207    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 10:28:43.608953 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605569    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 10:28:43.608953 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605574    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 10:28:43.608953 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605577    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 10:28:43.608953 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605580    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 10:28:43.608953 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605582    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 10:28:43.608953 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605585    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 10:28:43.608953 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605588    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 10:28:43.608953 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605591    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 10:28:43.608953 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605593    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 10:28:43.608953 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605596    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 10:28:43.608953 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605598    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 10:28:43.608953 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605602    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 10:28:43.608953 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605604    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 10:28:43.608953 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605607    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 10:28:43.608953 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605609    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 10:28:43.608953 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605612    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 10:28:43.608953 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605615    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 10:28:43.608953 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605620    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 10:28:43.609476 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605623    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 10:28:43.609476 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605627    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 10:28:43.609476 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605629    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 10:28:43.609476 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605633    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 10:28:43.609476 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605636    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 10:28:43.609476 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605639    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 10:28:43.609476 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605643    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 10:28:43.609476 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605646    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 10:28:43.609476 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605649    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 10:28:43.609476 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605651    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 10:28:43.609476 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605654    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 10:28:43.609476 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605656    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 10:28:43.609476 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605659    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 10:28:43.609476 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605661    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 10:28:43.609476 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605663    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 10:28:43.609476 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605666    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 10:28:43.609476 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605668    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 10:28:43.609476 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605671    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 10:28:43.609476 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605673    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 10:28:43.609942 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605676    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 10:28:43.609942 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605679    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 10:28:43.609942 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605681    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 10:28:43.609942 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605684    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 10:28:43.609942 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605686    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 10:28:43.609942 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605689    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 10:28:43.609942 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605691    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 10:28:43.609942 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605694    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 10:28:43.609942 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605696    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 10:28:43.609942 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605699    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 10:28:43.609942 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605702    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 10:28:43.609942 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605705    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 10:28:43.609942 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605708    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 10:28:43.609942 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605710    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 10:28:43.609942 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605713    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 10:28:43.609942 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605715    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 10:28:43.609942 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605718    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 10:28:43.609942 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605720    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 10:28:43.609942 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605723    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 10:28:43.609942 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605726    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 10:28:43.610434 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605728    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 10:28:43.610434 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605731    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 10:28:43.610434 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605734    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 10:28:43.610434 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605736    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 10:28:43.610434 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605738    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 10:28:43.610434 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605741    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 10:28:43.610434 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605745    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 10:28:43.610434 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605748    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 10:28:43.610434 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605751    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 10:28:43.610434 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605753    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 10:28:43.610434 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605756    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 10:28:43.610434 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605759    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 10:28:43.610434 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605762    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 10:28:43.610434 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605764    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 10:28:43.610434 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605767    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 10:28:43.610434 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605769    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 10:28:43.610434 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605772    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 10:28:43.610434 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605774    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 10:28:43.610434 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605777    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 10:28:43.610933 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605779    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 10:28:43.610933 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605782    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 10:28:43.610933 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605784    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 10:28:43.610933 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605787    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 10:28:43.610933 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605790    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 10:28:43.610933 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605793    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 10:28:43.610933 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605796    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 10:28:43.610933 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605799    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 10:28:43.610933 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605802    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 10:28:43.610933 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605804    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 10:28:43.610933 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605807    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 10:28:43.610933 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605809    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 10:28:43.610933 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605812    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 10:28:43.610933 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605814    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 10:28:43.610933 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605817    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 10:28:43.610933 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605819    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 10:28:43.610933 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605822    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 10:28:43.610933 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605824    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 10:28:43.610933 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605827    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 10:28:43.610933 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605829    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 10:28:43.611438 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605832    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 10:28:43.611438 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605837    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 10:28:43.611438 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.605840    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 10:28:43.611438 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.605909    2569 flags.go:64] FLAG: --address="0.0.0.0"
Jul 29 10:28:43.611438 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.605916    2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 29 10:28:43.611438 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.605923    2569 flags.go:64] FLAG: --anonymous-auth="true"
Jul 29 10:28:43.611438 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.605929    2569 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 29 10:28:43.611438 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.605935    2569 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 29 10:28:43.611438 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.605941    2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 29 10:28:43.611438 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.605948    2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 29 10:28:43.611438 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.605952    2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 29 10:28:43.611438 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.605956    2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 29 10:28:43.611438 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.605959    2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 29 10:28:43.611438 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.605962    2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 29 10:28:43.611438 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.605967    2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 29 10:28:43.611438 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.605971    2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 29 10:28:43.611438 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.605974    2569 flags.go:64] FLAG: --cgroup-root=""
Jul 29 10:28:43.611438 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.605977    2569 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 29 10:28:43.611438 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.605980    2569 flags.go:64] FLAG: --client-ca-file=""
Jul 29 10:28:43.611438 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.605982    2569 flags.go:64] FLAG: --cloud-provider="external"
Jul 29 10:28:43.611438 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.605985    2569 flags.go:64] FLAG: --cluster-dns="[]"
Jul 29 10:28:43.611438 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.605990    2569 flags.go:64] FLAG: --cluster-domain=""
Jul 29 10:28:43.611438 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.605993    2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 29 10:28:43.611438 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.605996    2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 29 10:28:43.612016 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.605999    2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 29 10:28:43.612016 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606002    2569 flags.go:64] FLAG: --container-log-max-files="5"
Jul 29 10:28:43.612016 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606006    2569 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 29 10:28:43.612016 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606008    2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 29 10:28:43.612016 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606011    2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 29 10:28:43.612016 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606014    2569 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 29 10:28:43.612016 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606017    2569 flags.go:64] FLAG: --contention-profiling="false"
Jul 29 10:28:43.612016 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606020    2569 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 29 10:28:43.612016 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606023    2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 29 10:28:43.612016 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606027    2569 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 29 10:28:43.612016 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606030    2569 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 29 10:28:43.612016 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606034    2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 29 10:28:43.612016 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606037    2569 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 29 10:28:43.612016 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606040    2569 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 29 10:28:43.612016 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606042    2569 flags.go:64] FLAG: --enable-load-reader="false"
Jul 29 10:28:43.612016 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606045    2569 flags.go:64] FLAG: --enable-server="true"
Jul 29 10:28:43.612016 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606048    2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 29 10:28:43.612016 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606052    2569 flags.go:64] FLAG: --event-burst="100"
Jul 29 10:28:43.612016 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606055    2569 flags.go:64] FLAG: --event-qps="50"
Jul 29 10:28:43.612016 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606058    2569 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 29 10:28:43.612016 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606061    2569 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 29 10:28:43.612016 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606065    2569 flags.go:64] FLAG: --eviction-hard=""
Jul 29 10:28:43.612016 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606080    2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 29 10:28:43.612016 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606084    2569 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 29 10:28:43.612635 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606088    2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 29 10:28:43.612635 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606091    2569 flags.go:64] FLAG: --eviction-soft=""
Jul 29 10:28:43.612635 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606094    2569 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 29 10:28:43.612635 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606097    2569 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 29 10:28:43.612635 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606099    2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 29 10:28:43.612635 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606102    2569 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 29 10:28:43.612635 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606105    2569 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 29 10:28:43.612635 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606108    2569 flags.go:64] FLAG: --fail-swap-on="true"
Jul 29 10:28:43.612635 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606111    2569 flags.go:64] FLAG: --feature-gates=""
Jul 29 10:28:43.612635 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606115    2569 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 29 10:28:43.612635 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606118    2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 29 10:28:43.612635 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606121    2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 29 10:28:43.612635 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606124    2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 29 10:28:43.612635 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606127    2569 flags.go:64] FLAG: --healthz-port="10248"
Jul 29 10:28:43.612635 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606130    2569 flags.go:64] FLAG: --help="false"
Jul 29 10:28:43.612635 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606132    2569 flags.go:64] FLAG: --hostname-override="ip-10-0-137-111.ec2.internal"
Jul 29 10:28:43.612635 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606135    2569 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 29 10:28:43.612635 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606140    2569 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 29 10:28:43.612635 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606142    2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 29 10:28:43.612635 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606146    2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 29 10:28:43.612635 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606149    2569 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 29 10:28:43.612635 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606152    2569 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 29 10:28:43.612635 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606155    2569 flags.go:64] FLAG: --image-service-endpoint=""
Jul 29 10:28:43.613310 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606157    2569 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 29 10:28:43.613310 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606160    2569 flags.go:64] FLAG: --kube-api-burst="100"
Jul 29 10:28:43.613310 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606163    2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 29 10:28:43.613310 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606166    2569 flags.go:64] FLAG: --kube-api-qps="50"
Jul 29 10:28:43.613310 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606169    2569 flags.go:64] FLAG: --kube-reserved=""
Jul 29 10:28:43.613310 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606171    2569 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 29 10:28:43.613310 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606176    2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 29 10:28:43.613310 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606179    2569 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 29 10:28:43.613310 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606182    2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 29 10:28:43.613310 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606185    2569 flags.go:64] FLAG: --lock-file=""
Jul 29 10:28:43.613310 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606188    2569 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 29 10:28:43.613310 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606191    2569 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 29 10:28:43.613310 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606194    2569 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 29 10:28:43.613310 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606206    2569 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 29 10:28:43.613310 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606209    2569 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 29 10:28:43.613310 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606212    2569 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 29 10:28:43.613310 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606215    2569 flags.go:64] FLAG: --logging-format="text"
Jul 29 10:28:43.613310 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606218    2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 29 10:28:43.613310 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606221    2569 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 29 10:28:43.613310 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606224    2569 flags.go:64] FLAG: --manifest-url=""
Jul 29 10:28:43.613310 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606227    2569 flags.go:64] FLAG: --manifest-url-header=""
Jul 29 10:28:43.613310 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606231    2569 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 29 10:28:43.613310 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606234    2569 flags.go:64] FLAG: --max-open-files="1000000"
Jul 29 10:28:43.613310 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606238    2569 flags.go:64] FLAG: --max-pods="110"
Jul 29 10:28:43.613310 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606241    2569 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 29 10:28:43.614237 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606244    2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 29 10:28:43.614237 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606248    2569 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 29 10:28:43.614237 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606251    2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 29 10:28:43.614237 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606254    2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 29 10:28:43.614237 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606257    2569 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 29 10:28:43.614237 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606260    2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 29 10:28:43.614237 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606267    2569 flags.go:64] FLAG: --node-status-max-images="50"
Jul 29 10:28:43.614237 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606270    2569 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 29 10:28:43.614237 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606273    2569 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 29 10:28:43.614237 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606276    2569 flags.go:64] FLAG: --pod-cidr=""
Jul 29 10:28:43.614237 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606280    2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 29 10:28:43.614237 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606285    2569 flags.go:64] FLAG: --pod-manifest-path=""
Jul 29 10:28:43.614237 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606288    2569 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 29 10:28:43.614237 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606291    2569 flags.go:64] FLAG: --pods-per-core="0"
Jul 29 10:28:43.614237 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606295    2569 flags.go:64] FLAG: --port="10250"
Jul 29 10:28:43.614237 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606298    2569 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 29 10:28:43.614237 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606300    2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0885019687de6d17e"
Jul 29 10:28:43.614237 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606305    2569 flags.go:64] FLAG: --qos-reserved=""
Jul 29 10:28:43.614237 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606308    2569 flags.go:64] FLAG: --read-only-port="10255"
Jul 29 10:28:43.614237 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606311    2569 flags.go:64] FLAG: --register-node="true"
Jul 29 10:28:43.614237 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606313    2569 flags.go:64] FLAG: --register-with-taints=""
Jul 29 10:28:43.614237 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606317    2569 flags.go:64] FLAG: --registry-burst="10"
Jul 29 10:28:43.614237 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606320    2569 flags.go:64] FLAG: --registry-qps="5"
Jul 29 10:28:43.614237 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606322    2569 flags.go:64] FLAG: --reserved-cpus=""
Jul 29 10:28:43.614874 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606325    2569 flags.go:64] FLAG: --reserved-memory=""
Jul 29 10:28:43.614874 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606329    2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 29 10:28:43.614874 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606332    2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 29 10:28:43.614874 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606335    2569 flags.go:64] FLAG: --rotate-certificates="false"
Jul 29 10:28:43.614874 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606338    2569 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 29 10:28:43.614874 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606341    2569 flags.go:64] FLAG: --runonce="false"
Jul 29 10:28:43.614874 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606344    2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 29 10:28:43.614874 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606347    2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 29 10:28:43.614874 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606350    2569 flags.go:64] FLAG: --seccomp-default="false"
Jul 29 10:28:43.614874 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606354    2569 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 29 10:28:43.614874 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606357    2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 29 10:28:43.614874 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606360    2569 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 29 10:28:43.614874 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606364    2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 29 10:28:43.614874 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606367    2569 flags.go:64] FLAG: --storage-driver-password="root"
Jul 29 10:28:43.614874 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606370    2569 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 29 10:28:43.614874 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606373    2569 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 29 10:28:43.614874 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606376    2569 flags.go:64] FLAG: --storage-driver-user="root"
Jul 29 10:28:43.614874 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606378    2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 29 10:28:43.614874 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606381    2569 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 29 10:28:43.614874 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606384    2569 flags.go:64] FLAG: --system-cgroups=""
Jul 29 10:28:43.614874 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606387    2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 29 10:28:43.614874 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606392    2569 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 29 10:28:43.614874 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606396    2569 flags.go:64] FLAG: --tls-cert-file=""
Jul 29 10:28:43.614874 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606399    2569 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 29 10:28:43.614874 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606403    2569 flags.go:64] FLAG: --tls-min-version=""
Jul 29 10:28:43.615504 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606406    2569 flags.go:64] FLAG: --tls-private-key-file=""
Jul 29 10:28:43.615504 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606409    2569 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 29 10:28:43.615504 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606412    2569 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 29 10:28:43.615504 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606414    2569 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 29 10:28:43.615504 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606417    2569 flags.go:64] FLAG: --v="2"
Jul 29 10:28:43.615504 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606421    2569 flags.go:64] FLAG: --version="false"
Jul 29 10:28:43.615504 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606425    2569 flags.go:64] FLAG: --vmodule=""
Jul 29 10:28:43.615504 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606429    2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 29 10:28:43.615504 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.606432    2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 29 10:28:43.615504 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606564    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 10:28:43.615504 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606568    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 10:28:43.615504 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606571    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 10:28:43.615504 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606574    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 10:28:43.615504 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606577    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 10:28:43.615504 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606580    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 10:28:43.615504 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606584    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 10:28:43.615504 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606589    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 10:28:43.615504 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606592    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 10:28:43.615504 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606595    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 10:28:43.615504 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606599    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 10:28:43.615504 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606603    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 10:28:43.616024 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606606    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 10:28:43.616024 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606609    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 10:28:43.616024 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606612    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 10:28:43.616024 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606615    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 10:28:43.616024 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606617    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 10:28:43.616024 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606620    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 10:28:43.616024 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606623    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 10:28:43.616024 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606626    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 10:28:43.616024 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606630    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 10:28:43.616024 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606632    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 10:28:43.616024 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606635    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 10:28:43.616024 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606638    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 10:28:43.616024 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606640    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 10:28:43.616024 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606643    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 10:28:43.616024 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606646    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 10:28:43.616024 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606648    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 10:28:43.616024 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606651    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 10:28:43.616024 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606654    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 10:28:43.616024 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606657    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 10:28:43.616024 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606659    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 10:28:43.616573 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606662    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 10:28:43.616573 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606664    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 10:28:43.616573 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606667    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 10:28:43.616573 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606669    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 10:28:43.616573 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606671    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 10:28:43.616573 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606674    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 10:28:43.616573 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606676    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 10:28:43.616573 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606680    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 10:28:43.616573 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606683    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 10:28:43.616573 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606686    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 10:28:43.616573 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606688    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 10:28:43.616573 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606691    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 10:28:43.616573 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606694    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 10:28:43.616573 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606696    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 10:28:43.616573 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606698    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 10:28:43.616573 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606701    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 10:28:43.616573 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606704    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 10:28:43.616573 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606706    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 10:28:43.616573 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606709    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 10:28:43.617037 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606711    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 10:28:43.617037 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606715    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 10:28:43.617037 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606718    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 10:28:43.617037 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606720    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 10:28:43.617037 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606723    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 10:28:43.617037 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606725    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 10:28:43.617037 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606728    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 10:28:43.617037 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606738    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 10:28:43.617037 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606741    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 10:28:43.617037 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606744    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 10:28:43.617037 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606746    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 10:28:43.617037 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606748    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 10:28:43.617037 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606751    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 10:28:43.617037 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606754    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 10:28:43.617037 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606756    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 10:28:43.617037 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606759    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 10:28:43.617037 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606761    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 10:28:43.617037 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606764    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 10:28:43.617037 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606766    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 10:28:43.617037 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606769    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 10:28:43.617541 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606773    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 10:28:43.617541 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606775    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 10:28:43.617541 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606778    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 10:28:43.617541 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606780    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 10:28:43.617541 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606783    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 10:28:43.617541 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606785    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 10:28:43.617541 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606788    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 10:28:43.617541 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606790    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 10:28:43.617541 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606793    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 10:28:43.617541 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606796    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 10:28:43.617541 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606798    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 10:28:43.617541 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606801    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 10:28:43.617541 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606803    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 10:28:43.617541 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606808    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 10:28:43.617541 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606810    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 10:28:43.617541 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606813    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 10:28:43.617541 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606816    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 10:28:43.617541 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606818    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 10:28:43.617541 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606820    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 10:28:43.617541 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606823    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 10:28:43.618040 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606827    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 10:28:43.618040 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606830    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 10:28:43.618040 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606833    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 10:28:43.618040 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606836    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 10:28:43.618040 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606839    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 10:28:43.618040 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606842    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 10:28:43.618040 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606844    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 10:28:43.618040 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.606847    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 10:28:43.618040 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.607379    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 10:28:43.618040 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.615406    2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 29 10:28:43.618040 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.615421    2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 29 10:28:43.618040 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.615434    2569 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 29 10:28:43.618040 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.615440    2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 29 10:28:43.618040 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615497    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 10:28:43.618399 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615502    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 10:28:43.618399 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615506    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 10:28:43.618399 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615510    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 10:28:43.618399 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615513    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 10:28:43.618399 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615516    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 10:28:43.618399 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615519    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 10:28:43.618399 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615521    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 10:28:43.618399 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615524    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 10:28:43.618399 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615527    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 10:28:43.618399 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615530    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 10:28:43.618399 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615533    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 10:28:43.618399 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615535    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 10:28:43.618399 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615538    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 10:28:43.618399 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615541    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 10:28:43.618399 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615543    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 10:28:43.618399 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615546    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 10:28:43.618399 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615549    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 10:28:43.618399 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615551    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 10:28:43.618399 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615554    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 10:28:43.618399 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615557    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 10:28:43.618886 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615560    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 10:28:43.618886 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615562    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 10:28:43.618886 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615565    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 10:28:43.618886 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615567    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 10:28:43.618886 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615570    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 10:28:43.618886 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615573    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 10:28:43.618886 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615576    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 10:28:43.618886 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615579    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 10:28:43.618886 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615582    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 10:28:43.618886 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615585    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 10:28:43.618886 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615588    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 10:28:43.618886 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615590    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 10:28:43.618886 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615593    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 10:28:43.618886 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615595    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 10:28:43.618886 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615599    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 10:28:43.618886 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615603    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 10:28:43.618886 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615606    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 10:28:43.618886 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615609    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 10:28:43.618886 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615611    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 10:28:43.619371 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615614    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 10:28:43.619371 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615617    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 10:28:43.619371 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615619    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 10:28:43.619371 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615622    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 10:28:43.619371 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615624    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 10:28:43.619371 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615627    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 10:28:43.619371 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615629    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 10:28:43.619371 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615632    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 10:28:43.619371 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615635    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 10:28:43.619371 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615637    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 10:28:43.619371 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615640    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 10:28:43.619371 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615643    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 10:28:43.619371 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615645    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 10:28:43.619371 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615648    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 10:28:43.619371 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615650    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 10:28:43.619371 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615654    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 10:28:43.619371 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615658    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 10:28:43.619371 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615661    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 10:28:43.619371 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615664    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 10:28:43.619843 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615667    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 10:28:43.619843 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615670    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 10:28:43.619843 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615674    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 10:28:43.619843 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615676    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 10:28:43.619843 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615679    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 10:28:43.619843 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615681    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 10:28:43.619843 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615684    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 10:28:43.619843 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615687    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 10:28:43.619843 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615690    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 10:28:43.619843 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615692    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 10:28:43.619843 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615695    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 10:28:43.619843 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615697    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 10:28:43.619843 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615700    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 10:28:43.619843 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615703    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 10:28:43.619843 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615705    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 10:28:43.619843 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615708    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 10:28:43.619843 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615711    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 10:28:43.619843 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615713    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 10:28:43.619843 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615716    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 10:28:43.619843 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615718    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 10:28:43.620340 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615721    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 10:28:43.620340 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615724    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 10:28:43.620340 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615727    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 10:28:43.620340 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615729    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 10:28:43.620340 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615733    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 10:28:43.620340 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615735    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 10:28:43.620340 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615738    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 10:28:43.620340 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615740    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 10:28:43.620340 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615743    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 10:28:43.620340 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615746    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 10:28:43.620340 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615750    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 10:28:43.620340 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615752    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 10:28:43.620340 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615756    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 10:28:43.620340 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615759    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 10:28:43.620340 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615762    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 10:28:43.620340 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615764    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 10:28:43.620340 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615767    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 10:28:43.620340 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615769    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 10:28:43.620340 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615772    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 10:28:43.620821 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615775    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 10:28:43.620821 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.615780    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 10:28:43.620821 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615892    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 10:28:43.620821 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615898    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 10:28:43.620821 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615901    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 10:28:43.620821 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615904    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 10:28:43.620821 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615906    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 10:28:43.620821 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615909    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 10:28:43.620821 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615912    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 10:28:43.620821 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615914    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 10:28:43.620821 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615917    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 10:28:43.620821 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615920    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 10:28:43.620821 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615923    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 10:28:43.620821 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615925    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 10:28:43.620821 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615928    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 10:28:43.621201 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615931    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 10:28:43.621201 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615934    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 10:28:43.621201 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615938    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 10:28:43.621201 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615942    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 10:28:43.621201 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615945    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 10:28:43.621201 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615948    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 10:28:43.621201 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615951    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 10:28:43.621201 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615954    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 10:28:43.621201 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615957    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 10:28:43.621201 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615960    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 10:28:43.621201 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615962    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 10:28:43.621201 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615966    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 10:28:43.621201 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615969    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 10:28:43.621201 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615971    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 10:28:43.621201 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615974    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 10:28:43.621201 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615977    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 10:28:43.621201 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615980    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 10:28:43.621201 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615983    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 10:28:43.621201 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615985    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 10:28:43.621657 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615988    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 10:28:43.621657 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615990    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 10:28:43.621657 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615993    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 10:28:43.621657 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615995    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 10:28:43.621657 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.615998    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 10:28:43.621657 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616001    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 10:28:43.621657 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616003    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 10:28:43.621657 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616006    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 10:28:43.621657 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616008    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 10:28:43.621657 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616011    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 10:28:43.621657 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616013    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 10:28:43.621657 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616016    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 10:28:43.621657 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616018    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 10:28:43.621657 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616022    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 10:28:43.621657 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616026    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 10:28:43.621657 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616029    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 10:28:43.621657 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616032    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 10:28:43.621657 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616035    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 10:28:43.621657 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616037    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 10:28:43.622128 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616040    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 10:28:43.622128 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616042    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 10:28:43.622128 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616045    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 10:28:43.622128 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616047    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 10:28:43.622128 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616050    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 10:28:43.622128 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616053    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 10:28:43.622128 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616056    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 10:28:43.622128 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616058    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 10:28:43.622128 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616061    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 10:28:43.622128 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616064    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 10:28:43.622128 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616066    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 10:28:43.622128 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616083    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 10:28:43.622128 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616086    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 10:28:43.622128 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616089    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 10:28:43.622128 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616093    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 10:28:43.622128 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616096    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 10:28:43.622128 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616099    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 10:28:43.622128 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616102    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 10:28:43.622128 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616105    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 10:28:43.622595 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616108    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 10:28:43.622595 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616110    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 10:28:43.622595 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616113    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 10:28:43.622595 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616115    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 10:28:43.622595 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616118    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 10:28:43.622595 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616121    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 10:28:43.622595 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616124    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 10:28:43.622595 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616126    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 10:28:43.622595 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616133    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 10:28:43.622595 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616136    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 10:28:43.622595 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616138    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 10:28:43.622595 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616141    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 10:28:43.622595 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616143    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 10:28:43.622595 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616146    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 10:28:43.622595 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616148    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 10:28:43.622595 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616151    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 10:28:43.622595 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616153    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 10:28:43.622595 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616156    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 10:28:43.622595 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616159    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 10:28:43.622595 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616162    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 10:28:43.623087 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616164    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 10:28:43.623087 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616167    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 10:28:43.623087 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616170    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 10:28:43.623087 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616172    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 10:28:43.623087 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616174    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 10:28:43.623087 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616177    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 10:28:43.623087 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616179    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 10:28:43.623087 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616182    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 10:28:43.623087 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:43.616185    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 10:28:43.623087 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.616189    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 10:28:43.623087 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.616298    2569 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 29 10:28:43.623087 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.620651    2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 29 10:28:43.623087 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.622189    2569 server.go:1019] "Starting client certificate rotation"
Jul 29 10:28:43.623087 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.622288    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 10:28:43.623438 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.622352    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 10:28:43.624902 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.624889    2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 10:28:43.628241 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.628213    2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 10:28:43.641957 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.641939    2569 log.go:25] "Validated CRI v1 runtime API"
Jul 29 10:28:43.644331 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.644316    2569 log.go:25] "Validated CRI v1 image API"
Jul 29 10:28:43.645214 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.645200    2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 29 10:28:43.650463 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.650445    2569 fs.go:135] Filesystem UUIDs: map[23b7b6b6-fa45-40b1-8100-4651a0ecd3a5:/dev/nvme0n1p4 4268ef42-9dc2-4676-9b87-5a34f1774e9b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 29 10:28:43.650520 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.650463    2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 29 10:28:43.654202 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.654183    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 10:28:43.657152 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.657022    2569 manager.go:216] Machine: {Timestamp:2026-07-29 10:28:43.655625464 +0000 UTC m=+0.380416128 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101769 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2402a12bac93b7125071952be6f310 SystemUUID:ec2402a1-2bac-93b7-1250-71952be6f310 BootID:585554af-a639-4803-86c9-a90b22272738 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b4:68:36:4c:e9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b4:68:36:4c:e9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:ad:60:39:6d:3b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 29 10:28:43.657152 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.657140    2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 29 10:28:43.657316 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.657251    2569 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 29 10:28:43.661226 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.661203    2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 29 10:28:43.661387 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.661229    2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-111.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 29 10:28:43.662132 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.662120    2569 topology_manager.go:138] "Creating topology manager with none policy"
Jul 29 10:28:43.662201 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.662139    2569 container_manager_linux.go:309] "Creating device plugin manager"
Jul 29 10:28:43.662201 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.662150    2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 10:28:43.663844 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.663830    2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 10:28:43.664923 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.664911    2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 29 10:28:43.666919 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.666906    2569 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 10:28:43.667061 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.667050    2569 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 29 10:28:43.667147 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.667110    2569 kubelet.go:486] "Attempting to sync node with API server"
Jul 29 10:28:43.667147 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.667126    2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 29 10:28:43.667147 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.667147    2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 29 10:28:43.667267 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.667160    2569 kubelet.go:398] "Adding apiserver pod source"
Jul 29 10:28:43.667267 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.667174    2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 29 10:28:43.667554 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.667541    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 10:28:43.667623 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.667564    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 10:28:43.671845 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.671818    2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 29 10:28:43.674201 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.674186    2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 29 10:28:43.674282 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.674215    2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 29 10:28:43.676104 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.676091    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 29 10:28:43.676168 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.676109    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 29 10:28:43.676168 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.676115    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 29 10:28:43.676168 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.676121    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 29 10:28:43.676168 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.676130    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 29 10:28:43.676168 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.676135    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 29 10:28:43.676168 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.676143    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 29 10:28:43.676168 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.676149    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 29 10:28:43.676168 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.676156    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 29 10:28:43.676168 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.676161    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 29 10:28:43.676168 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.676171    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 29 10:28:43.676168 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.676176    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 29 10:28:43.677050 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:43.677029    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 29 10:28:43.677050 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.677036    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 29 10:28:43.677194 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:43.677060    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-111.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 29 10:28:43.677194 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.677066    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 29 10:28:43.680175 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.680160    2569 server.go:1268] "Started kubelet"
Jul 29 10:28:43.680257 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.680229    2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 29 10:28:43.680667 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.680282    2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 29 10:28:43.680667 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.680366    2569 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 29 10:28:43.681096 ip-10-0-137-111 systemd[1]: Started Kubernetes Kubelet.
Jul 29 10:28:43.682173 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.682159    2569 server.go:310] "Adding debug handlers to kubelet server"
Jul 29 10:28:43.682484 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.682470    2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 29 10:28:43.686730 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:43.686286    2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 29 10:28:43.686796 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.686751    2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-111.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 29 10:28:43.687836 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:43.686700    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-111.ec2.internal.18c6bbebca06d815  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-111.ec2.internal,UID:ip-10-0-137-111.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-111.ec2.internal,},FirstTimestamp:2026-07-29 10:28:43.680135189 +0000 UTC m=+0.404925854,LastTimestamp:2026-07-29 10:28:43.680135189 +0000 UTC m=+0.404925854,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-111.ec2.internal,}"
Jul 29 10:28:43.691631 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.691616    2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 29 10:28:43.691720 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.691620    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 29 10:28:43.691720 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:43.691702    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-111.ec2.internal\" not found"
Jul 29 10:28:43.692785 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.692634    2569 factory.go:55] Registering systemd factory
Jul 29 10:28:43.692785 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.692545    2569 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 29 10:28:43.692785 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.692654    2569 factory.go:223] Registration of the systemd container factory successfully
Jul 29 10:28:43.692785 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.692534    2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 29 10:28:43.693440 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.693422    2569 factory.go:153] Registering CRI-O factory
Jul 29 10:28:43.693440 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.693441    2569 factory.go:223] Registration of the crio container factory successfully
Jul 29 10:28:43.693569 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.693489    2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 29 10:28:43.693569 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.693511    2569 factory.go:103] Registering Raw factory
Jul 29 10:28:43.693569 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.693534    2569 manager.go:1196] Started watching for new ooms in manager
Jul 29 10:28:43.693569 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.693544    2569 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 29 10:28:43.693737 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.693728    2569 reconstruct.go:97] "Volume reconstruction finished"
Jul 29 10:28:43.693782 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.693738    2569 reconciler.go:29] "Reconciler: start to sync state"
Jul 29 10:28:43.693940 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.693924    2569 manager.go:349] Starting recovery of all containers
Jul 29 10:28:43.696096 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:43.696031    2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-111.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 29 10:28:43.696183 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:43.696092    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 29 10:28:43.699782 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.699762    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nbssb"
Jul 29 10:28:43.700814 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.700790    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 29 10:28:43.705615 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.705597    2569 manager.go:354] Recovery completed
Jul 29 10:28:43.707654 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.707634    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nbssb"
Jul 29 10:28:43.710943 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.710929    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:28:43.712764 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.712744    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:28:43.712852 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.712779    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:28:43.712852 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.712794    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-111.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:28:43.713377 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.713359    2569 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 29 10:28:43.713448 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.713376    2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 29 10:28:43.713448 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.713392    2569 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 10:28:43.713503 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.713367    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:28:43.714122 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.714105    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:28:43.714178 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.714131    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:28:43.714178 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.714146    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-111.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:28:43.715085 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:43.714987    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-111.ec2.internal.18c6bbebcbf8b9e3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-111.ec2.internal,UID:ip-10-0-137-111.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-111.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-111.ec2.internal,},FirstTimestamp:2026-07-29 10:28:43.712764387 +0000 UTC m=+0.437555052,LastTimestamp:2026-07-29 10:28:43.712764387 +0000 UTC m=+0.437555052,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-111.ec2.internal,}"
Jul 29 10:28:43.715214 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.715201    2569 policy_none.go:49] "None policy: Start"
Jul 29 10:28:43.715281 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.715220    2569 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 29 10:28:43.715281 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.715233    2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 29 10:28:43.716127 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.716116    2569 policy_none.go:47] "Start"
Jul 29 10:28:43.763655 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.763639    2569 manager.go:337] "Starting Device Plugin manager"
Jul 29 10:28:43.763754 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:43.763677    2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 29 10:28:43.763754 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.763690    2569 server.go:87] "Starting device plugin registration server"
Jul 29 10:28:43.763963 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.763951    2569 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 29 10:28:43.764019 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.763965    2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 29 10:28:43.764105 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.764090    2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 29 10:28:43.764180 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.764169    2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 29 10:28:43.764295 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.764181    2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 29 10:28:43.764695 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:43.764675    2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 29 10:28:43.764764 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:43.764722    2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-111.ec2.internal\" not found"
Jul 29 10:28:43.831601 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.831569    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 29 10:28:43.832694 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.831611    2569 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 29 10:28:43.832694 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.831632    2569 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 29 10:28:43.832694 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:43.831669    2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 29 10:28:43.833624 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.833607    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 10:28:43.864437 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.864382    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:28:43.865265 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.865249    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:28:43.865350 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.865285    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:28:43.865350 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.865301    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-111.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:28:43.865350 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.865332    2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-111.ec2.internal"
Jul 29 10:28:43.876897 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.876881    2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-111.ec2.internal"
Jul 29 10:28:43.876980 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:43.876904    2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-111.ec2.internal\": node \"ip-10-0-137-111.ec2.internal\" not found"
Jul 29 10:28:43.900785 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:43.899663    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-111.ec2.internal\" not found"
Jul 29 10:28:43.932648 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.932603    2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-111.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-111.ec2.internal"]
Jul 29 10:28:43.932749 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.932691    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:28:43.933620 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.933600    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:28:43.933764 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.933636    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:28:43.933764 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.933649    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-111.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:28:43.935183 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.935169    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:28:43.935312 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.935293    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-111.ec2.internal"
Jul 29 10:28:43.935347 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.935322    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:28:43.935899 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.935884    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:28:43.935899 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.935890    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:28:43.936025 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.935911    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:28:43.936025 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.935914    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:28:43.936025 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.935924    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-111.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:28:43.936025 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.935924    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-111.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:28:43.937036 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.937023    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-111.ec2.internal"
Jul 29 10:28:43.937090 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.937048    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:28:43.937696 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.937682    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:28:43.937754 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.937711    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:28:43.937754 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.937723    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-111.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:28:43.960713 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:43.960696    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-111.ec2.internal\" not found" node="ip-10-0-137-111.ec2.internal"
Jul 29 10:28:43.964023 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:43.964006    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-111.ec2.internal\" not found" node="ip-10-0-137-111.ec2.internal"
Jul 29 10:28:43.995684 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.995659    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c1fc06c79a0f992a5584a2e3d3b6fcdb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-111.ec2.internal\" (UID: \"c1fc06c79a0f992a5584a2e3d3b6fcdb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-111.ec2.internal"
Jul 29 10:28:43.995765 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.995690    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1fc06c79a0f992a5584a2e3d3b6fcdb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-111.ec2.internal\" (UID: \"c1fc06c79a0f992a5584a2e3d3b6fcdb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-111.ec2.internal"
Jul 29 10:28:43.995765 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:43.995716    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e7e9628ca209a194bcb37f10a7461829-config\") pod \"kube-apiserver-proxy-ip-10-0-137-111.ec2.internal\" (UID: \"e7e9628ca209a194bcb37f10a7461829\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-111.ec2.internal"
Jul 29 10:28:44.000751 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:44.000736    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-111.ec2.internal\" not found"
Jul 29 10:28:44.096411 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:44.096376    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c1fc06c79a0f992a5584a2e3d3b6fcdb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-111.ec2.internal\" (UID: \"c1fc06c79a0f992a5584a2e3d3b6fcdb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-111.ec2.internal"
Jul 29 10:28:44.096411 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:44.096412    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1fc06c79a0f992a5584a2e3d3b6fcdb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-111.ec2.internal\" (UID: \"c1fc06c79a0f992a5584a2e3d3b6fcdb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-111.ec2.internal"
Jul 29 10:28:44.096581 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:44.096439    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e7e9628ca209a194bcb37f10a7461829-config\") pod \"kube-apiserver-proxy-ip-10-0-137-111.ec2.internal\" (UID: \"e7e9628ca209a194bcb37f10a7461829\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-111.ec2.internal"
Jul 29 10:28:44.096581 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:44.096481    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e7e9628ca209a194bcb37f10a7461829-config\") pod \"kube-apiserver-proxy-ip-10-0-137-111.ec2.internal\" (UID: \"e7e9628ca209a194bcb37f10a7461829\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-111.ec2.internal"
Jul 29 10:28:44.096581 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:44.096490    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1fc06c79a0f992a5584a2e3d3b6fcdb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-111.ec2.internal\" (UID: \"c1fc06c79a0f992a5584a2e3d3b6fcdb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-111.ec2.internal"
Jul 29 10:28:44.096581 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:44.096479    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c1fc06c79a0f992a5584a2e3d3b6fcdb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-111.ec2.internal\" (UID: \"c1fc06c79a0f992a5584a2e3d3b6fcdb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-111.ec2.internal"
Jul 29 10:28:44.101462 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:44.101445    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-111.ec2.internal\" not found"
Jul 29 10:28:44.202292 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:44.202262    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-111.ec2.internal\" not found"
Jul 29 10:28:44.263489 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:44.263448    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-111.ec2.internal"
Jul 29 10:28:44.267608 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:44.267457    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-111.ec2.internal"
Jul 29 10:28:44.302959 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:44.302930    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-111.ec2.internal\" not found"
Jul 29 10:28:44.403571 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:44.403521    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-111.ec2.internal\" not found"
Jul 29 10:28:44.504514 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:44.504442    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-111.ec2.internal\" not found"
Jul 29 10:28:44.605621 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:44.605585    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-111.ec2.internal\" not found"
Jul 29 10:28:44.621949 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:44.621927    2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 29 10:28:44.622129 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:44.622111    2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 10:28:44.695194 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:44.695159    2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 10:28:44.706246 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:44.706221    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-111.ec2.internal\" not found"
Jul 29 10:28:44.713590 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:44.713559    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 10:23:44 +0000 UTC" deadline="2027-12-25 02:16:54.763346177 +0000 UTC"
Jul 29 10:28:44.713654 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:44.713587    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12327h48m10.049763454s"
Jul 29 10:28:44.806367 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:44.806281    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-111.ec2.internal\" not found"
Jul 29 10:28:44.907169 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:44.907135    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-111.ec2.internal\" not found"
Jul 29 10:28:44.955803 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:44.955757    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7e9628ca209a194bcb37f10a7461829.slice/crio-d711c7bf93f5030915a94ac5efdb22ed6cfc608ae630016dc60598eb633a2e07 WatchSource:0}: Error finding container d711c7bf93f5030915a94ac5efdb22ed6cfc608ae630016dc60598eb633a2e07: Status 404 returned error can't find the container with id d711c7bf93f5030915a94ac5efdb22ed6cfc608ae630016dc60598eb633a2e07
Jul 29 10:28:44.956044 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:44.956022    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1fc06c79a0f992a5584a2e3d3b6fcdb.slice/crio-e51b7de66e64d33f93f08da4e024cdd05afec061d76eb84ad5f2cde4e9a4dbc0 WatchSource:0}: Error finding container e51b7de66e64d33f93f08da4e024cdd05afec061d76eb84ad5f2cde4e9a4dbc0: Status 404 returned error can't find the container with id e51b7de66e64d33f93f08da4e024cdd05afec061d76eb84ad5f2cde4e9a4dbc0
Jul 29 10:28:44.960462 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:44.960446    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 10:28:45.007987 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:45.007940    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-111.ec2.internal\" not found"
Jul 29 10:28:45.109199 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:45.109128    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-111.ec2.internal\" not found"
Jul 29 10:28:45.126351 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.126329    2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 10:28:45.192830 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.192794    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-111.ec2.internal"
Jul 29 10:28:45.208577 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.208558    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 10:28:45.210852 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.210833    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-111.ec2.internal"
Jul 29 10:28:45.220036 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.220020    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 10:28:45.285399 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.285373    2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 10:28:45.668755 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.668710    2569 apiserver.go:52] "Watching apiserver"
Jul 29 10:28:45.675406 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.675378    2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 29 10:28:45.675928 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.675904    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-r7xrd","openshift-multus/network-metrics-daemon-szr8n","openshift-network-operator/iptables-alerter-n5pfx","kube-system/kube-apiserver-proxy-ip-10-0-137-111.ec2.internal","openshift-cluster-node-tuning-operator/tuned-str4m","openshift-dns/node-resolver-hpmhx","openshift-multus/multus-wkrbj","openshift-network-diagnostics/network-check-target-mdqnc","openshift-ovn-kubernetes/ovnkube-node-6jzgn","kube-system/global-pull-secret-syncer-p8p95","kube-system/konnectivity-agent-6c2hd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvvkn","openshift-image-registry/node-ca-b6v6k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-111.ec2.internal"]
Jul 29 10:28:45.677596 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.677574    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.679029 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.679003    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szr8n"
Jul 29 10:28:45.679181 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:45.679126    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-szr8n" podUID="51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c"
Jul 29 10:28:45.680176 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.680156    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.680684 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.680663    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 29 10:28:45.680768 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.680710    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 29 10:28:45.681013 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.680994    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7996h\""
Jul 29 10:28:45.683220 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.683043    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 29 10:28:45.683220 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.683095    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 29 10:28:45.683220 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.683049    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 29 10:28:45.685217 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.685195    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hpmhx"
Jul 29 10:28:45.686123 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.685957    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 29 10:28:45.686123 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.686094    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-psxx7\""
Jul 29 10:28:45.686756 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.686736    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8p95"
Jul 29 10:28:45.686841 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:45.686815    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8p95" podUID="27e6f470-ad48-45f1-b171-ae66fac19573"
Jul 29 10:28:45.688966 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.688947    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 29 10:28:45.689094 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.688993    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lfc24\""
Jul 29 10:28:45.689094 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.689033    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 29 10:28:45.689308 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.689277    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-n5pfx"
Jul 29 10:28:45.689384 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.689343    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdqnc"
Jul 29 10:28:45.689451 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:45.689427    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdqnc" podUID="f7b4bba0-77fe-4240-9858-433b0f431c59"
Jul 29 10:28:45.691347 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.691326    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.692067 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.692048    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 29 10:28:45.692447 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.692431    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lhg9x\""
Jul 29 10:28:45.692447 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.692437    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 29 10:28:45.692704 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.692631    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 29 10:28:45.692704 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.692676    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 29 10:28:45.692897 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.692879    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r7xrd"
Jul 29 10:28:45.693781 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.693757    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 29 10:28:45.694750 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.694597    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 29 10:28:45.694750 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.694639    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 29 10:28:45.694941 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.694893    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 29 10:28:45.695087 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.695069    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 29 10:28:45.695298 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.695278    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9mv6h\""
Jul 29 10:28:45.695528 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.695508    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 29 10:28:45.695614 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.695586    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6c2hd"
Jul 29 10:28:45.695614 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.695599    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 29 10:28:45.695737 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.695624    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 29 10:28:45.695737 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.695635    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jknfd\""
Jul 29 10:28:45.697060 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.697009    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvvkn"
Jul 29 10:28:45.698552 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.698531    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-b6v6k"
Jul 29 10:28:45.699396 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.699067    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-dpbsw\""
Jul 29 10:28:45.699396 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.699192    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 29 10:28:45.699396 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.699196    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 29 10:28:45.700173 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.699813    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 29 10:28:45.700173 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.700083    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 29 10:28:45.700173 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.700102    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-db55q\""
Jul 29 10:28:45.700375 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.700318    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 29 10:28:45.702626 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.701136    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 29 10:28:45.702626 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.701223    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 29 10:28:45.702626 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.701436    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 29 10:28:45.702626 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.701633    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vmcxw\""
Jul 29 10:28:45.703225 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.702901    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 10:28:45.706896 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.706283    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/27e6f470-ad48-45f1-b171-ae66fac19573-dbus\") pod \"global-pull-secret-syncer-p8p95\" (UID: \"27e6f470-ad48-45f1-b171-ae66fac19573\") " pod="kube-system/global-pull-secret-syncer-p8p95"
Jul 29 10:28:45.706896 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.706318    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9d340b7-82a4-43bd-96b8-e19532b16b4b-etc-kubernetes\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.706896 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.706347    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b9d340b7-82a4-43bd-96b8-e19532b16b4b-etc-sysctl-d\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.706896 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.706371    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-var-lib-openvswitch\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.706896 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.706398    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b9d340b7-82a4-43bd-96b8-e19532b16b4b-etc-systemd\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.706896 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.706421    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9d340b7-82a4-43bd-96b8-e19532b16b4b-sys\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.706896 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.706444    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-system-cni-dir\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.706896 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.706480    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-host-var-lib-cni-bin\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.706896 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.706506    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-468zq\" (UniqueName: \"kubernetes.io/projected/faeca467-a198-4dd6-9da8-e4f72ac68c77-kube-api-access-468zq\") pod \"node-resolver-hpmhx\" (UID: \"faeca467-a198-4dd6-9da8-e4f72ac68c77\") " pod="openshift-dns/node-resolver-hpmhx"
Jul 29 10:28:45.706896 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.706530    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mntg7\" (UniqueName: \"kubernetes.io/projected/2e6f3639-fa84-4599-8a55-a6d7242cea3a-kube-api-access-mntg7\") pod \"multus-additional-cni-plugins-r7xrd\" (UID: \"2e6f3639-fa84-4599-8a55-a6d7242cea3a\") " pod="openshift-multus/multus-additional-cni-plugins-r7xrd"
Jul 29 10:28:45.706896 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.706576    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/770e3b65-661a-4dad-aaa9-5fbb68ae8dea-iptables-alerter-script\") pod \"iptables-alerter-n5pfx\" (UID: \"770e3b65-661a-4dad-aaa9-5fbb68ae8dea\") " pod="openshift-network-operator/iptables-alerter-n5pfx"
Jul 29 10:28:45.706896 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.706605    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b9d340b7-82a4-43bd-96b8-e19532b16b4b-lib-modules\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.706896 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.706634    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2e6f3639-fa84-4599-8a55-a6d7242cea3a-cni-binary-copy\") pod \"multus-additional-cni-plugins-r7xrd\" (UID: \"2e6f3639-fa84-4599-8a55-a6d7242cea3a\") " pod="openshift-multus/multus-additional-cni-plugins-r7xrd"
Jul 29 10:28:45.706896 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.706665    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/90f1789e-de1e-4cc2-a128-ee7621bf80ca-ovnkube-config\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.706896 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.706713    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27e6f470-ad48-45f1-b171-ae66fac19573-original-pull-secret\") pod \"global-pull-secret-syncer-p8p95\" (UID: \"27e6f470-ad48-45f1-b171-ae66fac19573\") " pod="kube-system/global-pull-secret-syncer-p8p95"
Jul 29 10:28:45.706896 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.706746    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-log-socket\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.707802 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.706770    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2e6f3639-fa84-4599-8a55-a6d7242cea3a-cnibin\") pod \"multus-additional-cni-plugins-r7xrd\" (UID: \"2e6f3639-fa84-4599-8a55-a6d7242cea3a\") " pod="openshift-multus/multus-additional-cni-plugins-r7xrd"
Jul 29 10:28:45.707802 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.706811    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-multus-socket-dir-parent\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.707802 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.706837    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-host-run-netns\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.707802 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.706869    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-host-var-lib-cni-multus\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.707802 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.706902    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/faeca467-a198-4dd6-9da8-e4f72ac68c77-hosts-file\") pod \"node-resolver-hpmhx\" (UID: \"faeca467-a198-4dd6-9da8-e4f72ac68c77\") " pod="openshift-dns/node-resolver-hpmhx"
Jul 29 10:28:45.707802 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.706941    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-host-run-netns\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.707802 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.706989    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/90f1789e-de1e-4cc2-a128-ee7621bf80ca-env-overrides\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.707802 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.707026    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b9d340b7-82a4-43bd-96b8-e19532b16b4b-etc-modprobe-d\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.707802 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.707057    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cffc10c9-fe6c-420b-b994-869307922201-cni-binary-copy\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.707802 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.707096    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-etc-kubernetes\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.707802 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.707121    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-host-cni-bin\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.707802 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.707147    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr78h\" (UniqueName: \"kubernetes.io/projected/770e3b65-661a-4dad-aaa9-5fbb68ae8dea-kube-api-access-tr78h\") pod \"iptables-alerter-n5pfx\" (UID: \"770e3b65-661a-4dad-aaa9-5fbb68ae8dea\") " pod="openshift-network-operator/iptables-alerter-n5pfx"
Jul 29 10:28:45.707802 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.707174    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-os-release\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.707802 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.707204    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-run-openvswitch\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.707802 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.707236    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/770e3b65-661a-4dad-aaa9-5fbb68ae8dea-host-slash\") pod \"iptables-alerter-n5pfx\" (UID: \"770e3b65-661a-4dad-aaa9-5fbb68ae8dea\") " pod="openshift-network-operator/iptables-alerter-n5pfx"
Jul 29 10:28:45.707802 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.707266    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-multus-cni-dir\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.707802 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.707307    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-host-slash\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.708650 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.707333    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-etc-openvswitch\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.708650 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.707372    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b9d340b7-82a4-43bd-96b8-e19532b16b4b-run\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.708650 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.707398    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c-metrics-certs\") pod \"network-metrics-daemon-szr8n\" (UID: \"51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c\") " pod="openshift-multus/network-metrics-daemon-szr8n"
Jul 29 10:28:45.708650 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.707424    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-cnibin\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.708650 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.707465    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-host-var-lib-kubelet\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.708650 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.707500    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-host-kubelet\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.708650 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.707524    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-run-systemd\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.708650 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.707553    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-node-log\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.708650 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.707577    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-host-run-ovn-kubernetes\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.708650 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.707604    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/90f1789e-de1e-4cc2-a128-ee7621bf80ca-ovnkube-script-lib\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.708650 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.707629    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e6f3639-fa84-4599-8a55-a6d7242cea3a-system-cni-dir\") pod \"multus-additional-cni-plugins-r7xrd\" (UID: \"2e6f3639-fa84-4599-8a55-a6d7242cea3a\") " pod="openshift-multus/multus-additional-cni-plugins-r7xrd"
Jul 29 10:28:45.708650 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.707655    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2e6f3639-fa84-4599-8a55-a6d7242cea3a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r7xrd\" (UID: \"2e6f3639-fa84-4599-8a55-a6d7242cea3a\") " pod="openshift-multus/multus-additional-cni-plugins-r7xrd"
Jul 29 10:28:45.708650 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.707680    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b9d340b7-82a4-43bd-96b8-e19532b16b4b-etc-tuned\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.708650 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.707704    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b9d340b7-82a4-43bd-96b8-e19532b16b4b-etc-sysctl-conf\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.708650 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.707727    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-hostroot\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.708650 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.707751    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-host-run-multus-certs\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.709489 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.707778    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2e6f3639-fa84-4599-8a55-a6d7242cea3a-os-release\") pod \"multus-additional-cni-plugins-r7xrd\" (UID: \"2e6f3639-fa84-4599-8a55-a6d7242cea3a\") " pod="openshift-multus/multus-additional-cni-plugins-r7xrd"
Jul 29 10:28:45.709489 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.707802    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2e6f3639-fa84-4599-8a55-a6d7242cea3a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r7xrd\" (UID: \"2e6f3639-fa84-4599-8a55-a6d7242cea3a\") " pod="openshift-multus/multus-additional-cni-plugins-r7xrd"
Jul 29 10:28:45.709489 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.707826    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9d340b7-82a4-43bd-96b8-e19532b16b4b-tmp\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.709489 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.707851    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cf5j\" (UniqueName: \"kubernetes.io/projected/b9d340b7-82a4-43bd-96b8-e19532b16b4b-kube-api-access-2cf5j\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.709489 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.707876    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/27e6f470-ad48-45f1-b171-ae66fac19573-kubelet-config\") pod \"global-pull-secret-syncer-p8p95\" (UID: \"27e6f470-ad48-45f1-b171-ae66fac19573\") " pod="kube-system/global-pull-secret-syncer-p8p95"
Jul 29 10:28:45.709489 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.707901    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-host-cni-netd\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.709489 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.707944    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc7gs\" (UniqueName: \"kubernetes.io/projected/90f1789e-de1e-4cc2-a128-ee7621bf80ca-kube-api-access-hc7gs\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.709489 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.707980    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-run-ovn\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.709489 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.708018    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.709489 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.708050    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/90f1789e-de1e-4cc2-a128-ee7621bf80ca-ovn-node-metrics-cert\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.709489 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.708145    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2e6f3639-fa84-4599-8a55-a6d7242cea3a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r7xrd\" (UID: \"2e6f3639-fa84-4599-8a55-a6d7242cea3a\") " pod="openshift-multus/multus-additional-cni-plugins-r7xrd"
Jul 29 10:28:45.709489 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.708173    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d8nk\" (UniqueName: \"kubernetes.io/projected/f7b4bba0-77fe-4240-9858-433b0f431c59-kube-api-access-4d8nk\") pod \"network-check-target-mdqnc\" (UID: \"f7b4bba0-77fe-4240-9858-433b0f431c59\") " pod="openshift-network-diagnostics/network-check-target-mdqnc"
Jul 29 10:28:45.709489 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.708198    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b9d340b7-82a4-43bd-96b8-e19532b16b4b-etc-sysconfig\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.709489 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.708221    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mtkc\" (UniqueName: \"kubernetes.io/projected/51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c-kube-api-access-8mtkc\") pod \"network-metrics-daemon-szr8n\" (UID: \"51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c\") " pod="openshift-multus/network-metrics-daemon-szr8n"
Jul 29 10:28:45.709489 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.708245    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-systemd-units\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.709489 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.708267    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b9d340b7-82a4-43bd-96b8-e19532b16b4b-host\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.710463 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.708291    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-host-run-k8s-cni-cncf-io\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.710463 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.708315    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-multus-conf-dir\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.710463 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.708339    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cffc10c9-fe6c-420b-b994-869307922201-multus-daemon-config\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.710463 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.708363    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxdz8\" (UniqueName: \"kubernetes.io/projected/cffc10c9-fe6c-420b-b994-869307922201-kube-api-access-rxdz8\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.710463 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.708415    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faeca467-a198-4dd6-9da8-e4f72ac68c77-tmp-dir\") pod \"node-resolver-hpmhx\" (UID: \"faeca467-a198-4dd6-9da8-e4f72ac68c77\") " pod="openshift-dns/node-resolver-hpmhx"
Jul 29 10:28:45.710463 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.708443    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9d340b7-82a4-43bd-96b8-e19532b16b4b-var-lib-kubelet\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.732920 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.732899    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-spqt9"
Jul 29 10:28:45.739918 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.739881    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-spqt9"
Jul 29 10:28:45.794125 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.794088    2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 29 10:28:45.809169 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809140    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-host-slash\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.809288 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809185    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-etc-openvswitch\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.809288 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809213    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b9d340b7-82a4-43bd-96b8-e19532b16b4b-run\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.809288 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809239    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c-metrics-certs\") pod \"network-metrics-daemon-szr8n\" (UID: \"51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c\") " pod="openshift-multus/network-metrics-daemon-szr8n"
Jul 29 10:28:45.809288 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809264    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-cnibin\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.809288 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809279    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-etc-openvswitch\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.809552 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809292    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-host-var-lib-kubelet\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.809552 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809239    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-host-slash\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.809552 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809317    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-host-kubelet\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.809552 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809343    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-run-systemd\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.809552 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809368    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-node-log\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.809552 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809392    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-host-run-ovn-kubernetes\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.809552 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:45.809417    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:45.809552 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809431    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/90f1789e-de1e-4cc2-a128-ee7621bf80ca-ovnkube-script-lib\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.809552 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809457    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e6f3639-fa84-4599-8a55-a6d7242cea3a-system-cni-dir\") pod \"multus-additional-cni-plugins-r7xrd\" (UID: \"2e6f3639-fa84-4599-8a55-a6d7242cea3a\") " pod="openshift-multus/multus-additional-cni-plugins-r7xrd"
Jul 29 10:28:45.809552 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809462    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-host-kubelet\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.809552 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:45.809485    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c-metrics-certs podName:51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c nodeName:}" failed. No retries permitted until 2026-07-29 10:28:46.309462384 +0000 UTC m=+3.034253041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c-metrics-certs") pod "network-metrics-daemon-szr8n" (UID: "51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:45.809552 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809411    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b9d340b7-82a4-43bd-96b8-e19532b16b4b-run\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.809552 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809510    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e6f3639-fa84-4599-8a55-a6d7242cea3a-system-cni-dir\") pod \"multus-additional-cni-plugins-r7xrd\" (UID: \"2e6f3639-fa84-4599-8a55-a6d7242cea3a\") " pod="openshift-multus/multus-additional-cni-plugins-r7xrd"
Jul 29 10:28:45.809552 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809525    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-cnibin\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.810304 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809562    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-run-systemd\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.810304 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809574    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-host-var-lib-kubelet\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.810304 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809609    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-node-log\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.810304 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809629    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2e6f3639-fa84-4599-8a55-a6d7242cea3a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r7xrd\" (UID: \"2e6f3639-fa84-4599-8a55-a6d7242cea3a\") " pod="openshift-multus/multus-additional-cni-plugins-r7xrd"
Jul 29 10:28:45.810304 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809656    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b9d340b7-82a4-43bd-96b8-e19532b16b4b-etc-tuned\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.810304 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809684    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b9d340b7-82a4-43bd-96b8-e19532b16b4b-etc-sysctl-conf\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.810304 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809709    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-hostroot\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.810304 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809735    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-host-run-multus-certs\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.810304 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809763    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2e6f3639-fa84-4599-8a55-a6d7242cea3a-os-release\") pod \"multus-additional-cni-plugins-r7xrd\" (UID: \"2e6f3639-fa84-4599-8a55-a6d7242cea3a\") " pod="openshift-multus/multus-additional-cni-plugins-r7xrd"
Jul 29 10:28:45.810304 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809786    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2e6f3639-fa84-4599-8a55-a6d7242cea3a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r7xrd\" (UID: \"2e6f3639-fa84-4599-8a55-a6d7242cea3a\") " pod="openshift-multus/multus-additional-cni-plugins-r7xrd"
Jul 29 10:28:45.810304 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809810    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9d340b7-82a4-43bd-96b8-e19532b16b4b-tmp\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.810304 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809834    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cf5j\" (UniqueName: \"kubernetes.io/projected/b9d340b7-82a4-43bd-96b8-e19532b16b4b-kube-api-access-2cf5j\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.810304 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809860    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/27e6f470-ad48-45f1-b171-ae66fac19573-kubelet-config\") pod \"global-pull-secret-syncer-p8p95\" (UID: \"27e6f470-ad48-45f1-b171-ae66fac19573\") " pod="kube-system/global-pull-secret-syncer-p8p95"
Jul 29 10:28:45.810304 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809884    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-host-cni-netd\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.810304 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809908    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc7gs\" (UniqueName: \"kubernetes.io/projected/90f1789e-de1e-4cc2-a128-ee7621bf80ca-kube-api-access-hc7gs\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.810304 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809933    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-run-ovn\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.810304 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809959    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.810304 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809974    2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 29 10:28:45.811249 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809987    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/90f1789e-de1e-4cc2-a128-ee7621bf80ca-ovn-node-metrics-cert\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.811249 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810012    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2e6f3639-fa84-4599-8a55-a6d7242cea3a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r7xrd\" (UID: \"2e6f3639-fa84-4599-8a55-a6d7242cea3a\") " pod="openshift-multus/multus-additional-cni-plugins-r7xrd"
Jul 29 10:28:45.811249 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810047    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d8nk\" (UniqueName: \"kubernetes.io/projected/f7b4bba0-77fe-4240-9858-433b0f431c59-kube-api-access-4d8nk\") pod \"network-check-target-mdqnc\" (UID: \"f7b4bba0-77fe-4240-9858-433b0f431c59\") " pod="openshift-network-diagnostics/network-check-target-mdqnc"
Jul 29 10:28:45.811249 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810071    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b9d340b7-82a4-43bd-96b8-e19532b16b4b-etc-sysconfig\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.811249 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810096    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mtkc\" (UniqueName: \"kubernetes.io/projected/51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c-kube-api-access-8mtkc\") pod \"network-metrics-daemon-szr8n\" (UID: \"51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c\") " pod="openshift-multus/network-metrics-daemon-szr8n"
Jul 29 10:28:45.811249 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810121    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-systemd-units\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.811249 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810166    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b9d340b7-82a4-43bd-96b8-e19532b16b4b-host\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.811249 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810192    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-host-run-k8s-cni-cncf-io\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.811249 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810219    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/90f1789e-de1e-4cc2-a128-ee7621bf80ca-ovnkube-script-lib\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.811249 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810219    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-multus-conf-dir\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.811249 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810257    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-multus-conf-dir\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.811249 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810278    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cffc10c9-fe6c-420b-b994-869307922201-multus-daemon-config\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.811249 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810303    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.811249 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810308    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxdz8\" (UniqueName: \"kubernetes.io/projected/cffc10c9-fe6c-420b-b994-869307922201-kube-api-access-rxdz8\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.811249 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810349    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/512bc3c2-b150-4eb6-ba9f-b02810556544-agent-certs\") pod \"konnectivity-agent-6c2hd\" (UID: \"512bc3c2-b150-4eb6-ba9f-b02810556544\") " pod="kube-system/konnectivity-agent-6c2hd"
Jul 29 10:28:45.811249 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810375    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/372aeb5f-78d6-427f-93a7-97d8db16e8d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xvvkn\" (UID: \"372aeb5f-78d6-427f-93a7-97d8db16e8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvvkn"
Jul 29 10:28:45.811249 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810402    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faeca467-a198-4dd6-9da8-e4f72ac68c77-tmp-dir\") pod \"node-resolver-hpmhx\" (UID: \"faeca467-a198-4dd6-9da8-e4f72ac68c77\") " pod="openshift-dns/node-resolver-hpmhx"
Jul 29 10:28:45.812106 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810426    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9d340b7-82a4-43bd-96b8-e19532b16b4b-var-lib-kubelet\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.812106 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810450    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/372aeb5f-78d6-427f-93a7-97d8db16e8d0-device-dir\") pod \"aws-ebs-csi-driver-node-xvvkn\" (UID: \"372aeb5f-78d6-427f-93a7-97d8db16e8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvvkn"
Jul 29 10:28:45.812106 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810477    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/27e6f470-ad48-45f1-b171-ae66fac19573-dbus\") pod \"global-pull-secret-syncer-p8p95\" (UID: \"27e6f470-ad48-45f1-b171-ae66fac19573\") " pod="kube-system/global-pull-secret-syncer-p8p95"
Jul 29 10:28:45.812106 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810500    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9d340b7-82a4-43bd-96b8-e19532b16b4b-etc-kubernetes\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.812106 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.809656    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-host-run-ovn-kubernetes\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.812106 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810532    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b9d340b7-82a4-43bd-96b8-e19532b16b4b-etc-sysctl-d\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.812106 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810586    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/372aeb5f-78d6-427f-93a7-97d8db16e8d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-xvvkn\" (UID: \"372aeb5f-78d6-427f-93a7-97d8db16e8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvvkn"
Jul 29 10:28:45.812106 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810621    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kst7w\" (UniqueName: \"kubernetes.io/projected/372aeb5f-78d6-427f-93a7-97d8db16e8d0-kube-api-access-kst7w\") pod \"aws-ebs-csi-driver-node-xvvkn\" (UID: \"372aeb5f-78d6-427f-93a7-97d8db16e8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvvkn"
Jul 29 10:28:45.812106 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810643    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b9d340b7-82a4-43bd-96b8-e19532b16b4b-etc-sysctl-d\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.812106 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810653    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bd747eb7-faeb-48ab-804a-96355d0b79c4-serviceca\") pod \"node-ca-b6v6k\" (UID: \"bd747eb7-faeb-48ab-804a-96355d0b79c4\") " pod="openshift-image-registry/node-ca-b6v6k"
Jul 29 10:28:45.812106 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810682    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkt22\" (UniqueName: \"kubernetes.io/projected/bd747eb7-faeb-48ab-804a-96355d0b79c4-kube-api-access-jkt22\") pod \"node-ca-b6v6k\" (UID: \"bd747eb7-faeb-48ab-804a-96355d0b79c4\") " pod="openshift-image-registry/node-ca-b6v6k"
Jul 29 10:28:45.812106 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810712    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-var-lib-openvswitch\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.812106 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810725    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b9d340b7-82a4-43bd-96b8-e19532b16b4b-host\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.812106 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810739    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b9d340b7-82a4-43bd-96b8-e19532b16b4b-etc-systemd\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.812106 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810778    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9d340b7-82a4-43bd-96b8-e19532b16b4b-sys\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.812106 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810790    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b9d340b7-82a4-43bd-96b8-e19532b16b4b-etc-systemd\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.812106 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810807    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-system-cni-dir\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.812938 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810838    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-host-var-lib-cni-bin\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.812938 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810865    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-468zq\" (UniqueName: \"kubernetes.io/projected/faeca467-a198-4dd6-9da8-e4f72ac68c77-kube-api-access-468zq\") pod \"node-resolver-hpmhx\" (UID: \"faeca467-a198-4dd6-9da8-e4f72ac68c77\") " pod="openshift-dns/node-resolver-hpmhx"
Jul 29 10:28:45.812938 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810892    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mntg7\" (UniqueName: \"kubernetes.io/projected/2e6f3639-fa84-4599-8a55-a6d7242cea3a-kube-api-access-mntg7\") pod \"multus-additional-cni-plugins-r7xrd\" (UID: \"2e6f3639-fa84-4599-8a55-a6d7242cea3a\") " pod="openshift-multus/multus-additional-cni-plugins-r7xrd"
Jul 29 10:28:45.812938 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810900    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b9d340b7-82a4-43bd-96b8-e19532b16b4b-etc-sysctl-conf\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.812938 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810946    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-hostroot\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.812938 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.810985    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-host-run-multus-certs\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.812938 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811042    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2e6f3639-fa84-4599-8a55-a6d7242cea3a-os-release\") pod \"multus-additional-cni-plugins-r7xrd\" (UID: \"2e6f3639-fa84-4599-8a55-a6d7242cea3a\") " pod="openshift-multus/multus-additional-cni-plugins-r7xrd"
Jul 29 10:28:45.812938 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811132    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2e6f3639-fa84-4599-8a55-a6d7242cea3a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r7xrd\" (UID: \"2e6f3639-fa84-4599-8a55-a6d7242cea3a\") " pod="openshift-multus/multus-additional-cni-plugins-r7xrd"
Jul 29 10:28:45.812938 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811144    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-systemd-units\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.812938 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811188    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9d340b7-82a4-43bd-96b8-e19532b16b4b-sys\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.812938 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811199    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2e6f3639-fa84-4599-8a55-a6d7242cea3a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r7xrd\" (UID: \"2e6f3639-fa84-4599-8a55-a6d7242cea3a\") " pod="openshift-multus/multus-additional-cni-plugins-r7xrd"
Jul 29 10:28:45.812938 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811272    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-host-var-lib-cni-bin\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.812938 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811385    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2e6f3639-fa84-4599-8a55-a6d7242cea3a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r7xrd\" (UID: \"2e6f3639-fa84-4599-8a55-a6d7242cea3a\") " pod="openshift-multus/multus-additional-cni-plugins-r7xrd"
Jul 29 10:28:45.812938 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811412    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/770e3b65-661a-4dad-aaa9-5fbb68ae8dea-iptables-alerter-script\") pod \"iptables-alerter-n5pfx\" (UID: \"770e3b65-661a-4dad-aaa9-5fbb68ae8dea\") " pod="openshift-network-operator/iptables-alerter-n5pfx"
Jul 29 10:28:45.812938 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811444    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b9d340b7-82a4-43bd-96b8-e19532b16b4b-lib-modules\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.812938 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811473    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/512bc3c2-b150-4eb6-ba9f-b02810556544-konnectivity-ca\") pod \"konnectivity-agent-6c2hd\" (UID: \"512bc3c2-b150-4eb6-ba9f-b02810556544\") " pod="kube-system/konnectivity-agent-6c2hd"
Jul 29 10:28:45.812938 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811481    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/27e6f470-ad48-45f1-b171-ae66fac19573-kubelet-config\") pod \"global-pull-secret-syncer-p8p95\" (UID: \"27e6f470-ad48-45f1-b171-ae66fac19573\") " pod="kube-system/global-pull-secret-syncer-p8p95"
Jul 29 10:28:45.813879 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811499    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/372aeb5f-78d6-427f-93a7-97d8db16e8d0-registration-dir\") pod \"aws-ebs-csi-driver-node-xvvkn\" (UID: \"372aeb5f-78d6-427f-93a7-97d8db16e8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvvkn"
Jul 29 10:28:45.813879 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811528    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2e6f3639-fa84-4599-8a55-a6d7242cea3a-cni-binary-copy\") pod \"multus-additional-cni-plugins-r7xrd\" (UID: \"2e6f3639-fa84-4599-8a55-a6d7242cea3a\") " pod="openshift-multus/multus-additional-cni-plugins-r7xrd"
Jul 29 10:28:45.813879 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811554    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/90f1789e-de1e-4cc2-a128-ee7621bf80ca-ovnkube-config\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.813879 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811562    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-run-ovn\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.813879 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811580    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27e6f470-ad48-45f1-b171-ae66fac19573-original-pull-secret\") pod \"global-pull-secret-syncer-p8p95\" (UID: \"27e6f470-ad48-45f1-b171-ae66fac19573\") " pod="kube-system/global-pull-secret-syncer-p8p95"
Jul 29 10:28:45.813879 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811617    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-log-socket\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.813879 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811627    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cffc10c9-fe6c-420b-b994-869307922201-multus-daemon-config\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.813879 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811646    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2e6f3639-fa84-4599-8a55-a6d7242cea3a-cnibin\") pod \"multus-additional-cni-plugins-r7xrd\" (UID: \"2e6f3639-fa84-4599-8a55-a6d7242cea3a\") " pod="openshift-multus/multus-additional-cni-plugins-r7xrd"
Jul 29 10:28:45.813879 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811659    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-system-cni-dir\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.813879 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811674    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-multus-socket-dir-parent\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.813879 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811702    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-host-run-netns\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.813879 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811710    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9d340b7-82a4-43bd-96b8-e19532b16b4b-var-lib-kubelet\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.813879 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811767    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-host-var-lib-cni-multus\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.813879 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811795    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/faeca467-a198-4dd6-9da8-e4f72ac68c77-hosts-file\") pod \"node-resolver-hpmhx\" (UID: \"faeca467-a198-4dd6-9da8-e4f72ac68c77\") " pod="openshift-dns/node-resolver-hpmhx"
Jul 29 10:28:45.813879 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811820    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-host-run-netns\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.813879 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811848    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/90f1789e-de1e-4cc2-a128-ee7621bf80ca-env-overrides\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.813879 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811879    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b9d340b7-82a4-43bd-96b8-e19532b16b4b-etc-modprobe-d\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.814729 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811904    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cffc10c9-fe6c-420b-b994-869307922201-cni-binary-copy\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.814729 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811930    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-etc-kubernetes\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.814729 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811949    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/27e6f470-ad48-45f1-b171-ae66fac19573-dbus\") pod \"global-pull-secret-syncer-p8p95\" (UID: \"27e6f470-ad48-45f1-b171-ae66fac19573\") " pod="kube-system/global-pull-secret-syncer-p8p95"
Jul 29 10:28:45.814729 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811960    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/372aeb5f-78d6-427f-93a7-97d8db16e8d0-socket-dir\") pod \"aws-ebs-csi-driver-node-xvvkn\" (UID: \"372aeb5f-78d6-427f-93a7-97d8db16e8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvvkn"
Jul 29 10:28:45.814729 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811988    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faeca467-a198-4dd6-9da8-e4f72ac68c77-tmp-dir\") pod \"node-resolver-hpmhx\" (UID: \"faeca467-a198-4dd6-9da8-e4f72ac68c77\") " pod="openshift-dns/node-resolver-hpmhx"
Jul 29 10:28:45.814729 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.812008    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9d340b7-82a4-43bd-96b8-e19532b16b4b-etc-kubernetes\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.814729 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811990    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-host-cni-bin\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.814729 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.812030    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-host-cni-bin\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.814729 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.812049    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr78h\" (UniqueName: \"kubernetes.io/projected/770e3b65-661a-4dad-aaa9-5fbb68ae8dea-kube-api-access-tr78h\") pod \"iptables-alerter-n5pfx\" (UID: \"770e3b65-661a-4dad-aaa9-5fbb68ae8dea\") " pod="openshift-network-operator/iptables-alerter-n5pfx"
Jul 29 10:28:45.814729 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811646    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-var-lib-openvswitch\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.814729 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.812078    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-os-release\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.814729 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.812108    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/372aeb5f-78d6-427f-93a7-97d8db16e8d0-sys-fs\") pod \"aws-ebs-csi-driver-node-xvvkn\" (UID: \"372aeb5f-78d6-427f-93a7-97d8db16e8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvvkn"
Jul 29 10:28:45.814729 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.812131    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bd747eb7-faeb-48ab-804a-96355d0b79c4-host\") pod \"node-ca-b6v6k\" (UID: \"bd747eb7-faeb-48ab-804a-96355d0b79c4\") " pod="openshift-image-registry/node-ca-b6v6k"
Jul 29 10:28:45.814729 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.812166    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b9d340b7-82a4-43bd-96b8-e19532b16b4b-lib-modules\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.814729 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.812181    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-run-openvswitch\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.814729 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.812235    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-os-release\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.814729 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.812243    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/770e3b65-661a-4dad-aaa9-5fbb68ae8dea-host-slash\") pod \"iptables-alerter-n5pfx\" (UID: \"770e3b65-661a-4dad-aaa9-5fbb68ae8dea\") " pod="openshift-network-operator/iptables-alerter-n5pfx"
Jul 29 10:28:45.814729 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.812274    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-multus-cni-dir\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.815685 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811613    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b9d340b7-82a4-43bd-96b8-e19532b16b4b-etc-sysconfig\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.815685 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.812051    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-host-run-k8s-cni-cncf-io\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.815685 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.812321    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-log-socket\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.815685 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.812351    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2e6f3639-fa84-4599-8a55-a6d7242cea3a-cnibin\") pod \"multus-additional-cni-plugins-r7xrd\" (UID: \"2e6f3639-fa84-4599-8a55-a6d7242cea3a\") " pod="openshift-multus/multus-additional-cni-plugins-r7xrd"
Jul 29 10:28:45.815685 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.812376    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-multus-cni-dir\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.815685 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.812415    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/770e3b65-661a-4dad-aaa9-5fbb68ae8dea-host-slash\") pod \"iptables-alerter-n5pfx\" (UID: \"770e3b65-661a-4dad-aaa9-5fbb68ae8dea\") " pod="openshift-network-operator/iptables-alerter-n5pfx"
Jul 29 10:28:45.815685 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:45.812497    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:45.815685 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:45.812547    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27e6f470-ad48-45f1-b171-ae66fac19573-original-pull-secret podName:27e6f470-ad48-45f1-b171-ae66fac19573 nodeName:}" failed. No retries permitted until 2026-07-29 10:28:46.312531096 +0000 UTC m=+3.037321767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27e6f470-ad48-45f1-b171-ae66fac19573-original-pull-secret") pod "global-pull-secret-syncer-p8p95" (UID: "27e6f470-ad48-45f1-b171-ae66fac19573") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:45.815685 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.812623    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-host-run-netns\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.815685 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.812683    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-multus-socket-dir-parent\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.815685 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.812725    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-host-run-netns\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.815685 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.812785    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-host-var-lib-cni-multus\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.815685 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.812788    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/90f1789e-de1e-4cc2-a128-ee7621bf80ca-ovnkube-config\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.815685 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.812837    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/faeca467-a198-4dd6-9da8-e4f72ac68c77-hosts-file\") pod \"node-resolver-hpmhx\" (UID: \"faeca467-a198-4dd6-9da8-e4f72ac68c77\") " pod="openshift-dns/node-resolver-hpmhx"
Jul 29 10:28:45.815685 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.812858    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-run-openvswitch\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.815685 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.812997    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/770e3b65-661a-4dad-aaa9-5fbb68ae8dea-iptables-alerter-script\") pod \"iptables-alerter-n5pfx\" (UID: \"770e3b65-661a-4dad-aaa9-5fbb68ae8dea\") " pod="openshift-network-operator/iptables-alerter-n5pfx"
Jul 29 10:28:45.815685 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.813067    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cffc10c9-fe6c-420b-b994-869307922201-etc-kubernetes\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.815685 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.813269    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/90f1789e-de1e-4cc2-a128-ee7621bf80ca-env-overrides\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.816490 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.813374    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b9d340b7-82a4-43bd-96b8-e19532b16b4b-etc-modprobe-d\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.816490 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.811529    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/90f1789e-de1e-4cc2-a128-ee7621bf80ca-host-cni-netd\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.816490 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.813425    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cffc10c9-fe6c-420b-b994-869307922201-cni-binary-copy\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.816490 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.813521    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2e6f3639-fa84-4599-8a55-a6d7242cea3a-cni-binary-copy\") pod \"multus-additional-cni-plugins-r7xrd\" (UID: \"2e6f3639-fa84-4599-8a55-a6d7242cea3a\") " pod="openshift-multus/multus-additional-cni-plugins-r7xrd"
Jul 29 10:28:45.816490 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.814992    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b9d340b7-82a4-43bd-96b8-e19532b16b4b-etc-tuned\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.816490 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.815863    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/90f1789e-de1e-4cc2-a128-ee7621bf80ca-ovn-node-metrics-cert\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.816490 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.815916    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9d340b7-82a4-43bd-96b8-e19532b16b4b-tmp\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.819259 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:45.818583    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:28:45.819259 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:45.818612    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:28:45.819259 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:45.818625    2569 projected.go:196] Error preparing data for projected volume kube-api-access-4d8nk for pod openshift-network-diagnostics/network-check-target-mdqnc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:45.819259 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:45.818676    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7b4bba0-77fe-4240-9858-433b0f431c59-kube-api-access-4d8nk podName:f7b4bba0-77fe-4240-9858-433b0f431c59 nodeName:}" failed. No retries permitted until 2026-07-29 10:28:46.318660957 +0000 UTC m=+3.043451643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4d8nk" (UniqueName: "kubernetes.io/projected/f7b4bba0-77fe-4240-9858-433b0f431c59-kube-api-access-4d8nk") pod "network-check-target-mdqnc" (UID: "f7b4bba0-77fe-4240-9858-433b0f431c59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:45.823091 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.823061    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-468zq\" (UniqueName: \"kubernetes.io/projected/faeca467-a198-4dd6-9da8-e4f72ac68c77-kube-api-access-468zq\") pod \"node-resolver-hpmhx\" (UID: \"faeca467-a198-4dd6-9da8-e4f72ac68c77\") " pod="openshift-dns/node-resolver-hpmhx"
Jul 29 10:28:45.823221 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.823061    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cf5j\" (UniqueName: \"kubernetes.io/projected/b9d340b7-82a4-43bd-96b8-e19532b16b4b-kube-api-access-2cf5j\") pod \"tuned-str4m\" (UID: \"b9d340b7-82a4-43bd-96b8-e19532b16b4b\") " pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:45.823283 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.823258    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr78h\" (UniqueName: \"kubernetes.io/projected/770e3b65-661a-4dad-aaa9-5fbb68ae8dea-kube-api-access-tr78h\") pod \"iptables-alerter-n5pfx\" (UID: \"770e3b65-661a-4dad-aaa9-5fbb68ae8dea\") " pod="openshift-network-operator/iptables-alerter-n5pfx"
Jul 29 10:28:45.823360 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.823339    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc7gs\" (UniqueName: \"kubernetes.io/projected/90f1789e-de1e-4cc2-a128-ee7621bf80ca-kube-api-access-hc7gs\") pod \"ovnkube-node-6jzgn\" (UID: \"90f1789e-de1e-4cc2-a128-ee7621bf80ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:45.823571 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.823547    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mtkc\" (UniqueName: \"kubernetes.io/projected/51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c-kube-api-access-8mtkc\") pod \"network-metrics-daemon-szr8n\" (UID: \"51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c\") " pod="openshift-multus/network-metrics-daemon-szr8n"
Jul 29 10:28:45.824434 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.824413    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxdz8\" (UniqueName: \"kubernetes.io/projected/cffc10c9-fe6c-420b-b994-869307922201-kube-api-access-rxdz8\") pod \"multus-wkrbj\" (UID: \"cffc10c9-fe6c-420b-b994-869307922201\") " pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:45.825631 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.825603    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mntg7\" (UniqueName: \"kubernetes.io/projected/2e6f3639-fa84-4599-8a55-a6d7242cea3a-kube-api-access-mntg7\") pod \"multus-additional-cni-plugins-r7xrd\" (UID: \"2e6f3639-fa84-4599-8a55-a6d7242cea3a\") " pod="openshift-multus/multus-additional-cni-plugins-r7xrd"
Jul 29 10:28:45.837630 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.837493    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-111.ec2.internal" event={"ID":"c1fc06c79a0f992a5584a2e3d3b6fcdb","Type":"ContainerStarted","Data":"e51b7de66e64d33f93f08da4e024cdd05afec061d76eb84ad5f2cde4e9a4dbc0"}
Jul 29 10:28:45.839343 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.839314    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-111.ec2.internal" event={"ID":"e7e9628ca209a194bcb37f10a7461829","Type":"ContainerStarted","Data":"d711c7bf93f5030915a94ac5efdb22ed6cfc608ae630016dc60598eb633a2e07"}
Jul 29 10:28:45.913312 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.913278    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/512bc3c2-b150-4eb6-ba9f-b02810556544-agent-certs\") pod \"konnectivity-agent-6c2hd\" (UID: \"512bc3c2-b150-4eb6-ba9f-b02810556544\") " pod="kube-system/konnectivity-agent-6c2hd"
Jul 29 10:28:45.913510 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.913324    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/372aeb5f-78d6-427f-93a7-97d8db16e8d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xvvkn\" (UID: \"372aeb5f-78d6-427f-93a7-97d8db16e8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvvkn"
Jul 29 10:28:45.913510 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.913351    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/372aeb5f-78d6-427f-93a7-97d8db16e8d0-device-dir\") pod \"aws-ebs-csi-driver-node-xvvkn\" (UID: \"372aeb5f-78d6-427f-93a7-97d8db16e8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvvkn"
Jul 29 10:28:45.913510 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.913376    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/372aeb5f-78d6-427f-93a7-97d8db16e8d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-xvvkn\" (UID: \"372aeb5f-78d6-427f-93a7-97d8db16e8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvvkn"
Jul 29 10:28:45.913510 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.913399    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kst7w\" (UniqueName: \"kubernetes.io/projected/372aeb5f-78d6-427f-93a7-97d8db16e8d0-kube-api-access-kst7w\") pod \"aws-ebs-csi-driver-node-xvvkn\" (UID: \"372aeb5f-78d6-427f-93a7-97d8db16e8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvvkn"
Jul 29 10:28:45.913510 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.913422    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bd747eb7-faeb-48ab-804a-96355d0b79c4-serviceca\") pod \"node-ca-b6v6k\" (UID: \"bd747eb7-faeb-48ab-804a-96355d0b79c4\") " pod="openshift-image-registry/node-ca-b6v6k"
Jul 29 10:28:45.913510 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.913446    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkt22\" (UniqueName: \"kubernetes.io/projected/bd747eb7-faeb-48ab-804a-96355d0b79c4-kube-api-access-jkt22\") pod \"node-ca-b6v6k\" (UID: \"bd747eb7-faeb-48ab-804a-96355d0b79c4\") " pod="openshift-image-registry/node-ca-b6v6k"
Jul 29 10:28:45.913510 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.913474    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/512bc3c2-b150-4eb6-ba9f-b02810556544-konnectivity-ca\") pod \"konnectivity-agent-6c2hd\" (UID: \"512bc3c2-b150-4eb6-ba9f-b02810556544\") " pod="kube-system/konnectivity-agent-6c2hd"
Jul 29 10:28:45.913510 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.913496    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/372aeb5f-78d6-427f-93a7-97d8db16e8d0-registration-dir\") pod \"aws-ebs-csi-driver-node-xvvkn\" (UID: \"372aeb5f-78d6-427f-93a7-97d8db16e8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvvkn"
Jul 29 10:28:45.913896 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.913542    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/372aeb5f-78d6-427f-93a7-97d8db16e8d0-socket-dir\") pod \"aws-ebs-csi-driver-node-xvvkn\" (UID: \"372aeb5f-78d6-427f-93a7-97d8db16e8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvvkn"
Jul 29 10:28:45.913896 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.913570    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/372aeb5f-78d6-427f-93a7-97d8db16e8d0-sys-fs\") pod \"aws-ebs-csi-driver-node-xvvkn\" (UID: \"372aeb5f-78d6-427f-93a7-97d8db16e8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvvkn"
Jul 29 10:28:45.913896 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.913595    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bd747eb7-faeb-48ab-804a-96355d0b79c4-host\") pod \"node-ca-b6v6k\" (UID: \"bd747eb7-faeb-48ab-804a-96355d0b79c4\") " pod="openshift-image-registry/node-ca-b6v6k"
Jul 29 10:28:45.913896 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.913685    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bd747eb7-faeb-48ab-804a-96355d0b79c4-host\") pod \"node-ca-b6v6k\" (UID: \"bd747eb7-faeb-48ab-804a-96355d0b79c4\") " pod="openshift-image-registry/node-ca-b6v6k"
Jul 29 10:28:45.914788 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.914293    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 10:28:45.914788 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.914292    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/372aeb5f-78d6-427f-93a7-97d8db16e8d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-xvvkn\" (UID: \"372aeb5f-78d6-427f-93a7-97d8db16e8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvvkn"
Jul 29 10:28:45.914788 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.914678    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/372aeb5f-78d6-427f-93a7-97d8db16e8d0-socket-dir\") pod \"aws-ebs-csi-driver-node-xvvkn\" (UID: \"372aeb5f-78d6-427f-93a7-97d8db16e8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvvkn"
Jul 29 10:28:45.914788 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.914685    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/372aeb5f-78d6-427f-93a7-97d8db16e8d0-registration-dir\") pod \"aws-ebs-csi-driver-node-xvvkn\" (UID: \"372aeb5f-78d6-427f-93a7-97d8db16e8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvvkn"
Jul 29 10:28:45.914788 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.914741    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/372aeb5f-78d6-427f-93a7-97d8db16e8d0-device-dir\") pod \"aws-ebs-csi-driver-node-xvvkn\" (UID: \"372aeb5f-78d6-427f-93a7-97d8db16e8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvvkn"
Jul 29 10:28:45.914788 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.914741    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/512bc3c2-b150-4eb6-ba9f-b02810556544-konnectivity-ca\") pod \"konnectivity-agent-6c2hd\" (UID: \"512bc3c2-b150-4eb6-ba9f-b02810556544\") " pod="kube-system/konnectivity-agent-6c2hd"
Jul 29 10:28:45.914788 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.914756    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/372aeb5f-78d6-427f-93a7-97d8db16e8d0-sys-fs\") pod \"aws-ebs-csi-driver-node-xvvkn\" (UID: \"372aeb5f-78d6-427f-93a7-97d8db16e8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvvkn"
Jul 29 10:28:45.915269 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.914812    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/372aeb5f-78d6-427f-93a7-97d8db16e8d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xvvkn\" (UID: \"372aeb5f-78d6-427f-93a7-97d8db16e8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvvkn"
Jul 29 10:28:45.915269 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.915161    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bd747eb7-faeb-48ab-804a-96355d0b79c4-serviceca\") pod \"node-ca-b6v6k\" (UID: \"bd747eb7-faeb-48ab-804a-96355d0b79c4\") " pod="openshift-image-registry/node-ca-b6v6k"
Jul 29 10:28:45.917400 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.917349    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/512bc3c2-b150-4eb6-ba9f-b02810556544-agent-certs\") pod \"konnectivity-agent-6c2hd\" (UID: \"512bc3c2-b150-4eb6-ba9f-b02810556544\") " pod="kube-system/konnectivity-agent-6c2hd"
Jul 29 10:28:45.925426 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.925364    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kst7w\" (UniqueName: \"kubernetes.io/projected/372aeb5f-78d6-427f-93a7-97d8db16e8d0-kube-api-access-kst7w\") pod \"aws-ebs-csi-driver-node-xvvkn\" (UID: \"372aeb5f-78d6-427f-93a7-97d8db16e8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvvkn"
Jul 29 10:28:45.925690 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.925665    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkt22\" (UniqueName: \"kubernetes.io/projected/bd747eb7-faeb-48ab-804a-96355d0b79c4-kube-api-access-jkt22\") pod \"node-ca-b6v6k\" (UID: \"bd747eb7-faeb-48ab-804a-96355d0b79c4\") " pod="openshift-image-registry/node-ca-b6v6k"
Jul 29 10:28:45.991307 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:45.991271    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-str4m"
Jul 29 10:28:46.004417 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:46.004384    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wkrbj"
Jul 29 10:28:46.015709 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:46.015684    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hpmhx"
Jul 29 10:28:46.023510 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:46.023489    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-n5pfx"
Jul 29 10:28:46.032728 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:46.032702    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:28:46.041657 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:46.041631    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r7xrd"
Jul 29 10:28:46.051429 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:46.051409    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6c2hd"
Jul 29 10:28:46.061319 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:46.061298    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvvkn"
Jul 29 10:28:46.067428 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:46.067404    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-b6v6k"
Jul 29 10:28:46.317007 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:46.316919    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27e6f470-ad48-45f1-b171-ae66fac19573-original-pull-secret\") pod \"global-pull-secret-syncer-p8p95\" (UID: \"27e6f470-ad48-45f1-b171-ae66fac19573\") " pod="kube-system/global-pull-secret-syncer-p8p95"
Jul 29 10:28:46.317007 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:46.316969    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c-metrics-certs\") pod \"network-metrics-daemon-szr8n\" (UID: \"51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c\") " pod="openshift-multus/network-metrics-daemon-szr8n"
Jul 29 10:28:46.317243 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:46.317082    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:46.317243 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:46.317113    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:46.317243 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:46.317170    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c-metrics-certs podName:51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c nodeName:}" failed. No retries permitted until 2026-07-29 10:28:47.317136455 +0000 UTC m=+4.041927131 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c-metrics-certs") pod "network-metrics-daemon-szr8n" (UID: "51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:46.317243 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:46.317186    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27e6f470-ad48-45f1-b171-ae66fac19573-original-pull-secret podName:27e6f470-ad48-45f1-b171-ae66fac19573 nodeName:}" failed. No retries permitted until 2026-07-29 10:28:47.317178916 +0000 UTC m=+4.041969569 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27e6f470-ad48-45f1-b171-ae66fac19573-original-pull-secret") pod "global-pull-secret-syncer-p8p95" (UID: "27e6f470-ad48-45f1-b171-ae66fac19573") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:46.418282 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:46.418246    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d8nk\" (UniqueName: \"kubernetes.io/projected/f7b4bba0-77fe-4240-9858-433b0f431c59-kube-api-access-4d8nk\") pod \"network-check-target-mdqnc\" (UID: \"f7b4bba0-77fe-4240-9858-433b0f431c59\") " pod="openshift-network-diagnostics/network-check-target-mdqnc"
Jul 29 10:28:46.418502 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:46.418403    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:28:46.418502 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:46.418426    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:28:46.418502 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:46.418441    2569 projected.go:196] Error preparing data for projected volume kube-api-access-4d8nk for pod openshift-network-diagnostics/network-check-target-mdqnc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:46.418662 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:46.418513    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7b4bba0-77fe-4240-9858-433b0f431c59-kube-api-access-4d8nk podName:f7b4bba0-77fe-4240-9858-433b0f431c59 nodeName:}" failed. No retries permitted until 2026-07-29 10:28:47.418491989 +0000 UTC m=+4.143282657 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4d8nk" (UniqueName: "kubernetes.io/projected/f7b4bba0-77fe-4240-9858-433b0f431c59-kube-api-access-4d8nk") pod "network-check-target-mdqnc" (UID: "f7b4bba0-77fe-4240-9858-433b0f431c59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:46.742886 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:46.742822    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 10:23:46 +0000 UTC" deadline="2028-03-03 13:55:19.90410407 +0000 UTC"
Jul 29 10:28:46.742886 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:46.742881    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13995h26m33.161228434s"
Jul 29 10:28:46.831946 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:46.831904    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szr8n"
Jul 29 10:28:46.832167 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:46.832057    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-szr8n" podUID="51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c"
Jul 29 10:28:47.003803 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:47.003723    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaeca467_a198_4dd6_9da8_e4f72ac68c77.slice/crio-fe17bd0612faec441f729968f5facc5bbd0f6c5a7fee547edf05382ab1d1ea26 WatchSource:0}: Error finding container fe17bd0612faec441f729968f5facc5bbd0f6c5a7fee547edf05382ab1d1ea26: Status 404 returned error can't find the container with id fe17bd0612faec441f729968f5facc5bbd0f6c5a7fee547edf05382ab1d1ea26
Jul 29 10:28:47.006508 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:47.006327    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcffc10c9_fe6c_420b_b994_869307922201.slice/crio-517b23a1c05aaac942623db827bb5f9f920dd0443d8bedaebddfec287e1b9592 WatchSource:0}: Error finding container 517b23a1c05aaac942623db827bb5f9f920dd0443d8bedaebddfec287e1b9592: Status 404 returned error can't find the container with id 517b23a1c05aaac942623db827bb5f9f920dd0443d8bedaebddfec287e1b9592
Jul 29 10:28:47.009651 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:47.009627    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d340b7_82a4_43bd_96b8_e19532b16b4b.slice/crio-3545afc485c62e7e2589e078d143275e8b0430704c0b029d5b50daed4982161c WatchSource:0}: Error finding container 3545afc485c62e7e2589e078d143275e8b0430704c0b029d5b50daed4982161c: Status 404 returned error can't find the container with id 3545afc485c62e7e2589e078d143275e8b0430704c0b029d5b50daed4982161c
Jul 29 10:28:47.011006 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:47.010982    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f1789e_de1e_4cc2_a128_ee7621bf80ca.slice/crio-9ffae9f961e6b10d5518725170910435ff7de8a17b571c25f455505c71f7dbd8 WatchSource:0}: Error finding container 9ffae9f961e6b10d5518725170910435ff7de8a17b571c25f455505c71f7dbd8: Status 404 returned error can't find the container with id 9ffae9f961e6b10d5518725170910435ff7de8a17b571c25f455505c71f7dbd8
Jul 29 10:28:47.011724 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:47.011702    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod372aeb5f_78d6_427f_93a7_97d8db16e8d0.slice/crio-97c4260029cb1099133435ff99497dcc40f4f93fb67dd791097ce83a5abcfa42 WatchSource:0}: Error finding container 97c4260029cb1099133435ff99497dcc40f4f93fb67dd791097ce83a5abcfa42: Status 404 returned error can't find the container with id 97c4260029cb1099133435ff99497dcc40f4f93fb67dd791097ce83a5abcfa42
Jul 29 10:28:47.013024 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:47.012914    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod770e3b65_661a_4dad_aaa9_5fbb68ae8dea.slice/crio-edf2fea3f489bd580803cff8e729c1fbdeb969811f49ef79e651c6fac3d99ab9 WatchSource:0}: Error finding container edf2fea3f489bd580803cff8e729c1fbdeb969811f49ef79e651c6fac3d99ab9: Status 404 returned error can't find the container with id edf2fea3f489bd580803cff8e729c1fbdeb969811f49ef79e651c6fac3d99ab9
Jul 29 10:28:47.013844 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:47.013822    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e6f3639_fa84_4599_8a55_a6d7242cea3a.slice/crio-f6282f949912a926ddb0c892450d0cb4740b197184b5ea9e488637ef1461efaf WatchSource:0}: Error finding container f6282f949912a926ddb0c892450d0cb4740b197184b5ea9e488637ef1461efaf: Status 404 returned error can't find the container with id f6282f949912a926ddb0c892450d0cb4740b197184b5ea9e488637ef1461efaf
Jul 29 10:28:47.015182 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:28:47.015060    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd747eb7_faeb_48ab_804a_96355d0b79c4.slice/crio-30bc6833becbeff4bb19190b3fbc4e70409e2a6715dc73f08f36efa9d54cdbf9 WatchSource:0}: Error finding container 30bc6833becbeff4bb19190b3fbc4e70409e2a6715dc73f08f36efa9d54cdbf9: Status 404 returned error can't find the container with id 30bc6833becbeff4bb19190b3fbc4e70409e2a6715dc73f08f36efa9d54cdbf9
Jul 29 10:28:47.325782 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:47.325574    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c-metrics-certs\") pod \"network-metrics-daemon-szr8n\" (UID: \"51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c\") " pod="openshift-multus/network-metrics-daemon-szr8n"
Jul 29 10:28:47.325966 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:47.325733    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:47.325966 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:47.325851    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27e6f470-ad48-45f1-b171-ae66fac19573-original-pull-secret\") pod \"global-pull-secret-syncer-p8p95\" (UID: \"27e6f470-ad48-45f1-b171-ae66fac19573\") " pod="kube-system/global-pull-secret-syncer-p8p95"
Jul 29 10:28:47.325966 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:47.325907    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c-metrics-certs podName:51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c nodeName:}" failed. No retries permitted until 2026-07-29 10:28:49.325889579 +0000 UTC m=+6.050680231 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c-metrics-certs") pod "network-metrics-daemon-szr8n" (UID: "51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:47.326141 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:47.325967    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:47.326141 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:47.326030    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27e6f470-ad48-45f1-b171-ae66fac19573-original-pull-secret podName:27e6f470-ad48-45f1-b171-ae66fac19573 nodeName:}" failed. No retries permitted until 2026-07-29 10:28:49.326013136 +0000 UTC m=+6.050803790 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27e6f470-ad48-45f1-b171-ae66fac19573-original-pull-secret") pod "global-pull-secret-syncer-p8p95" (UID: "27e6f470-ad48-45f1-b171-ae66fac19573") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:47.426828 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:47.426786    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d8nk\" (UniqueName: \"kubernetes.io/projected/f7b4bba0-77fe-4240-9858-433b0f431c59-kube-api-access-4d8nk\") pod \"network-check-target-mdqnc\" (UID: \"f7b4bba0-77fe-4240-9858-433b0f431c59\") " pod="openshift-network-diagnostics/network-check-target-mdqnc"
Jul 29 10:28:47.427015 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:47.426995    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:28:47.427082 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:47.427031    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:28:47.427082 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:47.427048    2569 projected.go:196] Error preparing data for projected volume kube-api-access-4d8nk for pod openshift-network-diagnostics/network-check-target-mdqnc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:47.427227 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:47.427137    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7b4bba0-77fe-4240-9858-433b0f431c59-kube-api-access-4d8nk podName:f7b4bba0-77fe-4240-9858-433b0f431c59 nodeName:}" failed. No retries permitted until 2026-07-29 10:28:49.427114769 +0000 UTC m=+6.151905429 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4d8nk" (UniqueName: "kubernetes.io/projected/f7b4bba0-77fe-4240-9858-433b0f431c59-kube-api-access-4d8nk") pod "network-check-target-mdqnc" (UID: "f7b4bba0-77fe-4240-9858-433b0f431c59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:47.743206 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:47.743118    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 10:23:46 +0000 UTC" deadline="2028-03-05 16:25:25.367414423 +0000 UTC"
Jul 29 10:28:47.743206 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:47.743161    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14045h56m37.624258829s"
Jul 29 10:28:47.833965 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:47.833307    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8p95"
Jul 29 10:28:47.833965 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:47.833457    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8p95" podUID="27e6f470-ad48-45f1-b171-ae66fac19573"
Jul 29 10:28:47.833965 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:47.833504    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdqnc"
Jul 29 10:28:47.833965 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:47.833589    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdqnc" podUID="f7b4bba0-77fe-4240-9858-433b0f431c59"
Jul 29 10:28:47.849924 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:47.849893    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-b6v6k" event={"ID":"bd747eb7-faeb-48ab-804a-96355d0b79c4","Type":"ContainerStarted","Data":"30bc6833becbeff4bb19190b3fbc4e70409e2a6715dc73f08f36efa9d54cdbf9"}
Jul 29 10:28:47.854565 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:47.854526    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r7xrd" event={"ID":"2e6f3639-fa84-4599-8a55-a6d7242cea3a","Type":"ContainerStarted","Data":"f6282f949912a926ddb0c892450d0cb4740b197184b5ea9e488637ef1461efaf"}
Jul 29 10:28:47.860852 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:47.860825    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-n5pfx" event={"ID":"770e3b65-661a-4dad-aaa9-5fbb68ae8dea","Type":"ContainerStarted","Data":"edf2fea3f489bd580803cff8e729c1fbdeb969811f49ef79e651c6fac3d99ab9"}
Jul 29 10:28:47.862877 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:47.862754    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvvkn" event={"ID":"372aeb5f-78d6-427f-93a7-97d8db16e8d0","Type":"ContainerStarted","Data":"97c4260029cb1099133435ff99497dcc40f4f93fb67dd791097ce83a5abcfa42"}
Jul 29 10:28:47.864205 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:47.864138    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-str4m" event={"ID":"b9d340b7-82a4-43bd-96b8-e19532b16b4b","Type":"ContainerStarted","Data":"3545afc485c62e7e2589e078d143275e8b0430704c0b029d5b50daed4982161c"}
Jul 29 10:28:47.865641 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:47.865554    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wkrbj" event={"ID":"cffc10c9-fe6c-420b-b994-869307922201","Type":"ContainerStarted","Data":"517b23a1c05aaac942623db827bb5f9f920dd0443d8bedaebddfec287e1b9592"}
Jul 29 10:28:47.867758 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:47.867686    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hpmhx" event={"ID":"faeca467-a198-4dd6-9da8-e4f72ac68c77","Type":"ContainerStarted","Data":"fe17bd0612faec441f729968f5facc5bbd0f6c5a7fee547edf05382ab1d1ea26"}
Jul 29 10:28:47.873024 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:47.872297    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-111.ec2.internal" event={"ID":"e7e9628ca209a194bcb37f10a7461829","Type":"ContainerStarted","Data":"b81982306aa8db7d673cf138e3510cc6d022f5b684211d21bd505c7feebcd2f1"}
Jul 29 10:28:47.878518 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:47.878483    2569 generic.go:356] "Generic (PLEG): container finished" podID="c1fc06c79a0f992a5584a2e3d3b6fcdb" containerID="a275775ba320fc4cf2cbb9277cbfa8353028b16c6aed9b96e54b305e3d4d49be" exitCode=0
Jul 29 10:28:47.878619 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:47.878580    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-111.ec2.internal" event={"ID":"c1fc06c79a0f992a5584a2e3d3b6fcdb","Type":"ContainerDied","Data":"a275775ba320fc4cf2cbb9277cbfa8353028b16c6aed9b96e54b305e3d4d49be"}
Jul 29 10:28:47.881212 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:47.881185    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn" event={"ID":"90f1789e-de1e-4cc2-a128-ee7621bf80ca","Type":"ContainerStarted","Data":"9ffae9f961e6b10d5518725170910435ff7de8a17b571c25f455505c71f7dbd8"}
Jul 29 10:28:47.884397 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:47.884326    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6c2hd" event={"ID":"512bc3c2-b150-4eb6-ba9f-b02810556544","Type":"ContainerStarted","Data":"55bf1fe71a7fb59bb4d46efd567f8d5551b1ab802bfefd83a8c4ab5431bafe39"}
Jul 29 10:28:47.892178 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:47.892111    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-111.ec2.internal" podStartSLOduration=1.89209577 podStartE2EDuration="1.89209577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:28:47.891162455 +0000 UTC m=+4.615953123" watchObservedRunningTime="2026-07-29 10:28:47.89209577 +0000 UTC m=+4.616886451"
Jul 29 10:28:48.832865 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:48.832334    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szr8n"
Jul 29 10:28:48.832865 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:48.832480    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-szr8n" podUID="51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c"
Jul 29 10:28:48.904003 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:48.903801    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-111.ec2.internal" event={"ID":"c1fc06c79a0f992a5584a2e3d3b6fcdb","Type":"ContainerStarted","Data":"892d8844a13fc10b7584739e6d9753eaaa9cd5264edc5012d83e967cec67f505"}
Jul 29 10:28:49.344569 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:49.344527    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27e6f470-ad48-45f1-b171-ae66fac19573-original-pull-secret\") pod \"global-pull-secret-syncer-p8p95\" (UID: \"27e6f470-ad48-45f1-b171-ae66fac19573\") " pod="kube-system/global-pull-secret-syncer-p8p95"
Jul 29 10:28:49.344788 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:49.344595    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c-metrics-certs\") pod \"network-metrics-daemon-szr8n\" (UID: \"51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c\") " pod="openshift-multus/network-metrics-daemon-szr8n"
Jul 29 10:28:49.344788 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:49.344751    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:49.344919 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:49.344818    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c-metrics-certs podName:51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c nodeName:}" failed. No retries permitted until 2026-07-29 10:28:53.344798557 +0000 UTC m=+10.069589210 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c-metrics-certs") pod "network-metrics-daemon-szr8n" (UID: "51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:49.345299 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:49.345274    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:49.345421 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:49.345334    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27e6f470-ad48-45f1-b171-ae66fac19573-original-pull-secret podName:27e6f470-ad48-45f1-b171-ae66fac19573 nodeName:}" failed. No retries permitted until 2026-07-29 10:28:53.345317269 +0000 UTC m=+10.070107925 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27e6f470-ad48-45f1-b171-ae66fac19573-original-pull-secret") pod "global-pull-secret-syncer-p8p95" (UID: "27e6f470-ad48-45f1-b171-ae66fac19573") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:49.445943 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:49.445888    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d8nk\" (UniqueName: \"kubernetes.io/projected/f7b4bba0-77fe-4240-9858-433b0f431c59-kube-api-access-4d8nk\") pod \"network-check-target-mdqnc\" (UID: \"f7b4bba0-77fe-4240-9858-433b0f431c59\") " pod="openshift-network-diagnostics/network-check-target-mdqnc"
Jul 29 10:28:49.446169 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:49.446088    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:28:49.446169 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:49.446132    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:28:49.446169 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:49.446149    2569 projected.go:196] Error preparing data for projected volume kube-api-access-4d8nk for pod openshift-network-diagnostics/network-check-target-mdqnc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:49.446347 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:49.446225    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7b4bba0-77fe-4240-9858-433b0f431c59-kube-api-access-4d8nk podName:f7b4bba0-77fe-4240-9858-433b0f431c59 nodeName:}" failed. No retries permitted until 2026-07-29 10:28:53.446205116 +0000 UTC m=+10.170995792 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4d8nk" (UniqueName: "kubernetes.io/projected/f7b4bba0-77fe-4240-9858-433b0f431c59-kube-api-access-4d8nk") pod "network-check-target-mdqnc" (UID: "f7b4bba0-77fe-4240-9858-433b0f431c59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:49.835156 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:49.834270    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8p95"
Jul 29 10:28:49.835156 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:49.834414    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8p95" podUID="27e6f470-ad48-45f1-b171-ae66fac19573"
Jul 29 10:28:49.835156 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:49.834831    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdqnc"
Jul 29 10:28:49.835156 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:49.834930    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdqnc" podUID="f7b4bba0-77fe-4240-9858-433b0f431c59"
Jul 29 10:28:50.832534 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:50.832284    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szr8n"
Jul 29 10:28:50.832534 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:50.832457    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-szr8n" podUID="51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c"
Jul 29 10:28:51.832189 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:51.832153    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8p95"
Jul 29 10:28:51.832728 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:51.832311    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8p95" podUID="27e6f470-ad48-45f1-b171-ae66fac19573"
Jul 29 10:28:51.832854 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:51.832831    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdqnc"
Jul 29 10:28:51.832994 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:51.832951    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdqnc" podUID="f7b4bba0-77fe-4240-9858-433b0f431c59"
Jul 29 10:28:52.831873 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:52.831834    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szr8n"
Jul 29 10:28:52.832079 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:52.831991    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-szr8n" podUID="51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c"
Jul 29 10:28:53.382811 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:53.382772    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27e6f470-ad48-45f1-b171-ae66fac19573-original-pull-secret\") pod \"global-pull-secret-syncer-p8p95\" (UID: \"27e6f470-ad48-45f1-b171-ae66fac19573\") " pod="kube-system/global-pull-secret-syncer-p8p95"
Jul 29 10:28:53.383335 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:53.382838    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c-metrics-certs\") pod \"network-metrics-daemon-szr8n\" (UID: \"51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c\") " pod="openshift-multus/network-metrics-daemon-szr8n"
Jul 29 10:28:53.383335 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:53.382883    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:53.383335 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:53.382945    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:53.383335 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:53.382959    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27e6f470-ad48-45f1-b171-ae66fac19573-original-pull-secret podName:27e6f470-ad48-45f1-b171-ae66fac19573 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:01.38293986 +0000 UTC m=+18.107730536 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27e6f470-ad48-45f1-b171-ae66fac19573-original-pull-secret") pod "global-pull-secret-syncer-p8p95" (UID: "27e6f470-ad48-45f1-b171-ae66fac19573") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:53.383335 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:53.382997    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c-metrics-certs podName:51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c nodeName:}" failed. No retries permitted until 2026-07-29 10:29:01.38297981 +0000 UTC m=+18.107770468 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c-metrics-certs") pod "network-metrics-daemon-szr8n" (UID: "51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:53.483608 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:53.483560    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d8nk\" (UniqueName: \"kubernetes.io/projected/f7b4bba0-77fe-4240-9858-433b0f431c59-kube-api-access-4d8nk\") pod \"network-check-target-mdqnc\" (UID: \"f7b4bba0-77fe-4240-9858-433b0f431c59\") " pod="openshift-network-diagnostics/network-check-target-mdqnc"
Jul 29 10:28:53.483807 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:53.483757    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:28:53.483807 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:53.483778    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:28:53.483807 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:53.483791    2569 projected.go:196] Error preparing data for projected volume kube-api-access-4d8nk for pod openshift-network-diagnostics/network-check-target-mdqnc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:53.483952 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:53.483858    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7b4bba0-77fe-4240-9858-433b0f431c59-kube-api-access-4d8nk podName:f7b4bba0-77fe-4240-9858-433b0f431c59 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:01.483839567 +0000 UTC m=+18.208630223 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4d8nk" (UniqueName: "kubernetes.io/projected/f7b4bba0-77fe-4240-9858-433b0f431c59-kube-api-access-4d8nk") pod "network-check-target-mdqnc" (UID: "f7b4bba0-77fe-4240-9858-433b0f431c59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:53.833489 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:53.833000    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8p95"
Jul 29 10:28:53.833489 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:53.833117    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8p95" podUID="27e6f470-ad48-45f1-b171-ae66fac19573"
Jul 29 10:28:53.833489 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:53.833267    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdqnc"
Jul 29 10:28:53.833489 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:53.833393    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdqnc" podUID="f7b4bba0-77fe-4240-9858-433b0f431c59"
Jul 29 10:28:54.832673 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:54.832250    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szr8n"
Jul 29 10:28:54.832673 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:54.832415    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-szr8n" podUID="51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c"
Jul 29 10:28:55.832812 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:55.832686    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8p95"
Jul 29 10:28:55.832812 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:55.832692    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdqnc"
Jul 29 10:28:55.833357 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:55.832831    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8p95" podUID="27e6f470-ad48-45f1-b171-ae66fac19573"
Jul 29 10:28:55.833357 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:55.833298    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdqnc" podUID="f7b4bba0-77fe-4240-9858-433b0f431c59"
Jul 29 10:28:56.832334 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:56.831882    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szr8n"
Jul 29 10:28:56.832334 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:56.831997    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-szr8n" podUID="51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c"
Jul 29 10:28:57.832872 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:57.832825    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdqnc"
Jul 29 10:28:57.833334 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:57.832825    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8p95"
Jul 29 10:28:57.833334 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:57.833008    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdqnc" podUID="f7b4bba0-77fe-4240-9858-433b0f431c59"
Jul 29 10:28:57.833334 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:57.833054    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8p95" podUID="27e6f470-ad48-45f1-b171-ae66fac19573"
Jul 29 10:28:58.832592 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:58.832553    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szr8n"
Jul 29 10:28:58.832773 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:58.832700    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-szr8n" podUID="51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c"
Jul 29 10:28:59.832256 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:59.832224    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8p95"
Jul 29 10:28:59.832671 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:59.832356    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8p95" podUID="27e6f470-ad48-45f1-b171-ae66fac19573"
Jul 29 10:28:59.832671 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:28:59.832387    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdqnc"
Jul 29 10:28:59.832671 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:28:59.832463    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdqnc" podUID="f7b4bba0-77fe-4240-9858-433b0f431c59"
Jul 29 10:29:00.832660 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:00.832621    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szr8n"
Jul 29 10:29:00.833193 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:00.832771    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-szr8n" podUID="51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c"
Jul 29 10:29:01.446988 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:01.446945    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c-metrics-certs\") pod \"network-metrics-daemon-szr8n\" (UID: \"51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c\") " pod="openshift-multus/network-metrics-daemon-szr8n"
Jul 29 10:29:01.447207 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:01.447049    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27e6f470-ad48-45f1-b171-ae66fac19573-original-pull-secret\") pod \"global-pull-secret-syncer-p8p95\" (UID: \"27e6f470-ad48-45f1-b171-ae66fac19573\") " pod="kube-system/global-pull-secret-syncer-p8p95"
Jul 29 10:29:01.447207 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:01.447162    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:29:01.447207 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:01.447194    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:29:01.447395 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:01.447245    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c-metrics-certs podName:51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c nodeName:}" failed. No retries permitted until 2026-07-29 10:29:17.44722279 +0000 UTC m=+34.172013459 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c-metrics-certs") pod "network-metrics-daemon-szr8n" (UID: "51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:29:01.447395 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:01.447267    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27e6f470-ad48-45f1-b171-ae66fac19573-original-pull-secret podName:27e6f470-ad48-45f1-b171-ae66fac19573 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:17.447255738 +0000 UTC m=+34.172046396 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27e6f470-ad48-45f1-b171-ae66fac19573-original-pull-secret") pod "global-pull-secret-syncer-p8p95" (UID: "27e6f470-ad48-45f1-b171-ae66fac19573") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:29:01.547902 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:01.547856    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d8nk\" (UniqueName: \"kubernetes.io/projected/f7b4bba0-77fe-4240-9858-433b0f431c59-kube-api-access-4d8nk\") pod \"network-check-target-mdqnc\" (UID: \"f7b4bba0-77fe-4240-9858-433b0f431c59\") " pod="openshift-network-diagnostics/network-check-target-mdqnc"
Jul 29 10:29:01.550449 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:01.548397    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:29:01.550449 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:01.548437    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:29:01.550449 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:01.548454    2569 projected.go:196] Error preparing data for projected volume kube-api-access-4d8nk for pod openshift-network-diagnostics/network-check-target-mdqnc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:29:01.550449 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:01.548534    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7b4bba0-77fe-4240-9858-433b0f431c59-kube-api-access-4d8nk podName:f7b4bba0-77fe-4240-9858-433b0f431c59 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:17.548508422 +0000 UTC m=+34.273299076 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4d8nk" (UniqueName: "kubernetes.io/projected/f7b4bba0-77fe-4240-9858-433b0f431c59-kube-api-access-4d8nk") pod "network-check-target-mdqnc" (UID: "f7b4bba0-77fe-4240-9858-433b0f431c59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:29:01.832954 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:01.832870    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdqnc"
Jul 29 10:29:01.833435 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:01.832870    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8p95"
Jul 29 10:29:01.833435 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:01.833007    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdqnc" podUID="f7b4bba0-77fe-4240-9858-433b0f431c59"
Jul 29 10:29:01.833435 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:01.833113    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8p95" podUID="27e6f470-ad48-45f1-b171-ae66fac19573"
Jul 29 10:29:02.832123 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:02.832084    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szr8n"
Jul 29 10:29:02.832327 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:02.832224    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-szr8n" podUID="51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c"
Jul 29 10:29:03.832695 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:03.832664    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8p95"
Jul 29 10:29:03.833233 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:03.832769    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8p95" podUID="27e6f470-ad48-45f1-b171-ae66fac19573"
Jul 29 10:29:03.833233 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:03.832879    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdqnc"
Jul 29 10:29:03.833233 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:03.833004    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdqnc" podUID="f7b4bba0-77fe-4240-9858-433b0f431c59"
Jul 29 10:29:04.832750 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:04.832708    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szr8n"
Jul 29 10:29:04.833168 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:04.832872    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-szr8n" podUID="51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c"
Jul 29 10:29:05.832724 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:05.832697    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8p95"
Jul 29 10:29:05.832871 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:05.832820    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8p95" podUID="27e6f470-ad48-45f1-b171-ae66fac19573"
Jul 29 10:29:05.833236 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:05.833181    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdqnc"
Jul 29 10:29:05.833295 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:05.833267    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdqnc" podUID="f7b4bba0-77fe-4240-9858-433b0f431c59"
Jul 29 10:29:05.944472 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:05.944229    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6c2hd" event={"ID":"512bc3c2-b150-4eb6-ba9f-b02810556544","Type":"ContainerStarted","Data":"4270f1c9fc7644b136345a6b16a801115d844e39e594921618901cb3ec12b6e2"}
Jul 29 10:29:05.946121 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:05.946066    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-b6v6k" event={"ID":"bd747eb7-faeb-48ab-804a-96355d0b79c4","Type":"ContainerStarted","Data":"da77943112d78c4c262f2275166719da72a0d49e22dce9b1d58c7b0986fc2c23"}
Jul 29 10:29:05.948179 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:05.948091    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvvkn" event={"ID":"372aeb5f-78d6-427f-93a7-97d8db16e8d0","Type":"ContainerStarted","Data":"d6c5e00112de0b6d9df72d9acc823c21c06fe566ad1452c53276e6c8af2caf63"}
Jul 29 10:29:05.950214 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:05.950066    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-str4m" event={"ID":"b9d340b7-82a4-43bd-96b8-e19532b16b4b","Type":"ContainerStarted","Data":"587abe347c1605ad8ac15712de07b5e6c990348fb9fae51881a3397d7e9fb73a"}
Jul 29 10:29:05.951868 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:05.951847    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wkrbj" event={"ID":"cffc10c9-fe6c-420b-b994-869307922201","Type":"ContainerStarted","Data":"c9651fdcd1502daad324843f94bf4c26a2adc9402cc46cdd2954108fd17f3aab"}
Jul 29 10:29:05.960296 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:05.960253    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6c2hd" podStartSLOduration=2.222649296 podStartE2EDuration="20.960239157s" totalImagesPullingTime="18.737589861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:28:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:05.959529018 +0000 UTC m=+22.684319690" watchObservedRunningTime="2026-07-29 10:29:05.960239157 +0000 UTC m=+22.685029830"
Jul 29 10:29:05.960429 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:05.960407    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-111.ec2.internal" podStartSLOduration=19.9604023 podStartE2EDuration="19.9604023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:28:48.928482932 +0000 UTC m=+5.653273592" watchObservedRunningTime="2026-07-29 10:29:05.9604023 +0000 UTC m=+22.685192973"
Jul 29 10:29:05.976476 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:05.976434    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wkrbj" podStartSLOduration=3.164212867 podStartE2EDuration="21.976419789s" totalImagesPullingTime="18.812206922s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:05.975964605 +0000 UTC m=+22.700755278" watchObservedRunningTime="2026-07-29 10:29:05.976419789 +0000 UTC m=+22.701210463"
Jul 29 10:29:05.994879 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:05.994809    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-str4m" podStartSLOduration=3.218165947 podStartE2EDuration="21.994786908s" totalImagesPullingTime="18.776620961s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:05.994336491 +0000 UTC m=+22.719127166" watchObservedRunningTime="2026-07-29 10:29:05.994786908 +0000 UTC m=+22.719577579"
Jul 29 10:29:06.010613 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:06.010465    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-b6v6k" podStartSLOduration=5.950701505 podStartE2EDuration="21.010441734s" totalImagesPullingTime="15.059740229s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:06.010098852 +0000 UTC m=+22.734889560" watchObservedRunningTime="2026-07-29 10:29:06.010441734 +0000 UTC m=+22.735232408"
Jul 29 10:29:06.832117 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:06.832064    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szr8n"
Jul 29 10:29:06.832332 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:06.832203    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-szr8n" podUID="51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c"
Jul 29 10:29:06.955630 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:06.955591    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hpmhx" event={"ID":"faeca467-a198-4dd6-9da8-e4f72ac68c77","Type":"ContainerStarted","Data":"0a0aa0eef5503a61d13d6509afda9a4db47538d2e6bf67e4c3dad2daf3dfcac5"}
Jul 29 10:29:06.958607 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:06.958580    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn" event={"ID":"90f1789e-de1e-4cc2-a128-ee7621bf80ca","Type":"ContainerStarted","Data":"87bac1af8d104f4dc3b139592e8a76646f76c4ba6201ba63b0c12cf70c552e4b"}
Jul 29 10:29:06.958715 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:06.958644    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn" event={"ID":"90f1789e-de1e-4cc2-a128-ee7621bf80ca","Type":"ContainerStarted","Data":"24d53dbac70b3bcdcb60910f48abd41115bc48aa33a3c564fd40b8e93474908c"}
Jul 29 10:29:06.958715 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:06.958676    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn" event={"ID":"90f1789e-de1e-4cc2-a128-ee7621bf80ca","Type":"ContainerStarted","Data":"4314271634e2e824c2614429743268a0c053bc7af72b0ca3f611f1c2bc487065"}
Jul 29 10:29:06.958799 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:06.958716    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn" event={"ID":"90f1789e-de1e-4cc2-a128-ee7621bf80ca","Type":"ContainerStarted","Data":"c7a06d0289d2f89504673339bbcc9c07aed0078eea0b383cb46659758e2ca228"}
Jul 29 10:29:06.958799 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:06.958749    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn" event={"ID":"90f1789e-de1e-4cc2-a128-ee7621bf80ca","Type":"ContainerStarted","Data":"8c167c0c0a42cad2d57f3c48dacfab141384cae4d43dd123e22ad69c531a13ed"}
Jul 29 10:29:06.958897 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:06.958830    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn" event={"ID":"90f1789e-de1e-4cc2-a128-ee7621bf80ca","Type":"ContainerStarted","Data":"b85edf20d779c13e215855bd5eefc0207f4d9b1f3d11c1434c3c89dbebbe7358"}
Jul 29 10:29:06.960024 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:06.960003    2569 generic.go:356] "Generic (PLEG): container finished" podID="2e6f3639-fa84-4599-8a55-a6d7242cea3a" containerID="b4e298c4a59dba8794d4cd38a7f8a217ba0a2f8f778b9e998a90e1eb430662b6" exitCode=0
Jul 29 10:29:06.960108 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:06.960090    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r7xrd" event={"ID":"2e6f3639-fa84-4599-8a55-a6d7242cea3a","Type":"ContainerDied","Data":"b4e298c4a59dba8794d4cd38a7f8a217ba0a2f8f778b9e998a90e1eb430662b6"}
Jul 29 10:29:06.973468 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:06.973435    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hpmhx" podStartSLOduration=4.222339806 podStartE2EDuration="22.973423394s" totalImagesPullingTime="18.751083588s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:06.972770145 +0000 UTC m=+23.697560819" watchObservedRunningTime="2026-07-29 10:29:06.973423394 +0000 UTC m=+23.698214061"
Jul 29 10:29:07.125443 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:07.125413    2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 29 10:29:07.646573 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:07.646537    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6c2hd"
Jul 29 10:29:07.647242 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:07.647218    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6c2hd"
Jul 29 10:29:07.777787 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:07.777619    2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T10:29:07.125439114Z","UUID":"3ea7ae1e-c06b-466c-8f5a-c8ff766f2c0f","Handler":null,"Name":"","Endpoint":""}
Jul 29 10:29:07.779256 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:07.779232    2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 29 10:29:07.779389 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:07.779277    2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 29 10:29:07.832328 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:07.832295    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdqnc"
Jul 29 10:29:07.832511 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:07.832335    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8p95"
Jul 29 10:29:07.832511 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:07.832426    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdqnc" podUID="f7b4bba0-77fe-4240-9858-433b0f431c59"
Jul 29 10:29:07.833013 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:07.832762    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8p95" podUID="27e6f470-ad48-45f1-b171-ae66fac19573"
Jul 29 10:29:07.964683 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:07.964632    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-n5pfx" event={"ID":"770e3b65-661a-4dad-aaa9-5fbb68ae8dea","Type":"ContainerStarted","Data":"3d47b09275cf55a04b8c53b9a9af3c6815918b1b3fd2193fdd5a9e907e11e502"}
Jul 29 10:29:07.967044 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:07.966977    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvvkn" event={"ID":"372aeb5f-78d6-427f-93a7-97d8db16e8d0","Type":"ContainerStarted","Data":"832e4fa56be43f9b35afe0ad91a26f8e2855046d2029824a7b77f785b49c8491"}
Jul 29 10:29:07.980134 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:07.980062    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-n5pfx" podStartSLOduration=4.232471872 podStartE2EDuration="22.980044588s" totalImagesPullingTime="18.747572716s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:07.979955975 +0000 UTC m=+24.704746650" watchObservedRunningTime="2026-07-29 10:29:07.980044588 +0000 UTC m=+24.704835254"
Jul 29 10:29:08.832552 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:08.832477    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szr8n"
Jul 29 10:29:08.832777 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:08.832607    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-szr8n" podUID="51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c"
Jul 29 10:29:08.972796 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:08.972755    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn" event={"ID":"90f1789e-de1e-4cc2-a128-ee7621bf80ca","Type":"ContainerStarted","Data":"e676fded1e6d0fcca33e2b4d41d288b0858c1dd25e0078aa8609066c51ce4daf"}
Jul 29 10:29:08.974902 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:08.974874    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvvkn" event={"ID":"372aeb5f-78d6-427f-93a7-97d8db16e8d0","Type":"ContainerStarted","Data":"85cad90c666503db420b5427585b991d682e7b9338346036c1bf02254ce4c9e2"}
Jul 29 10:29:08.975037 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:08.974909    2569 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 29 10:29:08.994793 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:08.994745    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvvkn" podStartSLOduration=3.015298249 podStartE2EDuration="23.994724634s" totalImagesPullingTime="20.979426385s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:08.993912563 +0000 UTC m=+25.718703236" watchObservedRunningTime="2026-07-29 10:29:08.994724634 +0000 UTC m=+25.719515308"
Jul 29 10:29:09.240764 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:09.240731    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6c2hd"
Jul 29 10:29:09.241449 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:09.241400    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6c2hd"
Jul 29 10:29:09.832259 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:09.832080    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdqnc"
Jul 29 10:29:09.832448 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:09.832122    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8p95"
Jul 29 10:29:09.832448 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:09.832371    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdqnc" podUID="f7b4bba0-77fe-4240-9858-433b0f431c59"
Jul 29 10:29:09.832448 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:09.832407    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8p95" podUID="27e6f470-ad48-45f1-b171-ae66fac19573"
Jul 29 10:29:10.832756 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:10.832727    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szr8n"
Jul 29 10:29:10.833165 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:10.832837    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-szr8n" podUID="51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c"
Jul 29 10:29:11.832542 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:11.832277    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8p95"
Jul 29 10:29:11.832710 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:11.832337    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdqnc"
Jul 29 10:29:11.832710 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:11.832658    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8p95" podUID="27e6f470-ad48-45f1-b171-ae66fac19573"
Jul 29 10:29:11.832778 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:11.832707    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdqnc" podUID="f7b4bba0-77fe-4240-9858-433b0f431c59"
Jul 29 10:29:11.983678 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:11.983646    2569 generic.go:356] "Generic (PLEG): container finished" podID="2e6f3639-fa84-4599-8a55-a6d7242cea3a" containerID="7015ed841bb19d08ea7c2e571f09a00778d058f5cde253e9b785395decde23cd" exitCode=0
Jul 29 10:29:11.984601 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:11.983736    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r7xrd" event={"ID":"2e6f3639-fa84-4599-8a55-a6d7242cea3a","Type":"ContainerDied","Data":"7015ed841bb19d08ea7c2e571f09a00778d058f5cde253e9b785395decde23cd"}
Jul 29 10:29:11.987446 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:11.987419    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn" event={"ID":"90f1789e-de1e-4cc2-a128-ee7621bf80ca","Type":"ContainerStarted","Data":"aa9ab13dc4ce3110e7da5b9d054249ba1d37caa9048bea033b7d967ffe9543c1"}
Jul 29 10:29:11.987711 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:11.987690    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:29:12.002727 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:12.002696    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:29:12.038523 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:12.038477    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn" podStartSLOduration=8.227436089 podStartE2EDuration="27.038458656s" totalImagesPullingTime="18.811022567s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:12.03781354 +0000 UTC m=+28.762604213" watchObservedRunningTime="2026-07-29 10:29:12.038458656 +0000 UTC m=+28.763249329"
Jul 29 10:29:12.417715 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:12.417678    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:29:12.832035 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:12.832002    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szr8n"
Jul 29 10:29:12.832194 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:12.832134    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-szr8n" podUID="51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c"
Jul 29 10:29:12.989970 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:12.989935    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:29:13.005191 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:13.005164    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:29:13.393699 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:13.393666    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p8p95"]
Jul 29 10:29:13.393866 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:13.393805    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8p95"
Jul 29 10:29:13.393921 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:13.393900    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8p95" podUID="27e6f470-ad48-45f1-b171-ae66fac19573"
Jul 29 10:29:13.397089 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:13.397043    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-szr8n"]
Jul 29 10:29:13.397244 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:13.397172    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szr8n"
Jul 29 10:29:13.397309 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:13.397281    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-szr8n" podUID="51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c"
Jul 29 10:29:13.397735 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:13.397710    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mdqnc"]
Jul 29 10:29:13.397846 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:13.397804    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdqnc"
Jul 29 10:29:13.397908 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:13.397883    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdqnc" podUID="f7b4bba0-77fe-4240-9858-433b0f431c59"
Jul 29 10:29:13.993132 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:13.993101    2569 generic.go:356] "Generic (PLEG): container finished" podID="2e6f3639-fa84-4599-8a55-a6d7242cea3a" containerID="0ebec5ae943aaf8412b7b6b9764a0152aede955aad6bf950cd19bb051641dc93" exitCode=0
Jul 29 10:29:13.993485 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:13.993194    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r7xrd" event={"ID":"2e6f3639-fa84-4599-8a55-a6d7242cea3a","Type":"ContainerDied","Data":"0ebec5ae943aaf8412b7b6b9764a0152aede955aad6bf950cd19bb051641dc93"}
Jul 29 10:29:14.832225 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:14.832198    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdqnc"
Jul 29 10:29:14.832384 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:14.832200    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8p95"
Jul 29 10:29:14.832384 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:14.832297    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdqnc" podUID="f7b4bba0-77fe-4240-9858-433b0f431c59"
Jul 29 10:29:14.832384 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:14.832200    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szr8n"
Jul 29 10:29:14.832503 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:14.832379    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8p95" podUID="27e6f470-ad48-45f1-b171-ae66fac19573"
Jul 29 10:29:14.832541 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:14.832499    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-szr8n" podUID="51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c"
Jul 29 10:29:16.000062 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:15.999883    2569 generic.go:356] "Generic (PLEG): container finished" podID="2e6f3639-fa84-4599-8a55-a6d7242cea3a" containerID="fb4c089fe8f78ecc723b329116f8e31e3d44e94d9d75c5b040a6c69b4f17a8cb" exitCode=0
Jul 29 10:29:16.000062 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:15.999962    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r7xrd" event={"ID":"2e6f3639-fa84-4599-8a55-a6d7242cea3a","Type":"ContainerDied","Data":"fb4c089fe8f78ecc723b329116f8e31e3d44e94d9d75c5b040a6c69b4f17a8cb"}
Jul 29 10:29:16.832620 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:16.832589    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8p95"
Jul 29 10:29:16.832806 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:16.832589    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdqnc"
Jul 29 10:29:16.832806 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:16.832701    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8p95" podUID="27e6f470-ad48-45f1-b171-ae66fac19573"
Jul 29 10:29:16.832806 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:16.832788    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdqnc" podUID="f7b4bba0-77fe-4240-9858-433b0f431c59"
Jul 29 10:29:16.832932 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:16.832597    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szr8n"
Jul 29 10:29:16.832932 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:16.832866    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-szr8n" podUID="51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c"
Jul 29 10:29:17.464529 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:17.464498    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c-metrics-certs\") pod \"network-metrics-daemon-szr8n\" (UID: \"51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c\") " pod="openshift-multus/network-metrics-daemon-szr8n"
Jul 29 10:29:17.464914 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:17.464670    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27e6f470-ad48-45f1-b171-ae66fac19573-original-pull-secret\") pod \"global-pull-secret-syncer-p8p95\" (UID: \"27e6f470-ad48-45f1-b171-ae66fac19573\") " pod="kube-system/global-pull-secret-syncer-p8p95"
Jul 29 10:29:17.464914 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:17.464669    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:29:17.464914 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:17.464735    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:29:17.464914 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:17.464767    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c-metrics-certs podName:51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c nodeName:}" failed. No retries permitted until 2026-07-29 10:29:49.464747765 +0000 UTC m=+66.189538430 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c-metrics-certs") pod "network-metrics-daemon-szr8n" (UID: "51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:29:17.464914 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:17.464792    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27e6f470-ad48-45f1-b171-ae66fac19573-original-pull-secret podName:27e6f470-ad48-45f1-b171-ae66fac19573 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:49.464773768 +0000 UTC m=+66.189564435 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27e6f470-ad48-45f1-b171-ae66fac19573-original-pull-secret") pod "global-pull-secret-syncer-p8p95" (UID: "27e6f470-ad48-45f1-b171-ae66fac19573") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:29:17.565238 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:17.565199    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d8nk\" (UniqueName: \"kubernetes.io/projected/f7b4bba0-77fe-4240-9858-433b0f431c59-kube-api-access-4d8nk\") pod \"network-check-target-mdqnc\" (UID: \"f7b4bba0-77fe-4240-9858-433b0f431c59\") " pod="openshift-network-diagnostics/network-check-target-mdqnc"
Jul 29 10:29:17.565437 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:17.565384    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:29:17.565437 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:17.565410    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:29:17.565437 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:17.565422    2569 projected.go:196] Error preparing data for projected volume kube-api-access-4d8nk for pod openshift-network-diagnostics/network-check-target-mdqnc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:29:17.565599 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:17.565484    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7b4bba0-77fe-4240-9858-433b0f431c59-kube-api-access-4d8nk podName:f7b4bba0-77fe-4240-9858-433b0f431c59 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:49.565458975 +0000 UTC m=+66.290249639 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4d8nk" (UniqueName: "kubernetes.io/projected/f7b4bba0-77fe-4240-9858-433b0f431c59-kube-api-access-4d8nk") pod "network-check-target-mdqnc" (UID: "f7b4bba0-77fe-4240-9858-433b0f431c59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:29:18.568562 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.568483    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-111.ec2.internal" event="NodeReady"
Jul 29 10:29:18.569135 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.568645    2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 29 10:29:18.609145 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.609115    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-66f4d7d896-fbvv2"]
Jul 29 10:29:18.611226 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.611207    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:29:18.614262 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.614225    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-c2kld\""
Jul 29 10:29:18.614360 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.614279    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 29 10:29:18.614360 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.614302    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 29 10:29:18.614661 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.614643    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 29 10:29:18.620664 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.619993    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 29 10:29:18.627566 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.627544    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ljsqp"]
Jul 29 10:29:18.629546 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.629529    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ljsqp"
Jul 29 10:29:18.632489 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.632450    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 29 10:29:18.632654 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.632485    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hm7tz\""
Jul 29 10:29:18.632749 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.632725    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66f4d7d896-fbvv2"]
Jul 29 10:29:18.632749 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.632740    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 29 10:29:18.633468 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.633448    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jvdpp"]
Jul 29 10:29:18.635343 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.635325    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jvdpp"
Jul 29 10:29:18.637932 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.637909    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 29 10:29:18.638022 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.637980    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 29 10:29:18.638095 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.638029    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 29 10:29:18.638195 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.638179    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9bsc6\""
Jul 29 10:29:18.646113 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.646064    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ljsqp"]
Jul 29 10:29:18.647096 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.647059    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jvdpp"]
Jul 29 10:29:18.773339 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.773309    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/662f9f8c-cd9a-4722-ab84-59cff17809ef-ca-trust-extracted\") pod \"image-registry-66f4d7d896-fbvv2\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") " pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:29:18.773339 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.773345    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/662f9f8c-cd9a-4722-ab84-59cff17809ef-installation-pull-secrets\") pod \"image-registry-66f4d7d896-fbvv2\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") " pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:29:18.773606 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.773422    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/662f9f8c-cd9a-4722-ab84-59cff17809ef-registry-certificates\") pod \"image-registry-66f4d7d896-fbvv2\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") " pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:29:18.773606 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.773458    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-bound-sa-token\") pod \"image-registry-66f4d7d896-fbvv2\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") " pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:29:18.773606 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.773482    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5edb4b57-220e-4f39-b481-164e813bbf3c-metrics-tls\") pod \"dns-default-ljsqp\" (UID: \"5edb4b57-220e-4f39-b481-164e813bbf3c\") " pod="openshift-dns/dns-default-ljsqp"
Jul 29 10:29:18.773606 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.773533    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-registry-tls\") pod \"image-registry-66f4d7d896-fbvv2\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") " pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:29:18.773606 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.773566    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5edb4b57-220e-4f39-b481-164e813bbf3c-tmp-dir\") pod \"dns-default-ljsqp\" (UID: \"5edb4b57-220e-4f39-b481-164e813bbf3c\") " pod="openshift-dns/dns-default-ljsqp"
Jul 29 10:29:18.773606 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.773583    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk648\" (UniqueName: \"kubernetes.io/projected/f3e9971d-59b4-4ea2-a9f3-5792b9e0b804-kube-api-access-bk648\") pod \"ingress-canary-jvdpp\" (UID: \"f3e9971d-59b4-4ea2-a9f3-5792b9e0b804\") " pod="openshift-ingress-canary/ingress-canary-jvdpp"
Jul 29 10:29:18.773606 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.773599    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/662f9f8c-cd9a-4722-ab84-59cff17809ef-trusted-ca\") pod \"image-registry-66f4d7d896-fbvv2\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") " pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:29:18.773928 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.773690    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5njhm\" (UniqueName: \"kubernetes.io/projected/5edb4b57-220e-4f39-b481-164e813bbf3c-kube-api-access-5njhm\") pod \"dns-default-ljsqp\" (UID: \"5edb4b57-220e-4f39-b481-164e813bbf3c\") " pod="openshift-dns/dns-default-ljsqp"
Jul 29 10:29:18.773928 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.773748    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s8qn\" (UniqueName: \"kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-kube-api-access-4s8qn\") pod \"image-registry-66f4d7d896-fbvv2\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") " pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:29:18.773928 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.773781    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/662f9f8c-cd9a-4722-ab84-59cff17809ef-image-registry-private-configuration\") pod \"image-registry-66f4d7d896-fbvv2\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") " pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:29:18.773928 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.773836    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5edb4b57-220e-4f39-b481-164e813bbf3c-config-volume\") pod \"dns-default-ljsqp\" (UID: \"5edb4b57-220e-4f39-b481-164e813bbf3c\") " pod="openshift-dns/dns-default-ljsqp"
Jul 29 10:29:18.773928 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.773868    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f3e9971d-59b4-4ea2-a9f3-5792b9e0b804-cert\") pod \"ingress-canary-jvdpp\" (UID: \"f3e9971d-59b4-4ea2-a9f3-5792b9e0b804\") " pod="openshift-ingress-canary/ingress-canary-jvdpp"
Jul 29 10:29:18.789491 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.789446    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:18.832620 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.832532    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8p95"
Jul 29 10:29:18.832620 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.832575    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdqnc"
Jul 29 10:29:18.832620 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.832585    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szr8n"
Jul 29 10:29:18.835597 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.835565    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 10:29:18.835810 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.835607    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2mpdj\""
Jul 29 10:29:18.835810 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.835767    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 10:29:18.835920 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.835898    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 10:29:18.835999 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.835984    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n9sqp\""
Jul 29 10:29:18.836053 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.836009    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 10:29:18.874628 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.874595    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/662f9f8c-cd9a-4722-ab84-59cff17809ef-registry-certificates\") pod \"image-registry-66f4d7d896-fbvv2\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") " pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:29:18.874778 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.874643    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-bound-sa-token\") pod \"image-registry-66f4d7d896-fbvv2\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") " pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:29:18.874841 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.874809    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5edb4b57-220e-4f39-b481-164e813bbf3c-metrics-tls\") pod \"dns-default-ljsqp\" (UID: \"5edb4b57-220e-4f39-b481-164e813bbf3c\") " pod="openshift-dns/dns-default-ljsqp"
Jul 29 10:29:18.874894 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.874858    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-registry-tls\") pod \"image-registry-66f4d7d896-fbvv2\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") " pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:29:18.874894 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.874889    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5edb4b57-220e-4f39-b481-164e813bbf3c-tmp-dir\") pod \"dns-default-ljsqp\" (UID: \"5edb4b57-220e-4f39-b481-164e813bbf3c\") " pod="openshift-dns/dns-default-ljsqp"
Jul 29 10:29:18.874976 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.874911    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk648\" (UniqueName: \"kubernetes.io/projected/f3e9971d-59b4-4ea2-a9f3-5792b9e0b804-kube-api-access-bk648\") pod \"ingress-canary-jvdpp\" (UID: \"f3e9971d-59b4-4ea2-a9f3-5792b9e0b804\") " pod="openshift-ingress-canary/ingress-canary-jvdpp"
Jul 29 10:29:18.874976 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:18.874934    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 10:29:18.875094 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.874937    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/662f9f8c-cd9a-4722-ab84-59cff17809ef-trusted-ca\") pod \"image-registry-66f4d7d896-fbvv2\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") " pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:29:18.875094 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:18.875009    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5edb4b57-220e-4f39-b481-164e813bbf3c-metrics-tls podName:5edb4b57-220e-4f39-b481-164e813bbf3c nodeName:}" failed. No retries permitted until 2026-07-29 10:29:19.374987245 +0000 UTC m=+36.099777911 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5edb4b57-220e-4f39-b481-164e813bbf3c-metrics-tls") pod "dns-default-ljsqp" (UID: "5edb4b57-220e-4f39-b481-164e813bbf3c") : secret "dns-default-metrics-tls" not found
Jul 29 10:29:18.875094 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:18.875030    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 10:29:18.875094 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:18.875045    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66f4d7d896-fbvv2: secret "image-registry-tls" not found
Jul 29 10:29:18.875094 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.875056    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5njhm\" (UniqueName: \"kubernetes.io/projected/5edb4b57-220e-4f39-b481-164e813bbf3c-kube-api-access-5njhm\") pod \"dns-default-ljsqp\" (UID: \"5edb4b57-220e-4f39-b481-164e813bbf3c\") " pod="openshift-dns/dns-default-ljsqp"
Jul 29 10:29:18.875345 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:18.875107    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-registry-tls podName:662f9f8c-cd9a-4722-ab84-59cff17809ef nodeName:}" failed. No retries permitted until 2026-07-29 10:29:19.375090967 +0000 UTC m=+36.099881633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-registry-tls") pod "image-registry-66f4d7d896-fbvv2" (UID: "662f9f8c-cd9a-4722-ab84-59cff17809ef") : secret "image-registry-tls" not found
Jul 29 10:29:18.875345 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.875151    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s8qn\" (UniqueName: \"kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-kube-api-access-4s8qn\") pod \"image-registry-66f4d7d896-fbvv2\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") " pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:29:18.875345 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.875184    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/662f9f8c-cd9a-4722-ab84-59cff17809ef-image-registry-private-configuration\") pod \"image-registry-66f4d7d896-fbvv2\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") " pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:29:18.875345 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.875210    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5edb4b57-220e-4f39-b481-164e813bbf3c-config-volume\") pod \"dns-default-ljsqp\" (UID: \"5edb4b57-220e-4f39-b481-164e813bbf3c\") " pod="openshift-dns/dns-default-ljsqp"
Jul 29 10:29:18.875345 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.875251    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5edb4b57-220e-4f39-b481-164e813bbf3c-tmp-dir\") pod \"dns-default-ljsqp\" (UID: \"5edb4b57-220e-4f39-b481-164e813bbf3c\") " pod="openshift-dns/dns-default-ljsqp"
Jul 29 10:29:18.875345 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.875275    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/662f9f8c-cd9a-4722-ab84-59cff17809ef-registry-certificates\") pod \"image-registry-66f4d7d896-fbvv2\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") " pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:29:18.875608 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.875379    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f3e9971d-59b4-4ea2-a9f3-5792b9e0b804-cert\") pod \"ingress-canary-jvdpp\" (UID: \"f3e9971d-59b4-4ea2-a9f3-5792b9e0b804\") " pod="openshift-ingress-canary/ingress-canary-jvdpp"
Jul 29 10:29:18.875608 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.875413    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/662f9f8c-cd9a-4722-ab84-59cff17809ef-ca-trust-extracted\") pod \"image-registry-66f4d7d896-fbvv2\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") " pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:29:18.875608 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.875440    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/662f9f8c-cd9a-4722-ab84-59cff17809ef-installation-pull-secrets\") pod \"image-registry-66f4d7d896-fbvv2\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") " pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:29:18.875743 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.875724    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5edb4b57-220e-4f39-b481-164e813bbf3c-config-volume\") pod \"dns-default-ljsqp\" (UID: \"5edb4b57-220e-4f39-b481-164e813bbf3c\") " pod="openshift-dns/dns-default-ljsqp"
Jul 29 10:29:18.875791 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.875746    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/662f9f8c-cd9a-4722-ab84-59cff17809ef-ca-trust-extracted\") pod \"image-registry-66f4d7d896-fbvv2\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") " pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:29:18.876169 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.876066    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/662f9f8c-cd9a-4722-ab84-59cff17809ef-trusted-ca\") pod \"image-registry-66f4d7d896-fbvv2\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") " pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:29:18.880056 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.880031    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/662f9f8c-cd9a-4722-ab84-59cff17809ef-image-registry-private-configuration\") pod \"image-registry-66f4d7d896-fbvv2\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") " pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:29:18.880173 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.880036    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/662f9f8c-cd9a-4722-ab84-59cff17809ef-installation-pull-secrets\") pod \"image-registry-66f4d7d896-fbvv2\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") " pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:29:18.880173 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.880040    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f3e9971d-59b4-4ea2-a9f3-5792b9e0b804-cert\") pod \"ingress-canary-jvdpp\" (UID: \"f3e9971d-59b4-4ea2-a9f3-5792b9e0b804\") " pod="openshift-ingress-canary/ingress-canary-jvdpp"
Jul 29 10:29:18.883530 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.883508    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-bound-sa-token\") pod \"image-registry-66f4d7d896-fbvv2\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") " pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:29:18.884019 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.883993    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5njhm\" (UniqueName: \"kubernetes.io/projected/5edb4b57-220e-4f39-b481-164e813bbf3c-kube-api-access-5njhm\") pod \"dns-default-ljsqp\" (UID: \"5edb4b57-220e-4f39-b481-164e813bbf3c\") " pod="openshift-dns/dns-default-ljsqp"
Jul 29 10:29:18.884019 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.884016    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk648\" (UniqueName: \"kubernetes.io/projected/f3e9971d-59b4-4ea2-a9f3-5792b9e0b804-kube-api-access-bk648\") pod \"ingress-canary-jvdpp\" (UID: \"f3e9971d-59b4-4ea2-a9f3-5792b9e0b804\") " pod="openshift-ingress-canary/ingress-canary-jvdpp"
Jul 29 10:29:18.884267 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.884249    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s8qn\" (UniqueName: \"kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-kube-api-access-4s8qn\") pod \"image-registry-66f4d7d896-fbvv2\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") " pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:29:18.949458 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:18.949417    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jvdpp"
Jul 29 10:29:19.380587 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:19.380546    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5edb4b57-220e-4f39-b481-164e813bbf3c-metrics-tls\") pod \"dns-default-ljsqp\" (UID: \"5edb4b57-220e-4f39-b481-164e813bbf3c\") " pod="openshift-dns/dns-default-ljsqp"
Jul 29 10:29:19.380750 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:19.380619    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-registry-tls\") pod \"image-registry-66f4d7d896-fbvv2\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") " pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:29:19.380750 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:19.380733    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 10:29:19.380839 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:19.380801    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 10:29:19.380839 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:19.380811    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5edb4b57-220e-4f39-b481-164e813bbf3c-metrics-tls podName:5edb4b57-220e-4f39-b481-164e813bbf3c nodeName:}" failed. No retries permitted until 2026-07-29 10:29:20.380787921 +0000 UTC m=+37.105578573 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5edb4b57-220e-4f39-b481-164e813bbf3c-metrics-tls") pod "dns-default-ljsqp" (UID: "5edb4b57-220e-4f39-b481-164e813bbf3c") : secret "dns-default-metrics-tls" not found
Jul 29 10:29:19.380839 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:19.380812    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66f4d7d896-fbvv2: secret "image-registry-tls" not found
Jul 29 10:29:19.380952 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:19.380856    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-registry-tls podName:662f9f8c-cd9a-4722-ab84-59cff17809ef nodeName:}" failed. No retries permitted until 2026-07-29 10:29:20.380845546 +0000 UTC m=+37.105636236 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-registry-tls") pod "image-registry-66f4d7d896-fbvv2" (UID: "662f9f8c-cd9a-4722-ab84-59cff17809ef") : secret "image-registry-tls" not found
Jul 29 10:29:20.327033 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:20.327003    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jvdpp"]
Jul 29 10:29:20.329870 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:29:20.329839    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3e9971d_59b4_4ea2_a9f3_5792b9e0b804.slice/crio-f76e86fe72ce16d6206e3bf79256e04c05da6ea8610340bff13432ba2efd5f3c WatchSource:0}: Error finding container f76e86fe72ce16d6206e3bf79256e04c05da6ea8610340bff13432ba2efd5f3c: Status 404 returned error can't find the container with id f76e86fe72ce16d6206e3bf79256e04c05da6ea8610340bff13432ba2efd5f3c
Jul 29 10:29:20.388826 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:20.388790    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5edb4b57-220e-4f39-b481-164e813bbf3c-metrics-tls\") pod \"dns-default-ljsqp\" (UID: \"5edb4b57-220e-4f39-b481-164e813bbf3c\") " pod="openshift-dns/dns-default-ljsqp"
Jul 29 10:29:20.388943 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:20.388853    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-registry-tls\") pod \"image-registry-66f4d7d896-fbvv2\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") " pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:29:20.388943 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:20.388927    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 10:29:20.389053 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:20.388964    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 10:29:20.389053 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:20.388975    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66f4d7d896-fbvv2: secret "image-registry-tls" not found
Jul 29 10:29:20.389053 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:20.389012    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5edb4b57-220e-4f39-b481-164e813bbf3c-metrics-tls podName:5edb4b57-220e-4f39-b481-164e813bbf3c nodeName:}" failed. No retries permitted until 2026-07-29 10:29:22.388991095 +0000 UTC m=+39.113781761 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5edb4b57-220e-4f39-b481-164e813bbf3c-metrics-tls") pod "dns-default-ljsqp" (UID: "5edb4b57-220e-4f39-b481-164e813bbf3c") : secret "dns-default-metrics-tls" not found
Jul 29 10:29:20.389053 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:20.389033    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-registry-tls podName:662f9f8c-cd9a-4722-ab84-59cff17809ef nodeName:}" failed. No retries permitted until 2026-07-29 10:29:22.389023446 +0000 UTC m=+39.113814100 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-registry-tls") pod "image-registry-66f4d7d896-fbvv2" (UID: "662f9f8c-cd9a-4722-ab84-59cff17809ef") : secret "image-registry-tls" not found
Jul 29 10:29:20.794971 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:20.794932    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:20.987409 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:20.987368    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:21.015807 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:21.015767    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jvdpp" event={"ID":"f3e9971d-59b4-4ea2-a9f3-5792b9e0b804","Type":"ContainerStarted","Data":"f76e86fe72ce16d6206e3bf79256e04c05da6ea8610340bff13432ba2efd5f3c"}
Jul 29 10:29:21.019065 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:21.019037    2569 generic.go:356] "Generic (PLEG): container finished" podID="2e6f3639-fa84-4599-8a55-a6d7242cea3a" containerID="b4150ab283a1b5b67c2997313db9b2d11b6dd43df2a58a4ad9095b69c8012ae2" exitCode=0
Jul 29 10:29:21.019210 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:21.019090    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r7xrd" event={"ID":"2e6f3639-fa84-4599-8a55-a6d7242cea3a","Type":"ContainerDied","Data":"b4150ab283a1b5b67c2997313db9b2d11b6dd43df2a58a4ad9095b69c8012ae2"}
Jul 29 10:29:21.046857 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:21.046771    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:22.026277 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:22.026229    2569 generic.go:356] "Generic (PLEG): container finished" podID="2e6f3639-fa84-4599-8a55-a6d7242cea3a" containerID="1f6c15e857e3cf16d56037b917b334dceb0f6bb1d9de61f8f1f1190e95b5819e" exitCode=0
Jul 29 10:29:22.026778 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:22.026304    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r7xrd" event={"ID":"2e6f3639-fa84-4599-8a55-a6d7242cea3a","Type":"ContainerDied","Data":"1f6c15e857e3cf16d56037b917b334dceb0f6bb1d9de61f8f1f1190e95b5819e"}
Jul 29 10:29:22.407356 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:22.407273    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5edb4b57-220e-4f39-b481-164e813bbf3c-metrics-tls\") pod \"dns-default-ljsqp\" (UID: \"5edb4b57-220e-4f39-b481-164e813bbf3c\") " pod="openshift-dns/dns-default-ljsqp"
Jul 29 10:29:22.407532 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:22.407407    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-registry-tls\") pod \"image-registry-66f4d7d896-fbvv2\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") " pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:29:22.407532 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:22.407416    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 10:29:22.407532 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:22.407481    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5edb4b57-220e-4f39-b481-164e813bbf3c-metrics-tls podName:5edb4b57-220e-4f39-b481-164e813bbf3c nodeName:}" failed. No retries permitted until 2026-07-29 10:29:26.407461228 +0000 UTC m=+43.132251896 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5edb4b57-220e-4f39-b481-164e813bbf3c-metrics-tls") pod "dns-default-ljsqp" (UID: "5edb4b57-220e-4f39-b481-164e813bbf3c") : secret "dns-default-metrics-tls" not found
Jul 29 10:29:22.407700 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:22.407549    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 10:29:22.407700 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:22.407566    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66f4d7d896-fbvv2: secret "image-registry-tls" not found
Jul 29 10:29:22.407700 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:22.407611    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-registry-tls podName:662f9f8c-cd9a-4722-ab84-59cff17809ef nodeName:}" failed. No retries permitted until 2026-07-29 10:29:26.407599797 +0000 UTC m=+43.132390448 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-registry-tls") pod "image-registry-66f4d7d896-fbvv2" (UID: "662f9f8c-cd9a-4722-ab84-59cff17809ef") : secret "image-registry-tls" not found
Jul 29 10:29:23.032015 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:23.031834    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r7xrd" event={"ID":"2e6f3639-fa84-4599-8a55-a6d7242cea3a","Type":"ContainerStarted","Data":"0a0d55ea5b151063cc334e5c065ecdb7e547f19963e46e401eb55cabc39a8934"}
Jul 29 10:29:23.061945 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:23.061892    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-r7xrd" podStartSLOduration=7.865414157 podStartE2EDuration="38.061876195s" totalImagesPullingTime="30.196462038s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:23.061493755 +0000 UTC m=+39.786284428" watchObservedRunningTime="2026-07-29 10:29:23.061876195 +0000 UTC m=+39.786666914"
Jul 29 10:29:23.065327 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:23.065301    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:23.137475 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:23.137443    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:23.207318 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:23.207233    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:23.280041 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:23.280007    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:23.379306 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:23.379270    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:24.027525 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:24.027492    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:24.035410 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:24.035378    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jvdpp" event={"ID":"f3e9971d-59b4-4ea2-a9f3-5792b9e0b804","Type":"ContainerStarted","Data":"9c60db91d04f7c931b87c6145236ca86f668dfd68d716b34eef4c439f107b4b0"}
Jul 29 10:29:24.054051 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:24.054001    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jvdpp" podStartSLOduration=3.457129278 podStartE2EDuration="6.053983464s" totalImagesPullingTime="2.596854186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:24.052755872 +0000 UTC m=+40.777546545" watchObservedRunningTime="2026-07-29 10:29:24.053983464 +0000 UTC m=+40.778774138"
Jul 29 10:29:24.418002 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:24.417967    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:24.474099 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:24.474050    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:24.955155 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:24.955124    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:26.435535 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:26.435500    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5edb4b57-220e-4f39-b481-164e813bbf3c-metrics-tls\") pod \"dns-default-ljsqp\" (UID: \"5edb4b57-220e-4f39-b481-164e813bbf3c\") " pod="openshift-dns/dns-default-ljsqp"
Jul 29 10:29:26.435931 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:26.435548    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-registry-tls\") pod \"image-registry-66f4d7d896-fbvv2\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") " pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:29:26.435931 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:26.435642    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 10:29:26.435931 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:26.435696    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5edb4b57-220e-4f39-b481-164e813bbf3c-metrics-tls podName:5edb4b57-220e-4f39-b481-164e813bbf3c nodeName:}" failed. No retries permitted until 2026-07-29 10:29:34.435682101 +0000 UTC m=+51.160472751 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5edb4b57-220e-4f39-b481-164e813bbf3c-metrics-tls") pod "dns-default-ljsqp" (UID: "5edb4b57-220e-4f39-b481-164e813bbf3c") : secret "dns-default-metrics-tls" not found
Jul 29 10:29:26.435931 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:26.435642    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 10:29:26.435931 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:26.435735    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66f4d7d896-fbvv2: secret "image-registry-tls" not found
Jul 29 10:29:26.435931 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:26.435774    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-registry-tls podName:662f9f8c-cd9a-4722-ab84-59cff17809ef nodeName:}" failed. No retries permitted until 2026-07-29 10:29:34.435763131 +0000 UTC m=+51.160553782 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-registry-tls") pod "image-registry-66f4d7d896-fbvv2" (UID: "662f9f8c-cd9a-4722-ab84-59cff17809ef") : secret "image-registry-tls" not found
Jul 29 10:29:27.705467 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:27.705431    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:27.804593 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:27.804560    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:27.881476 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:27.881443    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:34.500213 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:34.500165    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-registry-tls\") pod \"image-registry-66f4d7d896-fbvv2\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") " pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:29:34.500629 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:34.500282    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5edb4b57-220e-4f39-b481-164e813bbf3c-metrics-tls\") pod \"dns-default-ljsqp\" (UID: \"5edb4b57-220e-4f39-b481-164e813bbf3c\") " pod="openshift-dns/dns-default-ljsqp"
Jul 29 10:29:34.500629 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:34.500326    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 10:29:34.500629 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:34.500343    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66f4d7d896-fbvv2: secret "image-registry-tls" not found
Jul 29 10:29:34.500629 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:34.500365    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 10:29:34.500629 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:34.500415    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-registry-tls podName:662f9f8c-cd9a-4722-ab84-59cff17809ef nodeName:}" failed. No retries permitted until 2026-07-29 10:29:50.50039501 +0000 UTC m=+67.225185670 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-registry-tls") pod "image-registry-66f4d7d896-fbvv2" (UID: "662f9f8c-cd9a-4722-ab84-59cff17809ef") : secret "image-registry-tls" not found
Jul 29 10:29:34.500629 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:34.500429    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5edb4b57-220e-4f39-b481-164e813bbf3c-metrics-tls podName:5edb4b57-220e-4f39-b481-164e813bbf3c nodeName:}" failed. No retries permitted until 2026-07-29 10:29:50.500423431 +0000 UTC m=+67.225214082 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5edb4b57-220e-4f39-b481-164e813bbf3c-metrics-tls") pod "dns-default-ljsqp" (UID: "5edb4b57-220e-4f39-b481-164e813bbf3c") : secret "dns-default-metrics-tls" not found
Jul 29 10:29:35.207431 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:35.207395    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:35.269557 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:35.269523    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:39.114388 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:39.114348    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:39.187238 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:39.187207    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:40.791851 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:40.791799    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:40.857414 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:40.857384    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:45.013297 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:45.013241    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6jzgn"
Jul 29 10:29:45.173352 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:45.173317    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:45.252861 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:45.252832    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:48.684303 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:48.684261    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:48.785808 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:48.785774    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:49.508980 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:49.508923    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27e6f470-ad48-45f1-b171-ae66fac19573-original-pull-secret\") pod \"global-pull-secret-syncer-p8p95\" (UID: \"27e6f470-ad48-45f1-b171-ae66fac19573\") " pod="kube-system/global-pull-secret-syncer-p8p95"
Jul 29 10:29:49.509209 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:49.508997    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c-metrics-certs\") pod \"network-metrics-daemon-szr8n\" (UID: \"51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c\") " pod="openshift-multus/network-metrics-daemon-szr8n"
Jul 29 10:29:49.517594 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:49.517569    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 10:29:49.517705 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:49.517624    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 10:29:49.519695 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:49.519678    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 29 10:29:49.519746 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:49.519734    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c-metrics-certs podName:51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c nodeName:}" failed. No retries permitted until 2026-07-29 10:30:53.519719128 +0000 UTC m=+130.244509782 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c-metrics-certs") pod "network-metrics-daemon-szr8n" (UID: "51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c") : secret "metrics-daemon-secret" not found
Jul 29 10:29:49.523525 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:49.523501    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27e6f470-ad48-45f1-b171-ae66fac19573-original-pull-secret\") pod \"global-pull-secret-syncer-p8p95\" (UID: \"27e6f470-ad48-45f1-b171-ae66fac19573\") " pod="kube-system/global-pull-secret-syncer-p8p95"
Jul 29 10:29:49.610068 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:49.610009    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d8nk\" (UniqueName: \"kubernetes.io/projected/f7b4bba0-77fe-4240-9858-433b0f431c59-kube-api-access-4d8nk\") pod \"network-check-target-mdqnc\" (UID: \"f7b4bba0-77fe-4240-9858-433b0f431c59\") " pod="openshift-network-diagnostics/network-check-target-mdqnc"
Jul 29 10:29:49.612678 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:49.612661    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 10:29:49.623381 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:49.623362    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 10:29:49.633963 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:49.633938    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d8nk\" (UniqueName: \"kubernetes.io/projected/f7b4bba0-77fe-4240-9858-433b0f431c59-kube-api-access-4d8nk\") pod \"network-check-target-mdqnc\" (UID: \"f7b4bba0-77fe-4240-9858-433b0f431c59\") " pod="openshift-network-diagnostics/network-check-target-mdqnc"
Jul 29 10:29:49.744045 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:49.744008    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8p95"
Jul 29 10:29:49.753697 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:49.753677    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2mpdj\""
Jul 29 10:29:49.761931 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:49.761876    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdqnc"
Jul 29 10:29:49.900332 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:49.900301    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p8p95"]
Jul 29 10:29:49.902993 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:29:49.902967    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27e6f470_ad48_45f1_b171_ae66fac19573.slice/crio-702ab71b31a5eb050572e65f5307d909802b17fe9a4be54475f8baba5699b9ff WatchSource:0}: Error finding container 702ab71b31a5eb050572e65f5307d909802b17fe9a4be54475f8baba5699b9ff: Status 404 returned error can't find the container with id 702ab71b31a5eb050572e65f5307d909802b17fe9a4be54475f8baba5699b9ff
Jul 29 10:29:49.917989 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:49.917929    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mdqnc"]
Jul 29 10:29:49.922888 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:29:49.922864    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b4bba0_77fe_4240_9858_433b0f431c59.slice/crio-a9ddefcc2264e6573c318a50a0bc52ad349b464161b98b5f07377702a1a099d0 WatchSource:0}: Error finding container a9ddefcc2264e6573c318a50a0bc52ad349b464161b98b5f07377702a1a099d0: Status 404 returned error can't find the container with id a9ddefcc2264e6573c318a50a0bc52ad349b464161b98b5f07377702a1a099d0
Jul 29 10:29:50.098002 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:50.097916    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mdqnc" event={"ID":"f7b4bba0-77fe-4240-9858-433b0f431c59","Type":"ContainerStarted","Data":"a9ddefcc2264e6573c318a50a0bc52ad349b464161b98b5f07377702a1a099d0"}
Jul 29 10:29:50.098930 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:50.098910    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p8p95" event={"ID":"27e6f470-ad48-45f1-b171-ae66fac19573","Type":"ContainerStarted","Data":"702ab71b31a5eb050572e65f5307d909802b17fe9a4be54475f8baba5699b9ff"}
Jul 29 10:29:50.519501 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:50.519456    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-registry-tls\") pod \"image-registry-66f4d7d896-fbvv2\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") " pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:29:50.519779 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:50.519562    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5edb4b57-220e-4f39-b481-164e813bbf3c-metrics-tls\") pod \"dns-default-ljsqp\" (UID: \"5edb4b57-220e-4f39-b481-164e813bbf3c\") " pod="openshift-dns/dns-default-ljsqp"
Jul 29 10:29:50.519779 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:50.519692    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 10:29:50.519779 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:50.519757    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5edb4b57-220e-4f39-b481-164e813bbf3c-metrics-tls podName:5edb4b57-220e-4f39-b481-164e813bbf3c nodeName:}" failed. No retries permitted until 2026-07-29 10:30:22.519736353 +0000 UTC m=+99.244527025 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5edb4b57-220e-4f39-b481-164e813bbf3c-metrics-tls") pod "dns-default-ljsqp" (UID: "5edb4b57-220e-4f39-b481-164e813bbf3c") : secret "dns-default-metrics-tls" not found
Jul 29 10:29:50.520195 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:50.520173    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 10:29:50.520195 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:50.520190    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66f4d7d896-fbvv2: secret "image-registry-tls" not found
Jul 29 10:29:50.520338 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:29:50.520238    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-registry-tls podName:662f9f8c-cd9a-4722-ab84-59cff17809ef nodeName:}" failed. No retries permitted until 2026-07-29 10:30:22.520221747 +0000 UTC m=+99.245012418 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-registry-tls") pod "image-registry-66f4d7d896-fbvv2" (UID: "662f9f8c-cd9a-4722-ab84-59cff17809ef") : secret "image-registry-tls" not found
Jul 29 10:29:55.115023 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.114984    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mdqnc" event={"ID":"f7b4bba0-77fe-4240-9858-433b0f431c59","Type":"ContainerStarted","Data":"440956d8b06b6b76cdb624e4166cf102f5583bb53275075af0af4321414d9134"}
Jul 29 10:29:55.115483 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.115082    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mdqnc"
Jul 29 10:29:55.116412 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.116386    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p8p95" event={"ID":"27e6f470-ad48-45f1-b171-ae66fac19573","Type":"ContainerStarted","Data":"f0b36e2d65c1eba6bc220952fa124e69f260036997c6b73df97827c5eb513e0b"}
Jul 29 10:29:55.142809 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.142763    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mdqnc" podStartSLOduration=65.804413327 podStartE2EDuration="1m10.142751033s" totalImagesPullingTime="4.338337706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:28:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:55.142539568 +0000 UTC m=+71.867330244" watchObservedRunningTime="2026-07-29 10:29:55.142751033 +0000 UTC m=+71.867541757"
Jul 29 10:29:55.158887 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.158847    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-p8p95" podStartSLOduration=65.795000036 podStartE2EDuration="1m10.158835749s" totalImagesPullingTime="4.363835713s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:55.15828688 +0000 UTC m=+71.883077552" watchObservedRunningTime="2026-07-29 10:29:55.158835749 +0000 UTC m=+71.883626421"
Jul 29 10:29:55.296736 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.296703    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-fgw4v"]
Jul 29 10:29:55.298722 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.298699    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-fgw4v"
Jul 29 10:29:55.299539 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.299421    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ns69c"]
Jul 29 10:29:55.301888 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.301864    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-r68g5\""
Jul 29 10:29:55.304053 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.304033    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-pvdgw"]
Jul 29 10:29:55.304166 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.304140    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ns69c"
Jul 29 10:29:55.305744 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.305729    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-pvdgw"
Jul 29 10:29:55.309428 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.309413    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 29 10:29:55.309536 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.309520    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 29 10:29:55.310249 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.310228    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-rn4xb\""
Jul 29 10:29:55.310317 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.310290    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-nn5tk\""
Jul 29 10:29:55.310400 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.310383    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 29 10:29:55.310615 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.310601    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 29 10:29:55.316560 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.316539    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 29 10:29:55.316651 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.316546    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 29 10:29:55.316651 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.316595    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 29 10:29:55.316651 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.316620    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 29 10:29:55.325516 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.325496    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-pvdgw"]
Jul 29 10:29:55.326491 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.326471    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-fgw4v"]
Jul 29 10:29:55.327506 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.327485    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ns69c"]
Jul 29 10:29:55.331682 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.331665    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 29 10:29:55.345658 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.345632    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:55.353542 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.353518    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/afd0d65c-bbfc-49f0-81b6-e855fd81da7f-trusted-ca\") pod \"console-operator-584dcb6b54-pvdgw\" (UID: \"afd0d65c-bbfc-49f0-81b6-e855fd81da7f\") " pod="openshift-console-operator/console-operator-584dcb6b54-pvdgw"
Jul 29 10:29:55.353598 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.353552    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv6fd\" (UniqueName: \"kubernetes.io/projected/223649ac-cf58-4cbe-827d-9b16b0ee0359-kube-api-access-gv6fd\") pod \"network-check-source-7d88cb55d8-fgw4v\" (UID: \"223649ac-cf58-4cbe-827d-9b16b0ee0359\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-fgw4v"
Jul 29 10:29:55.353598 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.353572    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nwnl\" (UniqueName: \"kubernetes.io/projected/3aaafa2b-ae84-417a-905b-62bbc49a287c-kube-api-access-6nwnl\") pod \"service-ca-operator-5d6c75f84f-ns69c\" (UID: \"3aaafa2b-ae84-417a-905b-62bbc49a287c\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ns69c"
Jul 29 10:29:55.353671 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.353652    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/afd0d65c-bbfc-49f0-81b6-e855fd81da7f-config\") pod \"console-operator-584dcb6b54-pvdgw\" (UID: \"afd0d65c-bbfc-49f0-81b6-e855fd81da7f\") " pod="openshift-console-operator/console-operator-584dcb6b54-pvdgw"
Jul 29 10:29:55.353705 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.353678    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls6px\" (UniqueName: \"kubernetes.io/projected/afd0d65c-bbfc-49f0-81b6-e855fd81da7f-kube-api-access-ls6px\") pod \"console-operator-584dcb6b54-pvdgw\" (UID: \"afd0d65c-bbfc-49f0-81b6-e855fd81da7f\") " pod="openshift-console-operator/console-operator-584dcb6b54-pvdgw"
Jul 29 10:29:55.353741 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.353703    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3aaafa2b-ae84-417a-905b-62bbc49a287c-serving-cert\") pod \"service-ca-operator-5d6c75f84f-ns69c\" (UID: \"3aaafa2b-ae84-417a-905b-62bbc49a287c\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ns69c"
Jul 29 10:29:55.353741 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.353722    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3aaafa2b-ae84-417a-905b-62bbc49a287c-config\") pod \"service-ca-operator-5d6c75f84f-ns69c\" (UID: \"3aaafa2b-ae84-417a-905b-62bbc49a287c\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ns69c"
Jul 29 10:29:55.353800 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.353780    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/afd0d65c-bbfc-49f0-81b6-e855fd81da7f-serving-cert\") pod \"console-operator-584dcb6b54-pvdgw\" (UID: \"afd0d65c-bbfc-49f0-81b6-e855fd81da7f\") " pod="openshift-console-operator/console-operator-584dcb6b54-pvdgw"
Jul 29 10:29:55.435109 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.435082    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:55.455008 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.454979    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/afd0d65c-bbfc-49f0-81b6-e855fd81da7f-serving-cert\") pod \"console-operator-584dcb6b54-pvdgw\" (UID: \"afd0d65c-bbfc-49f0-81b6-e855fd81da7f\") " pod="openshift-console-operator/console-operator-584dcb6b54-pvdgw"
Jul 29 10:29:55.455138 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.455043    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/afd0d65c-bbfc-49f0-81b6-e855fd81da7f-trusted-ca\") pod \"console-operator-584dcb6b54-pvdgw\" (UID: \"afd0d65c-bbfc-49f0-81b6-e855fd81da7f\") " pod="openshift-console-operator/console-operator-584dcb6b54-pvdgw"
Jul 29 10:29:55.455138 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.455092    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv6fd\" (UniqueName: \"kubernetes.io/projected/223649ac-cf58-4cbe-827d-9b16b0ee0359-kube-api-access-gv6fd\") pod \"network-check-source-7d88cb55d8-fgw4v\" (UID: \"223649ac-cf58-4cbe-827d-9b16b0ee0359\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-fgw4v"
Jul 29 10:29:55.455138 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.455118    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nwnl\" (UniqueName: \"kubernetes.io/projected/3aaafa2b-ae84-417a-905b-62bbc49a287c-kube-api-access-6nwnl\") pod \"service-ca-operator-5d6c75f84f-ns69c\" (UID: \"3aaafa2b-ae84-417a-905b-62bbc49a287c\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ns69c"
Jul 29 10:29:55.455313 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.455286    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/afd0d65c-bbfc-49f0-81b6-e855fd81da7f-config\") pod \"console-operator-584dcb6b54-pvdgw\" (UID: \"afd0d65c-bbfc-49f0-81b6-e855fd81da7f\") " pod="openshift-console-operator/console-operator-584dcb6b54-pvdgw"
Jul 29 10:29:55.455409 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.455334    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls6px\" (UniqueName: \"kubernetes.io/projected/afd0d65c-bbfc-49f0-81b6-e855fd81da7f-kube-api-access-ls6px\") pod \"console-operator-584dcb6b54-pvdgw\" (UID: \"afd0d65c-bbfc-49f0-81b6-e855fd81da7f\") " pod="openshift-console-operator/console-operator-584dcb6b54-pvdgw"
Jul 29 10:29:55.455409 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.455377    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3aaafa2b-ae84-417a-905b-62bbc49a287c-serving-cert\") pod \"service-ca-operator-5d6c75f84f-ns69c\" (UID: \"3aaafa2b-ae84-417a-905b-62bbc49a287c\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ns69c"
Jul 29 10:29:55.455508 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.455407    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3aaafa2b-ae84-417a-905b-62bbc49a287c-config\") pod \"service-ca-operator-5d6c75f84f-ns69c\" (UID: \"3aaafa2b-ae84-417a-905b-62bbc49a287c\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ns69c"
Jul 29 10:29:55.455918 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.455895    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/afd0d65c-bbfc-49f0-81b6-e855fd81da7f-config\") pod \"console-operator-584dcb6b54-pvdgw\" (UID: \"afd0d65c-bbfc-49f0-81b6-e855fd81da7f\") " pod="openshift-console-operator/console-operator-584dcb6b54-pvdgw"
Jul 29 10:29:55.456017 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.455899    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3aaafa2b-ae84-417a-905b-62bbc49a287c-config\") pod \"service-ca-operator-5d6c75f84f-ns69c\" (UID: \"3aaafa2b-ae84-417a-905b-62bbc49a287c\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ns69c"
Jul 29 10:29:55.456017 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.455940    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/afd0d65c-bbfc-49f0-81b6-e855fd81da7f-trusted-ca\") pod \"console-operator-584dcb6b54-pvdgw\" (UID: \"afd0d65c-bbfc-49f0-81b6-e855fd81da7f\") " pod="openshift-console-operator/console-operator-584dcb6b54-pvdgw"
Jul 29 10:29:55.457480 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.457460    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3aaafa2b-ae84-417a-905b-62bbc49a287c-serving-cert\") pod \"service-ca-operator-5d6c75f84f-ns69c\" (UID: \"3aaafa2b-ae84-417a-905b-62bbc49a287c\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ns69c"
Jul 29 10:29:55.457480 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.457467    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/afd0d65c-bbfc-49f0-81b6-e855fd81da7f-serving-cert\") pod \"console-operator-584dcb6b54-pvdgw\" (UID: \"afd0d65c-bbfc-49f0-81b6-e855fd81da7f\") " pod="openshift-console-operator/console-operator-584dcb6b54-pvdgw"
Jul 29 10:29:55.473278 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.473256    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nwnl\" (UniqueName: \"kubernetes.io/projected/3aaafa2b-ae84-417a-905b-62bbc49a287c-kube-api-access-6nwnl\") pod \"service-ca-operator-5d6c75f84f-ns69c\" (UID: \"3aaafa2b-ae84-417a-905b-62bbc49a287c\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ns69c"
Jul 29 10:29:55.475375 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.475357    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls6px\" (UniqueName: \"kubernetes.io/projected/afd0d65c-bbfc-49f0-81b6-e855fd81da7f-kube-api-access-ls6px\") pod \"console-operator-584dcb6b54-pvdgw\" (UID: \"afd0d65c-bbfc-49f0-81b6-e855fd81da7f\") " pod="openshift-console-operator/console-operator-584dcb6b54-pvdgw"
Jul 29 10:29:55.478142 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.478122    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv6fd\" (UniqueName: \"kubernetes.io/projected/223649ac-cf58-4cbe-827d-9b16b0ee0359-kube-api-access-gv6fd\") pod \"network-check-source-7d88cb55d8-fgw4v\" (UID: \"223649ac-cf58-4cbe-827d-9b16b0ee0359\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-fgw4v"
Jul 29 10:29:55.610647 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.610613    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-fgw4v"
Jul 29 10:29:55.616374 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.616357    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ns69c"
Jul 29 10:29:55.621404 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.621378    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-pvdgw"
Jul 29 10:29:55.753256 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.753227    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-fgw4v"]
Jul 29 10:29:55.756262 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:29:55.756237    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod223649ac_cf58_4cbe_827d_9b16b0ee0359.slice/crio-b7735b20680c3dce88c6aa8c1e181026611f33d266d5e23beddae0bf2885e176 WatchSource:0}: Error finding container b7735b20680c3dce88c6aa8c1e181026611f33d266d5e23beddae0bf2885e176: Status 404 returned error can't find the container with id b7735b20680c3dce88c6aa8c1e181026611f33d266d5e23beddae0bf2885e176
Jul 29 10:29:55.988140 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.988034    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ns69c"]
Jul 29 10:29:55.991721 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:29:55.991686    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aaafa2b_ae84_417a_905b_62bbc49a287c.slice/crio-7f468a47e85005545bcece120661fa272d5a1baec62fb29aada2c2cd107895cc WatchSource:0}: Error finding container 7f468a47e85005545bcece120661fa272d5a1baec62fb29aada2c2cd107895cc: Status 404 returned error can't find the container with id 7f468a47e85005545bcece120661fa272d5a1baec62fb29aada2c2cd107895cc
Jul 29 10:29:55.992787 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:55.992764    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-pvdgw"]
Jul 29 10:29:55.995900 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:29:55.995871    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd0d65c_bbfc_49f0_81b6_e855fd81da7f.slice/crio-0a00af71a10ebd0c37a03b5732eb266de2914a9cab560019c5090d2d0f949b5d WatchSource:0}: Error finding container 0a00af71a10ebd0c37a03b5732eb266de2914a9cab560019c5090d2d0f949b5d: Status 404 returned error can't find the container with id 0a00af71a10ebd0c37a03b5732eb266de2914a9cab560019c5090d2d0f949b5d
Jul 29 10:29:56.120362 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:56.120322    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ns69c" event={"ID":"3aaafa2b-ae84-417a-905b-62bbc49a287c","Type":"ContainerStarted","Data":"7f468a47e85005545bcece120661fa272d5a1baec62fb29aada2c2cd107895cc"}
Jul 29 10:29:56.121854 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:56.121825    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-fgw4v" event={"ID":"223649ac-cf58-4cbe-827d-9b16b0ee0359","Type":"ContainerStarted","Data":"8f5de87504ba9ef2ac3035ef23d6a086173d7f7af20113a165b7c7dcd62adf99"}
Jul 29 10:29:56.121982 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:56.121873    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-fgw4v" event={"ID":"223649ac-cf58-4cbe-827d-9b16b0ee0359","Type":"ContainerStarted","Data":"b7735b20680c3dce88c6aa8c1e181026611f33d266d5e23beddae0bf2885e176"}
Jul 29 10:29:56.122900 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:56.122876    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-pvdgw" event={"ID":"afd0d65c-bbfc-49f0-81b6-e855fd81da7f","Type":"ContainerStarted","Data":"0a00af71a10ebd0c37a03b5732eb266de2914a9cab560019c5090d2d0f949b5d"}
Jul 29 10:29:59.134704 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:59.134675    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-pvdgw_afd0d65c-bbfc-49f0-81b6-e855fd81da7f/console-operator/0.log"
Jul 29 10:29:59.135171 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:59.134717    2569 generic.go:356] "Generic (PLEG): container finished" podID="afd0d65c-bbfc-49f0-81b6-e855fd81da7f" containerID="15d680c77f8205d5f1ef924f9929d9965d2458f24fa5a677d0171593fab48213" exitCode=255
Jul 29 10:29:59.135171 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:59.134794    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-pvdgw" event={"ID":"afd0d65c-bbfc-49f0-81b6-e855fd81da7f","Type":"ContainerDied","Data":"15d680c77f8205d5f1ef924f9929d9965d2458f24fa5a677d0171593fab48213"}
Jul 29 10:29:59.135171 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:59.135008    2569 scope.go:117] "RemoveContainer" containerID="15d680c77f8205d5f1ef924f9929d9965d2458f24fa5a677d0171593fab48213"
Jul 29 10:29:59.136370 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:59.136335    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ns69c" event={"ID":"3aaafa2b-ae84-417a-905b-62bbc49a287c","Type":"ContainerStarted","Data":"e04277c27d3da3824aa21a68ef6d87a1e2cbafa9eaca19eecfba239337ad3c35"}
Jul 29 10:29:59.158684 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:59.158642    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-fgw4v" podStartSLOduration=4.158628802 podStartE2EDuration="4.158628802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:56.144152216 +0000 UTC m=+72.868942891" watchObservedRunningTime="2026-07-29 10:29:59.158628802 +0000 UTC m=+75.883419474"
Jul 29 10:29:59.181693 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:59.181646    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ns69c" podStartSLOduration=1.8734684289999999 podStartE2EDuration="4.181627345s" totalImagesPullingTime="2.308158916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:59.180573679 +0000 UTC m=+75.905364362" watchObservedRunningTime="2026-07-29 10:29:59.181627345 +0000 UTC m=+75.906418019"
Jul 29 10:29:59.342123 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:59.342064    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:59.442995 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:59.442963    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:59.584982 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:59.584946    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:59.649088 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:29:59.649043    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:00.054443 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:00.054410    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:00.139539 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:00.139501    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:00.142373 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:00.142351    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-pvdgw_afd0d65c-bbfc-49f0-81b6-e855fd81da7f/console-operator/1.log"
Jul 29 10:30:00.142758 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:00.142741    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-pvdgw_afd0d65c-bbfc-49f0-81b6-e855fd81da7f/console-operator/0.log"
Jul 29 10:30:00.142817 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:00.142777    2569 generic.go:356] "Generic (PLEG): container finished" podID="afd0d65c-bbfc-49f0-81b6-e855fd81da7f" containerID="e620005f6f3fe05236768eabbc52540b1c3397fcb50b135b45be799901a6a280" exitCode=255
Jul 29 10:30:00.142863 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:00.142827    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-pvdgw" event={"ID":"afd0d65c-bbfc-49f0-81b6-e855fd81da7f","Type":"ContainerDied","Data":"e620005f6f3fe05236768eabbc52540b1c3397fcb50b135b45be799901a6a280"}
Jul 29 10:30:00.142904 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:00.142891    2569 scope.go:117] "RemoveContainer" containerID="15d680c77f8205d5f1ef924f9929d9965d2458f24fa5a677d0171593fab48213"
Jul 29 10:30:00.143247 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:00.143227    2569 scope.go:117] "RemoveContainer" containerID="e620005f6f3fe05236768eabbc52540b1c3397fcb50b135b45be799901a6a280"
Jul 29 10:30:00.143395 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:30:00.143376    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-pvdgw_openshift-console-operator(afd0d65c-bbfc-49f0-81b6-e855fd81da7f)\"" pod="openshift-console-operator/console-operator-584dcb6b54-pvdgw" podUID="afd0d65c-bbfc-49f0-81b6-e855fd81da7f"
Jul 29 10:30:01.091205 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.091171    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:01.147345 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.147318    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-pvdgw_afd0d65c-bbfc-49f0-81b6-e855fd81da7f/console-operator/1.log"
Jul 29 10:30:01.147700 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.147665    2569 scope.go:117] "RemoveContainer" containerID="e620005f6f3fe05236768eabbc52540b1c3397fcb50b135b45be799901a6a280"
Jul 29 10:30:01.147853 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:30:01.147832    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-pvdgw_openshift-console-operator(afd0d65c-bbfc-49f0-81b6-e855fd81da7f)\"" pod="openshift-console-operator/console-operator-584dcb6b54-pvdgw" podUID="afd0d65c-bbfc-49f0-81b6-e855fd81da7f"
Jul 29 10:30:01.174874 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.174844    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:01.642164 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.642130    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-nslw9"]
Jul 29 10:30:01.644655 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.644639    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-nslw9"
Jul 29 10:30:01.647356 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.647333    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 29 10:30:01.648304 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.648285    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-qvtnh\""
Jul 29 10:30:01.648418 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.648335    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 29 10:30:01.648418 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.648335    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 29 10:30:01.648418 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.648414    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 29 10:30:01.655910 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.655889    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-nslw9"]
Jul 29 10:30:01.706597 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.706567    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0b327881-88ad-4d8d-b7b7-d73bc8c59e2d-signing-key\") pod \"service-ca-658d7b7c78-nslw9\" (UID: \"0b327881-88ad-4d8d-b7b7-d73bc8c59e2d\") " pod="openshift-service-ca/service-ca-658d7b7c78-nslw9"
Jul 29 10:30:01.706767 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.706631    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49l8j\" (UniqueName: \"kubernetes.io/projected/0b327881-88ad-4d8d-b7b7-d73bc8c59e2d-kube-api-access-49l8j\") pod \"service-ca-658d7b7c78-nslw9\" (UID: \"0b327881-88ad-4d8d-b7b7-d73bc8c59e2d\") " pod="openshift-service-ca/service-ca-658d7b7c78-nslw9"
Jul 29 10:30:01.706767 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.706716    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0b327881-88ad-4d8d-b7b7-d73bc8c59e2d-signing-cabundle\") pod \"service-ca-658d7b7c78-nslw9\" (UID: \"0b327881-88ad-4d8d-b7b7-d73bc8c59e2d\") " pod="openshift-service-ca/service-ca-658d7b7c78-nslw9"
Jul 29 10:30:01.807287 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.807242    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0b327881-88ad-4d8d-b7b7-d73bc8c59e2d-signing-cabundle\") pod \"service-ca-658d7b7c78-nslw9\" (UID: \"0b327881-88ad-4d8d-b7b7-d73bc8c59e2d\") " pod="openshift-service-ca/service-ca-658d7b7c78-nslw9"
Jul 29 10:30:01.807482 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.807459    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0b327881-88ad-4d8d-b7b7-d73bc8c59e2d-signing-key\") pod \"service-ca-658d7b7c78-nslw9\" (UID: \"0b327881-88ad-4d8d-b7b7-d73bc8c59e2d\") " pod="openshift-service-ca/service-ca-658d7b7c78-nslw9"
Jul 29 10:30:01.807633 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.807617    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49l8j\" (UniqueName: \"kubernetes.io/projected/0b327881-88ad-4d8d-b7b7-d73bc8c59e2d-kube-api-access-49l8j\") pod \"service-ca-658d7b7c78-nslw9\" (UID: \"0b327881-88ad-4d8d-b7b7-d73bc8c59e2d\") " pod="openshift-service-ca/service-ca-658d7b7c78-nslw9"
Jul 29 10:30:01.807998 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.807969    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0b327881-88ad-4d8d-b7b7-d73bc8c59e2d-signing-cabundle\") pod \"service-ca-658d7b7c78-nslw9\" (UID: \"0b327881-88ad-4d8d-b7b7-d73bc8c59e2d\") " pod="openshift-service-ca/service-ca-658d7b7c78-nslw9"
Jul 29 10:30:01.809774 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.809752    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0b327881-88ad-4d8d-b7b7-d73bc8c59e2d-signing-key\") pod \"service-ca-658d7b7c78-nslw9\" (UID: \"0b327881-88ad-4d8d-b7b7-d73bc8c59e2d\") " pod="openshift-service-ca/service-ca-658d7b7c78-nslw9"
Jul 29 10:30:01.824860 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.824834    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49l8j\" (UniqueName: \"kubernetes.io/projected/0b327881-88ad-4d8d-b7b7-d73bc8c59e2d-kube-api-access-49l8j\") pod \"service-ca-658d7b7c78-nslw9\" (UID: \"0b327881-88ad-4d8d-b7b7-d73bc8c59e2d\") " pod="openshift-service-ca/service-ca-658d7b7c78-nslw9"
Jul 29 10:30:01.836732 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.836694    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:01.857591 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.857554    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-mhnng"]
Jul 29 10:30:01.859699 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.859681    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-mhnng"
Jul 29 10:30:01.862514 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.862485    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 29 10:30:01.862641 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.862484    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 29 10:30:01.862867 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.862850    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2kx4l\""
Jul 29 10:30:01.869969 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.869943    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-mhnng"]
Jul 29 10:30:01.908214 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.908123    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgsvr\" (UniqueName: \"kubernetes.io/projected/192911bd-5538-4220-8429-4e435554e1ef-kube-api-access-qgsvr\") pod \"migrator-6865c9967-mhnng\" (UID: \"192911bd-5538-4220-8429-4e435554e1ef\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-mhnng"
Jul 29 10:30:01.935861 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.935830    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:01.953312 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.953281    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-nslw9"
Jul 29 10:30:01.977313 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.977280    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-llm74"]
Jul 29 10:30:01.980113 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.980091    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-llm74"
Jul 29 10:30:01.983264 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.983242    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 29 10:30:01.983384 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.983288    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 29 10:30:01.983529 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.983512    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 29 10:30:01.983819 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.983805    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-27qr7\""
Jul 29 10:30:01.983866 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.983808    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 29 10:30:01.992569 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:01.992540    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-llm74"]
Jul 29 10:30:02.009153 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:02.009116    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgsvr\" (UniqueName: \"kubernetes.io/projected/192911bd-5538-4220-8429-4e435554e1ef-kube-api-access-qgsvr\") pod \"migrator-6865c9967-mhnng\" (UID: \"192911bd-5538-4220-8429-4e435554e1ef\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-mhnng"
Jul 29 10:30:02.028594 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:02.028536    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgsvr\" (UniqueName: \"kubernetes.io/projected/192911bd-5538-4220-8429-4e435554e1ef-kube-api-access-qgsvr\") pod \"migrator-6865c9967-mhnng\" (UID: \"192911bd-5538-4220-8429-4e435554e1ef\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-mhnng"
Jul 29 10:30:02.062596 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:02.062559    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:02.092862 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:02.092826    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-nslw9"]
Jul 29 10:30:02.097262 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:30:02.097232    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b327881_88ad_4d8d_b7b7_d73bc8c59e2d.slice/crio-6b5b8989fa38b4e984e0c07952df3cbef002ac45e721f8ea4c6bfa88ed4ad6a2 WatchSource:0}: Error finding container 6b5b8989fa38b4e984e0c07952df3cbef002ac45e721f8ea4c6bfa88ed4ad6a2: Status 404 returned error can't find the container with id 6b5b8989fa38b4e984e0c07952df3cbef002ac45e721f8ea4c6bfa88ed4ad6a2
Jul 29 10:30:02.110100 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:02.110060    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7687cc83-d880-467f-a270-32bba1bba683-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llm74\" (UID: \"7687cc83-d880-467f-a270-32bba1bba683\") " pod="openshift-insights/insights-runtime-extractor-llm74"
Jul 29 10:30:02.110173 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:02.110140    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7fww\" (UniqueName: \"kubernetes.io/projected/7687cc83-d880-467f-a270-32bba1bba683-kube-api-access-q7fww\") pod \"insights-runtime-extractor-llm74\" (UID: \"7687cc83-d880-467f-a270-32bba1bba683\") " pod="openshift-insights/insights-runtime-extractor-llm74"
Jul 29 10:30:02.110225 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:02.110173    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7687cc83-d880-467f-a270-32bba1bba683-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-llm74\" (UID: \"7687cc83-d880-467f-a270-32bba1bba683\") " pod="openshift-insights/insights-runtime-extractor-llm74"
Jul 29 10:30:02.110225 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:02.110198    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7687cc83-d880-467f-a270-32bba1bba683-data-volume\") pod \"insights-runtime-extractor-llm74\" (UID: \"7687cc83-d880-467f-a270-32bba1bba683\") " pod="openshift-insights/insights-runtime-extractor-llm74"
Jul 29 10:30:02.110318 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:02.110253    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7687cc83-d880-467f-a270-32bba1bba683-crio-socket\") pod \"insights-runtime-extractor-llm74\" (UID: \"7687cc83-d880-467f-a270-32bba1bba683\") " pod="openshift-insights/insights-runtime-extractor-llm74"
Jul 29 10:30:02.135224 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:02.135196    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:02.152116 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:02.152064    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-nslw9" event={"ID":"0b327881-88ad-4d8d-b7b7-d73bc8c59e2d","Type":"ContainerStarted","Data":"6b5b8989fa38b4e984e0c07952df3cbef002ac45e721f8ea4c6bfa88ed4ad6a2"}
Jul 29 10:30:02.168859 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:02.168831    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-mhnng"
Jul 29 10:30:02.211287 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:02.211252    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7687cc83-d880-467f-a270-32bba1bba683-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llm74\" (UID: \"7687cc83-d880-467f-a270-32bba1bba683\") " pod="openshift-insights/insights-runtime-extractor-llm74"
Jul 29 10:30:02.211434 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:02.211300    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7fww\" (UniqueName: \"kubernetes.io/projected/7687cc83-d880-467f-a270-32bba1bba683-kube-api-access-q7fww\") pod \"insights-runtime-extractor-llm74\" (UID: \"7687cc83-d880-467f-a270-32bba1bba683\") " pod="openshift-insights/insights-runtime-extractor-llm74"
Jul 29 10:30:02.211434 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:02.211339    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7687cc83-d880-467f-a270-32bba1bba683-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-llm74\" (UID: \"7687cc83-d880-467f-a270-32bba1bba683\") " pod="openshift-insights/insights-runtime-extractor-llm74"
Jul 29 10:30:02.211434 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:02.211367    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7687cc83-d880-467f-a270-32bba1bba683-data-volume\") pod \"insights-runtime-extractor-llm74\" (UID: \"7687cc83-d880-467f-a270-32bba1bba683\") " pod="openshift-insights/insights-runtime-extractor-llm74"
Jul 29 10:30:02.211434 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:30:02.211412    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 10:30:02.211622 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:30:02.211468    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7687cc83-d880-467f-a270-32bba1bba683-insights-runtime-extractor-tls podName:7687cc83-d880-467f-a270-32bba1bba683 nodeName:}" failed. No retries permitted until 2026-07-29 10:30:02.711451827 +0000 UTC m=+79.436242492 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7687cc83-d880-467f-a270-32bba1bba683-insights-runtime-extractor-tls") pod "insights-runtime-extractor-llm74" (UID: "7687cc83-d880-467f-a270-32bba1bba683") : secret "insights-runtime-extractor-tls" not found
Jul 29 10:30:02.211622 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:30:02.211469    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 10:30:02.211622 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:30:02.211541    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7687cc83-d880-467f-a270-32bba1bba683-kube-rbac-proxy-cm podName:7687cc83-d880-467f-a270-32bba1bba683 nodeName:}" failed. No retries permitted until 2026-07-29 10:30:02.711523644 +0000 UTC m=+79.436314313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7687cc83-d880-467f-a270-32bba1bba683-kube-rbac-proxy-cm") pod "insights-runtime-extractor-llm74" (UID: "7687cc83-d880-467f-a270-32bba1bba683") : configmap "kube-rbac-proxy" not found
Jul 29 10:30:02.211820 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:02.211629    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7687cc83-d880-467f-a270-32bba1bba683-crio-socket\") pod \"insights-runtime-extractor-llm74\" (UID: \"7687cc83-d880-467f-a270-32bba1bba683\") " pod="openshift-insights/insights-runtime-extractor-llm74"
Jul 29 10:30:02.211820 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:02.211698    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7687cc83-d880-467f-a270-32bba1bba683-crio-socket\") pod \"insights-runtime-extractor-llm74\" (UID: \"7687cc83-d880-467f-a270-32bba1bba683\") " pod="openshift-insights/insights-runtime-extractor-llm74"
Jul 29 10:30:02.211888 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:02.211875    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7687cc83-d880-467f-a270-32bba1bba683-data-volume\") pod \"insights-runtime-extractor-llm74\" (UID: \"7687cc83-d880-467f-a270-32bba1bba683\") " pod="openshift-insights/insights-runtime-extractor-llm74"
Jul 29 10:30:02.221688 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:02.221665    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7fww\" (UniqueName: \"kubernetes.io/projected/7687cc83-d880-467f-a270-32bba1bba683-kube-api-access-q7fww\") pod \"insights-runtime-extractor-llm74\" (UID: \"7687cc83-d880-467f-a270-32bba1bba683\") " pod="openshift-insights/insights-runtime-extractor-llm74"
Jul 29 10:30:02.298474 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:02.298442    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-mhnng"]
Jul 29 10:30:02.301413 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:30:02.301386    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod192911bd_5538_4220_8429_4e435554e1ef.slice/crio-3e185a2688ceb2dac30f07aa5258d3a49ef1a720bc47657dbea09e1376bdbc57 WatchSource:0}: Error finding container 3e185a2688ceb2dac30f07aa5258d3a49ef1a720bc47657dbea09e1376bdbc57: Status 404 returned error can't find the container with id 3e185a2688ceb2dac30f07aa5258d3a49ef1a720bc47657dbea09e1376bdbc57
Jul 29 10:30:02.640201 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:02.640112    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:02.719177 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:02.719137    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7687cc83-d880-467f-a270-32bba1bba683-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llm74\" (UID: \"7687cc83-d880-467f-a270-32bba1bba683\") " pod="openshift-insights/insights-runtime-extractor-llm74"
Jul 29 10:30:02.719390 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:30:02.719305    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 10:30:02.719390 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:02.719315    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7687cc83-d880-467f-a270-32bba1bba683-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-llm74\" (UID: \"7687cc83-d880-467f-a270-32bba1bba683\") " pod="openshift-insights/insights-runtime-extractor-llm74"
Jul 29 10:30:02.719390 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:30:02.719379    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7687cc83-d880-467f-a270-32bba1bba683-insights-runtime-extractor-tls podName:7687cc83-d880-467f-a270-32bba1bba683 nodeName:}" failed. No retries permitted until 2026-07-29 10:30:03.719359603 +0000 UTC m=+80.444150268 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7687cc83-d880-467f-a270-32bba1bba683-insights-runtime-extractor-tls") pod "insights-runtime-extractor-llm74" (UID: "7687cc83-d880-467f-a270-32bba1bba683") : secret "insights-runtime-extractor-tls" not found
Jul 29 10:30:02.719525 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:30:02.719401    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 10:30:02.719525 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:30:02.719447    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7687cc83-d880-467f-a270-32bba1bba683-kube-rbac-proxy-cm podName:7687cc83-d880-467f-a270-32bba1bba683 nodeName:}" failed. No retries permitted until 2026-07-29 10:30:03.719433258 +0000 UTC m=+80.444223934 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7687cc83-d880-467f-a270-32bba1bba683-kube-rbac-proxy-cm") pod "insights-runtime-extractor-llm74" (UID: "7687cc83-d880-467f-a270-32bba1bba683") : configmap "kube-rbac-proxy" not found
Jul 29 10:30:03.161629 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:03.161326    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-nslw9" event={"ID":"0b327881-88ad-4d8d-b7b7-d73bc8c59e2d","Type":"ContainerStarted","Data":"02b66095e36accb1eca466aa6178286776920d4e22777a1e79730b208d68e150"}
Jul 29 10:30:03.163561 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:03.163469    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-mhnng" event={"ID":"192911bd-5538-4220-8429-4e435554e1ef","Type":"ContainerStarted","Data":"3e185a2688ceb2dac30f07aa5258d3a49ef1a720bc47657dbea09e1376bdbc57"}
Jul 29 10:30:03.181112 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:03.179963    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-nslw9" podStartSLOduration=2.179944182 podStartE2EDuration="2.179944182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:30:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:03.17840155 +0000 UTC m=+79.903192229" watchObservedRunningTime="2026-07-29 10:30:03.179944182 +0000 UTC m=+79.904734856"
Jul 29 10:30:03.189861 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:03.189822    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:03.474384 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:03.474348    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:03.588655 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:03.586292    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:03.670821 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:03.670788    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:03.739696 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:03.739584    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7687cc83-d880-467f-a270-32bba1bba683-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llm74\" (UID: \"7687cc83-d880-467f-a270-32bba1bba683\") " pod="openshift-insights/insights-runtime-extractor-llm74"
Jul 29 10:30:03.739696 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:03.739671    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7687cc83-d880-467f-a270-32bba1bba683-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-llm74\" (UID: \"7687cc83-d880-467f-a270-32bba1bba683\") " pod="openshift-insights/insights-runtime-extractor-llm74"
Jul 29 10:30:03.739931 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:30:03.739762    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 10:30:03.739931 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:30:03.739838    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7687cc83-d880-467f-a270-32bba1bba683-kube-rbac-proxy-cm podName:7687cc83-d880-467f-a270-32bba1bba683 nodeName:}" failed. No retries permitted until 2026-07-29 10:30:05.73982233 +0000 UTC m=+82.464613012 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7687cc83-d880-467f-a270-32bba1bba683-kube-rbac-proxy-cm") pod "insights-runtime-extractor-llm74" (UID: "7687cc83-d880-467f-a270-32bba1bba683") : configmap "kube-rbac-proxy" not found
Jul 29 10:30:03.739931 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:30:03.739762    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 10:30:03.739931 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:30:03.739913    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7687cc83-d880-467f-a270-32bba1bba683-insights-runtime-extractor-tls podName:7687cc83-d880-467f-a270-32bba1bba683 nodeName:}" failed. No retries permitted until 2026-07-29 10:30:05.739894818 +0000 UTC m=+82.464685484 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7687cc83-d880-467f-a270-32bba1bba683-insights-runtime-extractor-tls") pod "insights-runtime-extractor-llm74" (UID: "7687cc83-d880-467f-a270-32bba1bba683") : secret "insights-runtime-extractor-tls" not found
Jul 29 10:30:03.800663 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:03.800628    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:04.017659 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:04.017564    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:04.168842 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:04.168801    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-mhnng" event={"ID":"192911bd-5538-4220-8429-4e435554e1ef","Type":"ContainerStarted","Data":"1075710a89076a754922dd263d2fb3f178ba2942b7f913b3ddfdde4a1a13ddea"}
Jul 29 10:30:04.169348 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:04.168875    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-mhnng" event={"ID":"192911bd-5538-4220-8429-4e435554e1ef","Type":"ContainerStarted","Data":"5f4a3fdff382e98729a5a477d432ae0525fe800fc187e3fb8a210944c980645a"}
Jul 29 10:30:04.187408 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:04.187349    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-mhnng" podStartSLOduration=1.985205026 podStartE2EDuration="3.187327625s" totalImagesPullingTime="1.202122599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:30:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:04.185727087 +0000 UTC m=+80.910517759" watchObservedRunningTime="2026-07-29 10:30:04.187327625 +0000 UTC m=+80.912118299"
Jul 29 10:30:04.461764 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:04.461729    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:04.760238 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:04.760163    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hpmhx_faeca467-a198-4dd6-9da8-e4f72ac68c77/dns-node-resolver/0.log"
Jul 29 10:30:04.867429 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:04.867392    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:05.471188 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:05.471152    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:05.560117 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:05.560063    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-b6v6k_bd747eb7-faeb-48ab-804a-96355d0b79c4/node-ca/0.log"
Jul 29 10:30:05.622223 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:05.622191    2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-pvdgw"
Jul 29 10:30:05.622367 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:05.622244    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-pvdgw"
Jul 29 10:30:05.622581 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:05.622570    2569 scope.go:117] "RemoveContainer" containerID="e620005f6f3fe05236768eabbc52540b1c3397fcb50b135b45be799901a6a280"
Jul 29 10:30:05.622728 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:30:05.622709    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-pvdgw_openshift-console-operator(afd0d65c-bbfc-49f0-81b6-e855fd81da7f)\"" pod="openshift-console-operator/console-operator-584dcb6b54-pvdgw" podUID="afd0d65c-bbfc-49f0-81b6-e855fd81da7f"
Jul 29 10:30:05.756608 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:05.756513    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7687cc83-d880-467f-a270-32bba1bba683-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llm74\" (UID: \"7687cc83-d880-467f-a270-32bba1bba683\") " pod="openshift-insights/insights-runtime-extractor-llm74"
Jul 29 10:30:05.756608 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:05.756577    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7687cc83-d880-467f-a270-32bba1bba683-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-llm74\" (UID: \"7687cc83-d880-467f-a270-32bba1bba683\") " pod="openshift-insights/insights-runtime-extractor-llm74"
Jul 29 10:30:05.756838 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:30:05.756629    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 10:30:05.756838 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:30:05.756653    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 10:30:05.756838 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:30:05.756680    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7687cc83-d880-467f-a270-32bba1bba683-kube-rbac-proxy-cm podName:7687cc83-d880-467f-a270-32bba1bba683 nodeName:}" failed. No retries permitted until 2026-07-29 10:30:09.756662803 +0000 UTC m=+86.481453454 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7687cc83-d880-467f-a270-32bba1bba683-kube-rbac-proxy-cm") pod "insights-runtime-extractor-llm74" (UID: "7687cc83-d880-467f-a270-32bba1bba683") : configmap "kube-rbac-proxy" not found
Jul 29 10:30:05.756838 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:30:05.756726    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7687cc83-d880-467f-a270-32bba1bba683-insights-runtime-extractor-tls podName:7687cc83-d880-467f-a270-32bba1bba683 nodeName:}" failed. No retries permitted until 2026-07-29 10:30:09.756708336 +0000 UTC m=+86.481498986 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7687cc83-d880-467f-a270-32bba1bba683-insights-runtime-extractor-tls") pod "insights-runtime-extractor-llm74" (UID: "7687cc83-d880-467f-a270-32bba1bba683") : secret "insights-runtime-extractor-tls" not found
Jul 29 10:30:06.066329 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:06.066186    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:06.166813 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:06.166785    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:06.269630 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:06.269588    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:06.560996 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:06.560963    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jvdpp_f3e9971d-59b4-4ea2-a9f3-5792b9e0b804/serve-healthcheck-canary/0.log"
Jul 29 10:30:06.685592 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:06.685561    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:06.868337 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:06.868251    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:06.936406 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:06.936375    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:06.961496 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:06.961470    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-mhnng_192911bd-5538-4220-8429-4e435554e1ef/migrator/0.log"
Jul 29 10:30:07.160515 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:07.160444    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-mhnng_192911bd-5538-4220-8429-4e435554e1ef/graceful-termination/0.log"
Jul 29 10:30:07.367523 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:07.367475    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:07.474247 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:07.474211    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:08.080975 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:08.080943    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:08.270265 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:08.270230    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:08.467265 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:08.467228    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:08.868761 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:08.868670    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:09.473722 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:09.473690    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:09.573164 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:09.573136    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:09.787767 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:09.787683    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7687cc83-d880-467f-a270-32bba1bba683-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llm74\" (UID: \"7687cc83-d880-467f-a270-32bba1bba683\") " pod="openshift-insights/insights-runtime-extractor-llm74"
Jul 29 10:30:09.787767 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:09.787731    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7687cc83-d880-467f-a270-32bba1bba683-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-llm74\" (UID: \"7687cc83-d880-467f-a270-32bba1bba683\") " pod="openshift-insights/insights-runtime-extractor-llm74"
Jul 29 10:30:09.787941 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:30:09.787801    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 10:30:09.787941 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:30:09.787820    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 10:30:09.787941 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:30:09.787852    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7687cc83-d880-467f-a270-32bba1bba683-kube-rbac-proxy-cm podName:7687cc83-d880-467f-a270-32bba1bba683 nodeName:}" failed. No retries permitted until 2026-07-29 10:30:17.787839157 +0000 UTC m=+94.512629808 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7687cc83-d880-467f-a270-32bba1bba683-kube-rbac-proxy-cm") pod "insights-runtime-extractor-llm74" (UID: "7687cc83-d880-467f-a270-32bba1bba683") : configmap "kube-rbac-proxy" not found
Jul 29 10:30:09.787941 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:30:09.787865    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7687cc83-d880-467f-a270-32bba1bba683-insights-runtime-extractor-tls podName:7687cc83-d880-467f-a270-32bba1bba683 nodeName:}" failed. No retries permitted until 2026-07-29 10:30:17.787859189 +0000 UTC m=+94.512649839 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7687cc83-d880-467f-a270-32bba1bba683-insights-runtime-extractor-tls") pod "insights-runtime-extractor-llm74" (UID: "7687cc83-d880-467f-a270-32bba1bba683") : secret "insights-runtime-extractor-tls" not found
Jul 29 10:30:10.065368 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:10.065290    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:10.603483 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:10.603451    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:10.704669 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:10.704641    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:10.870397 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:10.870314    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:11.475951 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:11.475914    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:12.069396 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:12.069365    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:12.271598 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:12.271563    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:12.467718 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:12.467681    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:12.777538 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:12.777459    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:12.876149 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:12.876115    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:13.269481 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:13.269448    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:13.603439 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:13.603353    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:14.132542 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:14.132502    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:14.199082 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:14.199038    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:14.300801 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:14.300766    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:16.832264 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:16.832232    2569 scope.go:117] "RemoveContainer" containerID="e620005f6f3fe05236768eabbc52540b1c3397fcb50b135b45be799901a6a280"
Jul 29 10:30:17.209627 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:17.209589    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-pvdgw_afd0d65c-bbfc-49f0-81b6-e855fd81da7f/console-operator/1.log"
Jul 29 10:30:17.209780 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:17.209685    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-pvdgw" event={"ID":"afd0d65c-bbfc-49f0-81b6-e855fd81da7f","Type":"ContainerStarted","Data":"d1b79f94785672979a7dff02eeddb5eab9589a0b222a26e95a643dd0f12f10da"}
Jul 29 10:30:17.210065 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:17.210043    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-pvdgw"
Jul 29 10:30:17.229101 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:17.229038    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-pvdgw" podStartSLOduration=19.931511185 podStartE2EDuration="22.229023697s" totalImagesPullingTime="2.297512512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:17.228605519 +0000 UTC m=+93.953396192" watchObservedRunningTime="2026-07-29 10:30:17.229023697 +0000 UTC m=+93.953814389"
Jul 29 10:30:17.241791 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:17.241767    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-pvdgw"
Jul 29 10:30:17.421567 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:17.421531    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:17.523012 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:17.522932    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:17.640315 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:17.640281    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:17.734924 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:17.734889    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:17.853447 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:17.853364    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7687cc83-d880-467f-a270-32bba1bba683-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llm74\" (UID: \"7687cc83-d880-467f-a270-32bba1bba683\") " pod="openshift-insights/insights-runtime-extractor-llm74"
Jul 29 10:30:17.853447 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:17.853418    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7687cc83-d880-467f-a270-32bba1bba683-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-llm74\" (UID: \"7687cc83-d880-467f-a270-32bba1bba683\") " pod="openshift-insights/insights-runtime-extractor-llm74"
Jul 29 10:30:17.853905 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:17.853887    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7687cc83-d880-467f-a270-32bba1bba683-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-llm74\" (UID: \"7687cc83-d880-467f-a270-32bba1bba683\") " pod="openshift-insights/insights-runtime-extractor-llm74"
Jul 29 10:30:17.855655 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:17.855635    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7687cc83-d880-467f-a270-32bba1bba683-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llm74\" (UID: \"7687cc83-d880-467f-a270-32bba1bba683\") " pod="openshift-insights/insights-runtime-extractor-llm74"
Jul 29 10:30:17.891836 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:17.891805    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-llm74"
Jul 29 10:30:17.913884 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:17.913856    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:17.989721 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:17.989693    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:18.028594 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:18.028558    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-llm74"]
Jul 29 10:30:18.031748 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:30:18.031707    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7687cc83_d880_467f_a270_32bba1bba683.slice/crio-9386e221a3784dd6e3cb5be5660aa1920caeb097cdadf151e37234bd4c989e61 WatchSource:0}: Error finding container 9386e221a3784dd6e3cb5be5660aa1920caeb097cdadf151e37234bd4c989e61: Status 404 returned error can't find the container with id 9386e221a3784dd6e3cb5be5660aa1920caeb097cdadf151e37234bd4c989e61
Jul 29 10:30:18.138933 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:18.138849    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:18.213572 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:18.213539    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-llm74" event={"ID":"7687cc83-d880-467f-a270-32bba1bba683","Type":"ContainerStarted","Data":"9386e221a3784dd6e3cb5be5660aa1920caeb097cdadf151e37234bd4c989e61"}
Jul 29 10:30:18.239093 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:18.239036    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:18.347944 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:18.347914    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:18.416170 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:18.416136    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:19.116222 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:19.116186    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:19.321293 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:19.321243    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:19.514319 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:19.514282    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:19.581516 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:19.581480    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:20.539153 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:20.539103    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:20.643043 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:20.643009    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:20.978432 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:20.978397    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:21.037518 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:21.037486    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:22.592270 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:22.592226    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-registry-tls\") pod \"image-registry-66f4d7d896-fbvv2\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") " pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:30:22.592834 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:22.592314    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5edb4b57-220e-4f39-b481-164e813bbf3c-metrics-tls\") pod \"dns-default-ljsqp\" (UID: \"5edb4b57-220e-4f39-b481-164e813bbf3c\") " pod="openshift-dns/dns-default-ljsqp"
Jul 29 10:30:22.594793 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:22.594760    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-registry-tls\") pod \"image-registry-66f4d7d896-fbvv2\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") " pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:30:22.595042 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:22.595019    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5edb4b57-220e-4f39-b481-164e813bbf3c-metrics-tls\") pod \"dns-default-ljsqp\" (UID: \"5edb4b57-220e-4f39-b481-164e813bbf3c\") " pod="openshift-dns/dns-default-ljsqp"
Jul 29 10:30:22.825660 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:22.825627    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-c2kld\""
Jul 29 10:30:22.833892 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:22.833868    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:30:22.844638 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:22.844620    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hm7tz\""
Jul 29 10:30:22.851822 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:22.851797    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ljsqp"
Jul 29 10:30:22.980246 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:22.980216    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66f4d7d896-fbvv2"]
Jul 29 10:30:22.983473 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:30:22.983446    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod662f9f8c_cd9a_4722_ab84_59cff17809ef.slice/crio-32df146a0df4166563abe967194ebd874b1f968bc8693d612b6da1cbe345a3af WatchSource:0}: Error finding container 32df146a0df4166563abe967194ebd874b1f968bc8693d612b6da1cbe345a3af: Status 404 returned error can't find the container with id 32df146a0df4166563abe967194ebd874b1f968bc8693d612b6da1cbe345a3af
Jul 29 10:30:22.995850 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:22.995823    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ljsqp"]
Jul 29 10:30:22.999181 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:30:22.999158    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5edb4b57_220e_4f39_b481_164e813bbf3c.slice/crio-2ac16b5d0c1dac3ab7da5f18ff638490186c6c587fc0e8a81782f9fab0658a31 WatchSource:0}: Error finding container 2ac16b5d0c1dac3ab7da5f18ff638490186c6c587fc0e8a81782f9fab0658a31: Status 404 returned error can't find the container with id 2ac16b5d0c1dac3ab7da5f18ff638490186c6c587fc0e8a81782f9fab0658a31
Jul 29 10:30:23.237287 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:23.237117    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2" event={"ID":"662f9f8c-cd9a-4722-ab84-59cff17809ef","Type":"ContainerStarted","Data":"16b1e17f5f02a64b6b26f19c19b0b3ae61f92a22ae365b4cb244228e4f4a5c44"}
Jul 29 10:30:23.237287 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:23.237182    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2" event={"ID":"662f9f8c-cd9a-4722-ab84-59cff17809ef","Type":"ContainerStarted","Data":"32df146a0df4166563abe967194ebd874b1f968bc8693d612b6da1cbe345a3af"}
Jul 29 10:30:23.237287 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:23.237240    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:30:23.238965 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:23.238936    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-llm74" event={"ID":"7687cc83-d880-467f-a270-32bba1bba683","Type":"ContainerStarted","Data":"f0397b09e9b5e82ae73580eba909e67b53355a4d1d9713fd321b3e80077a676a"}
Jul 29 10:30:23.240162 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:23.240128    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ljsqp" event={"ID":"5edb4b57-220e-4f39-b481-164e813bbf3c","Type":"ContainerStarted","Data":"2ac16b5d0c1dac3ab7da5f18ff638490186c6c587fc0e8a81782f9fab0658a31"}
Jul 29 10:30:23.283707 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:23.283344    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2" podStartSLOduration=98.283322031 podStartE2EDuration="1m38.283322031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:23.277889595 +0000 UTC m=+100.002680269" watchObservedRunningTime="2026-07-29 10:30:23.283322031 +0000 UTC m=+100.008112757"
Jul 29 10:30:23.335685 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:23.335646    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:23.362516 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:23.362480    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-7gk4j"]
Jul 29 10:30:23.365776 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:23.365749    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-7gk4j"
Jul 29 10:30:23.368916 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:23.368675    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-j8mgs\""
Jul 29 10:30:23.368916 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:23.368720    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 29 10:30:23.368916 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:23.368754    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 29 10:30:23.382165 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:23.381724    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-7gk4j"]
Jul 29 10:30:23.397526 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:23.397454    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7x7c\" (UniqueName: \"kubernetes.io/projected/f0278c1e-7a6a-472f-aaba-09c9a22c9fa9-kube-api-access-t7x7c\") pod \"downloads-7c97dc4555-7gk4j\" (UID: \"f0278c1e-7a6a-472f-aaba-09c9a22c9fa9\") " pod="openshift-console/downloads-7c97dc4555-7gk4j"
Jul 29 10:30:23.458168 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:23.458043    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:23.498376 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:23.497992    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7x7c\" (UniqueName: \"kubernetes.io/projected/f0278c1e-7a6a-472f-aaba-09c9a22c9fa9-kube-api-access-t7x7c\") pod \"downloads-7c97dc4555-7gk4j\" (UID: \"f0278c1e-7a6a-472f-aaba-09c9a22c9fa9\") " pod="openshift-console/downloads-7c97dc4555-7gk4j"
Jul 29 10:30:23.511679 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:23.511614    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7x7c\" (UniqueName: \"kubernetes.io/projected/f0278c1e-7a6a-472f-aaba-09c9a22c9fa9-kube-api-access-t7x7c\") pod \"downloads-7c97dc4555-7gk4j\" (UID: \"f0278c1e-7a6a-472f-aaba-09c9a22c9fa9\") " pod="openshift-console/downloads-7c97dc4555-7gk4j"
Jul 29 10:30:23.603723 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:23.603645    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:23.677802 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:23.677771    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-7gk4j"
Jul 29 10:30:23.868490 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:23.868419    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-7gk4j"]
Jul 29 10:30:23.873214 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:30:23.873178    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0278c1e_7a6a_472f_aaba_09c9a22c9fa9.slice/crio-8d62b831481569dc9d23f1f5602155198ace830dbd85865d3ccafab82983340e WatchSource:0}: Error finding container 8d62b831481569dc9d23f1f5602155198ace830dbd85865d3ccafab82983340e: Status 404 returned error can't find the container with id 8d62b831481569dc9d23f1f5602155198ace830dbd85865d3ccafab82983340e
Jul 29 10:30:24.247176 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:24.247128    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-7gk4j" event={"ID":"f0278c1e-7a6a-472f-aaba-09c9a22c9fa9","Type":"ContainerStarted","Data":"8d62b831481569dc9d23f1f5602155198ace830dbd85865d3ccafab82983340e"}
Jul 29 10:30:26.125970 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:26.125934    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mdqnc"
Jul 29 10:30:28.268003 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:28.267969    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ljsqp" event={"ID":"5edb4b57-220e-4f39-b481-164e813bbf3c","Type":"ContainerStarted","Data":"16769326d99af0b82b8367017010adf53fa69a9a354851b7b8eb4f2326993233"}
Jul 29 10:30:29.275630 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:29.275591    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ljsqp" event={"ID":"5edb4b57-220e-4f39-b481-164e813bbf3c","Type":"ContainerStarted","Data":"f32e8c9412aa581cc2551b2d0921a125d177c59a0e297f0ef74fe5162542b479"}
Jul 29 10:30:29.276114 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:29.275681    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ljsqp"
Jul 29 10:30:29.277631 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:29.277589    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-llm74" event={"ID":"7687cc83-d880-467f-a270-32bba1bba683","Type":"ContainerStarted","Data":"a7397c2c9edd4eb07aee85aaf9af3248e2ceb00918af17042a4d34a5b85436ce"}
Jul 29 10:30:29.298436 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:29.298383    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ljsqp" podStartSLOduration=66.403011743 podStartE2EDuration="1m11.298362046s" totalImagesPullingTime="4.895350303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:29.298161784 +0000 UTC m=+106.022952458" watchObservedRunningTime="2026-07-29 10:30:29.298362046 +0000 UTC m=+106.023152720"
Jul 29 10:30:29.946280 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:29.946244    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7cdb5fcdf7-wclz4"]
Jul 29 10:30:29.963803 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:29.963769    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:29.967997 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:29.967960    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cdb5fcdf7-wclz4"]
Jul 29 10:30:29.968166 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:29.968091    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cdb5fcdf7-wclz4"
Jul 29 10:30:29.973043 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:29.973016    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 29 10:30:29.973339 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:29.973320    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 29 10:30:29.973959 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:29.973933    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 29 10:30:29.974711 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:29.974690    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 29 10:30:29.974877 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:29.974859    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-nkcld\""
Jul 29 10:30:29.974949 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:29.974859    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 29 10:30:29.983222 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:29.983194    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 29 10:30:30.042655 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.042611    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0cd86203-fe78-4a76-ab30-d13780b1a3db-service-ca\") pod \"console-7cdb5fcdf7-wclz4\" (UID: \"0cd86203-fe78-4a76-ab30-d13780b1a3db\") " pod="openshift-console/console-7cdb5fcdf7-wclz4"
Jul 29 10:30:30.042655 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.042657    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cd86203-fe78-4a76-ab30-d13780b1a3db-trusted-ca-bundle\") pod \"console-7cdb5fcdf7-wclz4\" (UID: \"0cd86203-fe78-4a76-ab30-d13780b1a3db\") " pod="openshift-console/console-7cdb5fcdf7-wclz4"
Jul 29 10:30:30.042896 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.042745    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0cd86203-fe78-4a76-ab30-d13780b1a3db-console-oauth-config\") pod \"console-7cdb5fcdf7-wclz4\" (UID: \"0cd86203-fe78-4a76-ab30-d13780b1a3db\") " pod="openshift-console/console-7cdb5fcdf7-wclz4"
Jul 29 10:30:30.042896 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.042795    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0cd86203-fe78-4a76-ab30-d13780b1a3db-console-config\") pod \"console-7cdb5fcdf7-wclz4\" (UID: \"0cd86203-fe78-4a76-ab30-d13780b1a3db\") " pod="openshift-console/console-7cdb5fcdf7-wclz4"
Jul 29 10:30:30.042896 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.042852    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqrjv\" (UniqueName: \"kubernetes.io/projected/0cd86203-fe78-4a76-ab30-d13780b1a3db-kube-api-access-gqrjv\") pod \"console-7cdb5fcdf7-wclz4\" (UID: \"0cd86203-fe78-4a76-ab30-d13780b1a3db\") " pod="openshift-console/console-7cdb5fcdf7-wclz4"
Jul 29 10:30:30.042896 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.042870    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0cd86203-fe78-4a76-ab30-d13780b1a3db-oauth-serving-cert\") pod \"console-7cdb5fcdf7-wclz4\" (UID: \"0cd86203-fe78-4a76-ab30-d13780b1a3db\") " pod="openshift-console/console-7cdb5fcdf7-wclz4"
Jul 29 10:30:30.043129 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.042912    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cd86203-fe78-4a76-ab30-d13780b1a3db-console-serving-cert\") pod \"console-7cdb5fcdf7-wclz4\" (UID: \"0cd86203-fe78-4a76-ab30-d13780b1a3db\") " pod="openshift-console/console-7cdb5fcdf7-wclz4"
Jul 29 10:30:30.095048 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.095013    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-qzzx2"]
Jul 29 10:30:30.118267 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.118237    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-qzzx2"]
Jul 29 10:30:30.118464 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.118360    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-qzzx2"
Jul 29 10:30:30.121619 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.121587    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 29 10:30:30.121843 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.121825    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-q8vqg\""
Jul 29 10:30:30.122138 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.122051    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 29 10:30:30.122235 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.122214    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 29 10:30:30.123052 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.122533    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 29 10:30:30.123052 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.122550    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 29 10:30:30.143876 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.143847    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqrjv\" (UniqueName: \"kubernetes.io/projected/0cd86203-fe78-4a76-ab30-d13780b1a3db-kube-api-access-gqrjv\") pod \"console-7cdb5fcdf7-wclz4\" (UID: \"0cd86203-fe78-4a76-ab30-d13780b1a3db\") " pod="openshift-console/console-7cdb5fcdf7-wclz4"
Jul 29 10:30:30.143994 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.143893    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0cd86203-fe78-4a76-ab30-d13780b1a3db-oauth-serving-cert\") pod \"console-7cdb5fcdf7-wclz4\" (UID: \"0cd86203-fe78-4a76-ab30-d13780b1a3db\") " pod="openshift-console/console-7cdb5fcdf7-wclz4"
Jul 29 10:30:30.143994 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.143938    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/70bc0d3f-ea52-4c82-b033-961c7fd5fc8c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-qzzx2\" (UID: \"70bc0d3f-ea52-4c82-b033-961c7fd5fc8c\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-qzzx2"
Jul 29 10:30:30.143994 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.143974    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70bc0d3f-ea52-4c82-b033-961c7fd5fc8c-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-qzzx2\" (UID: \"70bc0d3f-ea52-4c82-b033-961c7fd5fc8c\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-qzzx2"
Jul 29 10:30:30.145875 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.144003    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cd86203-fe78-4a76-ab30-d13780b1a3db-console-serving-cert\") pod \"console-7cdb5fcdf7-wclz4\" (UID: \"0cd86203-fe78-4a76-ab30-d13780b1a3db\") " pod="openshift-console/console-7cdb5fcdf7-wclz4"
Jul 29 10:30:30.145875 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.144033    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/70bc0d3f-ea52-4c82-b033-961c7fd5fc8c-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-qzzx2\" (UID: \"70bc0d3f-ea52-4c82-b033-961c7fd5fc8c\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-qzzx2"
Jul 29 10:30:30.145875 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.144067    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0cd86203-fe78-4a76-ab30-d13780b1a3db-service-ca\") pod \"console-7cdb5fcdf7-wclz4\" (UID: \"0cd86203-fe78-4a76-ab30-d13780b1a3db\") " pod="openshift-console/console-7cdb5fcdf7-wclz4"
Jul 29 10:30:30.145875 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.144114    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cd86203-fe78-4a76-ab30-d13780b1a3db-trusted-ca-bundle\") pod \"console-7cdb5fcdf7-wclz4\" (UID: \"0cd86203-fe78-4a76-ab30-d13780b1a3db\") " pod="openshift-console/console-7cdb5fcdf7-wclz4"
Jul 29 10:30:30.145875 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.144307    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5tzd\" (UniqueName: \"kubernetes.io/projected/70bc0d3f-ea52-4c82-b033-961c7fd5fc8c-kube-api-access-x5tzd\") pod \"prometheus-operator-84d69f794d-qzzx2\" (UID: \"70bc0d3f-ea52-4c82-b033-961c7fd5fc8c\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-qzzx2"
Jul 29 10:30:30.145875 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.144358    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0cd86203-fe78-4a76-ab30-d13780b1a3db-console-oauth-config\") pod \"console-7cdb5fcdf7-wclz4\" (UID: \"0cd86203-fe78-4a76-ab30-d13780b1a3db\") " pod="openshift-console/console-7cdb5fcdf7-wclz4"
Jul 29 10:30:30.145875 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.144388    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0cd86203-fe78-4a76-ab30-d13780b1a3db-console-config\") pod \"console-7cdb5fcdf7-wclz4\" (UID: \"0cd86203-fe78-4a76-ab30-d13780b1a3db\") " pod="openshift-console/console-7cdb5fcdf7-wclz4"
Jul 29 10:30:30.145875 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.144884    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0cd86203-fe78-4a76-ab30-d13780b1a3db-oauth-serving-cert\") pod \"console-7cdb5fcdf7-wclz4\" (UID: \"0cd86203-fe78-4a76-ab30-d13780b1a3db\") " pod="openshift-console/console-7cdb5fcdf7-wclz4"
Jul 29 10:30:30.145875 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.145036    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0cd86203-fe78-4a76-ab30-d13780b1a3db-console-config\") pod \"console-7cdb5fcdf7-wclz4\" (UID: \"0cd86203-fe78-4a76-ab30-d13780b1a3db\") " pod="openshift-console/console-7cdb5fcdf7-wclz4"
Jul 29 10:30:30.145875 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.145285    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0cd86203-fe78-4a76-ab30-d13780b1a3db-service-ca\") pod \"console-7cdb5fcdf7-wclz4\" (UID: \"0cd86203-fe78-4a76-ab30-d13780b1a3db\") " pod="openshift-console/console-7cdb5fcdf7-wclz4"
Jul 29 10:30:30.145875 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.145729    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cd86203-fe78-4a76-ab30-d13780b1a3db-trusted-ca-bundle\") pod \"console-7cdb5fcdf7-wclz4\" (UID: \"0cd86203-fe78-4a76-ab30-d13780b1a3db\") " pod="openshift-console/console-7cdb5fcdf7-wclz4"
Jul 29 10:30:30.147728 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.147685    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0cd86203-fe78-4a76-ab30-d13780b1a3db-console-oauth-config\") pod \"console-7cdb5fcdf7-wclz4\" (UID: \"0cd86203-fe78-4a76-ab30-d13780b1a3db\") " pod="openshift-console/console-7cdb5fcdf7-wclz4"
Jul 29 10:30:30.151466 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.151441    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cd86203-fe78-4a76-ab30-d13780b1a3db-console-serving-cert\") pod \"console-7cdb5fcdf7-wclz4\" (UID: \"0cd86203-fe78-4a76-ab30-d13780b1a3db\") " pod="openshift-console/console-7cdb5fcdf7-wclz4"
Jul 29 10:30:30.154985 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.154962    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqrjv\" (UniqueName: \"kubernetes.io/projected/0cd86203-fe78-4a76-ab30-d13780b1a3db-kube-api-access-gqrjv\") pod \"console-7cdb5fcdf7-wclz4\" (UID: \"0cd86203-fe78-4a76-ab30-d13780b1a3db\") " pod="openshift-console/console-7cdb5fcdf7-wclz4"
Jul 29 10:30:30.244838 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.244749    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/70bc0d3f-ea52-4c82-b033-961c7fd5fc8c-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-qzzx2\" (UID: \"70bc0d3f-ea52-4c82-b033-961c7fd5fc8c\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-qzzx2"
Jul 29 10:30:30.244838 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.244817    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5tzd\" (UniqueName: \"kubernetes.io/projected/70bc0d3f-ea52-4c82-b033-961c7fd5fc8c-kube-api-access-x5tzd\") pod \"prometheus-operator-84d69f794d-qzzx2\" (UID: \"70bc0d3f-ea52-4c82-b033-961c7fd5fc8c\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-qzzx2"
Jul 29 10:30:30.245104 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.244982    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/70bc0d3f-ea52-4c82-b033-961c7fd5fc8c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-qzzx2\" (UID: \"70bc0d3f-ea52-4c82-b033-961c7fd5fc8c\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-qzzx2"
Jul 29 10:30:30.245104 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.245021    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70bc0d3f-ea52-4c82-b033-961c7fd5fc8c-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-qzzx2\" (UID: \"70bc0d3f-ea52-4c82-b033-961c7fd5fc8c\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-qzzx2"
Jul 29 10:30:30.246853 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.246798    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70bc0d3f-ea52-4c82-b033-961c7fd5fc8c-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-qzzx2\" (UID: \"70bc0d3f-ea52-4c82-b033-961c7fd5fc8c\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-qzzx2"
Jul 29 10:30:30.248173 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.247818    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/70bc0d3f-ea52-4c82-b033-961c7fd5fc8c-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-qzzx2\" (UID: \"70bc0d3f-ea52-4c82-b033-961c7fd5fc8c\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-qzzx2"
Jul 29 10:30:30.249560 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.249515    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/70bc0d3f-ea52-4c82-b033-961c7fd5fc8c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-qzzx2\" (UID: \"70bc0d3f-ea52-4c82-b033-961c7fd5fc8c\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-qzzx2"
Jul 29 10:30:30.254671 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.254627    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5tzd\" (UniqueName: \"kubernetes.io/projected/70bc0d3f-ea52-4c82-b033-961c7fd5fc8c-kube-api-access-x5tzd\") pod \"prometheus-operator-84d69f794d-qzzx2\" (UID: \"70bc0d3f-ea52-4c82-b033-961c7fd5fc8c\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-qzzx2"
Jul 29 10:30:30.280360 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.280311    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cdb5fcdf7-wclz4"
Jul 29 10:30:30.374559 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.374531    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:30.430725 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.430477    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-qzzx2"
Jul 29 10:30:30.444506 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.444418    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:30.471387 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.471350    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cdb5fcdf7-wclz4"]
Jul 29 10:30:30.474741 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:30:30.474707    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cd86203_fe78_4a76_ab30_d13780b1a3db.slice/crio-4754380593fbb15e507fb4ef7549b44fab048ca227f26fdfb130edccd3622316 WatchSource:0}: Error finding container 4754380593fbb15e507fb4ef7549b44fab048ca227f26fdfb130edccd3622316: Status 404 returned error can't find the container with id 4754380593fbb15e507fb4ef7549b44fab048ca227f26fdfb130edccd3622316
Jul 29 10:30:30.617189 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:30.617144    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-qzzx2"]
Jul 29 10:30:31.287670 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:31.287630    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cdb5fcdf7-wclz4" event={"ID":"0cd86203-fe78-4a76-ab30-d13780b1a3db","Type":"ContainerStarted","Data":"4754380593fbb15e507fb4ef7549b44fab048ca227f26fdfb130edccd3622316"}
Jul 29 10:30:31.289033 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:31.288993    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-qzzx2" event={"ID":"70bc0d3f-ea52-4c82-b033-961c7fd5fc8c","Type":"ContainerStarted","Data":"289cba41bbb54997f9042085bce1d15b11c304c8db41519a56cc6c5c199e9cc1"}
Jul 29 10:30:32.328340 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:32.328299    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:33.137198 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:33.137062    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:33.248227 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:33.248150    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:33.299631 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:33.299589    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-llm74" event={"ID":"7687cc83-d880-467f-a270-32bba1bba683","Type":"ContainerStarted","Data":"d96c7bed35812fac2476552b19cb36a30dfd86202c3b2907abe296ad89adbb46"}
Jul 29 10:30:33.319557 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:33.319491    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-llm74" podStartSLOduration=17.819128318 podStartE2EDuration="32.319467135s" totalImagesPullingTime="14.500338817s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:30:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:33.317770426 +0000 UTC m=+110.042561106" watchObservedRunningTime="2026-07-29 10:30:33.319467135 +0000 UTC m=+110.044257809"
Jul 29 10:30:34.069492 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:34.069406    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:34.308537 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:34.308498    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-qzzx2" event={"ID":"70bc0d3f-ea52-4c82-b033-961c7fd5fc8c","Type":"ContainerStarted","Data":"a14dc1a9d2f2b070097a10c7800f846543af8533be7eca6b6b4de23c7e1f0f7d"}
Jul 29 10:30:34.308730 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:34.308561    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-qzzx2" event={"ID":"70bc0d3f-ea52-4c82-b033-961c7fd5fc8c","Type":"ContainerStarted","Data":"f2de4fa60c1022974b01491902ae13ace61b402febe505bc8ce95273759d14c8"}
Jul 29 10:30:34.327318 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:34.327215    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-qzzx2" podStartSLOduration=1.17789719 podStartE2EDuration="4.327193073s" totalImagesPullingTime="3.149295883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:30:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:34.326505481 +0000 UTC m=+111.051296156" watchObservedRunningTime="2026-07-29 10:30:34.327193073 +0000 UTC m=+111.051983749"
Jul 29 10:30:36.178387 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.178348    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:36.274219 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.274175    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:36.451104 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.450546    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dn2fp"]
Jul 29 10:30:36.454841 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.454818    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dn2fp"
Jul 29 10:30:36.458617 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.458021    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-22g7n\""
Jul 29 10:30:36.458617 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.458263    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 29 10:30:36.458617 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.457768    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 29 10:30:36.458787 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.458652    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 29 10:30:36.491416 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.491379    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:36.499063 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.499035    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m9tf\" (UniqueName: \"kubernetes.io/projected/683dda88-94e8-4960-be54-7e26765bd876-kube-api-access-4m9tf\") pod \"node-exporter-dn2fp\" (UID: \"683dda88-94e8-4960-be54-7e26765bd876\") " pod="openshift-monitoring/node-exporter-dn2fp"
Jul 29 10:30:36.499206 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.499109    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/683dda88-94e8-4960-be54-7e26765bd876-sys\") pod \"node-exporter-dn2fp\" (UID: \"683dda88-94e8-4960-be54-7e26765bd876\") " pod="openshift-monitoring/node-exporter-dn2fp"
Jul 29 10:30:36.499206 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.499130    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/683dda88-94e8-4960-be54-7e26765bd876-node-exporter-textfile\") pod \"node-exporter-dn2fp\" (UID: \"683dda88-94e8-4960-be54-7e26765bd876\") " pod="openshift-monitoring/node-exporter-dn2fp"
Jul 29 10:30:36.499206 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.499157    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/683dda88-94e8-4960-be54-7e26765bd876-root\") pod \"node-exporter-dn2fp\" (UID: \"683dda88-94e8-4960-be54-7e26765bd876\") " pod="openshift-monitoring/node-exporter-dn2fp"
Jul 29 10:30:36.499206 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.499194    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/683dda88-94e8-4960-be54-7e26765bd876-metrics-client-ca\") pod \"node-exporter-dn2fp\" (UID: \"683dda88-94e8-4960-be54-7e26765bd876\") " pod="openshift-monitoring/node-exporter-dn2fp"
Jul 29 10:30:36.499412 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.499211    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/683dda88-94e8-4960-be54-7e26765bd876-node-exporter-tls\") pod \"node-exporter-dn2fp\" (UID: \"683dda88-94e8-4960-be54-7e26765bd876\") " pod="openshift-monitoring/node-exporter-dn2fp"
Jul 29 10:30:36.499412 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.499234    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/683dda88-94e8-4960-be54-7e26765bd876-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dn2fp\" (UID: \"683dda88-94e8-4960-be54-7e26765bd876\") " pod="openshift-monitoring/node-exporter-dn2fp"
Jul 29 10:30:36.499412 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.499261    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/683dda88-94e8-4960-be54-7e26765bd876-node-exporter-wtmp\") pod \"node-exporter-dn2fp\" (UID: \"683dda88-94e8-4960-be54-7e26765bd876\") " pod="openshift-monitoring/node-exporter-dn2fp"
Jul 29 10:30:36.499412 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.499313    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/683dda88-94e8-4960-be54-7e26765bd876-node-exporter-accelerators-collector-config\") pod \"node-exporter-dn2fp\" (UID: \"683dda88-94e8-4960-be54-7e26765bd876\") " pod="openshift-monitoring/node-exporter-dn2fp"
Jul 29 10:30:36.568549 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.568515    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:36.600683 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.600637    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/683dda88-94e8-4960-be54-7e26765bd876-sys\") pod \"node-exporter-dn2fp\" (UID: \"683dda88-94e8-4960-be54-7e26765bd876\") " pod="openshift-monitoring/node-exporter-dn2fp"
Jul 29 10:30:36.600846 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.600691    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/683dda88-94e8-4960-be54-7e26765bd876-node-exporter-textfile\") pod \"node-exporter-dn2fp\" (UID: \"683dda88-94e8-4960-be54-7e26765bd876\") " pod="openshift-monitoring/node-exporter-dn2fp"
Jul 29 10:30:36.600846 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.600729    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/683dda88-94e8-4960-be54-7e26765bd876-root\") pod \"node-exporter-dn2fp\" (UID: \"683dda88-94e8-4960-be54-7e26765bd876\") " pod="openshift-monitoring/node-exporter-dn2fp"
Jul 29 10:30:36.600846 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.600749    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/683dda88-94e8-4960-be54-7e26765bd876-sys\") pod \"node-exporter-dn2fp\" (UID: \"683dda88-94e8-4960-be54-7e26765bd876\") " pod="openshift-monitoring/node-exporter-dn2fp"
Jul 29 10:30:36.600846 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.600811    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/683dda88-94e8-4960-be54-7e26765bd876-root\") pod \"node-exporter-dn2fp\" (UID: \"683dda88-94e8-4960-be54-7e26765bd876\") " pod="openshift-monitoring/node-exporter-dn2fp"
Jul 29 10:30:36.600846 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.600757    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/683dda88-94e8-4960-be54-7e26765bd876-metrics-client-ca\") pod \"node-exporter-dn2fp\" (UID: \"683dda88-94e8-4960-be54-7e26765bd876\") " pod="openshift-monitoring/node-exporter-dn2fp"
Jul 29 10:30:36.601129 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.600857    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/683dda88-94e8-4960-be54-7e26765bd876-node-exporter-tls\") pod \"node-exporter-dn2fp\" (UID: \"683dda88-94e8-4960-be54-7e26765bd876\") " pod="openshift-monitoring/node-exporter-dn2fp"
Jul 29 10:30:36.601129 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.600881    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/683dda88-94e8-4960-be54-7e26765bd876-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dn2fp\" (UID: \"683dda88-94e8-4960-be54-7e26765bd876\") " pod="openshift-monitoring/node-exporter-dn2fp"
Jul 29 10:30:36.601129 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.600908    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/683dda88-94e8-4960-be54-7e26765bd876-node-exporter-wtmp\") pod \"node-exporter-dn2fp\" (UID: \"683dda88-94e8-4960-be54-7e26765bd876\") " pod="openshift-monitoring/node-exporter-dn2fp"
Jul 29 10:30:36.601129 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.600943    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/683dda88-94e8-4960-be54-7e26765bd876-node-exporter-accelerators-collector-config\") pod \"node-exporter-dn2fp\" (UID: \"683dda88-94e8-4960-be54-7e26765bd876\") " pod="openshift-monitoring/node-exporter-dn2fp"
Jul 29 10:30:36.601129 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.601023    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m9tf\" (UniqueName: \"kubernetes.io/projected/683dda88-94e8-4960-be54-7e26765bd876-kube-api-access-4m9tf\") pod \"node-exporter-dn2fp\" (UID: \"683dda88-94e8-4960-be54-7e26765bd876\") " pod="openshift-monitoring/node-exporter-dn2fp"
Jul 29 10:30:36.601129 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.601124    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/683dda88-94e8-4960-be54-7e26765bd876-node-exporter-textfile\") pod \"node-exporter-dn2fp\" (UID: \"683dda88-94e8-4960-be54-7e26765bd876\") " pod="openshift-monitoring/node-exporter-dn2fp"
Jul 29 10:30:36.601915 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:30:36.601496    2569 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 29 10:30:36.601915 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.601507    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/683dda88-94e8-4960-be54-7e26765bd876-node-exporter-wtmp\") pod \"node-exporter-dn2fp\" (UID: \"683dda88-94e8-4960-be54-7e26765bd876\") " pod="openshift-monitoring/node-exporter-dn2fp"
Jul 29 10:30:36.601915 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:30:36.601564    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/683dda88-94e8-4960-be54-7e26765bd876-node-exporter-tls podName:683dda88-94e8-4960-be54-7e26765bd876 nodeName:}" failed. No retries permitted until 2026-07-29 10:30:37.101541311 +0000 UTC m=+113.826331980 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/683dda88-94e8-4960-be54-7e26765bd876-node-exporter-tls") pod "node-exporter-dn2fp" (UID: "683dda88-94e8-4960-be54-7e26765bd876") : secret "node-exporter-tls" not found
Jul 29 10:30:36.601915 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.601854    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/683dda88-94e8-4960-be54-7e26765bd876-metrics-client-ca\") pod \"node-exporter-dn2fp\" (UID: \"683dda88-94e8-4960-be54-7e26765bd876\") " pod="openshift-monitoring/node-exporter-dn2fp"
Jul 29 10:30:36.602212 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.601952    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/683dda88-94e8-4960-be54-7e26765bd876-node-exporter-accelerators-collector-config\") pod \"node-exporter-dn2fp\" (UID: \"683dda88-94e8-4960-be54-7e26765bd876\") " pod="openshift-monitoring/node-exporter-dn2fp"
Jul 29 10:30:36.605997 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.605972    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/683dda88-94e8-4960-be54-7e26765bd876-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dn2fp\" (UID: \"683dda88-94e8-4960-be54-7e26765bd876\") " pod="openshift-monitoring/node-exporter-dn2fp"
Jul 29 10:30:36.612428 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.612385    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m9tf\" (UniqueName: \"kubernetes.io/projected/683dda88-94e8-4960-be54-7e26765bd876-kube-api-access-4m9tf\") pod \"node-exporter-dn2fp\" (UID: \"683dda88-94e8-4960-be54-7e26765bd876\") " pod="openshift-monitoring/node-exporter-dn2fp"
Jul 29 10:30:36.702174 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.702060    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:36.803501 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.803457    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:36.903332 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:36.903296    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:37.090560 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:37.090468    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:37.104799 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:37.104767    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/683dda88-94e8-4960-be54-7e26765bd876-node-exporter-tls\") pod \"node-exporter-dn2fp\" (UID: \"683dda88-94e8-4960-be54-7e26765bd876\") " pod="openshift-monitoring/node-exporter-dn2fp"
Jul 29 10:30:37.104984 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:30:37.104944    2569 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 29 10:30:37.105051 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:30:37.105023    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/683dda88-94e8-4960-be54-7e26765bd876-node-exporter-tls podName:683dda88-94e8-4960-be54-7e26765bd876 nodeName:}" failed. No retries permitted until 2026-07-29 10:30:38.105003244 +0000 UTC m=+114.829793899 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/683dda88-94e8-4960-be54-7e26765bd876-node-exporter-tls") pod "node-exporter-dn2fp" (UID: "683dda88-94e8-4960-be54-7e26765bd876") : secret "node-exporter-tls" not found
Jul 29 10:30:37.252089 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:37.252034    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:37.410580 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:37.410493    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:37.502494 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:37.502448    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:37.589412 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:37.589375    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:37.706050 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:37.706015    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:37.807726 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:37.807689    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:38.113878 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:38.113792    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/683dda88-94e8-4960-be54-7e26765bd876-node-exporter-tls\") pod \"node-exporter-dn2fp\" (UID: \"683dda88-94e8-4960-be54-7e26765bd876\") " pod="openshift-monitoring/node-exporter-dn2fp"
Jul 29 10:30:38.116439 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:38.116411    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/683dda88-94e8-4960-be54-7e26765bd876-node-exporter-tls\") pod \"node-exporter-dn2fp\" (UID: \"683dda88-94e8-4960-be54-7e26765bd876\") " pod="openshift-monitoring/node-exporter-dn2fp"
Jul 29 10:30:38.267881 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:38.267838    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dn2fp"
Jul 29 10:30:39.284774 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:39.284741    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ljsqp"
Jul 29 10:30:39.361899 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:39.361855    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:39.430506 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:39.430465    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:39.534584 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:39.534543    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:39.635174 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:39.635065    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:40.734661 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:40.734629    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-9dd45858d-4m6ts"]
Jul 29 10:30:40.762858 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:40.762823    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-9dd45858d-4m6ts"]
Jul 29 10:30:40.763023 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:40.762921    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-9dd45858d-4m6ts"
Jul 29 10:30:40.765858 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:40.765833    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 29 10:30:40.766979 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:40.766955    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-b7dss3hgsi11h\""
Jul 29 10:30:40.767132 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:40.767014    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 29 10:30:40.767132 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:40.767040    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-xrlsz\""
Jul 29 10:30:40.767132 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:40.766961    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 29 10:30:40.767310 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:40.767207    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 29 10:30:40.840192 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:40.840144    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be-secret-metrics-server-tls\") pod \"metrics-server-9dd45858d-4m6ts\" (UID: \"c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be\") " pod="openshift-monitoring/metrics-server-9dd45858d-4m6ts"
Jul 29 10:30:40.840377 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:40.840211    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9dd45858d-4m6ts\" (UID: \"c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be\") " pod="openshift-monitoring/metrics-server-9dd45858d-4m6ts"
Jul 29 10:30:40.840377 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:40.840262    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be-client-ca-bundle\") pod \"metrics-server-9dd45858d-4m6ts\" (UID: \"c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be\") " pod="openshift-monitoring/metrics-server-9dd45858d-4m6ts"
Jul 29 10:30:40.840377 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:40.840315    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k9gv\" (UniqueName: \"kubernetes.io/projected/c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be-kube-api-access-6k9gv\") pod \"metrics-server-9dd45858d-4m6ts\" (UID: \"c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be\") " pod="openshift-monitoring/metrics-server-9dd45858d-4m6ts"
Jul 29 10:30:40.840377 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:40.840342    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be-secret-metrics-server-client-certs\") pod \"metrics-server-9dd45858d-4m6ts\" (UID: \"c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be\") " pod="openshift-monitoring/metrics-server-9dd45858d-4m6ts"
Jul 29 10:30:40.840377 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:40.840364    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be-audit-log\") pod \"metrics-server-9dd45858d-4m6ts\" (UID: \"c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be\") " pod="openshift-monitoring/metrics-server-9dd45858d-4m6ts"
Jul 29 10:30:40.840618 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:40.840406    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be-metrics-server-audit-profiles\") pod \"metrics-server-9dd45858d-4m6ts\" (UID: \"c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be\") " pod="openshift-monitoring/metrics-server-9dd45858d-4m6ts"
Jul 29 10:30:40.941426 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:40.941388    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k9gv\" (UniqueName: \"kubernetes.io/projected/c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be-kube-api-access-6k9gv\") pod \"metrics-server-9dd45858d-4m6ts\" (UID: \"c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be\") " pod="openshift-monitoring/metrics-server-9dd45858d-4m6ts"
Jul 29 10:30:40.941617 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:40.941439    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be-secret-metrics-server-client-certs\") pod \"metrics-server-9dd45858d-4m6ts\" (UID: \"c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be\") " pod="openshift-monitoring/metrics-server-9dd45858d-4m6ts"
Jul 29 10:30:40.941617 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:40.941524    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be-audit-log\") pod \"metrics-server-9dd45858d-4m6ts\" (UID: \"c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be\") " pod="openshift-monitoring/metrics-server-9dd45858d-4m6ts"
Jul 29 10:30:40.941617 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:40.941577    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be-metrics-server-audit-profiles\") pod \"metrics-server-9dd45858d-4m6ts\" (UID: \"c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be\") " pod="openshift-monitoring/metrics-server-9dd45858d-4m6ts"
Jul 29 10:30:40.941787 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:40.941631    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be-secret-metrics-server-tls\") pod \"metrics-server-9dd45858d-4m6ts\" (UID: \"c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be\") " pod="openshift-monitoring/metrics-server-9dd45858d-4m6ts"
Jul 29 10:30:40.941787 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:40.941678    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9dd45858d-4m6ts\" (UID: \"c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be\") " pod="openshift-monitoring/metrics-server-9dd45858d-4m6ts"
Jul 29 10:30:40.941787 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:40.941716    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be-client-ca-bundle\") pod \"metrics-server-9dd45858d-4m6ts\" (UID: \"c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be\") " pod="openshift-monitoring/metrics-server-9dd45858d-4m6ts"
Jul 29 10:30:40.942066 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:40.942030    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be-audit-log\") pod \"metrics-server-9dd45858d-4m6ts\" (UID: \"c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be\") " pod="openshift-monitoring/metrics-server-9dd45858d-4m6ts"
Jul 29 10:30:40.942688 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:40.942622    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9dd45858d-4m6ts\" (UID: \"c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be\") " pod="openshift-monitoring/metrics-server-9dd45858d-4m6ts"
Jul 29 10:30:40.942820 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:40.942765    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be-metrics-server-audit-profiles\") pod \"metrics-server-9dd45858d-4m6ts\" (UID: \"c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be\") " pod="openshift-monitoring/metrics-server-9dd45858d-4m6ts"
Jul 29 10:30:40.944821 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:40.944784    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be-secret-metrics-server-tls\") pod \"metrics-server-9dd45858d-4m6ts\" (UID: \"c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be\") " pod="openshift-monitoring/metrics-server-9dd45858d-4m6ts"
Jul 29 10:30:40.944821 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:40.944813    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be-client-ca-bundle\") pod \"metrics-server-9dd45858d-4m6ts\" (UID: \"c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be\") " pod="openshift-monitoring/metrics-server-9dd45858d-4m6ts"
Jul 29 10:30:40.944988 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:40.944901    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be-secret-metrics-server-client-certs\") pod \"metrics-server-9dd45858d-4m6ts\" (UID: \"c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be\") " pod="openshift-monitoring/metrics-server-9dd45858d-4m6ts"
Jul 29 10:30:40.949931 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:40.949910    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k9gv\" (UniqueName: \"kubernetes.io/projected/c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be-kube-api-access-6k9gv\") pod \"metrics-server-9dd45858d-4m6ts\" (UID: \"c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be\") " pod="openshift-monitoring/metrics-server-9dd45858d-4m6ts"
Jul 29 10:30:41.074813 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:41.074730    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-9dd45858d-4m6ts"
Jul 29 10:30:41.203704 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:41.203655    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:41.217669 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:41.217637    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-q7dn5"]
Jul 29 10:30:41.236355 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:41.236326    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-q7dn5"]
Jul 29 10:30:41.236558 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:41.236476    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-q7dn5"
Jul 29 10:30:41.239343 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:41.239136    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 29 10:30:41.239540 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:41.239491    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-7hmkx\""
Jul 29 10:30:41.291257 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:41.291224    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:41.346671 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:41.346586    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7e7009ad-7d80-45c3-aff0-22e228496341-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-q7dn5\" (UID: \"7e7009ad-7d80-45c3-aff0-22e228496341\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-q7dn5"
Jul 29 10:30:41.346820 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:41.346691    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6tr2\" (UniqueName: \"kubernetes.io/projected/7e7009ad-7d80-45c3-aff0-22e228496341-kube-api-access-n6tr2\") pod \"monitoring-plugin-68b7fc9574-q7dn5\" (UID: \"7e7009ad-7d80-45c3-aff0-22e228496341\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-q7dn5"
Jul 29 10:30:41.387294 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:41.387261    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:41.447495 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:41.447456    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6tr2\" (UniqueName: \"kubernetes.io/projected/7e7009ad-7d80-45c3-aff0-22e228496341-kube-api-access-n6tr2\") pod \"monitoring-plugin-68b7fc9574-q7dn5\" (UID: \"7e7009ad-7d80-45c3-aff0-22e228496341\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-q7dn5"
Jul 29 10:30:41.447699 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:41.447551    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7e7009ad-7d80-45c3-aff0-22e228496341-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-q7dn5\" (UID: \"7e7009ad-7d80-45c3-aff0-22e228496341\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-q7dn5"
Jul 29 10:30:41.450407 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:41.450379    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7e7009ad-7d80-45c3-aff0-22e228496341-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-q7dn5\" (UID: \"7e7009ad-7d80-45c3-aff0-22e228496341\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-q7dn5"
Jul 29 10:30:41.456320 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:41.456292    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6tr2\" (UniqueName: \"kubernetes.io/projected/7e7009ad-7d80-45c3-aff0-22e228496341-kube-api-access-n6tr2\") pod \"monitoring-plugin-68b7fc9574-q7dn5\" (UID: \"7e7009ad-7d80-45c3-aff0-22e228496341\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-q7dn5"
Jul 29 10:30:41.549474 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:41.549440    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-q7dn5"
Jul 29 10:30:42.605300 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.605257    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:42.612042 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.612011    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 10:30:42.629061 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.629036    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.632994 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.632952    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 10:30:42.633681 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.633208    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 29 10:30:42.633681 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.633265    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 29 10:30:42.633681 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.633547    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 29 10:30:42.633681 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.633563    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 29 10:30:42.633681 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.633586    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 29 10:30:42.634006 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.633811    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 29 10:30:42.634006 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.633916    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 29 10:30:42.634135 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.634029    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-tfx9c\""
Jul 29 10:30:42.634222 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.634201    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 29 10:30:42.634398 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.634274    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-22qdcjn05jf7j\""
Jul 29 10:30:42.634398 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.634208    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 29 10:30:42.634590 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.634559    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 29 10:30:42.634590 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.634558    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 29 10:30:42.634755 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.634721    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 29 10:30:42.634856 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.634836    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 29 10:30:42.636089 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.636048    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 29 10:30:42.759377 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.759341    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.759377 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.759376    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.759602 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.759396    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.759602 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.759421    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.759602 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.759478    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.759602 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.759514    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.759602 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.759539    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.759602 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.759555    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.759602 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.759585    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.759873 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.759611    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.759873 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.759628    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-config-out\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.759873 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.759643    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.759873 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.759664    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-config\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.759873 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.759711    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.759873 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.759778    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.759873 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.759801    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.759873 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.759855    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.760231 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.759885    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.760231 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.759912    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-web-config\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.760231 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.759960    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd2f6\" (UniqueName: \"kubernetes.io/projected/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-kube-api-access-pd2f6\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.861275 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.861196    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd2f6\" (UniqueName: \"kubernetes.io/projected/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-kube-api-access-pd2f6\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.861275 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.861237    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.861275 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.861256    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.861554 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.861283    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.861554 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.861312    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.861554 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.861345    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.861554 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.861380    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.861554 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.861420    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.861554 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.861445    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.862213 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.862179    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.862357 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.862258    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.862357 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.862321    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.863668 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.862640    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.863668 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.862863    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.863668 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.862906    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-config-out\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.863668 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.862913    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.863668 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.862940    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.863668 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.862970    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-config\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.863668 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.863008    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.863668 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.863058    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.863668 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.863121    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.863668 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.863166    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.863668 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.863200    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.863668 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.863233    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-web-config\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.863668 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.863422    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.865227 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.865199    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.865351 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.865302    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.866985 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.865630    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.866985 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.866090    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.866985 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.866549    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.866985 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.866916    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-web-config\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.869742 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.869694    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.870135 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.870095    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.870462 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.870435    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-config-out\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.870559 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.870488    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.870622 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.870561    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.870759 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.870722    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-config\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.871051 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.871033    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.872566 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.872505    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.872913 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.872894    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd2f6\" (UniqueName: \"kubernetes.io/projected/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-kube-api-access-pd2f6\") pod \"prometheus-k8s-0\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.892287 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:30:42.892257    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod683dda88_94e8_4960_be54_7e26765bd876.slice/crio-eea0f7f048a29726efaa949c7a756b2ebc2e3d7b6f265f8e412da7e8373bc2c2 WatchSource:0}: Error finding container eea0f7f048a29726efaa949c7a756b2ebc2e3d7b6f265f8e412da7e8373bc2c2: Status 404 returned error can't find the container with id eea0f7f048a29726efaa949c7a756b2ebc2e3d7b6f265f8e412da7e8373bc2c2
Jul 29 10:30:42.944590 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.944276    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:42.953789 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:42.953675    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cdb5fcdf7-wclz4"]
Jul 29 10:30:43.061152 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:43.060673    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-9dd45858d-4m6ts"]
Jul 29 10:30:43.065053 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:30:43.064623    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0e19b31_af9a_4a7b_8dcb_b4c76bd6d3be.slice/crio-b29a52f97dd2efa05dcf1b4b608ea6c095305b47f4d686e3f296ac9ac2071bc6 WatchSource:0}: Error finding container b29a52f97dd2efa05dcf1b4b608ea6c095305b47f4d686e3f296ac9ac2071bc6: Status 404 returned error can't find the container with id b29a52f97dd2efa05dcf1b4b608ea6c095305b47f4d686e3f296ac9ac2071bc6
Jul 29 10:30:43.073596 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:43.073571    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-q7dn5"]
Jul 29 10:30:43.077875 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:30:43.077851    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e7009ad_7d80_45c3_aff0_22e228496341.slice/crio-d1d27703b76bff5cba2a249604a010ac947b0f00c8984293ef8280261dddc8f8 WatchSource:0}: Error finding container d1d27703b76bff5cba2a249604a010ac947b0f00c8984293ef8280261dddc8f8: Status 404 returned error can't find the container with id d1d27703b76bff5cba2a249604a010ac947b0f00c8984293ef8280261dddc8f8
Jul 29 10:30:43.161854 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:43.161770    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 10:30:43.165114 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:30:43.165088    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd20dcaa9_3476_4fd3_bb2b_6a936d2d9c26.slice/crio-43d48d4ad623f2be32c9bffc77f313363dcf0fb8d6fc519db9ed3e0feba2c5d6 WatchSource:0}: Error finding container 43d48d4ad623f2be32c9bffc77f313363dcf0fb8d6fc519db9ed3e0feba2c5d6: Status 404 returned error can't find the container with id 43d48d4ad623f2be32c9bffc77f313363dcf0fb8d6fc519db9ed3e0feba2c5d6
Jul 29 10:30:43.343994 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:43.343950    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26","Type":"ContainerStarted","Data":"43d48d4ad623f2be32c9bffc77f313363dcf0fb8d6fc519db9ed3e0feba2c5d6"}
Jul 29 10:30:43.345268 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:43.345242    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-9dd45858d-4m6ts" event={"ID":"c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be","Type":"ContainerStarted","Data":"b29a52f97dd2efa05dcf1b4b608ea6c095305b47f4d686e3f296ac9ac2071bc6"}
Jul 29 10:30:43.346380 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:43.346354    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-q7dn5" event={"ID":"7e7009ad-7d80-45c3-aff0-22e228496341","Type":"ContainerStarted","Data":"d1d27703b76bff5cba2a249604a010ac947b0f00c8984293ef8280261dddc8f8"}
Jul 29 10:30:43.347976 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:43.347945    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cdb5fcdf7-wclz4" event={"ID":"0cd86203-fe78-4a76-ab30-d13780b1a3db","Type":"ContainerStarted","Data":"1a0546b6527f84ecd5eba58ba1a7e7aed952abde1747af20974c2d2f0fe9e934"}
Jul 29 10:30:43.349355 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:43.349329    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dn2fp" event={"ID":"683dda88-94e8-4960-be54-7e26765bd876","Type":"ContainerStarted","Data":"eea0f7f048a29726efaa949c7a756b2ebc2e3d7b6f265f8e412da7e8373bc2c2"}
Jul 29 10:30:43.370798 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:43.370735    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7cdb5fcdf7-wclz4" podStartSLOduration=1.9509793530000001 podStartE2EDuration="14.370718158s" totalImagesPullingTime="12.419738805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:30:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:43.370571758 +0000 UTC m=+120.095362435" watchObservedRunningTime="2026-07-29 10:30:43.370718158 +0000 UTC m=+120.095508833"
Jul 29 10:30:44.144743 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:44.144653    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:44.245191 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:44.245119    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:44.253564 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:44.253533    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:30:44.360278 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:44.360229    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-7gk4j" event={"ID":"f0278c1e-7a6a-472f-aaba-09c9a22c9fa9","Type":"ContainerStarted","Data":"a4977e7f3a048df8f102e9835bad9b091cdbca93f6e423be17b6e16cf846e489"}
Jul 29 10:30:44.360720 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:44.360698    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-7gk4j"
Jul 29 10:30:44.369375 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:44.369323    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-7gk4j"
Jul 29 10:30:44.384813 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:44.382768    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-7gk4j" podStartSLOduration=1.953850603 podStartE2EDuration="21.382750406s" totalImagesPullingTime="19.428899803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:30:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:44.381823321 +0000 UTC m=+121.106613995" watchObservedRunningTime="2026-07-29 10:30:44.382750406 +0000 UTC m=+121.107541081"
Jul 29 10:30:44.430850 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:44.430773    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:44.529994 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:44.529953    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:45.170342 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:45.170297    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66f4d7d896-fbvv2"]
Jul 29 10:30:45.264373 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:45.264331    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:45.364456 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:45.364391    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:45.635234 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:45.635160    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:45.738057 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:45.738020    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:47.376517 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:47.376477    2569 generic.go:356] "Generic (PLEG): container finished" podID="683dda88-94e8-4960-be54-7e26765bd876" containerID="5cf28024b3c59a5bb17be8bba7e2875f6890e28b1decea9ec692c903783d0569" exitCode=0
Jul 29 10:30:47.377107 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:47.376564    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dn2fp" event={"ID":"683dda88-94e8-4960-be54-7e26765bd876","Type":"ContainerDied","Data":"5cf28024b3c59a5bb17be8bba7e2875f6890e28b1decea9ec692c903783d0569"}
Jul 29 10:30:47.378791 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:47.378764    2569 generic.go:356] "Generic (PLEG): container finished" podID="d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" containerID="f5c4d4e4bb6113e012b897496df839cf92ba01e508c30720e7ae641e9251e51e" exitCode=0
Jul 29 10:30:47.378903 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:47.378856    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26","Type":"ContainerDied","Data":"f5c4d4e4bb6113e012b897496df839cf92ba01e508c30720e7ae641e9251e51e"}
Jul 29 10:30:47.382408 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:47.382376    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-9dd45858d-4m6ts" event={"ID":"c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be","Type":"ContainerStarted","Data":"e82a1db7f82fb81566776446db44820e0dcaa336cdb2ec911c323e3b3cdc0872"}
Jul 29 10:30:47.384894 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:47.384865    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-q7dn5" event={"ID":"7e7009ad-7d80-45c3-aff0-22e228496341","Type":"ContainerStarted","Data":"8dcf3001dcc893d27fd7a93642269500d3acad41e09460c40a8cfc0665a0cbf7"}
Jul 29 10:30:47.385123 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:47.385102    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-q7dn5"
Jul 29 10:30:47.390839 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:47.390814    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-q7dn5"
Jul 29 10:30:47.416497 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:47.416437    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-9dd45858d-4m6ts" podStartSLOduration=3.705842346 podStartE2EDuration="7.416414792s" totalImagesPullingTime="3.710572446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:30:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:47.41479282 +0000 UTC m=+124.139583496" watchObservedRunningTime="2026-07-29 10:30:47.416414792 +0000 UTC m=+124.141205468"
Jul 29 10:30:47.431778 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:47.431723    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-q7dn5" podStartSLOduration=2.746051198 podStartE2EDuration="6.431702948s" totalImagesPullingTime="3.68565175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:30:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:47.431286084 +0000 UTC m=+124.156076759" watchObservedRunningTime="2026-07-29 10:30:47.431702948 +0000 UTC m=+124.156493622"
Jul 29 10:30:48.393684 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:48.393193    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dn2fp" event={"ID":"683dda88-94e8-4960-be54-7e26765bd876","Type":"ContainerStarted","Data":"b5896d516ca0b14dc67206137b2fc4180924683c601c11c69997f843583fedf5"}
Jul 29 10:30:48.393684 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:48.393268    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dn2fp" event={"ID":"683dda88-94e8-4960-be54-7e26765bd876","Type":"ContainerStarted","Data":"515d48228f6cf8e9e4039b1be34ab69b7b75e82c225503efb2e731232f4048a4"}
Jul 29 10:30:48.412963 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:48.412901    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dn2fp" podStartSLOduration=8.466427718 podStartE2EDuration="12.412878676s" totalImagesPullingTime="3.841869744s" totalInitContainerRuntime="104.581214ms" isStatefulPod=true podCreationTimestamp="2026-07-29 10:30:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:48.411731038 +0000 UTC m=+125.136521709" watchObservedRunningTime="2026-07-29 10:30:48.412878676 +0000 UTC m=+125.137669352"
Jul 29 10:30:48.423433 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:48.423380    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:48.505818 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:48.505764    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:50.281593 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:50.281504    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7cdb5fcdf7-wclz4"
Jul 29 10:30:50.991592 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:50.991548    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:51.059876 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:51.059825    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:51.536165 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:51.536064    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:51.636743 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:51.636708    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:52.059533 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:52.059492    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:52.127881 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:52.127847    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:52.414724 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:52.414690    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26","Type":"ContainerStarted","Data":"8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3"}
Jul 29 10:30:52.414894 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:52.414770    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26","Type":"ContainerStarted","Data":"a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77"}
Jul 29 10:30:53.585466 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:53.585423    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c-metrics-certs\") pod \"network-metrics-daemon-szr8n\" (UID: \"51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c\") " pod="openshift-multus/network-metrics-daemon-szr8n"
Jul 29 10:30:53.587754 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:53.587727    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c-metrics-certs\") pod \"network-metrics-daemon-szr8n\" (UID: \"51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c\") " pod="openshift-multus/network-metrics-daemon-szr8n"
Jul 29 10:30:53.658797 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:53.658771    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n9sqp\""
Jul 29 10:30:53.666885 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:53.666864    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szr8n"
Jul 29 10:30:53.798658 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:53.798629    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-szr8n"]
Jul 29 10:30:53.802001 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:30:53.801974    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51dcc151_74b3_46ad_9ef5_5d1e5acc9f4c.slice/crio-63e4f1b6ca5bb6f44bb85b36952c39036fd7de46d5a5055ecdcc8f8f90e724a0 WatchSource:0}: Error finding container 63e4f1b6ca5bb6f44bb85b36952c39036fd7de46d5a5055ecdcc8f8f90e724a0: Status 404 returned error can't find the container with id 63e4f1b6ca5bb6f44bb85b36952c39036fd7de46d5a5055ecdcc8f8f90e724a0
Jul 29 10:30:54.427153 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:54.427110    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-szr8n" event={"ID":"51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c","Type":"ContainerStarted","Data":"63e4f1b6ca5bb6f44bb85b36952c39036fd7de46d5a5055ecdcc8f8f90e724a0"}
Jul 29 10:30:54.431408 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:54.431358    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26","Type":"ContainerStarted","Data":"967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23"}
Jul 29 10:30:54.431561 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:54.431534    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26","Type":"ContainerStarted","Data":"46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47"}
Jul 29 10:30:54.431629 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:54.431586    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26","Type":"ContainerStarted","Data":"40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902"}
Jul 29 10:30:54.431684 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:54.431631    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26","Type":"ContainerStarted","Data":"976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60"}
Jul 29 10:30:54.463533 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:54.463438    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.535867681 podStartE2EDuration="12.463421002s" totalImagesPullingTime="9.879484543s" totalInitContainerRuntime="48.068778ms" isStatefulPod=false podCreationTimestamp="2026-07-29 10:30:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:54.462178629 +0000 UTC m=+131.186969302" watchObservedRunningTime="2026-07-29 10:30:54.463421002 +0000 UTC m=+131.188211675"
Jul 29 10:30:56.441229 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:56.441187    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-szr8n" event={"ID":"51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c","Type":"ContainerStarted","Data":"e1fa3c223e8fe1f3a171d2421e58009218bbef25cc59c6f269840c9720d1e46c"}
Jul 29 10:30:56.441623 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:56.441246    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-szr8n" event={"ID":"51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c","Type":"ContainerStarted","Data":"bd1eb1882d8122eddbe7d766dd2677287d06ed0207e5b62ce20efba9306d322c"}
Jul 29 10:30:56.458784 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:56.458734    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-szr8n" podStartSLOduration=130.880322626 podStartE2EDuration="2m12.458720554s" totalImagesPullingTime="1.578397928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:28:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:56.456766942 +0000 UTC m=+133.181557615" watchObservedRunningTime="2026-07-29 10:30:56.458720554 +0000 UTC m=+133.183511227"
Jul 29 10:30:56.556325 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:56.556285    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:56.677724 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:56.677690    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:57.443717 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:57.443681    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:57.945206 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:30:57.945158    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:31:00.034619 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:00.034579    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:31:00.147092 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:00.147036    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:31:00.478707 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:00.478670    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:31:00.579832 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:00.579784    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:31:00.680273 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:00.680239    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:31:01.075274 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:01.075236    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-9dd45858d-4m6ts"
Jul 29 10:31:01.075723 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:01.075321    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-9dd45858d-4m6ts"
Jul 29 10:31:06.042395 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:06.042321    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:31:06.108389 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:06.108355    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:31:08.378023 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:08.377959    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7cdb5fcdf7-wclz4" podUID="0cd86203-fe78-4a76-ab30-d13780b1a3db" containerName="console" containerID="cri-o://1a0546b6527f84ecd5eba58ba1a7e7aed952abde1747af20974c2d2f0fe9e934" gracePeriod=15
Jul 29 10:31:08.648955 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:08.648891    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cdb5fcdf7-wclz4"
Jul 29 10:31:08.813566 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:08.813530    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cd86203-fe78-4a76-ab30-d13780b1a3db-trusted-ca-bundle\") pod \"0cd86203-fe78-4a76-ab30-d13780b1a3db\" (UID: \"0cd86203-fe78-4a76-ab30-d13780b1a3db\") "
Jul 29 10:31:08.813769 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:08.813620    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0cd86203-fe78-4a76-ab30-d13780b1a3db-oauth-serving-cert\") pod \"0cd86203-fe78-4a76-ab30-d13780b1a3db\" (UID: \"0cd86203-fe78-4a76-ab30-d13780b1a3db\") "
Jul 29 10:31:08.813769 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:08.813642    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0cd86203-fe78-4a76-ab30-d13780b1a3db-service-ca\") pod \"0cd86203-fe78-4a76-ab30-d13780b1a3db\" (UID: \"0cd86203-fe78-4a76-ab30-d13780b1a3db\") "
Jul 29 10:31:08.813769 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:08.813659    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0cd86203-fe78-4a76-ab30-d13780b1a3db-console-config\") pod \"0cd86203-fe78-4a76-ab30-d13780b1a3db\" (UID: \"0cd86203-fe78-4a76-ab30-d13780b1a3db\") "
Jul 29 10:31:08.813769 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:08.813675    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqrjv\" (UniqueName: \"kubernetes.io/projected/0cd86203-fe78-4a76-ab30-d13780b1a3db-kube-api-access-gqrjv\") pod \"0cd86203-fe78-4a76-ab30-d13780b1a3db\" (UID: \"0cd86203-fe78-4a76-ab30-d13780b1a3db\") "
Jul 29 10:31:08.813769 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:08.813694    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0cd86203-fe78-4a76-ab30-d13780b1a3db-console-oauth-config\") pod \"0cd86203-fe78-4a76-ab30-d13780b1a3db\" (UID: \"0cd86203-fe78-4a76-ab30-d13780b1a3db\") "
Jul 29 10:31:08.813769 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:08.813722    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cd86203-fe78-4a76-ab30-d13780b1a3db-console-serving-cert\") pod \"0cd86203-fe78-4a76-ab30-d13780b1a3db\" (UID: \"0cd86203-fe78-4a76-ab30-d13780b1a3db\") "
Jul 29 10:31:08.814187 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:08.814015    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cd86203-fe78-4a76-ab30-d13780b1a3db-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0cd86203-fe78-4a76-ab30-d13780b1a3db" (UID: "0cd86203-fe78-4a76-ab30-d13780b1a3db"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:31:08.814187 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:08.814150    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cd86203-fe78-4a76-ab30-d13780b1a3db-service-ca" (OuterVolumeSpecName: "service-ca") pod "0cd86203-fe78-4a76-ab30-d13780b1a3db" (UID: "0cd86203-fe78-4a76-ab30-d13780b1a3db"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:31:08.814374 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:08.814219    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cd86203-fe78-4a76-ab30-d13780b1a3db-console-config" (OuterVolumeSpecName: "console-config") pod "0cd86203-fe78-4a76-ab30-d13780b1a3db" (UID: "0cd86203-fe78-4a76-ab30-d13780b1a3db"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:31:08.814374 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:08.814233    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cd86203-fe78-4a76-ab30-d13780b1a3db-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0cd86203-fe78-4a76-ab30-d13780b1a3db" (UID: "0cd86203-fe78-4a76-ab30-d13780b1a3db"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:31:08.816003 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:08.815971    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cd86203-fe78-4a76-ab30-d13780b1a3db-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0cd86203-fe78-4a76-ab30-d13780b1a3db" (UID: "0cd86203-fe78-4a76-ab30-d13780b1a3db"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:31:08.816003 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:08.815995    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd86203-fe78-4a76-ab30-d13780b1a3db-kube-api-access-gqrjv" (OuterVolumeSpecName: "kube-api-access-gqrjv") pod "0cd86203-fe78-4a76-ab30-d13780b1a3db" (UID: "0cd86203-fe78-4a76-ab30-d13780b1a3db"). InnerVolumeSpecName "kube-api-access-gqrjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:31:08.816160 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:08.816090    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cd86203-fe78-4a76-ab30-d13780b1a3db-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0cd86203-fe78-4a76-ab30-d13780b1a3db" (UID: "0cd86203-fe78-4a76-ab30-d13780b1a3db"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:31:08.915251 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:08.915219    2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0cd86203-fe78-4a76-ab30-d13780b1a3db-oauth-serving-cert\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:31:08.915251 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:08.915248    2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0cd86203-fe78-4a76-ab30-d13780b1a3db-service-ca\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:31:08.915441 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:08.915261    2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0cd86203-fe78-4a76-ab30-d13780b1a3db-console-config\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:31:08.915441 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:08.915273    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqrjv\" (UniqueName: \"kubernetes.io/projected/0cd86203-fe78-4a76-ab30-d13780b1a3db-kube-api-access-gqrjv\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:31:08.915441 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:08.915285    2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0cd86203-fe78-4a76-ab30-d13780b1a3db-console-oauth-config\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:31:08.915441 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:08.915295    2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cd86203-fe78-4a76-ab30-d13780b1a3db-console-serving-cert\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:31:08.915441 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:08.915308    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cd86203-fe78-4a76-ab30-d13780b1a3db-trusted-ca-bundle\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:31:09.495508 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:09.495474    2569 generic.go:356] "Generic (PLEG): container finished" podID="0cd86203-fe78-4a76-ab30-d13780b1a3db" containerID="1a0546b6527f84ecd5eba58ba1a7e7aed952abde1747af20974c2d2f0fe9e934" exitCode=0
Jul 29 10:31:09.495998 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:09.495516    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cdb5fcdf7-wclz4"
Jul 29 10:31:09.495998 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:09.495551    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cdb5fcdf7-wclz4" event={"ID":"0cd86203-fe78-4a76-ab30-d13780b1a3db","Type":"ContainerDied","Data":"1a0546b6527f84ecd5eba58ba1a7e7aed952abde1747af20974c2d2f0fe9e934"}
Jul 29 10:31:09.495998 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:09.495595    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cdb5fcdf7-wclz4" event={"ID":"0cd86203-fe78-4a76-ab30-d13780b1a3db","Type":"ContainerDied","Data":"4754380593fbb15e507fb4ef7549b44fab048ca227f26fdfb130edccd3622316"}
Jul 29 10:31:09.495998 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:09.495616    2569 scope.go:117] "RemoveContainer" containerID="1a0546b6527f84ecd5eba58ba1a7e7aed952abde1747af20974c2d2f0fe9e934"
Jul 29 10:31:09.497193 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:09.497171    2569 generic.go:356] "Generic (PLEG): container finished" podID="3aaafa2b-ae84-417a-905b-62bbc49a287c" containerID="e04277c27d3da3824aa21a68ef6d87a1e2cbafa9eaca19eecfba239337ad3c35" exitCode=0
Jul 29 10:31:09.497296 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:09.497208    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ns69c" event={"ID":"3aaafa2b-ae84-417a-905b-62bbc49a287c","Type":"ContainerDied","Data":"e04277c27d3da3824aa21a68ef6d87a1e2cbafa9eaca19eecfba239337ad3c35"}
Jul 29 10:31:09.497531 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:09.497515    2569 scope.go:117] "RemoveContainer" containerID="e04277c27d3da3824aa21a68ef6d87a1e2cbafa9eaca19eecfba239337ad3c35"
Jul 29 10:31:09.504569 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:09.504551    2569 scope.go:117] "RemoveContainer" containerID="1a0546b6527f84ecd5eba58ba1a7e7aed952abde1747af20974c2d2f0fe9e934"
Jul 29 10:31:09.504833 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:31:09.504813    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a0546b6527f84ecd5eba58ba1a7e7aed952abde1747af20974c2d2f0fe9e934\": container with ID starting with 1a0546b6527f84ecd5eba58ba1a7e7aed952abde1747af20974c2d2f0fe9e934 not found: ID does not exist" containerID="1a0546b6527f84ecd5eba58ba1a7e7aed952abde1747af20974c2d2f0fe9e934"
Jul 29 10:31:09.504881 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:09.504840    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a0546b6527f84ecd5eba58ba1a7e7aed952abde1747af20974c2d2f0fe9e934"} err="failed to get container status \"1a0546b6527f84ecd5eba58ba1a7e7aed952abde1747af20974c2d2f0fe9e934\": rpc error: code = NotFound desc = could not find container \"1a0546b6527f84ecd5eba58ba1a7e7aed952abde1747af20974c2d2f0fe9e934\": container with ID starting with 1a0546b6527f84ecd5eba58ba1a7e7aed952abde1747af20974c2d2f0fe9e934 not found: ID does not exist"
Jul 29 10:31:09.526943 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:09.526915    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cdb5fcdf7-wclz4"]
Jul 29 10:31:09.533604 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:09.533580    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7cdb5fcdf7-wclz4"]
Jul 29 10:31:09.837541 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:09.837506    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cd86203-fe78-4a76-ab30-d13780b1a3db" path="/var/lib/kubelet/pods/0cd86203-fe78-4a76-ab30-d13780b1a3db/volumes"
Jul 29 10:31:10.192853 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:10.192814    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2" podUID="662f9f8c-cd9a-4722-ab84-59cff17809ef" containerName="registry" containerID="cri-o://16b1e17f5f02a64b6b26f19c19b0b3ae61f92a22ae365b4cb244228e4f4a5c44" gracePeriod=30
Jul 29 10:31:10.462333 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:10.462265    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:31:10.503722 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:10.503682    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ns69c" event={"ID":"3aaafa2b-ae84-417a-905b-62bbc49a287c","Type":"ContainerStarted","Data":"5f101db864b570bd8a980ed8fd602b688fe0f64b7849400fe44a43c13fa0b4b7"}
Jul 29 10:31:10.505086 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:10.505051    2569 generic.go:356] "Generic (PLEG): container finished" podID="662f9f8c-cd9a-4722-ab84-59cff17809ef" containerID="16b1e17f5f02a64b6b26f19c19b0b3ae61f92a22ae365b4cb244228e4f4a5c44" exitCode=0
Jul 29 10:31:10.505208 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:10.505106    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2" event={"ID":"662f9f8c-cd9a-4722-ab84-59cff17809ef","Type":"ContainerDied","Data":"16b1e17f5f02a64b6b26f19c19b0b3ae61f92a22ae365b4cb244228e4f4a5c44"}
Jul 29 10:31:10.505208 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:10.505171    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2" event={"ID":"662f9f8c-cd9a-4722-ab84-59cff17809ef","Type":"ContainerDied","Data":"32df146a0df4166563abe967194ebd874b1f968bc8693d612b6da1cbe345a3af"}
Jul 29 10:31:10.505208 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:10.505201    2569 scope.go:117] "RemoveContainer" containerID="16b1e17f5f02a64b6b26f19c19b0b3ae61f92a22ae365b4cb244228e4f4a5c44"
Jul 29 10:31:10.505208 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:10.505113    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66f4d7d896-fbvv2"
Jul 29 10:31:10.514598 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:10.514580    2569 scope.go:117] "RemoveContainer" containerID="16b1e17f5f02a64b6b26f19c19b0b3ae61f92a22ae365b4cb244228e4f4a5c44"
Jul 29 10:31:10.514880 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:31:10.514861    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16b1e17f5f02a64b6b26f19c19b0b3ae61f92a22ae365b4cb244228e4f4a5c44\": container with ID starting with 16b1e17f5f02a64b6b26f19c19b0b3ae61f92a22ae365b4cb244228e4f4a5c44 not found: ID does not exist" containerID="16b1e17f5f02a64b6b26f19c19b0b3ae61f92a22ae365b4cb244228e4f4a5c44"
Jul 29 10:31:10.514933 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:10.514888    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16b1e17f5f02a64b6b26f19c19b0b3ae61f92a22ae365b4cb244228e4f4a5c44"} err="failed to get container status \"16b1e17f5f02a64b6b26f19c19b0b3ae61f92a22ae365b4cb244228e4f4a5c44\": rpc error: code = NotFound desc = could not find container \"16b1e17f5f02a64b6b26f19c19b0b3ae61f92a22ae365b4cb244228e4f4a5c44\": container with ID starting with 16b1e17f5f02a64b6b26f19c19b0b3ae61f92a22ae365b4cb244228e4f4a5c44 not found: ID does not exist"
Jul 29 10:31:10.630487 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:10.630453    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4s8qn\" (UniqueName: \"kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-kube-api-access-4s8qn\") pod \"662f9f8c-cd9a-4722-ab84-59cff17809ef\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") "
Jul 29 10:31:10.630656 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:10.630515    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/662f9f8c-cd9a-4722-ab84-59cff17809ef-registry-certificates\") pod \"662f9f8c-cd9a-4722-ab84-59cff17809ef\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") "
Jul 29 10:31:10.630656 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:10.630584    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/662f9f8c-cd9a-4722-ab84-59cff17809ef-installation-pull-secrets\") pod \"662f9f8c-cd9a-4722-ab84-59cff17809ef\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") "
Jul 29 10:31:10.630656 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:10.630613    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-bound-sa-token\") pod \"662f9f8c-cd9a-4722-ab84-59cff17809ef\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") "
Jul 29 10:31:10.630656 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:10.630640    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/662f9f8c-cd9a-4722-ab84-59cff17809ef-image-registry-private-configuration\") pod \"662f9f8c-cd9a-4722-ab84-59cff17809ef\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") "
Jul 29 10:31:10.630858 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:10.630677    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/662f9f8c-cd9a-4722-ab84-59cff17809ef-trusted-ca\") pod \"662f9f8c-cd9a-4722-ab84-59cff17809ef\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") "
Jul 29 10:31:10.630858 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:10.630796    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/662f9f8c-cd9a-4722-ab84-59cff17809ef-ca-trust-extracted\") pod \"662f9f8c-cd9a-4722-ab84-59cff17809ef\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") "
Jul 29 10:31:10.630958 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:10.630866    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-registry-tls\") pod \"662f9f8c-cd9a-4722-ab84-59cff17809ef\" (UID: \"662f9f8c-cd9a-4722-ab84-59cff17809ef\") "
Jul 29 10:31:10.630958 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:10.630923    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/662f9f8c-cd9a-4722-ab84-59cff17809ef-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "662f9f8c-cd9a-4722-ab84-59cff17809ef" (UID: "662f9f8c-cd9a-4722-ab84-59cff17809ef"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:31:10.631164 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:10.631113    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/662f9f8c-cd9a-4722-ab84-59cff17809ef-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "662f9f8c-cd9a-4722-ab84-59cff17809ef" (UID: "662f9f8c-cd9a-4722-ab84-59cff17809ef"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:31:10.631640 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:10.631297    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/662f9f8c-cd9a-4722-ab84-59cff17809ef-trusted-ca\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:31:10.631640 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:10.631320    2569 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/662f9f8c-cd9a-4722-ab84-59cff17809ef-registry-certificates\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:31:10.633220 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:10.633191    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/662f9f8c-cd9a-4722-ab84-59cff17809ef-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "662f9f8c-cd9a-4722-ab84-59cff17809ef" (UID: "662f9f8c-cd9a-4722-ab84-59cff17809ef"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:31:10.633343 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:10.633257    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-kube-api-access-4s8qn" (OuterVolumeSpecName: "kube-api-access-4s8qn") pod "662f9f8c-cd9a-4722-ab84-59cff17809ef" (UID: "662f9f8c-cd9a-4722-ab84-59cff17809ef"). InnerVolumeSpecName "kube-api-access-4s8qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:31:10.633343 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:10.633303    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "662f9f8c-cd9a-4722-ab84-59cff17809ef" (UID: "662f9f8c-cd9a-4722-ab84-59cff17809ef"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:31:10.633458 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:10.633351    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "662f9f8c-cd9a-4722-ab84-59cff17809ef" (UID: "662f9f8c-cd9a-4722-ab84-59cff17809ef"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:31:10.633683 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:10.633664    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/662f9f8c-cd9a-4722-ab84-59cff17809ef-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "662f9f8c-cd9a-4722-ab84-59cff17809ef" (UID: "662f9f8c-cd9a-4722-ab84-59cff17809ef"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:31:10.641464 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:10.641436    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/662f9f8c-cd9a-4722-ab84-59cff17809ef-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "662f9f8c-cd9a-4722-ab84-59cff17809ef" (UID: "662f9f8c-cd9a-4722-ab84-59cff17809ef"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:31:10.731926 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:10.731845    2569 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/662f9f8c-cd9a-4722-ab84-59cff17809ef-installation-pull-secrets\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:31:10.731926 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:10.731880    2569 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-bound-sa-token\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:31:10.731926 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:10.731892    2569 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/662f9f8c-cd9a-4722-ab84-59cff17809ef-image-registry-private-configuration\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:31:10.731926 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:10.731902    2569 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/662f9f8c-cd9a-4722-ab84-59cff17809ef-ca-trust-extracted\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:31:10.731926 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:10.731910    2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-registry-tls\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:31:10.731926 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:10.731919    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s8qn\" (UniqueName: \"kubernetes.io/projected/662f9f8c-cd9a-4722-ab84-59cff17809ef-kube-api-access-4s8qn\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:31:10.827570 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:10.827533    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66f4d7d896-fbvv2"]
Jul 29 10:31:10.831668 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:10.831643    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-66f4d7d896-fbvv2"]
Jul 29 10:31:11.837483 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:11.837451    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="662f9f8c-cd9a-4722-ab84-59cff17809ef" path="/var/lib/kubelet/pods/662f9f8c-cd9a-4722-ab84-59cff17809ef/volumes"
Jul 29 10:31:13.486251 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:13.486217    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:31:13.557192 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:13.557160    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:31:21.072452 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:21.072413    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:31:21.080786 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:21.080759    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-9dd45858d-4m6ts"
Jul 29 10:31:21.084624 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:21.084597    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-9dd45858d-4m6ts"
Jul 29 10:31:21.173750 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:21.173719    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:31:21.274865 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:21.274836    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:31:42.945312 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:42.945225    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:31:42.964386 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:42.964361    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:31:43.642181 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:43.642153    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:31:52.555015 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:52.554975    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:31:52.656639 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:52.656603    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:31:55.104795 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:31:55.104760    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:32:01.052491 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.052449    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 10:32:01.053017 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.052841    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" containerName="prometheus" containerID="cri-o://a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77" gracePeriod=600
Jul 29 10:32:01.053017 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.052864    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" containerName="kube-rbac-proxy" containerID="cri-o://46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47" gracePeriod=600
Jul 29 10:32:01.053017 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.052904    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" containerName="config-reloader" containerID="cri-o://8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3" gracePeriod=600
Jul 29 10:32:01.053017 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.052927    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" containerName="kube-rbac-proxy-thanos" containerID="cri-o://967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23" gracePeriod=600
Jul 29 10:32:01.053017 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.052894    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" containerName="kube-rbac-proxy-web" containerID="cri-o://40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902" gracePeriod=600
Jul 29 10:32:01.053270 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.052881    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" containerName="thanos-sidecar" containerID="cri-o://976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60" gracePeriod=600
Jul 29 10:32:01.316993 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.316965    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.446884 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.446843    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-config-out\") pod \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") "
Jul 29 10:32:01.447111 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.446902    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-prometheus-trusted-ca-bundle\") pod \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") "
Jul 29 10:32:01.447111 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.446928    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") "
Jul 29 10:32:01.447111 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.446969    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-prometheus-k8s-rulefiles-0\") pod \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") "
Jul 29 10:32:01.447111 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.446991    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-secret-kube-rbac-proxy\") pod \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") "
Jul 29 10:32:01.447111 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.447017    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-secret-metrics-client-certs\") pod \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") "
Jul 29 10:32:01.447111 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.447043    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-config\") pod \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") "
Jul 29 10:32:01.447426 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.447117    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-secret-prometheus-k8s-tls\") pod \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") "
Jul 29 10:32:01.447426 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.447148    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-prometheus-k8s-rulefiles-1\") pod \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") "
Jul 29 10:32:01.447426 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.447199    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-prometheus-k8s-rulefiles-2\") pod \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") "
Jul 29 10:32:01.447426 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.447229    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-web-config\") pod \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") "
Jul 29 10:32:01.447426 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.447254    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pd2f6\" (UniqueName: \"kubernetes.io/projected/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-kube-api-access-pd2f6\") pod \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") "
Jul 29 10:32:01.447426 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.447285    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-tls-assets\") pod \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") "
Jul 29 10:32:01.447426 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.447311    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-prometheus-k8s-db\") pod \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") "
Jul 29 10:32:01.447426 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.447310    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" (UID: "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:32:01.447426 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.447340    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-configmap-metrics-client-ca\") pod \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") "
Jul 29 10:32:01.447426 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.447367    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-configmap-serving-certs-ca-bundle\") pod \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") "
Jul 29 10:32:01.447426 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.447393    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") "
Jul 29 10:32:01.447941 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.447432    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-thanos-prometheus-http-client-file\") pod \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") "
Jul 29 10:32:01.447941 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.447460    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-secret-grpc-tls\") pod \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") "
Jul 29 10:32:01.447941 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.447494    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-configmap-kubelet-serving-ca-bundle\") pod \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\" (UID: \"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26\") "
Jul 29 10:32:01.447941 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.447589    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" (UID: "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:32:01.447941 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.447762    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:32:01.447941 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.447781    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:32:01.447941 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.447929    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" (UID: "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:32:01.448325 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.448036    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" (UID: "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:32:01.448325 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.448045    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" (UID: "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:32:01.448325 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.448233    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" (UID: "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:32:01.448825 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.448573    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" (UID: "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:32:01.450065 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.450022    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" (UID: "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:32:01.451037 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.451007    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-kube-api-access-pd2f6" (OuterVolumeSpecName: "kube-api-access-pd2f6") pod "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" (UID: "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26"). InnerVolumeSpecName "kube-api-access-pd2f6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:32:01.451156 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.451018    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-config-out" (OuterVolumeSpecName: "config-out") pod "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" (UID: "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:32:01.451156 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.451095    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" (UID: "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:32:01.451529 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.451501    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" (UID: "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:32:01.451608 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.451526    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" (UID: "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:32:01.451608 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.451567    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" (UID: "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:32:01.451608 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.451578    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" (UID: "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:32:01.451726 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.451598    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" (UID: "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:32:01.451865 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.451838    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" (UID: "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:32:01.452030 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.452009    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" (UID: "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:32:01.452645 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.452631    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-config" (OuterVolumeSpecName: "config") pod "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" (UID: "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:32:01.462010 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.461986    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-web-config" (OuterVolumeSpecName: "web-config") pod "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" (UID: "d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:32:01.548728 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.548680    2569 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-web-config\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:32:01.548728 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.548721    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd2f6\" (UniqueName: \"kubernetes.io/projected/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-kube-api-access-pd2f6\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:32:01.548728 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.548733    2569 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-tls-assets\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:32:01.548728 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.548742    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-prometheus-k8s-db\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:32:01.548987 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.548751    2569 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-configmap-metrics-client-ca\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:32:01.548987 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.548762    2569 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:32:01.548987 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.548770    2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:32:01.548987 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.548780    2569 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:32:01.548987 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.548789    2569 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-secret-grpc-tls\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:32:01.548987 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.548798    2569 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:32:01.548987 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.548805    2569 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-config-out\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:32:01.548987 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.548813    2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:32:01.548987 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.548822    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:32:01.548987 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.548830    2569 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-secret-kube-rbac-proxy\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:32:01.548987 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.548838    2569 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-secret-metrics-client-certs\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:32:01.548987 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.548847    2569 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-config\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:32:01.548987 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.548854    2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:32:01.548987 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.548863    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:32:01.706239 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.706204    2569 generic.go:356] "Generic (PLEG): container finished" podID="d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" containerID="967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23" exitCode=0
Jul 29 10:32:01.706239 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.706229    2569 generic.go:356] "Generic (PLEG): container finished" podID="d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" containerID="46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47" exitCode=0
Jul 29 10:32:01.706239 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.706236    2569 generic.go:356] "Generic (PLEG): container finished" podID="d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" containerID="40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902" exitCode=0
Jul 29 10:32:01.706239 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.706242    2569 generic.go:356] "Generic (PLEG): container finished" podID="d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" containerID="976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60" exitCode=0
Jul 29 10:32:01.706239 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.706247    2569 generic.go:356] "Generic (PLEG): container finished" podID="d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" containerID="8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3" exitCode=0
Jul 29 10:32:01.706522 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.706252    2569 generic.go:356] "Generic (PLEG): container finished" podID="d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" containerID="a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77" exitCode=0
Jul 29 10:32:01.706522 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.706281    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26","Type":"ContainerDied","Data":"967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23"}
Jul 29 10:32:01.706522 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.706303    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.706522 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.706348    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26","Type":"ContainerDied","Data":"46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47"}
Jul 29 10:32:01.706522 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.706377    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26","Type":"ContainerDied","Data":"40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902"}
Jul 29 10:32:01.706522 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.706404    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26","Type":"ContainerDied","Data":"976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60"}
Jul 29 10:32:01.706522 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.706414    2569 scope.go:117] "RemoveContainer" containerID="967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23"
Jul 29 10:32:01.706522 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.706431    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26","Type":"ContainerDied","Data":"8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3"}
Jul 29 10:32:01.706522 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.706455    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26","Type":"ContainerDied","Data":"a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77"}
Jul 29 10:32:01.706522 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.706480    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26","Type":"ContainerDied","Data":"43d48d4ad623f2be32c9bffc77f313363dcf0fb8d6fc519db9ed3e0feba2c5d6"}
Jul 29 10:32:01.714354 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.714335    2569 scope.go:117] "RemoveContainer" containerID="46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47"
Jul 29 10:32:01.721173 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.721156    2569 scope.go:117] "RemoveContainer" containerID="40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902"
Jul 29 10:32:01.727959 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.727924    2569 scope.go:117] "RemoveContainer" containerID="976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60"
Jul 29 10:32:01.730292 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.730271    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 10:32:01.733878 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.733858    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 10:32:01.737672 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.737656    2569 scope.go:117] "RemoveContainer" containerID="8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3"
Jul 29 10:32:01.744394 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.744377    2569 scope.go:117] "RemoveContainer" containerID="a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77"
Jul 29 10:32:01.751941 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.751925    2569 scope.go:117] "RemoveContainer" containerID="f5c4d4e4bb6113e012b897496df839cf92ba01e508c30720e7ae641e9251e51e"
Jul 29 10:32:01.758747 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.758726    2569 scope.go:117] "RemoveContainer" containerID="967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23"
Jul 29 10:32:01.759040 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:32:01.759022    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23\": container with ID starting with 967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23 not found: ID does not exist" containerID="967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23"
Jul 29 10:32:01.759129 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.759047    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23"} err="failed to get container status \"967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23\": rpc error: code = NotFound desc = could not find container \"967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23\": container with ID starting with 967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23 not found: ID does not exist"
Jul 29 10:32:01.759129 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.759065    2569 scope.go:117] "RemoveContainer" containerID="46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47"
Jul 29 10:32:01.759363 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:32:01.759342    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47\": container with ID starting with 46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47 not found: ID does not exist" containerID="46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47"
Jul 29 10:32:01.759473 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.759367    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47"} err="failed to get container status \"46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47\": rpc error: code = NotFound desc = could not find container \"46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47\": container with ID starting with 46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47 not found: ID does not exist"
Jul 29 10:32:01.759473 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.759382    2569 scope.go:117] "RemoveContainer" containerID="40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902"
Jul 29 10:32:01.759576 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.759551    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 10:32:01.759630 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:32:01.759605    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902\": container with ID starting with 40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902 not found: ID does not exist" containerID="40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902"
Jul 29 10:32:01.759630 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.759619    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902"} err="failed to get container status \"40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902\": rpc error: code = NotFound desc = could not find container \"40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902\": container with ID starting with 40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902 not found: ID does not exist"
Jul 29 10:32:01.759743 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.759631    2569 scope.go:117] "RemoveContainer" containerID="976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60"
Jul 29 10:32:01.759859 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:32:01.759840    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60\": container with ID starting with 976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60 not found: ID does not exist" containerID="976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60"
Jul 29 10:32:01.759914 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.759863    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60"} err="failed to get container status \"976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60\": rpc error: code = NotFound desc = could not find container \"976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60\": container with ID starting with 976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60 not found: ID does not exist"
Jul 29 10:32:01.759914 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.759881    2569 scope.go:117] "RemoveContainer" containerID="8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3"
Jul 29 10:32:01.759914 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.759887    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cd86203-fe78-4a76-ab30-d13780b1a3db" containerName="console"
Jul 29 10:32:01.759914 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.759902    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cd86203-fe78-4a76-ab30-d13780b1a3db" containerName="console"
Jul 29 10:32:01.760285 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.759922    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" containerName="kube-rbac-proxy-web"
Jul 29 10:32:01.760285 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.759932    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" containerName="kube-rbac-proxy-web"
Jul 29 10:32:01.760285 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.759945    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" containerName="init-config-reloader"
Jul 29 10:32:01.760285 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.759954    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" containerName="init-config-reloader"
Jul 29 10:32:01.760285 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.759967    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="662f9f8c-cd9a-4722-ab84-59cff17809ef" containerName="registry"
Jul 29 10:32:01.760285 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.759975    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="662f9f8c-cd9a-4722-ab84-59cff17809ef" containerName="registry"
Jul 29 10:32:01.760285 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.759992    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" containerName="kube-rbac-proxy-thanos"
Jul 29 10:32:01.760285 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.760001    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" containerName="kube-rbac-proxy-thanos"
Jul 29 10:32:01.760285 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.760013    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" containerName="kube-rbac-proxy"
Jul 29 10:32:01.760285 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.760022    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" containerName="kube-rbac-proxy"
Jul 29 10:32:01.760285 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.760030    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" containerName="config-reloader"
Jul 29 10:32:01.760285 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.760039    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" containerName="config-reloader"
Jul 29 10:32:01.760285 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.760055    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" containerName="thanos-sidecar"
Jul 29 10:32:01.760285 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.760063    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" containerName="thanos-sidecar"
Jul 29 10:32:01.760285 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.760105    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" containerName="prometheus"
Jul 29 10:32:01.760285 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.760114    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" containerName="prometheus"
Jul 29 10:32:01.760285 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:32:01.760145    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3\": container with ID starting with 8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3 not found: ID does not exist" containerID="8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3"
Jul 29 10:32:01.760285 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.760167    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3"} err="failed to get container status \"8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3\": rpc error: code = NotFound desc = could not find container \"8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3\": container with ID starting with 8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3 not found: ID does not exist"
Jul 29 10:32:01.760285 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.760181    2569 scope.go:117] "RemoveContainer" containerID="a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77"
Jul 29 10:32:01.760285 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.760183    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" containerName="prometheus"
Jul 29 10:32:01.760285 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.760234    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" containerName="thanos-sidecar"
Jul 29 10:32:01.760285 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.760240    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" containerName="kube-rbac-proxy"
Jul 29 10:32:01.760285 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.760250    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cd86203-fe78-4a76-ab30-d13780b1a3db" containerName="console"
Jul 29 10:32:01.760285 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.760256    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" containerName="kube-rbac-proxy-thanos"
Jul 29 10:32:01.760285 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.760264    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="662f9f8c-cd9a-4722-ab84-59cff17809ef" containerName="registry"
Jul 29 10:32:01.760285 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.760272    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" containerName="config-reloader"
Jul 29 10:32:01.760285 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.760280    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" containerName="kube-rbac-proxy-web"
Jul 29 10:32:01.761207 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:32:01.760465    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77\": container with ID starting with a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77 not found: ID does not exist" containerID="a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77"
Jul 29 10:32:01.761207 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.760485    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77"} err="failed to get container status \"a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77\": rpc error: code = NotFound desc = could not find container \"a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77\": container with ID starting with a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77 not found: ID does not exist"
Jul 29 10:32:01.761207 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.760498    2569 scope.go:117] "RemoveContainer" containerID="f5c4d4e4bb6113e012b897496df839cf92ba01e508c30720e7ae641e9251e51e"
Jul 29 10:32:01.761207 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:32:01.760708    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5c4d4e4bb6113e012b897496df839cf92ba01e508c30720e7ae641e9251e51e\": container with ID starting with f5c4d4e4bb6113e012b897496df839cf92ba01e508c30720e7ae641e9251e51e not found: ID does not exist" containerID="f5c4d4e4bb6113e012b897496df839cf92ba01e508c30720e7ae641e9251e51e"
Jul 29 10:32:01.761207 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.760722    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5c4d4e4bb6113e012b897496df839cf92ba01e508c30720e7ae641e9251e51e"} err="failed to get container status \"f5c4d4e4bb6113e012b897496df839cf92ba01e508c30720e7ae641e9251e51e\": rpc error: code = NotFound desc = could not find container \"f5c4d4e4bb6113e012b897496df839cf92ba01e508c30720e7ae641e9251e51e\": container with ID starting with f5c4d4e4bb6113e012b897496df839cf92ba01e508c30720e7ae641e9251e51e not found: ID does not exist"
Jul 29 10:32:01.761207 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.760734    2569 scope.go:117] "RemoveContainer" containerID="967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23"
Jul 29 10:32:01.761207 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.760927    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23"} err="failed to get container status \"967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23\": rpc error: code = NotFound desc = could not find container \"967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23\": container with ID starting with 967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23 not found: ID does not exist"
Jul 29 10:32:01.761207 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.760945    2569 scope.go:117] "RemoveContainer" containerID="46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47"
Jul 29 10:32:01.761207 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.761173    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47"} err="failed to get container status \"46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47\": rpc error: code = NotFound desc = could not find container \"46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47\": container with ID starting with 46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47 not found: ID does not exist"
Jul 29 10:32:01.761207 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.761189    2569 scope.go:117] "RemoveContainer" containerID="40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902"
Jul 29 10:32:01.761563 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.761366    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902"} err="failed to get container status \"40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902\": rpc error: code = NotFound desc = could not find container \"40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902\": container with ID starting with 40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902 not found: ID does not exist"
Jul 29 10:32:01.761563 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.761380    2569 scope.go:117] "RemoveContainer" containerID="976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60"
Jul 29 10:32:01.761636 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.761587    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60"} err="failed to get container status \"976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60\": rpc error: code = NotFound desc = could not find container \"976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60\": container with ID starting with 976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60 not found: ID does not exist"
Jul 29 10:32:01.761636 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.761601    2569 scope.go:117] "RemoveContainer" containerID="8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3"
Jul 29 10:32:01.761847 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.761827    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3"} err="failed to get container status \"8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3\": rpc error: code = NotFound desc = could not find container \"8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3\": container with ID starting with 8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3 not found: ID does not exist"
Jul 29 10:32:01.761847 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.761847    2569 scope.go:117] "RemoveContainer" containerID="a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77"
Jul 29 10:32:01.762125 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.762103    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77"} err="failed to get container status \"a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77\": rpc error: code = NotFound desc = could not find container \"a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77\": container with ID starting with a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77 not found: ID does not exist"
Jul 29 10:32:01.762202 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.762127    2569 scope.go:117] "RemoveContainer" containerID="f5c4d4e4bb6113e012b897496df839cf92ba01e508c30720e7ae641e9251e51e"
Jul 29 10:32:01.762347 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.762331    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5c4d4e4bb6113e012b897496df839cf92ba01e508c30720e7ae641e9251e51e"} err="failed to get container status \"f5c4d4e4bb6113e012b897496df839cf92ba01e508c30720e7ae641e9251e51e\": rpc error: code = NotFound desc = could not find container \"f5c4d4e4bb6113e012b897496df839cf92ba01e508c30720e7ae641e9251e51e\": container with ID starting with f5c4d4e4bb6113e012b897496df839cf92ba01e508c30720e7ae641e9251e51e not found: ID does not exist"
Jul 29 10:32:01.762413 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.762349    2569 scope.go:117] "RemoveContainer" containerID="967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23"
Jul 29 10:32:01.762567 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.762550    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23"} err="failed to get container status \"967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23\": rpc error: code = NotFound desc = could not find container \"967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23\": container with ID starting with 967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23 not found: ID does not exist"
Jul 29 10:32:01.762621 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.762568    2569 scope.go:117] "RemoveContainer" containerID="46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47"
Jul 29 10:32:01.762759 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.762744    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47"} err="failed to get container status \"46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47\": rpc error: code = NotFound desc = could not find container \"46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47\": container with ID starting with 46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47 not found: ID does not exist"
Jul 29 10:32:01.762807 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.762759    2569 scope.go:117] "RemoveContainer" containerID="40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902"
Jul 29 10:32:01.762975 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.762954    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902"} err="failed to get container status \"40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902\": rpc error: code = NotFound desc = could not find container \"40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902\": container with ID starting with 40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902 not found: ID does not exist"
Jul 29 10:32:01.763032 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.762975    2569 scope.go:117] "RemoveContainer" containerID="976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60"
Jul 29 10:32:01.763189 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.763173    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60"} err="failed to get container status \"976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60\": rpc error: code = NotFound desc = could not find container \"976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60\": container with ID starting with 976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60 not found: ID does not exist"
Jul 29 10:32:01.763240 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.763189    2569 scope.go:117] "RemoveContainer" containerID="8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3"
Jul 29 10:32:01.763380 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.763366    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3"} err="failed to get container status \"8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3\": rpc error: code = NotFound desc = could not find container \"8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3\": container with ID starting with 8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3 not found: ID does not exist"
Jul 29 10:32:01.763848 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.763837    2569 scope.go:117] "RemoveContainer" containerID="a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77"
Jul 29 10:32:01.764014 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.763996    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77"} err="failed to get container status \"a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77\": rpc error: code = NotFound desc = could not find container \"a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77\": container with ID starting with a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77 not found: ID does not exist"
Jul 29 10:32:01.764055 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.764015    2569 scope.go:117] "RemoveContainer" containerID="f5c4d4e4bb6113e012b897496df839cf92ba01e508c30720e7ae641e9251e51e"
Jul 29 10:32:01.764055 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.764029    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.764449 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.764252    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5c4d4e4bb6113e012b897496df839cf92ba01e508c30720e7ae641e9251e51e"} err="failed to get container status \"f5c4d4e4bb6113e012b897496df839cf92ba01e508c30720e7ae641e9251e51e\": rpc error: code = NotFound desc = could not find container \"f5c4d4e4bb6113e012b897496df839cf92ba01e508c30720e7ae641e9251e51e\": container with ID starting with f5c4d4e4bb6113e012b897496df839cf92ba01e508c30720e7ae641e9251e51e not found: ID does not exist"
Jul 29 10:32:01.764449 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.764274    2569 scope.go:117] "RemoveContainer" containerID="967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23"
Jul 29 10:32:01.764725 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.764700    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23"} err="failed to get container status \"967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23\": rpc error: code = NotFound desc = could not find container \"967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23\": container with ID starting with 967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23 not found: ID does not exist"
Jul 29 10:32:01.764812 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.764728    2569 scope.go:117] "RemoveContainer" containerID="46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47"
Jul 29 10:32:01.764960 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.764940    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47"} err="failed to get container status \"46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47\": rpc error: code = NotFound desc = could not find container \"46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47\": container with ID starting with 46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47 not found: ID does not exist"
Jul 29 10:32:01.765048 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.764960    2569 scope.go:117] "RemoveContainer" containerID="40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902"
Jul 29 10:32:01.765282 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.765255    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902"} err="failed to get container status \"40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902\": rpc error: code = NotFound desc = could not find container \"40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902\": container with ID starting with 40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902 not found: ID does not exist"
Jul 29 10:32:01.765385 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.765283    2569 scope.go:117] "RemoveContainer" containerID="976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60"
Jul 29 10:32:01.765606 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.765587    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60"} err="failed to get container status \"976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60\": rpc error: code = NotFound desc = could not find container \"976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60\": container with ID starting with 976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60 not found: ID does not exist"
Jul 29 10:32:01.765717 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.765609    2569 scope.go:117] "RemoveContainer" containerID="8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3"
Jul 29 10:32:01.765894 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.765873    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3"} err="failed to get container status \"8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3\": rpc error: code = NotFound desc = could not find container \"8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3\": container with ID starting with 8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3 not found: ID does not exist"
Jul 29 10:32:01.765894 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.765894    2569 scope.go:117] "RemoveContainer" containerID="a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77"
Jul 29 10:32:01.766186 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.766161    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77"} err="failed to get container status \"a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77\": rpc error: code = NotFound desc = could not find container \"a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77\": container with ID starting with a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77 not found: ID does not exist"
Jul 29 10:32:01.766258 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.766188    2569 scope.go:117] "RemoveContainer" containerID="f5c4d4e4bb6113e012b897496df839cf92ba01e508c30720e7ae641e9251e51e"
Jul 29 10:32:01.766476 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.766458    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5c4d4e4bb6113e012b897496df839cf92ba01e508c30720e7ae641e9251e51e"} err="failed to get container status \"f5c4d4e4bb6113e012b897496df839cf92ba01e508c30720e7ae641e9251e51e\": rpc error: code = NotFound desc = could not find container \"f5c4d4e4bb6113e012b897496df839cf92ba01e508c30720e7ae641e9251e51e\": container with ID starting with f5c4d4e4bb6113e012b897496df839cf92ba01e508c30720e7ae641e9251e51e not found: ID does not exist"
Jul 29 10:32:01.766523 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.766477    2569 scope.go:117] "RemoveContainer" containerID="967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23"
Jul 29 10:32:01.766705 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.766689    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23"} err="failed to get container status \"967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23\": rpc error: code = NotFound desc = could not find container \"967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23\": container with ID starting with 967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23 not found: ID does not exist"
Jul 29 10:32:01.766764 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.766705    2569 scope.go:117] "RemoveContainer" containerID="46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47"
Jul 29 10:32:01.766937 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.766908    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47"} err="failed to get container status \"46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47\": rpc error: code = NotFound desc = could not find container \"46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47\": container with ID starting with 46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47 not found: ID does not exist"
Jul 29 10:32:01.767033 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.766938    2569 scope.go:117] "RemoveContainer" containerID="40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902"
Jul 29 10:32:01.767128 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.767102    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 29 10:32:01.767128 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.767114    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 29 10:32:01.767242 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.767154    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 29 10:32:01.767294 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.767235    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902"} err="failed to get container status \"40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902\": rpc error: code = NotFound desc = could not find container \"40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902\": container with ID starting with 40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902 not found: ID does not exist"
Jul 29 10:32:01.767294 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.767257    2569 scope.go:117] "RemoveContainer" containerID="976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60"
Jul 29 10:32:01.767517 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.767495    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60"} err="failed to get container status \"976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60\": rpc error: code = NotFound desc = could not find container \"976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60\": container with ID starting with 976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60 not found: ID does not exist"
Jul 29 10:32:01.767517 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.767515    2569 scope.go:117] "RemoveContainer" containerID="8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3"
Jul 29 10:32:01.767778 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.767760    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3"} err="failed to get container status \"8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3\": rpc error: code = NotFound desc = could not find container \"8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3\": container with ID starting with 8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3 not found: ID does not exist"
Jul 29 10:32:01.767778 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.767777    2569 scope.go:117] "RemoveContainer" containerID="a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77"
Jul 29 10:32:01.768009 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.767989    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77"} err="failed to get container status \"a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77\": rpc error: code = NotFound desc = could not find container \"a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77\": container with ID starting with a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77 not found: ID does not exist"
Jul 29 10:32:01.768108 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.768012    2569 scope.go:117] "RemoveContainer" containerID="f5c4d4e4bb6113e012b897496df839cf92ba01e508c30720e7ae641e9251e51e"
Jul 29 10:32:01.768173 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.768146    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 29 10:32:01.768336 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.768296    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 29 10:32:01.768447 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.768324    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5c4d4e4bb6113e012b897496df839cf92ba01e508c30720e7ae641e9251e51e"} err="failed to get container status \"f5c4d4e4bb6113e012b897496df839cf92ba01e508c30720e7ae641e9251e51e\": rpc error: code = NotFound desc = could not find container \"f5c4d4e4bb6113e012b897496df839cf92ba01e508c30720e7ae641e9251e51e\": container with ID starting with f5c4d4e4bb6113e012b897496df839cf92ba01e508c30720e7ae641e9251e51e not found: ID does not exist"
Jul 29 10:32:01.768447 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.768384    2569 scope.go:117] "RemoveContainer" containerID="967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23"
Jul 29 10:32:01.768447 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.768396    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 29 10:32:01.768620 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.768600    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 29 10:32:01.768726 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.768705    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23"} err="failed to get container status \"967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23\": rpc error: code = NotFound desc = could not find container \"967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23\": container with ID starting with 967043985b97285439019066feadc2c880d51d47c5d43ba88012a4a110497e23 not found: ID does not exist"
Jul 29 10:32:01.768799 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.768729    2569 scope.go:117] "RemoveContainer" containerID="46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47"
Jul 29 10:32:01.768799 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.768784    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-22qdcjn05jf7j\""
Jul 29 10:32:01.768954 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.768845    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 29 10:32:01.768954 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.768852    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 29 10:32:01.769096 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.768990    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47"} err="failed to get container status \"46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47\": rpc error: code = NotFound desc = could not find container \"46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47\": container with ID starting with 46b1c4241679a793994f3c4132f388f49266241e22a4b4afd512186e7f37ee47 not found: ID does not exist"
Jul 29 10:32:01.769096 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.769013    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 29 10:32:01.769208 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.769097    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 29 10:32:01.769208 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.769158    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-tfx9c\""
Jul 29 10:32:01.769302 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.769016    2569 scope.go:117] "RemoveContainer" containerID="40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902"
Jul 29 10:32:01.769436 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.769419    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 29 10:32:01.769700 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.769673    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902"} err="failed to get container status \"40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902\": rpc error: code = NotFound desc = could not find container \"40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902\": container with ID starting with 40d49dd1781cf7dab00d7a7e8f7b6206184aa23af6bc9434bdb664de2b92f902 not found: ID does not exist"
Jul 29 10:32:01.769700 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.769698    2569 scope.go:117] "RemoveContainer" containerID="976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60"
Jul 29 10:32:01.770375 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.770331    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60"} err="failed to get container status \"976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60\": rpc error: code = NotFound desc = could not find container \"976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60\": container with ID starting with 976605ae99ce21c34fdc2e2d02ec795bd273ac9741d44e6eaef24a47a3e0ef60 not found: ID does not exist"
Jul 29 10:32:01.770375 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.770358    2569 scope.go:117] "RemoveContainer" containerID="8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3"
Jul 29 10:32:01.770660 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.770640    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3"} err="failed to get container status \"8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3\": rpc error: code = NotFound desc = could not find container \"8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3\": container with ID starting with 8a622e33addb3383f12d4b80d876b07c9c8d0b749e7b7da6a714f324e8af40b3 not found: ID does not exist"
Jul 29 10:32:01.770660 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.770660    2569 scope.go:117] "RemoveContainer" containerID="a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77"
Jul 29 10:32:01.770936 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.770897    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77"} err="failed to get container status \"a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77\": rpc error: code = NotFound desc = could not find container \"a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77\": container with ID starting with a549d5511b7284184b645669a3d4008ae95050672ba08a9ca56b266148adbb77 not found: ID does not exist"
Jul 29 10:32:01.770936 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.770921    2569 scope.go:117] "RemoveContainer" containerID="f5c4d4e4bb6113e012b897496df839cf92ba01e508c30720e7ae641e9251e51e"
Jul 29 10:32:01.771311 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.771199    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5c4d4e4bb6113e012b897496df839cf92ba01e508c30720e7ae641e9251e51e"} err="failed to get container status \"f5c4d4e4bb6113e012b897496df839cf92ba01e508c30720e7ae641e9251e51e\": rpc error: code = NotFound desc = could not find container \"f5c4d4e4bb6113e012b897496df839cf92ba01e508c30720e7ae641e9251e51e\": container with ID starting with f5c4d4e4bb6113e012b897496df839cf92ba01e508c30720e7ae641e9251e51e not found: ID does not exist"
Jul 29 10:32:01.771528 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.771496    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 29 10:32:01.777115 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.775503    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 29 10:32:01.787586 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.787559    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 10:32:01.837514 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.837482    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26" path="/var/lib/kubelet/pods/d20dcaa9-3476-4fd3-bb2b-6a936d2d9c26/volumes"
Jul 29 10:32:01.851956 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.851930    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/672f11b9-45a4-4641-be41-e585b857b1ac-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.852046 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.851962    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/672f11b9-45a4-4641-be41-e585b857b1ac-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.852046 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.851983    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/672f11b9-45a4-4641-be41-e585b857b1ac-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.852046 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.852006    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/672f11b9-45a4-4641-be41-e585b857b1ac-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.852167 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.852099    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/672f11b9-45a4-4641-be41-e585b857b1ac-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.852167 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.852131    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/672f11b9-45a4-4641-be41-e585b857b1ac-config-out\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.852167 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.852151    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/672f11b9-45a4-4641-be41-e585b857b1ac-web-config\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.852254 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.852168    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/672f11b9-45a4-4641-be41-e585b857b1ac-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.852254 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.852194    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/672f11b9-45a4-4641-be41-e585b857b1ac-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.852254 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.852219    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/672f11b9-45a4-4641-be41-e585b857b1ac-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.852254 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.852238    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/672f11b9-45a4-4641-be41-e585b857b1ac-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.852367 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.852257    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/672f11b9-45a4-4641-be41-e585b857b1ac-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.852367 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.852273    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/672f11b9-45a4-4641-be41-e585b857b1ac-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.852367 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.852303    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/672f11b9-45a4-4641-be41-e585b857b1ac-config\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.852367 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.852338    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/672f11b9-45a4-4641-be41-e585b857b1ac-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.852367 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.852354    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/672f11b9-45a4-4641-be41-e585b857b1ac-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.852532 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.852375    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/672f11b9-45a4-4641-be41-e585b857b1ac-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.852532 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.852424    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/672f11b9-45a4-4641-be41-e585b857b1ac-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.852532 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.852442    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xlnw\" (UniqueName: \"kubernetes.io/projected/672f11b9-45a4-4641-be41-e585b857b1ac-kube-api-access-7xlnw\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.852532 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.852458    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/672f11b9-45a4-4641-be41-e585b857b1ac-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.953628 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.953590    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/672f11b9-45a4-4641-be41-e585b857b1ac-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.953820 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.953637    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/672f11b9-45a4-4641-be41-e585b857b1ac-config-out\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.953820 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.953664    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/672f11b9-45a4-4641-be41-e585b857b1ac-web-config\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.953820 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.953678    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/672f11b9-45a4-4641-be41-e585b857b1ac-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.953820 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.953696    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/672f11b9-45a4-4641-be41-e585b857b1ac-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.953820 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.953726    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/672f11b9-45a4-4641-be41-e585b857b1ac-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.953820 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.953751    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/672f11b9-45a4-4641-be41-e585b857b1ac-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.953820 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.953778    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/672f11b9-45a4-4641-be41-e585b857b1ac-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.953820 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.953801    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/672f11b9-45a4-4641-be41-e585b857b1ac-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.954255 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.953826    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/672f11b9-45a4-4641-be41-e585b857b1ac-config\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.954255 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.953859    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/672f11b9-45a4-4641-be41-e585b857b1ac-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.954255 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.953884    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/672f11b9-45a4-4641-be41-e585b857b1ac-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.954255 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.953912    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/672f11b9-45a4-4641-be41-e585b857b1ac-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.954255 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.953951    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/672f11b9-45a4-4641-be41-e585b857b1ac-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.954255 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.953981    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xlnw\" (UniqueName: \"kubernetes.io/projected/672f11b9-45a4-4641-be41-e585b857b1ac-kube-api-access-7xlnw\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.954255 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.954008    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/672f11b9-45a4-4641-be41-e585b857b1ac-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.954255 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.954038    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/672f11b9-45a4-4641-be41-e585b857b1ac-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.954255 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.954094    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/672f11b9-45a4-4641-be41-e585b857b1ac-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.954255 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.954125    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/672f11b9-45a4-4641-be41-e585b857b1ac-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.954255 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.954154    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/672f11b9-45a4-4641-be41-e585b857b1ac-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.954818 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.954393    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/672f11b9-45a4-4641-be41-e585b857b1ac-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.954818 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.954540    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/672f11b9-45a4-4641-be41-e585b857b1ac-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.954921 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.954825    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/672f11b9-45a4-4641-be41-e585b857b1ac-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.954921 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.954909    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/672f11b9-45a4-4641-be41-e585b857b1ac-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.955364 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.955280    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/672f11b9-45a4-4641-be41-e585b857b1ac-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.955364 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.955318    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/672f11b9-45a4-4641-be41-e585b857b1ac-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.956216 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.956194    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/672f11b9-45a4-4641-be41-e585b857b1ac-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.956813 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.956752    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/672f11b9-45a4-4641-be41-e585b857b1ac-web-config\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.956932 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.956913    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/672f11b9-45a4-4641-be41-e585b857b1ac-config-out\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.957186 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.957156    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/672f11b9-45a4-4641-be41-e585b857b1ac-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.957308 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.957278    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/672f11b9-45a4-4641-be41-e585b857b1ac-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.957908 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.957862    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/672f11b9-45a4-4641-be41-e585b857b1ac-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.958615 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.958589    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/672f11b9-45a4-4641-be41-e585b857b1ac-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.958807 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.958783    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/672f11b9-45a4-4641-be41-e585b857b1ac-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.958881 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.958807    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/672f11b9-45a4-4641-be41-e585b857b1ac-config\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.959473 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.959443    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/672f11b9-45a4-4641-be41-e585b857b1ac-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.959707 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.959686    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/672f11b9-45a4-4641-be41-e585b857b1ac-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.959745 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.959686    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/672f11b9-45a4-4641-be41-e585b857b1ac-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.959804 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.959788    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/672f11b9-45a4-4641-be41-e585b857b1ac-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:01.964171 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:01.964153    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xlnw\" (UniqueName: \"kubernetes.io/projected/672f11b9-45a4-4641-be41-e585b857b1ac-kube-api-access-7xlnw\") pod \"prometheus-k8s-0\" (UID: \"672f11b9-45a4-4641-be41-e585b857b1ac\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:02.078698 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:02.078668    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:02.215803 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:02.215723    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 10:32:02.219195 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:32:02.219167    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod672f11b9_45a4_4641_be41_e585b857b1ac.slice/crio-b27b762caa0c5ef79b31a80c5ad5e3862a07e9a56511d941f3fb94c9e84ea16f WatchSource:0}: Error finding container b27b762caa0c5ef79b31a80c5ad5e3862a07e9a56511d941f3fb94c9e84ea16f: Status 404 returned error can't find the container with id b27b762caa0c5ef79b31a80c5ad5e3862a07e9a56511d941f3fb94c9e84ea16f
Jul 29 10:32:02.711302 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:02.711267    2569 generic.go:356] "Generic (PLEG): container finished" podID="672f11b9-45a4-4641-be41-e585b857b1ac" containerID="c46f9f395a64e6ca66dd14991a84149d90cb738d8dbf5b03056daa5e84957e27" exitCode=0
Jul 29 10:32:02.711500 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:02.711358    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"672f11b9-45a4-4641-be41-e585b857b1ac","Type":"ContainerDied","Data":"c46f9f395a64e6ca66dd14991a84149d90cb738d8dbf5b03056daa5e84957e27"}
Jul 29 10:32:02.711500 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:02.711442    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"672f11b9-45a4-4641-be41-e585b857b1ac","Type":"ContainerStarted","Data":"b27b762caa0c5ef79b31a80c5ad5e3862a07e9a56511d941f3fb94c9e84ea16f"}
Jul 29 10:32:03.720472 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:03.720435    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"672f11b9-45a4-4641-be41-e585b857b1ac","Type":"ContainerStarted","Data":"7e33b0c602e8d133ad5e31a2b4fd0abe014b32bea27c3d14dda60fc85e529a9b"}
Jul 29 10:32:03.720869 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:03.720504    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"672f11b9-45a4-4641-be41-e585b857b1ac","Type":"ContainerStarted","Data":"f3dbb7e2cb91a4b61f8fbb8450ad429515f372ed54696ec483c36eb645cc1d82"}
Jul 29 10:32:03.720869 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:03.720530    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"672f11b9-45a4-4641-be41-e585b857b1ac","Type":"ContainerStarted","Data":"75cf0a0b13074df6c5bcbb2b28426d65a7a4a78e6638b67b81f38bb22a8cf398"}
Jul 29 10:32:03.720869 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:03.720555    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"672f11b9-45a4-4641-be41-e585b857b1ac","Type":"ContainerStarted","Data":"60cbda1546b7d8ac926f48bf4ce098457591a2d1371b52ca52e875af98c09a33"}
Jul 29 10:32:03.720869 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:03.720583    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"672f11b9-45a4-4641-be41-e585b857b1ac","Type":"ContainerStarted","Data":"db69cc0d3979b4eaed2744d9e3dcb00af9ce5ae89ffd01bf0a9325ddd7a5fbde"}
Jul 29 10:32:03.720869 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:03.720606    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"672f11b9-45a4-4641-be41-e585b857b1ac","Type":"ContainerStarted","Data":"f004bc6c4c52ba9388bbc10c5edb372df2b0c03437ef178872ceb7e66978a734"}
Jul 29 10:32:03.751050 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:03.750991    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.733003452 podStartE2EDuration="2.75096569s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.962238ms" isStatefulPod=false podCreationTimestamp="2026-07-29 10:32:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:32:03.749860715 +0000 UTC m=+200.474651388" watchObservedRunningTime="2026-07-29 10:32:03.75096569 +0000 UTC m=+200.475756365"
Jul 29 10:32:04.489224 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:04.489178    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:32:04.554628 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:04.554580    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:32:05.764347 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:05.764313    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:32:05.864772 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:05.864739    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:32:05.928526 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:05.928496    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:32:07.079826 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:07.079789    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:32:20.984306 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:20.984267    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:32:21.084275 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:32:21.084244    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:33:02.078958 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:33:02.078888    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:33:02.094396 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:33:02.094369    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:33:02.956774 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:33:02.956747    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:33:09.376737 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:33:09.376649    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:33:21.743251 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:33:21.743209    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:33:43.699708 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:33:43.699672    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-pvdgw_afd0d65c-bbfc-49f0-81b6-e855fd81da7f/console-operator/1.log"
Jul 29 10:33:43.706409 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:33:43.706384    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-pvdgw_afd0d65c-bbfc-49f0-81b6-e855fd81da7f/console-operator/1.log"
Jul 29 10:33:43.710544 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:33:43.710511    2569 kubelet.go:1648] "Image garbage collection succeeded"
Jul 29 10:34:02.086669 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:34:02.086633    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:34:22.907669 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:34:22.907629    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:34:24.445964 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:34:24.445930    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:35:01.093752 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:35:01.093670    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:35:01.187589 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:35:01.187555    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:35:23.446977 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:35:23.446939    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:35:23.703316 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:35:23.703236    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:35:23.906485 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:35:23.906455    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:35:24.002252 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:35:24.002163    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:35:24.102738 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:35:24.102702    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:35:24.670029 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:35:24.669994    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:35:24.729089 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:35:24.729042    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:35:24.960923 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:35:24.960830    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:35:25.061923 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:35:25.061885    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:35:25.167917 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:35:25.167877    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:35:25.268666 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:35:25.268569    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:35:25.368669 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:35:25.368637    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:35:51.272597 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:35:51.272562    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:35:51.360187 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:35:51.360147    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:35:51.431352 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:35:51.431323    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:35:52.008960 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:35:52.008930    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:35:52.074640 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:35:52.074592    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:35:59.119837 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:35:59.119773    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:36:01.767542 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:36:01.767478    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:36:01.859859 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:36:01.859819    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:36:01.946343 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:36:01.946308    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:36:40.429944 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:36:40.429894    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:36:40.505373 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:36:40.505327    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:36:40.566883 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:36:40.566846    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:36:40.669374 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:36:40.669342    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:36:40.768121 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:36:40.768018    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:36:41.256765 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:36:41.256726    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:36:41.354422 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:36:41.354384    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:58.252614 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:37:58.252534    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:58.295083 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:37:58.295043    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-pz52z"]
Jul 29 10:37:58.298813 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:37:58.298794    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-pz52z"
Jul 29 10:37:58.301391 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:37:58.301363    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 29 10:37:58.301534 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:37:58.301479    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 29 10:37:58.302476 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:37:58.302455    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-c9dkk\""
Jul 29 10:37:58.302589 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:37:58.302482    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 29 10:37:58.305708 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:37:58.305686    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-pz52z"]
Jul 29 10:37:58.322702 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:37:58.322670    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:58.383566 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:37:58.383532    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:58.406714 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:37:58.406679    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a2002a3-039c-4654-b534-0dff05f3c704-tls-certs\") pod \"model-serving-api-7b954ffc95-pz52z\" (UID: \"7a2002a3-039c-4654-b534-0dff05f3c704\") " pod="kserve/model-serving-api-7b954ffc95-pz52z"
Jul 29 10:37:58.406848 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:37:58.406742    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk4hr\" (UniqueName: \"kubernetes.io/projected/7a2002a3-039c-4654-b534-0dff05f3c704-kube-api-access-qk4hr\") pod \"model-serving-api-7b954ffc95-pz52z\" (UID: \"7a2002a3-039c-4654-b534-0dff05f3c704\") " pod="kserve/model-serving-api-7b954ffc95-pz52z"
Jul 29 10:37:58.507758 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:37:58.507685    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a2002a3-039c-4654-b534-0dff05f3c704-tls-certs\") pod \"model-serving-api-7b954ffc95-pz52z\" (UID: \"7a2002a3-039c-4654-b534-0dff05f3c704\") " pod="kserve/model-serving-api-7b954ffc95-pz52z"
Jul 29 10:37:58.507758 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:37:58.507725    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk4hr\" (UniqueName: \"kubernetes.io/projected/7a2002a3-039c-4654-b534-0dff05f3c704-kube-api-access-qk4hr\") pod \"model-serving-api-7b954ffc95-pz52z\" (UID: \"7a2002a3-039c-4654-b534-0dff05f3c704\") " pod="kserve/model-serving-api-7b954ffc95-pz52z"
Jul 29 10:37:58.510295 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:37:58.510271    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a2002a3-039c-4654-b534-0dff05f3c704-tls-certs\") pod \"model-serving-api-7b954ffc95-pz52z\" (UID: \"7a2002a3-039c-4654-b534-0dff05f3c704\") " pod="kserve/model-serving-api-7b954ffc95-pz52z"
Jul 29 10:37:58.515688 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:37:58.515667    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk4hr\" (UniqueName: \"kubernetes.io/projected/7a2002a3-039c-4654-b534-0dff05f3c704-kube-api-access-qk4hr\") pod \"model-serving-api-7b954ffc95-pz52z\" (UID: \"7a2002a3-039c-4654-b534-0dff05f3c704\") " pod="kserve/model-serving-api-7b954ffc95-pz52z"
Jul 29 10:37:58.553108 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:37:58.549698    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:58.611045 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:37:58.610999    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-pz52z"
Jul 29 10:37:58.738294 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:37:58.738115    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-pz52z"]
Jul 29 10:37:58.741022 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:37:58.740983    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a2002a3_039c_4654_b534_0dff05f3c704.slice/crio-915949867dda99dd9472dba068f54ad4c207feea74e8de81cc24fb706958ecfc WatchSource:0}: Error finding container 915949867dda99dd9472dba068f54ad4c207feea74e8de81cc24fb706958ecfc: Status 404 returned error can't find the container with id 915949867dda99dd9472dba068f54ad4c207feea74e8de81cc24fb706958ecfc
Jul 29 10:37:58.742940 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:37:58.742920    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 10:37:58.978590 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:37:58.978558    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:59.029609 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:37:59.029578    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-pz52z" event={"ID":"7a2002a3-039c-4654-b534-0dff05f3c704","Type":"ContainerStarted","Data":"915949867dda99dd9472dba068f54ad4c207feea74e8de81cc24fb706958ecfc"}
Jul 29 10:37:59.037599 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:37:59.037571    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:38:01.038352 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:38:01.038316    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-pz52z" event={"ID":"7a2002a3-039c-4654-b534-0dff05f3c704","Type":"ContainerStarted","Data":"b0683803ab84365940c9f5c0e9eeccf3bf9ce01824eb159e3f77a63f16068d9d"}
Jul 29 10:38:01.038746 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:38:01.038545    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-pz52z"
Jul 29 10:38:01.056643 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:38:01.056546    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-pz52z" podStartSLOduration=0.994311059 podStartE2EDuration="3.05653057s" totalImagesPullingTime="2.062219511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:38:01.055236994 +0000 UTC m=+557.780027679" watchObservedRunningTime="2026-07-29 10:38:01.05653057 +0000 UTC m=+557.781321242"
Jul 29 10:38:12.046778 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:38:12.046732    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-pz52z"
Jul 29 10:38:13.917910 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:38:13.917869    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-ttxsr"]
Jul 29 10:38:13.921270 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:38:13.921250    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-ttxsr"
Jul 29 10:38:13.923539 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:38:13.923520    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 29 10:38:13.923896 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:38:13.923882    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-bxbkh\""
Jul 29 10:38:13.929060 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:38:13.929038    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-ttxsr"]
Jul 29 10:38:13.950186 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:38:13.950147    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0c13552-5b54-4754-b970-65a8307a4698-model-cache\") pod \"s3-init-ttxsr\" (UID: \"f0c13552-5b54-4754-b970-65a8307a4698\") " pod="kserve/s3-init-ttxsr"
Jul 29 10:38:13.950186 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:38:13.950188    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9css\" (UniqueName: \"kubernetes.io/projected/f0c13552-5b54-4754-b970-65a8307a4698-kube-api-access-l9css\") pod \"s3-init-ttxsr\" (UID: \"f0c13552-5b54-4754-b970-65a8307a4698\") " pod="kserve/s3-init-ttxsr"
Jul 29 10:38:14.050622 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:38:14.050581    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0c13552-5b54-4754-b970-65a8307a4698-model-cache\") pod \"s3-init-ttxsr\" (UID: \"f0c13552-5b54-4754-b970-65a8307a4698\") " pod="kserve/s3-init-ttxsr"
Jul 29 10:38:14.050622 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:38:14.050623    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9css\" (UniqueName: \"kubernetes.io/projected/f0c13552-5b54-4754-b970-65a8307a4698-kube-api-access-l9css\") pod \"s3-init-ttxsr\" (UID: \"f0c13552-5b54-4754-b970-65a8307a4698\") " pod="kserve/s3-init-ttxsr"
Jul 29 10:38:14.050980 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:38:14.050959    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0c13552-5b54-4754-b970-65a8307a4698-model-cache\") pod \"s3-init-ttxsr\" (UID: \"f0c13552-5b54-4754-b970-65a8307a4698\") " pod="kserve/s3-init-ttxsr"
Jul 29 10:38:14.059348 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:38:14.059326    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9css\" (UniqueName: \"kubernetes.io/projected/f0c13552-5b54-4754-b970-65a8307a4698-kube-api-access-l9css\") pod \"s3-init-ttxsr\" (UID: \"f0c13552-5b54-4754-b970-65a8307a4698\") " pod="kserve/s3-init-ttxsr"
Jul 29 10:38:14.241709 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:38:14.241619    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-ttxsr"
Jul 29 10:38:14.371123 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:38:14.371093    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-ttxsr"]
Jul 29 10:38:14.373994 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:38:14.373969    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0c13552_5b54_4754_b970_65a8307a4698.slice/crio-8614ac38a3a5c5dc45070883b40c1a5cd103d6891a7414c28d86cdb74039e851 WatchSource:0}: Error finding container 8614ac38a3a5c5dc45070883b40c1a5cd103d6891a7414c28d86cdb74039e851: Status 404 returned error can't find the container with id 8614ac38a3a5c5dc45070883b40c1a5cd103d6891a7414c28d86cdb74039e851
Jul 29 10:38:15.095097 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:38:15.095039    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-ttxsr" event={"ID":"f0c13552-5b54-4754-b970-65a8307a4698","Type":"ContainerStarted","Data":"8614ac38a3a5c5dc45070883b40c1a5cd103d6891a7414c28d86cdb74039e851"}
Jul 29 10:38:18.110744 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:38:18.110703    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-ttxsr" event={"ID":"f0c13552-5b54-4754-b970-65a8307a4698","Type":"ContainerStarted","Data":"61d7daa1c452572e95456ac3f91ecc65cac58e22f79bb748fb135c4bc9d7cc37"}
Jul 29 10:38:18.376761 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:38:18.376639    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:38:18.477833 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:38:18.477687    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:38:19.116604 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:38:19.116508    2569 generic.go:356] "Generic (PLEG): container finished" podID="f0c13552-5b54-4754-b970-65a8307a4698" containerID="61d7daa1c452572e95456ac3f91ecc65cac58e22f79bb748fb135c4bc9d7cc37" exitCode=0
Jul 29 10:38:19.116604 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:38:19.116592    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-ttxsr" event={"ID":"f0c13552-5b54-4754-b970-65a8307a4698","Type":"ContainerDied","Data":"61d7daa1c452572e95456ac3f91ecc65cac58e22f79bb748fb135c4bc9d7cc37"}
Jul 29 10:38:24.147003 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:38:24.146964    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-ttxsr" event={"ID":"f0c13552-5b54-4754-b970-65a8307a4698","Type":"ContainerStarted","Data":"66273d6f2eac4fd3a3f4304debaef6485da1067ee7c7f985ccd8bfb6b7a3cb52"}
Jul 29 10:38:24.165014 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:38:24.164965    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-ttxsr" podStartSLOduration=2.354447465 podStartE2EDuration="11.164948886s" totalImagesPullingTime="7.45477148s" totalInitContainerRuntime="1.355729941s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:38:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:38:24.163246696 +0000 UTC m=+580.888037371" watchObservedRunningTime="2026-07-29 10:38:24.164948886 +0000 UTC m=+580.889739559"
Jul 29 10:38:36.136034 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:38:36.135999    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:38:43.726466 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:38:43.726437    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-pvdgw_afd0d65c-bbfc-49f0-81b6-e855fd81da7f/console-operator/1.log"
Jul 29 10:38:43.729458 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:38:43.729434    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-pvdgw_afd0d65c-bbfc-49f0-81b6-e855fd81da7f/console-operator/1.log"
Jul 29 10:39:04.308680 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:04.308637    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:39:04.366921 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:04.366889    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:39:28.389379 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:28.389290    2569 generic.go:356] "Generic (PLEG): container finished" podID="f0c13552-5b54-4754-b970-65a8307a4698" containerID="66273d6f2eac4fd3a3f4304debaef6485da1067ee7c7f985ccd8bfb6b7a3cb52" exitCode=0
Jul 29 10:39:28.389379 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:28.389339    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-ttxsr" event={"ID":"f0c13552-5b54-4754-b970-65a8307a4698","Type":"ContainerDied","Data":"66273d6f2eac4fd3a3f4304debaef6485da1067ee7c7f985ccd8bfb6b7a3cb52"}
Jul 29 10:39:29.515339 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:29.515312    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-ttxsr"
Jul 29 10:39:29.585982 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:29.585944    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0c13552-5b54-4754-b970-65a8307a4698-model-cache\") pod \"f0c13552-5b54-4754-b970-65a8307a4698\" (UID: \"f0c13552-5b54-4754-b970-65a8307a4698\") "
Jul 29 10:39:29.586201 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:29.586031    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9css\" (UniqueName: \"kubernetes.io/projected/f0c13552-5b54-4754-b970-65a8307a4698-kube-api-access-l9css\") pod \"f0c13552-5b54-4754-b970-65a8307a4698\" (UID: \"f0c13552-5b54-4754-b970-65a8307a4698\") "
Jul 29 10:39:29.588162 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:29.588129    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0c13552-5b54-4754-b970-65a8307a4698-kube-api-access-l9css" (OuterVolumeSpecName: "kube-api-access-l9css") pod "f0c13552-5b54-4754-b970-65a8307a4698" (UID: "f0c13552-5b54-4754-b970-65a8307a4698"). InnerVolumeSpecName "kube-api-access-l9css". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:39:29.608486 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:29.608461    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c13552-5b54-4754-b970-65a8307a4698-model-cache" (OuterVolumeSpecName: "model-cache") pod "f0c13552-5b54-4754-b970-65a8307a4698" (UID: "f0c13552-5b54-4754-b970-65a8307a4698"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:39:29.688514 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:29.688444    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0c13552-5b54-4754-b970-65a8307a4698-model-cache\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:39:29.688514 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:29.688482    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9css\" (UniqueName: \"kubernetes.io/projected/f0c13552-5b54-4754-b970-65a8307a4698-kube-api-access-l9css\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:39:30.398214 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:30.398183    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-ttxsr"
Jul 29 10:39:30.398398 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:30.398186    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-ttxsr" event={"ID":"f0c13552-5b54-4754-b970-65a8307a4698","Type":"ContainerDied","Data":"8614ac38a3a5c5dc45070883b40c1a5cd103d6891a7414c28d86cdb74039e851"}
Jul 29 10:39:30.398398 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:30.398299    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8614ac38a3a5c5dc45070883b40c1a5cd103d6891a7414c28d86cdb74039e851"
Jul 29 10:39:30.805160 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:30.805119    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:39:32.136501 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:32.136457    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:39:42.550480 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:42.550443    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:39:42.606870 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:42.606831    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x"]
Jul 29 10:39:42.607161 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:42.607148    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0c13552-5b54-4754-b970-65a8307a4698" containerName="download-hf-model"
Jul 29 10:39:42.607161 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:42.607162    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0c13552-5b54-4754-b970-65a8307a4698" containerName="download-hf-model"
Jul 29 10:39:42.607260 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:42.607172    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0c13552-5b54-4754-b970-65a8307a4698" containerName="s3-init"
Jul 29 10:39:42.607260 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:42.607180    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0c13552-5b54-4754-b970-65a8307a4698" containerName="s3-init"
Jul 29 10:39:42.607260 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:42.607233    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0c13552-5b54-4754-b970-65a8307a4698" containerName="s3-init"
Jul 29 10:39:42.609729 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:42.609698    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x"
Jul 29 10:39:42.612691 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:42.612444    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-61cfd-predictor-serving-cert\""
Jul 29 10:39:42.612691 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:42.612467    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 10:39:42.612691 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:42.612480    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 10:39:42.612691 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:42.612487    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 10:39:42.612691 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:42.612484    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 10:39:42.612993 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:42.612697    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6nnxs\""
Jul 29 10:39:42.612993 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:42.612913    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-61cfd-kube-rbac-proxy-sar-config\""
Jul 29 10:39:42.623952 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:42.623928    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x"]
Jul 29 10:39:42.698870 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:42.698829    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:39:42.704953 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:42.704924    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ceacbb5-bfd0-4035-9684-1630ce0cdc7e-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x\" (UID: \"8ceacbb5-bfd0-4035-9684-1630ce0cdc7e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x"
Jul 29 10:39:42.705105 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:42.704958    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ceacbb5-bfd0-4035-9684-1630ce0cdc7e-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x\" (UID: \"8ceacbb5-bfd0-4035-9684-1630ce0cdc7e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x"
Jul 29 10:39:42.705105 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:42.704986    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ceacbb5-bfd0-4035-9684-1630ce0cdc7e-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x\" (UID: \"8ceacbb5-bfd0-4035-9684-1630ce0cdc7e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x"
Jul 29 10:39:42.705105 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:42.705092    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ceacbb5-bfd0-4035-9684-1630ce0cdc7e-proxy-tls\") pod \"isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x\" (UID: \"8ceacbb5-bfd0-4035-9684-1630ce0cdc7e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x"
Jul 29 10:39:42.705210 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:42.705146    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zskj\" (UniqueName: \"kubernetes.io/projected/8ceacbb5-bfd0-4035-9684-1630ce0cdc7e-kube-api-access-5zskj\") pod \"isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x\" (UID: \"8ceacbb5-bfd0-4035-9684-1630ce0cdc7e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x"
Jul 29 10:39:42.806366 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:42.806283    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ceacbb5-bfd0-4035-9684-1630ce0cdc7e-proxy-tls\") pod \"isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x\" (UID: \"8ceacbb5-bfd0-4035-9684-1630ce0cdc7e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x"
Jul 29 10:39:42.806366 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:42.806341    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zskj\" (UniqueName: \"kubernetes.io/projected/8ceacbb5-bfd0-4035-9684-1630ce0cdc7e-kube-api-access-5zskj\") pod \"isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x\" (UID: \"8ceacbb5-bfd0-4035-9684-1630ce0cdc7e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x"
Jul 29 10:39:42.806568 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:42.806370    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ceacbb5-bfd0-4035-9684-1630ce0cdc7e-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x\" (UID: \"8ceacbb5-bfd0-4035-9684-1630ce0cdc7e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x"
Jul 29 10:39:42.806568 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:42.806389    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ceacbb5-bfd0-4035-9684-1630ce0cdc7e-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x\" (UID: \"8ceacbb5-bfd0-4035-9684-1630ce0cdc7e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x"
Jul 29 10:39:42.806568 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:42.806427    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ceacbb5-bfd0-4035-9684-1630ce0cdc7e-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x\" (UID: \"8ceacbb5-bfd0-4035-9684-1630ce0cdc7e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x"
Jul 29 10:39:42.806917 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:42.806894    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ceacbb5-bfd0-4035-9684-1630ce0cdc7e-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x\" (UID: \"8ceacbb5-bfd0-4035-9684-1630ce0cdc7e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x"
Jul 29 10:39:42.807136 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:42.807117    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ceacbb5-bfd0-4035-9684-1630ce0cdc7e-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x\" (UID: \"8ceacbb5-bfd0-4035-9684-1630ce0cdc7e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x"
Jul 29 10:39:42.807217 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:42.807122    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ceacbb5-bfd0-4035-9684-1630ce0cdc7e-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x\" (UID: \"8ceacbb5-bfd0-4035-9684-1630ce0cdc7e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x"
Jul 29 10:39:42.808783 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:42.808763    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ceacbb5-bfd0-4035-9684-1630ce0cdc7e-proxy-tls\") pod \"isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x\" (UID: \"8ceacbb5-bfd0-4035-9684-1630ce0cdc7e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x"
Jul 29 10:39:42.814928 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:42.814910    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zskj\" (UniqueName: \"kubernetes.io/projected/8ceacbb5-bfd0-4035-9684-1630ce0cdc7e-kube-api-access-5zskj\") pod \"isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x\" (UID: \"8ceacbb5-bfd0-4035-9684-1630ce0cdc7e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x"
Jul 29 10:39:42.921107 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:42.921038    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x"
Jul 29 10:39:43.054244 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:43.054113    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x"]
Jul 29 10:39:43.056530 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:39:43.056474    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ceacbb5_bfd0_4035_9684_1630ce0cdc7e.slice/crio-831cef77721073b3e6210606e3280afcbe189e0c9a54a02b516a75c923be34a4 WatchSource:0}: Error finding container 831cef77721073b3e6210606e3280afcbe189e0c9a54a02b516a75c923be34a4: Status 404 returned error can't find the container with id 831cef77721073b3e6210606e3280afcbe189e0c9a54a02b516a75c923be34a4
Jul 29 10:39:43.446779 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:43.446737    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" event={"ID":"8ceacbb5-bfd0-4035-9684-1630ce0cdc7e","Type":"ContainerStarted","Data":"eebe5534fbcba241d6e4937361e03f45ef5584a7e15b319418cf725da69ad528"}
Jul 29 10:39:43.446936 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:43.446820    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" event={"ID":"8ceacbb5-bfd0-4035-9684-1630ce0cdc7e","Type":"ContainerStarted","Data":"831cef77721073b3e6210606e3280afcbe189e0c9a54a02b516a75c923be34a4"}
Jul 29 10:39:47.463975 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:47.463930    2569 generic.go:356] "Generic (PLEG): container finished" podID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerID="eebe5534fbcba241d6e4937361e03f45ef5584a7e15b319418cf725da69ad528" exitCode=0
Jul 29 10:39:47.463975 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:39:47.463967    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" event={"ID":"8ceacbb5-bfd0-4035-9684-1630ce0cdc7e","Type":"ContainerDied","Data":"eebe5534fbcba241d6e4937361e03f45ef5584a7e15b319418cf725da69ad528"}
Jul 29 10:40:01.540637 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:40:01.540594    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" event={"ID":"8ceacbb5-bfd0-4035-9684-1630ce0cdc7e","Type":"ContainerStarted","Data":"7ff99fe8a89bd822f508dc17b87a95a1bd08e01545201e271c189f0307c82bbc"}
Jul 29 10:40:03.550943 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:40:03.550906    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" event={"ID":"8ceacbb5-bfd0-4035-9684-1630ce0cdc7e","Type":"ContainerStarted","Data":"d61501e85a6ea12767b54240547be05c9da588a15a84e8b9a22248f8cb746edf"}
Jul 29 10:40:06.566233 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:40:06.566188    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" event={"ID":"8ceacbb5-bfd0-4035-9684-1630ce0cdc7e","Type":"ContainerStarted","Data":"f1b4d8a26bc9319fcd5c3297fe7bc4cb9e1b9678784b38f24e0ae4204ebf240b"}
Jul 29 10:40:06.566649 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:40:06.566315    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x"
Jul 29 10:40:06.590831 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:40:06.590778    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" podStartSLOduration=2.993208246 podStartE2EDuration="24.590761081s" totalImagesPullingTime="18.008489685s" totalInitContainerRuntime="3.58906315s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:39:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:40:06.588448098 +0000 UTC m=+683.313238770" watchObservedRunningTime="2026-07-29 10:40:06.590761081 +0000 UTC m=+683.315551753"
Jul 29 10:40:07.570945 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:40:07.570902    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x"
Jul 29 10:40:07.571328 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:40:07.570983    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x"
Jul 29 10:40:07.572259 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:40:07.572225    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 29 10:40:07.572931 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:40:07.572907    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:40:07.575759 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:40:07.575741    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x"
Jul 29 10:40:08.576795 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:40:08.576757    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 29 10:40:08.577276 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:40:08.577114    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:40:09.580884 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:40:09.580833    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 29 10:40:09.581328 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:40:09.581173    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:40:19.581543 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:40:19.581469    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 29 10:40:19.581999 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:40:19.581983    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:40:29.581471 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:40:29.581423    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 29 10:40:29.581979 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:40:29.581955    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:40:39.581497 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:40:39.581258    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 29 10:40:39.582491 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:40:39.582451    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:40:44.394622 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:40:44.394572    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:40:49.581144 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:40:49.581065    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 29 10:40:49.581557 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:40:49.581447    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:40:59.581394 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:40:59.581344    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 29 10:40:59.581813 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:40:59.581729    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:41:05.364926 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:05.364887    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:41:09.580843 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:09.580789    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 29 10:41:09.581292 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:09.581241    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:41:19.582175 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:19.582134    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x"
Jul 29 10:41:19.582635 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:19.582255    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x"
Jul 29 10:41:27.739532 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:27.739499    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:41:27.895826 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:27.895793    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x"]
Jul 29 10:41:27.898888 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:27.898867    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x"
Jul 29 10:41:27.903492 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:27.903464    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-877ef-kube-rbac-proxy-sar-config\""
Jul 29 10:41:27.903714 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:27.903694    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-877ef-predictor-serving-cert\""
Jul 29 10:41:27.925528 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:27.925502    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x"]
Jul 29 10:41:27.957464 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:27.957436    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:41:27.974908 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:27.974885    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x"]
Jul 29 10:41:27.975177 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:27.975148    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="agent" containerID="cri-o://f1b4d8a26bc9319fcd5c3297fe7bc4cb9e1b9678784b38f24e0ae4204ebf240b" gracePeriod=30
Jul 29 10:41:27.975245 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:27.975161    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="kube-rbac-proxy" containerID="cri-o://d61501e85a6ea12767b54240547be05c9da588a15a84e8b9a22248f8cb746edf" gracePeriod=30
Jul 29 10:41:27.975317 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:27.975136    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="kserve-container" containerID="cri-o://7ff99fe8a89bd822f508dc17b87a95a1bd08e01545201e271c189f0307c82bbc" gracePeriod=30
Jul 29 10:41:28.002086 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.002022    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5a024eb-9344-4ae0-8a17-87ce32cfa61f-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x\" (UID: \"c5a024eb-9344-4ae0-8a17-87ce32cfa61f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x"
Jul 29 10:41:28.002177 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.002098    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5a024eb-9344-4ae0-8a17-87ce32cfa61f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x\" (UID: \"c5a024eb-9344-4ae0-8a17-87ce32cfa61f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x"
Jul 29 10:41:28.002177 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.002136    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg445\" (UniqueName: \"kubernetes.io/projected/c5a024eb-9344-4ae0-8a17-87ce32cfa61f-kube-api-access-dg445\") pod \"isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x\" (UID: \"c5a024eb-9344-4ae0-8a17-87ce32cfa61f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x"
Jul 29 10:41:28.002177 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.002157    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5a024eb-9344-4ae0-8a17-87ce32cfa61f-cabundle-cert\") pod \"isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x\" (UID: \"c5a024eb-9344-4ae0-8a17-87ce32cfa61f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x"
Jul 29 10:41:28.002282 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.002203    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5a024eb-9344-4ae0-8a17-87ce32cfa61f-proxy-tls\") pod \"isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x\" (UID: \"c5a024eb-9344-4ae0-8a17-87ce32cfa61f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x"
Jul 29 10:41:28.050771 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.050744    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9"]
Jul 29 10:41:28.054562 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.054545    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9"
Jul 29 10:41:28.057246 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.057227    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-877ef-predictor-serving-cert\""
Jul 29 10:41:28.057712 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.057698    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-877ef-kube-rbac-proxy-sar-config\""
Jul 29 10:41:28.078163 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.078141    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9"]
Jul 29 10:41:28.089812 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.089790    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:41:28.102928 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.102904    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5a024eb-9344-4ae0-8a17-87ce32cfa61f-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x\" (UID: \"c5a024eb-9344-4ae0-8a17-87ce32cfa61f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x"
Jul 29 10:41:28.103025 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.102938    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ghnx\" (UniqueName: \"kubernetes.io/projected/269bc236-82b6-4cb1-b594-13b1711573b0-kube-api-access-4ghnx\") pod \"isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9\" (UID: \"269bc236-82b6-4cb1-b594-13b1711573b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9"
Jul 29 10:41:28.103025 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.102969    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/269bc236-82b6-4cb1-b594-13b1711573b0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9\" (UID: \"269bc236-82b6-4cb1-b594-13b1711573b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9"
Jul 29 10:41:28.103025 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.102991    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/269bc236-82b6-4cb1-b594-13b1711573b0-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9\" (UID: \"269bc236-82b6-4cb1-b594-13b1711573b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9"
Jul 29 10:41:28.103259 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.103035    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5a024eb-9344-4ae0-8a17-87ce32cfa61f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x\" (UID: \"c5a024eb-9344-4ae0-8a17-87ce32cfa61f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x"
Jul 29 10:41:28.103259 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.103067    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg445\" (UniqueName: \"kubernetes.io/projected/c5a024eb-9344-4ae0-8a17-87ce32cfa61f-kube-api-access-dg445\") pod \"isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x\" (UID: \"c5a024eb-9344-4ae0-8a17-87ce32cfa61f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x"
Jul 29 10:41:28.103259 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.103112    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5a024eb-9344-4ae0-8a17-87ce32cfa61f-cabundle-cert\") pod \"isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x\" (UID: \"c5a024eb-9344-4ae0-8a17-87ce32cfa61f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x"
Jul 29 10:41:28.103259 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.103138    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5a024eb-9344-4ae0-8a17-87ce32cfa61f-proxy-tls\") pod \"isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x\" (UID: \"c5a024eb-9344-4ae0-8a17-87ce32cfa61f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x"
Jul 29 10:41:28.103458 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.103316    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/269bc236-82b6-4cb1-b594-13b1711573b0-cabundle-cert\") pod \"isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9\" (UID: \"269bc236-82b6-4cb1-b594-13b1711573b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9"
Jul 29 10:41:28.103458 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.103373    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/269bc236-82b6-4cb1-b594-13b1711573b0-proxy-tls\") pod \"isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9\" (UID: \"269bc236-82b6-4cb1-b594-13b1711573b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9"
Jul 29 10:41:28.103458 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.103392    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5a024eb-9344-4ae0-8a17-87ce32cfa61f-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x\" (UID: \"c5a024eb-9344-4ae0-8a17-87ce32cfa61f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x"
Jul 29 10:41:28.103700 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.103682    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5a024eb-9344-4ae0-8a17-87ce32cfa61f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x\" (UID: \"c5a024eb-9344-4ae0-8a17-87ce32cfa61f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x"
Jul 29 10:41:28.103767 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.103748    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5a024eb-9344-4ae0-8a17-87ce32cfa61f-cabundle-cert\") pod \"isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x\" (UID: \"c5a024eb-9344-4ae0-8a17-87ce32cfa61f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x"
Jul 29 10:41:28.105465 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.105446    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5a024eb-9344-4ae0-8a17-87ce32cfa61f-proxy-tls\") pod \"isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x\" (UID: \"c5a024eb-9344-4ae0-8a17-87ce32cfa61f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x"
Jul 29 10:41:28.112728 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.112709    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg445\" (UniqueName: \"kubernetes.io/projected/c5a024eb-9344-4ae0-8a17-87ce32cfa61f-kube-api-access-dg445\") pod \"isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x\" (UID: \"c5a024eb-9344-4ae0-8a17-87ce32cfa61f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x"
Jul 29 10:41:28.204280 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.204244    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/269bc236-82b6-4cb1-b594-13b1711573b0-cabundle-cert\") pod \"isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9\" (UID: \"269bc236-82b6-4cb1-b594-13b1711573b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9"
Jul 29 10:41:28.204280 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.204286    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/269bc236-82b6-4cb1-b594-13b1711573b0-proxy-tls\") pod \"isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9\" (UID: \"269bc236-82b6-4cb1-b594-13b1711573b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9"
Jul 29 10:41:28.204465 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.204313    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ghnx\" (UniqueName: \"kubernetes.io/projected/269bc236-82b6-4cb1-b594-13b1711573b0-kube-api-access-4ghnx\") pod \"isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9\" (UID: \"269bc236-82b6-4cb1-b594-13b1711573b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9"
Jul 29 10:41:28.204465 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.204343    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/269bc236-82b6-4cb1-b594-13b1711573b0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9\" (UID: \"269bc236-82b6-4cb1-b594-13b1711573b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9"
Jul 29 10:41:28.204465 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.204359    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/269bc236-82b6-4cb1-b594-13b1711573b0-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9\" (UID: \"269bc236-82b6-4cb1-b594-13b1711573b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9"
Jul 29 10:41:28.204903 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.204859    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/269bc236-82b6-4cb1-b594-13b1711573b0-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9\" (UID: \"269bc236-82b6-4cb1-b594-13b1711573b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9"
Jul 29 10:41:28.205038 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.205018    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/269bc236-82b6-4cb1-b594-13b1711573b0-cabundle-cert\") pod \"isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9\" (UID: \"269bc236-82b6-4cb1-b594-13b1711573b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9"
Jul 29 10:41:28.205113 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.205049    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/269bc236-82b6-4cb1-b594-13b1711573b0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9\" (UID: \"269bc236-82b6-4cb1-b594-13b1711573b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9"
Jul 29 10:41:28.206681 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.206661    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/269bc236-82b6-4cb1-b594-13b1711573b0-proxy-tls\") pod \"isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9\" (UID: \"269bc236-82b6-4cb1-b594-13b1711573b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9"
Jul 29 10:41:28.209643 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.209620    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x"
Jul 29 10:41:28.213986 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.213960    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ghnx\" (UniqueName: \"kubernetes.io/projected/269bc236-82b6-4cb1-b594-13b1711573b0-kube-api-access-4ghnx\") pod \"isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9\" (UID: \"269bc236-82b6-4cb1-b594-13b1711573b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9"
Jul 29 10:41:28.341767 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.341742    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x"]
Jul 29 10:41:28.344214 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:41:28.344187    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a024eb_9344_4ae0_8a17_87ce32cfa61f.slice/crio-8027951f324f6ff0a837efe433e3723f60e8dfb1a3baaa79ea82bd413ef22e6b WatchSource:0}: Error finding container 8027951f324f6ff0a837efe433e3723f60e8dfb1a3baaa79ea82bd413ef22e6b: Status 404 returned error can't find the container with id 8027951f324f6ff0a837efe433e3723f60e8dfb1a3baaa79ea82bd413ef22e6b
Jul 29 10:41:28.364218 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.364191    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9"
Jul 29 10:41:28.500301 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.500260    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9"]
Jul 29 10:41:28.508059 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:41:28.507992    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod269bc236_82b6_4cb1_b594_13b1711573b0.slice/crio-1b69c4e4460bce385add53d3729addafb495917fb654d02fbe8c77c8642c5097 WatchSource:0}: Error finding container 1b69c4e4460bce385add53d3729addafb495917fb654d02fbe8c77c8642c5097: Status 404 returned error can't find the container with id 1b69c4e4460bce385add53d3729addafb495917fb654d02fbe8c77c8642c5097
Jul 29 10:41:28.887287 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.887195    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9" event={"ID":"269bc236-82b6-4cb1-b594-13b1711573b0","Type":"ContainerStarted","Data":"9e33b9f1e06aa58e8f45bcd58007182e5e76d801d9f67dee934683de854da133"}
Jul 29 10:41:28.887287 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.887269    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9" event={"ID":"269bc236-82b6-4cb1-b594-13b1711573b0","Type":"ContainerStarted","Data":"1b69c4e4460bce385add53d3729addafb495917fb654d02fbe8c77c8642c5097"}
Jul 29 10:41:28.889981 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.889948    2569 generic.go:356] "Generic (PLEG): container finished" podID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerID="d61501e85a6ea12767b54240547be05c9da588a15a84e8b9a22248f8cb746edf" exitCode=2
Jul 29 10:41:28.890123 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.890028    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" event={"ID":"8ceacbb5-bfd0-4035-9684-1630ce0cdc7e","Type":"ContainerDied","Data":"d61501e85a6ea12767b54240547be05c9da588a15a84e8b9a22248f8cb746edf"}
Jul 29 10:41:28.891725 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.891689    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x" event={"ID":"c5a024eb-9344-4ae0-8a17-87ce32cfa61f","Type":"ContainerStarted","Data":"4da79cfe2ef4276b991fcc49a7935041578de7eef0dc4c86df8dc4b2e2ccdd05"}
Jul 29 10:41:28.891837 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:28.891761    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x" event={"ID":"c5a024eb-9344-4ae0-8a17-87ce32cfa61f","Type":"ContainerStarted","Data":"8027951f324f6ff0a837efe433e3723f60e8dfb1a3baaa79ea82bd413ef22e6b"}
Jul 29 10:41:29.581100 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:29.581038    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 29 10:41:29.581356 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:29.581330    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:41:32.571434 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:32.571385    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Jul 29 10:41:32.909274 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:32.909244    2569 generic.go:356] "Generic (PLEG): container finished" podID="269bc236-82b6-4cb1-b594-13b1711573b0" containerID="9e33b9f1e06aa58e8f45bcd58007182e5e76d801d9f67dee934683de854da133" exitCode=0
Jul 29 10:41:32.909274 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:32.909262    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9" event={"ID":"269bc236-82b6-4cb1-b594-13b1711573b0","Type":"ContainerDied","Data":"9e33b9f1e06aa58e8f45bcd58007182e5e76d801d9f67dee934683de854da133"}
Jul 29 10:41:32.911943 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:32.911925    2569 generic.go:356] "Generic (PLEG): container finished" podID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerID="7ff99fe8a89bd822f508dc17b87a95a1bd08e01545201e271c189f0307c82bbc" exitCode=0
Jul 29 10:41:32.912047 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:32.912001    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" event={"ID":"8ceacbb5-bfd0-4035-9684-1630ce0cdc7e","Type":"ContainerDied","Data":"7ff99fe8a89bd822f508dc17b87a95a1bd08e01545201e271c189f0307c82bbc"}
Jul 29 10:41:32.913475 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:32.913455    2569 generic.go:356] "Generic (PLEG): container finished" podID="c5a024eb-9344-4ae0-8a17-87ce32cfa61f" containerID="4da79cfe2ef4276b991fcc49a7935041578de7eef0dc4c86df8dc4b2e2ccdd05" exitCode=0
Jul 29 10:41:32.913565 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:32.913486    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x" event={"ID":"c5a024eb-9344-4ae0-8a17-87ce32cfa61f","Type":"ContainerDied","Data":"4da79cfe2ef4276b991fcc49a7935041578de7eef0dc4c86df8dc4b2e2ccdd05"}
Jul 29 10:41:33.921146 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:33.921105    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x" event={"ID":"c5a024eb-9344-4ae0-8a17-87ce32cfa61f","Type":"ContainerStarted","Data":"6b114532d6755f5c6dc7be56a23ee31ed326e378579193cdd8c79445d5c44fbe"}
Jul 29 10:41:33.921555 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:33.921191    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x" event={"ID":"c5a024eb-9344-4ae0-8a17-87ce32cfa61f","Type":"ContainerStarted","Data":"28261c143cd7d2b7047c82c089f5c9a28cafb1ab04cb9712dfaba99eb4ed7714"}
Jul 29 10:41:33.921623 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:33.921574    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x"
Jul 29 10:41:33.940777 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:33.940721    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x" podStartSLOduration=3.019591262 podStartE2EDuration="6.94069999s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.921108728s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:41:33.9404614 +0000 UTC m=+770.665252074" watchObservedRunningTime="2026-07-29 10:41:33.94069999 +0000 UTC m=+770.665490667"
Jul 29 10:41:34.928032 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:34.927943    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x"
Jul 29 10:41:34.929209 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:34.929178    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x" podUID="c5a024eb-9344-4ae0-8a17-87ce32cfa61f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 29 10:41:35.933005 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:35.932924    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x" podUID="c5a024eb-9344-4ae0-8a17-87ce32cfa61f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 29 10:41:37.572249 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:37.572052    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Jul 29 10:41:39.581020 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:39.580960    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 29 10:41:39.581471 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:39.581306    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:41:40.938700 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:40.938670    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x"
Jul 29 10:41:40.939464 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:40.939429    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x" podUID="c5a024eb-9344-4ae0-8a17-87ce32cfa61f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 29 10:41:42.571622 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:42.571550    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Jul 29 10:41:42.572098 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:42.571769    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x"
Jul 29 10:41:43.341969 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:43.341932    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:41:43.451574 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:43.451537    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:41:47.571275 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:47.571233    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Jul 29 10:41:49.581250 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:49.581201    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 29 10:41:49.581678 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:49.581380    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x"
Jul 29 10:41:49.581678 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:49.581462    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:41:49.581678 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:49.581552    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x"
Jul 29 10:41:50.939309 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:50.939265    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x" podUID="c5a024eb-9344-4ae0-8a17-87ce32cfa61f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 29 10:41:51.001619 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:51.001588    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9" event={"ID":"269bc236-82b6-4cb1-b594-13b1711573b0","Type":"ContainerStarted","Data":"f77703714867b74f755655b3354f8d886e9f8dda9efd39c1919a66c0fd3a47eb"}
Jul 29 10:41:51.001790 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:51.001659    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9" event={"ID":"269bc236-82b6-4cb1-b594-13b1711573b0","Type":"ContainerStarted","Data":"7e2a6742565d7f6d4fb56bd17141e3f942b7b8a7e6678af97a79c354221b19e5"}
Jul 29 10:41:51.001894 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:51.001873    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9"
Jul 29 10:41:51.020730 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:51.020671    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9" podStartSLOduration=2.171933968 podStartE2EDuration="23.020657202s" totalImagesPullingTime="17.064723103s" totalInitContainerRuntime="3.784000131s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:41:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:41:51.019053791 +0000 UTC m=+787.743844464" watchObservedRunningTime="2026-07-29 10:41:51.020657202 +0000 UTC m=+787.745447875"
Jul 29 10:41:52.005693 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:52.005665    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9"
Jul 29 10:41:52.007139 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:52.007110    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9" podUID="269bc236-82b6-4cb1-b594-13b1711573b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 29 10:41:52.571403 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:52.571366    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Jul 29 10:41:53.009488 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:53.009449    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9" podUID="269bc236-82b6-4cb1-b594-13b1711573b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 29 10:41:57.571497 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:57.571449    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Jul 29 10:41:58.013968 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:58.013944    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9"
Jul 29 10:41:58.014521 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:58.014491    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9" podUID="269bc236-82b6-4cb1-b594-13b1711573b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 29 10:41:58.642035 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:58.642006    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x"
Jul 29 10:41:58.784845 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:58.784811    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ceacbb5-bfd0-4035-9684-1630ce0cdc7e-kserve-provision-location\") pod \"8ceacbb5-bfd0-4035-9684-1630ce0cdc7e\" (UID: \"8ceacbb5-bfd0-4035-9684-1630ce0cdc7e\") "
Jul 29 10:41:58.785046 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:58.784895    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ceacbb5-bfd0-4035-9684-1630ce0cdc7e-proxy-tls\") pod \"8ceacbb5-bfd0-4035-9684-1630ce0cdc7e\" (UID: \"8ceacbb5-bfd0-4035-9684-1630ce0cdc7e\") "
Jul 29 10:41:58.785046 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:58.784942    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ceacbb5-bfd0-4035-9684-1630ce0cdc7e-cabundle-cert\") pod \"8ceacbb5-bfd0-4035-9684-1630ce0cdc7e\" (UID: \"8ceacbb5-bfd0-4035-9684-1630ce0cdc7e\") "
Jul 29 10:41:58.785046 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:58.784970    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ceacbb5-bfd0-4035-9684-1630ce0cdc7e-kube-rbac-proxy-sar-config\") pod \"8ceacbb5-bfd0-4035-9684-1630ce0cdc7e\" (UID: \"8ceacbb5-bfd0-4035-9684-1630ce0cdc7e\") "
Jul 29 10:41:58.785046 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:58.785034    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zskj\" (UniqueName: \"kubernetes.io/projected/8ceacbb5-bfd0-4035-9684-1630ce0cdc7e-kube-api-access-5zskj\") pod \"8ceacbb5-bfd0-4035-9684-1630ce0cdc7e\" (UID: \"8ceacbb5-bfd0-4035-9684-1630ce0cdc7e\") "
Jul 29 10:41:58.785320 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:58.785218    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ceacbb5-bfd0-4035-9684-1630ce0cdc7e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" (UID: "8ceacbb5-bfd0-4035-9684-1630ce0cdc7e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:41:58.785376 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:58.785303    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ceacbb5-bfd0-4035-9684-1630ce0cdc7e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" (UID: "8ceacbb5-bfd0-4035-9684-1630ce0cdc7e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:41:58.785412 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:58.785381    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ceacbb5-bfd0-4035-9684-1630ce0cdc7e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" (UID: "8ceacbb5-bfd0-4035-9684-1630ce0cdc7e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:41:58.787176 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:58.787148    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ceacbb5-bfd0-4035-9684-1630ce0cdc7e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" (UID: "8ceacbb5-bfd0-4035-9684-1630ce0cdc7e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:41:58.787290 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:58.787253    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ceacbb5-bfd0-4035-9684-1630ce0cdc7e-kube-api-access-5zskj" (OuterVolumeSpecName: "kube-api-access-5zskj") pod "8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" (UID: "8ceacbb5-bfd0-4035-9684-1630ce0cdc7e"). InnerVolumeSpecName "kube-api-access-5zskj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:41:58.885865 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:58.885788    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ceacbb5-bfd0-4035-9684-1630ce0cdc7e-proxy-tls\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:41:58.885865 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:58.885816    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ceacbb5-bfd0-4035-9684-1630ce0cdc7e-cabundle-cert\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:41:58.885865 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:58.885827    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ceacbb5-bfd0-4035-9684-1630ce0cdc7e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:41:58.885865 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:58.885835    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zskj\" (UniqueName: \"kubernetes.io/projected/8ceacbb5-bfd0-4035-9684-1630ce0cdc7e-kube-api-access-5zskj\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:41:58.885865 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:58.885844    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ceacbb5-bfd0-4035-9684-1630ce0cdc7e-kserve-provision-location\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:41:59.044111 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:59.044064    2569 generic.go:356] "Generic (PLEG): container finished" podID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerID="f1b4d8a26bc9319fcd5c3297fe7bc4cb9e1b9678784b38f24e0ae4204ebf240b" exitCode=0
Jul 29 10:41:59.044111 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:59.044094    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" event={"ID":"8ceacbb5-bfd0-4035-9684-1630ce0cdc7e","Type":"ContainerDied","Data":"f1b4d8a26bc9319fcd5c3297fe7bc4cb9e1b9678784b38f24e0ae4204ebf240b"}
Jul 29 10:41:59.044345 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:59.044145    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x"
Jul 29 10:41:59.044345 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:59.044181    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x" event={"ID":"8ceacbb5-bfd0-4035-9684-1630ce0cdc7e","Type":"ContainerDied","Data":"831cef77721073b3e6210606e3280afcbe189e0c9a54a02b516a75c923be34a4"}
Jul 29 10:41:59.044345 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:59.044213    2569 scope.go:117] "RemoveContainer" containerID="f1b4d8a26bc9319fcd5c3297fe7bc4cb9e1b9678784b38f24e0ae4204ebf240b"
Jul 29 10:41:59.053524 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:59.053506    2569 scope.go:117] "RemoveContainer" containerID="d61501e85a6ea12767b54240547be05c9da588a15a84e8b9a22248f8cb746edf"
Jul 29 10:41:59.060747 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:59.060728    2569 scope.go:117] "RemoveContainer" containerID="7ff99fe8a89bd822f508dc17b87a95a1bd08e01545201e271c189f0307c82bbc"
Jul 29 10:41:59.066622 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:59.066594    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x"]
Jul 29 10:41:59.068030 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:59.068014    2569 scope.go:117] "RemoveContainer" containerID="eebe5534fbcba241d6e4937361e03f45ef5584a7e15b319418cf725da69ad528"
Jul 29 10:41:59.071858 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:59.071837    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61cfd-predictor-556558959-jdg9x"]
Jul 29 10:41:59.075411 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:59.075396    2569 scope.go:117] "RemoveContainer" containerID="f1b4d8a26bc9319fcd5c3297fe7bc4cb9e1b9678784b38f24e0ae4204ebf240b"
Jul 29 10:41:59.075612 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:41:59.075595    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1b4d8a26bc9319fcd5c3297fe7bc4cb9e1b9678784b38f24e0ae4204ebf240b\": container with ID starting with f1b4d8a26bc9319fcd5c3297fe7bc4cb9e1b9678784b38f24e0ae4204ebf240b not found: ID does not exist" containerID="f1b4d8a26bc9319fcd5c3297fe7bc4cb9e1b9678784b38f24e0ae4204ebf240b"
Jul 29 10:41:59.075668 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:59.075620    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1b4d8a26bc9319fcd5c3297fe7bc4cb9e1b9678784b38f24e0ae4204ebf240b"} err="failed to get container status \"f1b4d8a26bc9319fcd5c3297fe7bc4cb9e1b9678784b38f24e0ae4204ebf240b\": rpc error: code = NotFound desc = could not find container \"f1b4d8a26bc9319fcd5c3297fe7bc4cb9e1b9678784b38f24e0ae4204ebf240b\": container with ID starting with f1b4d8a26bc9319fcd5c3297fe7bc4cb9e1b9678784b38f24e0ae4204ebf240b not found: ID does not exist"
Jul 29 10:41:59.075668 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:59.075636    2569 scope.go:117] "RemoveContainer" containerID="d61501e85a6ea12767b54240547be05c9da588a15a84e8b9a22248f8cb746edf"
Jul 29 10:41:59.075833 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:41:59.075819    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d61501e85a6ea12767b54240547be05c9da588a15a84e8b9a22248f8cb746edf\": container with ID starting with d61501e85a6ea12767b54240547be05c9da588a15a84e8b9a22248f8cb746edf not found: ID does not exist" containerID="d61501e85a6ea12767b54240547be05c9da588a15a84e8b9a22248f8cb746edf"
Jul 29 10:41:59.075878 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:59.075836    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d61501e85a6ea12767b54240547be05c9da588a15a84e8b9a22248f8cb746edf"} err="failed to get container status \"d61501e85a6ea12767b54240547be05c9da588a15a84e8b9a22248f8cb746edf\": rpc error: code = NotFound desc = could not find container \"d61501e85a6ea12767b54240547be05c9da588a15a84e8b9a22248f8cb746edf\": container with ID starting with d61501e85a6ea12767b54240547be05c9da588a15a84e8b9a22248f8cb746edf not found: ID does not exist"
Jul 29 10:41:59.075878 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:59.075847    2569 scope.go:117] "RemoveContainer" containerID="7ff99fe8a89bd822f508dc17b87a95a1bd08e01545201e271c189f0307c82bbc"
Jul 29 10:41:59.076061 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:41:59.076044    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ff99fe8a89bd822f508dc17b87a95a1bd08e01545201e271c189f0307c82bbc\": container with ID starting with 7ff99fe8a89bd822f508dc17b87a95a1bd08e01545201e271c189f0307c82bbc not found: ID does not exist" containerID="7ff99fe8a89bd822f508dc17b87a95a1bd08e01545201e271c189f0307c82bbc"
Jul 29 10:41:59.076125 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:59.076085    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ff99fe8a89bd822f508dc17b87a95a1bd08e01545201e271c189f0307c82bbc"} err="failed to get container status \"7ff99fe8a89bd822f508dc17b87a95a1bd08e01545201e271c189f0307c82bbc\": rpc error: code = NotFound desc = could not find container \"7ff99fe8a89bd822f508dc17b87a95a1bd08e01545201e271c189f0307c82bbc\": container with ID starting with 7ff99fe8a89bd822f508dc17b87a95a1bd08e01545201e271c189f0307c82bbc not found: ID does not exist"
Jul 29 10:41:59.076125 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:59.076102    2569 scope.go:117] "RemoveContainer" containerID="eebe5534fbcba241d6e4937361e03f45ef5584a7e15b319418cf725da69ad528"
Jul 29 10:41:59.076329 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:41:59.076315    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eebe5534fbcba241d6e4937361e03f45ef5584a7e15b319418cf725da69ad528\": container with ID starting with eebe5534fbcba241d6e4937361e03f45ef5584a7e15b319418cf725da69ad528 not found: ID does not exist" containerID="eebe5534fbcba241d6e4937361e03f45ef5584a7e15b319418cf725da69ad528"
Jul 29 10:41:59.076373 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:59.076331    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eebe5534fbcba241d6e4937361e03f45ef5584a7e15b319418cf725da69ad528"} err="failed to get container status \"eebe5534fbcba241d6e4937361e03f45ef5584a7e15b319418cf725da69ad528\": rpc error: code = NotFound desc = could not find container \"eebe5534fbcba241d6e4937361e03f45ef5584a7e15b319418cf725da69ad528\": container with ID starting with eebe5534fbcba241d6e4937361e03f45ef5584a7e15b319418cf725da69ad528 not found: ID does not exist"
Jul 29 10:41:59.838646 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:41:59.838612    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" path="/var/lib/kubelet/pods/8ceacbb5-bfd0-4035-9684-1630ce0cdc7e/volumes"
Jul 29 10:42:00.940218 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:42:00.940176    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x" podUID="c5a024eb-9344-4ae0-8a17-87ce32cfa61f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 29 10:42:08.015240 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:42:08.015157    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9" podUID="269bc236-82b6-4cb1-b594-13b1711573b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 29 10:42:10.939840 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:42:10.939796    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x" podUID="c5a024eb-9344-4ae0-8a17-87ce32cfa61f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 29 10:42:18.015408 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:42:18.015366    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9" podUID="269bc236-82b6-4cb1-b594-13b1711573b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 29 10:42:20.939650 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:42:20.939604    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x" podUID="c5a024eb-9344-4ae0-8a17-87ce32cfa61f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 29 10:42:28.014930 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:42:28.014892    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9" podUID="269bc236-82b6-4cb1-b594-13b1711573b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 29 10:42:30.939573 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:42:30.939534    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x" podUID="c5a024eb-9344-4ae0-8a17-87ce32cfa61f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 29 10:42:38.014541 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:42:38.014501    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9" podUID="269bc236-82b6-4cb1-b594-13b1711573b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 29 10:42:40.939416 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:42:40.939372    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x" podUID="c5a024eb-9344-4ae0-8a17-87ce32cfa61f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 29 10:42:48.015387 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:42:48.015347    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9" podUID="269bc236-82b6-4cb1-b594-13b1711573b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 29 10:42:50.939968 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:42:50.939939    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x"
Jul 29 10:42:58.015130 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:42:58.015096    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9"
Jul 29 10:43:07.818687 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:07.818649    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:43:08.081371 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:08.081281    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:43:17.901627 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:17.901591    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:43:18.026309 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.026268    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:43:18.050217 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.050192    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x"]
Jul 29 10:43:18.050459 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.050433    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x" podUID="c5a024eb-9344-4ae0-8a17-87ce32cfa61f" containerName="kserve-container" containerID="cri-o://28261c143cd7d2b7047c82c089f5c9a28cafb1ab04cb9712dfaba99eb4ed7714" gracePeriod=30
Jul 29 10:43:18.050525 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.050448    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x" podUID="c5a024eb-9344-4ae0-8a17-87ce32cfa61f" containerName="kube-rbac-proxy" containerID="cri-o://6b114532d6755f5c6dc7be56a23ee31ed326e378579193cdd8c79445d5c44fbe" gracePeriod=30
Jul 29 10:43:18.080223 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.080196    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5"]
Jul 29 10:43:18.080522 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.080508    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="storage-initializer"
Jul 29 10:43:18.080522 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.080520    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="storage-initializer"
Jul 29 10:43:18.080658 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.080529    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="kserve-container"
Jul 29 10:43:18.080658 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.080535    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="kserve-container"
Jul 29 10:43:18.080658 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.080544    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="kube-rbac-proxy"
Jul 29 10:43:18.080658 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.080551    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="kube-rbac-proxy"
Jul 29 10:43:18.080658 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.080561    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="agent"
Jul 29 10:43:18.080658 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.080566    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="agent"
Jul 29 10:43:18.080658 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.080610    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="kube-rbac-proxy"
Jul 29 10:43:18.080658 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.080621    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="kserve-container"
Jul 29 10:43:18.080658 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.080627    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ceacbb5-bfd0-4035-9684-1630ce0cdc7e" containerName="agent"
Jul 29 10:43:18.082918 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.082894    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5"
Jul 29 10:43:18.085584 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.085561    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-2d850-predictor-serving-cert\""
Jul 29 10:43:18.085761 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.085573    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-2d850-kube-rbac-proxy-sar-config\""
Jul 29 10:43:18.098266 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.098245    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5"]
Jul 29 10:43:18.119370 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.119346    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:43:18.155117 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.155040    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw"]
Jul 29 10:43:18.158167 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.158147    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw"
Jul 29 10:43:18.161118 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.161053    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-2d850-kube-rbac-proxy-sar-config\""
Jul 29 10:43:18.161265 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.161230    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-2d850-predictor-serving-cert\""
Jul 29 10:43:18.169332 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.169309    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw"]
Jul 29 10:43:18.188347 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.188322    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:43:18.224804 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.224780    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q66h\" (UniqueName: \"kubernetes.io/projected/59ae549e-eeb1-49eb-8862-3cd5709a2efe-kube-api-access-9q66h\") pod \"isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5\" (UID: \"59ae549e-eeb1-49eb-8862-3cd5709a2efe\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5"
Jul 29 10:43:18.224910 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.224830    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59ae549e-eeb1-49eb-8862-3cd5709a2efe-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5\" (UID: \"59ae549e-eeb1-49eb-8862-3cd5709a2efe\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5"
Jul 29 10:43:18.224910 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.224865    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59ae549e-eeb1-49eb-8862-3cd5709a2efe-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5\" (UID: \"59ae549e-eeb1-49eb-8862-3cd5709a2efe\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5"
Jul 29 10:43:18.225003 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.224936    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59ae549e-eeb1-49eb-8862-3cd5709a2efe-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5\" (UID: \"59ae549e-eeb1-49eb-8862-3cd5709a2efe\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5"
Jul 29 10:43:18.225003 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.224964    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59ae549e-eeb1-49eb-8862-3cd5709a2efe-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5\" (UID: \"59ae549e-eeb1-49eb-8862-3cd5709a2efe\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5"
Jul 29 10:43:18.260896 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.260866    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9"]
Jul 29 10:43:18.261221 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.261166    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9" podUID="269bc236-82b6-4cb1-b594-13b1711573b0" containerName="kserve-container" containerID="cri-o://7e2a6742565d7f6d4fb56bd17141e3f942b7b8a7e6678af97a79c354221b19e5" gracePeriod=30
Jul 29 10:43:18.261221 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.261205    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9" podUID="269bc236-82b6-4cb1-b594-13b1711573b0" containerName="kube-rbac-proxy" containerID="cri-o://f77703714867b74f755655b3354f8d886e9f8dda9efd39c1919a66c0fd3a47eb" gracePeriod=30
Jul 29 10:43:18.307262 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.307231    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:43:18.326326 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.326299    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q66h\" (UniqueName: \"kubernetes.io/projected/59ae549e-eeb1-49eb-8862-3cd5709a2efe-kube-api-access-9q66h\") pod \"isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5\" (UID: \"59ae549e-eeb1-49eb-8862-3cd5709a2efe\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5"
Jul 29 10:43:18.326432 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.326343    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59ae549e-eeb1-49eb-8862-3cd5709a2efe-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5\" (UID: \"59ae549e-eeb1-49eb-8862-3cd5709a2efe\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5"
Jul 29 10:43:18.326432 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.326373    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59ae549e-eeb1-49eb-8862-3cd5709a2efe-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5\" (UID: \"59ae549e-eeb1-49eb-8862-3cd5709a2efe\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5"
Jul 29 10:43:18.326432 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.326409    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw\" (UID: \"e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw"
Jul 29 10:43:18.326582 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.326463    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2kpc\" (UniqueName: \"kubernetes.io/projected/e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01-kube-api-access-k2kpc\") pod \"isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw\" (UID: \"e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw"
Jul 29 10:43:18.326582 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.326557    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw\" (UID: \"e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw"
Jul 29 10:43:18.326672 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.326614    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59ae549e-eeb1-49eb-8862-3cd5709a2efe-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5\" (UID: \"59ae549e-eeb1-49eb-8862-3cd5709a2efe\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5"
Jul 29 10:43:18.326672 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.326649    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59ae549e-eeb1-49eb-8862-3cd5709a2efe-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5\" (UID: \"59ae549e-eeb1-49eb-8862-3cd5709a2efe\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5"
Jul 29 10:43:18.326748 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.326693    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw\" (UID: \"e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw"
Jul 29 10:43:18.326748 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.326723    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw\" (UID: \"e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw"
Jul 29 10:43:18.326837 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:43:18.326800    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-serving-cert: secret "isvc-sklearn-graph-raw-hpa-2d850-predictor-serving-cert" not found
Jul 29 10:43:18.326900 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:43:18.326879    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59ae549e-eeb1-49eb-8862-3cd5709a2efe-proxy-tls podName:59ae549e-eeb1-49eb-8862-3cd5709a2efe nodeName:}" failed. No retries permitted until 2026-07-29 10:43:18.82685836 +0000 UTC m=+875.551649024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/59ae549e-eeb1-49eb-8862-3cd5709a2efe-proxy-tls") pod "isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5" (UID: "59ae549e-eeb1-49eb-8862-3cd5709a2efe") : secret "isvc-sklearn-graph-raw-hpa-2d850-predictor-serving-cert" not found
Jul 29 10:43:18.327003 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.326972    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59ae549e-eeb1-49eb-8862-3cd5709a2efe-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5\" (UID: \"59ae549e-eeb1-49eb-8862-3cd5709a2efe\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5"
Jul 29 10:43:18.327062 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.327041    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59ae549e-eeb1-49eb-8862-3cd5709a2efe-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5\" (UID: \"59ae549e-eeb1-49eb-8862-3cd5709a2efe\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5"
Jul 29 10:43:18.327386 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.327369    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59ae549e-eeb1-49eb-8862-3cd5709a2efe-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5\" (UID: \"59ae549e-eeb1-49eb-8862-3cd5709a2efe\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5"
Jul 29 10:43:18.335061 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.335036    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q66h\" (UniqueName: \"kubernetes.io/projected/59ae549e-eeb1-49eb-8862-3cd5709a2efe-kube-api-access-9q66h\") pod \"isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5\" (UID: \"59ae549e-eeb1-49eb-8862-3cd5709a2efe\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5"
Jul 29 10:43:18.372797 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.372772    2569 generic.go:356] "Generic (PLEG): container finished" podID="c5a024eb-9344-4ae0-8a17-87ce32cfa61f" containerID="6b114532d6755f5c6dc7be56a23ee31ed326e378579193cdd8c79445d5c44fbe" exitCode=2
Jul 29 10:43:18.372884 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.372807    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x" event={"ID":"c5a024eb-9344-4ae0-8a17-87ce32cfa61f","Type":"ContainerDied","Data":"6b114532d6755f5c6dc7be56a23ee31ed326e378579193cdd8c79445d5c44fbe"}
Jul 29 10:43:18.427297 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.427269    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw\" (UID: \"e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw"
Jul 29 10:43:18.427416 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.427335    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2kpc\" (UniqueName: \"kubernetes.io/projected/e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01-kube-api-access-k2kpc\") pod \"isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw\" (UID: \"e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw"
Jul 29 10:43:18.427416 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.427379    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw\" (UID: \"e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw"
Jul 29 10:43:18.427511 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.427445    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw\" (UID: \"e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw"
Jul 29 10:43:18.427511 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.427472    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw\" (UID: \"e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw"
Jul 29 10:43:18.427767 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.427735    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw\" (UID: \"e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw"
Jul 29 10:43:18.427972 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.427948    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw\" (UID: \"e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw"
Jul 29 10:43:18.428176 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.428158    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw\" (UID: \"e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw"
Jul 29 10:43:18.429817 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.429798    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw\" (UID: \"e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw"
Jul 29 10:43:18.435059 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.435034    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2kpc\" (UniqueName: \"kubernetes.io/projected/e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01-kube-api-access-k2kpc\") pod \"isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw\" (UID: \"e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw"
Jul 29 10:43:18.463117 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.463093    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:43:18.470004 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.469986    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw"
Jul 29 10:43:18.608406 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:43:18.608375    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ea49c6_c3ff_4bc8_8f72_d9ad35d3cc01.slice/crio-f78668349de7ed64646da32093c97bddc38c346899ff0f6841ea8252eb63aa00 WatchSource:0}: Error finding container f78668349de7ed64646da32093c97bddc38c346899ff0f6841ea8252eb63aa00: Status 404 returned error can't find the container with id f78668349de7ed64646da32093c97bddc38c346899ff0f6841ea8252eb63aa00
Jul 29 10:43:18.608609 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.608587    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw"]
Jul 29 10:43:18.629468 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.629431    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:43:18.831523 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.831424    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59ae549e-eeb1-49eb-8862-3cd5709a2efe-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5\" (UID: \"59ae549e-eeb1-49eb-8862-3cd5709a2efe\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5"
Jul 29 10:43:18.834424 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.834387    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59ae549e-eeb1-49eb-8862-3cd5709a2efe-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5\" (UID: \"59ae549e-eeb1-49eb-8862-3cd5709a2efe\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5"
Jul 29 10:43:18.993489 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:18.993454    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5"
Jul 29 10:43:19.126397 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:19.126110    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5"]
Jul 29 10:43:19.128908 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:43:19.128876    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ae549e_eeb1_49eb_8862_3cd5709a2efe.slice/crio-7b6c1109f3f40a67000c974086800c9f75547928d2d12687b54ec9976d113bc6 WatchSource:0}: Error finding container 7b6c1109f3f40a67000c974086800c9f75547928d2d12687b54ec9976d113bc6: Status 404 returned error can't find the container with id 7b6c1109f3f40a67000c974086800c9f75547928d2d12687b54ec9976d113bc6
Jul 29 10:43:19.379495 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:19.379389    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5" event={"ID":"59ae549e-eeb1-49eb-8862-3cd5709a2efe","Type":"ContainerStarted","Data":"225b4026987ddfa6054123ad1ca0ada6821c27d3683e627cbb50c0f26d7a3d10"}
Jul 29 10:43:19.379495 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:19.379471    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5" event={"ID":"59ae549e-eeb1-49eb-8862-3cd5709a2efe","Type":"ContainerStarted","Data":"7b6c1109f3f40a67000c974086800c9f75547928d2d12687b54ec9976d113bc6"}
Jul 29 10:43:19.381141 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:19.381104    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw" event={"ID":"e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01","Type":"ContainerStarted","Data":"1b427a8203bc27af0acee7c874a9bfdc4882273d11a1c242a99a317fec6b52d7"}
Jul 29 10:43:19.381256 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:19.381169    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw" event={"ID":"e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01","Type":"ContainerStarted","Data":"f78668349de7ed64646da32093c97bddc38c346899ff0f6841ea8252eb63aa00"}
Jul 29 10:43:19.383246 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:19.383211    2569 generic.go:356] "Generic (PLEG): container finished" podID="269bc236-82b6-4cb1-b594-13b1711573b0" containerID="f77703714867b74f755655b3354f8d886e9f8dda9efd39c1919a66c0fd3a47eb" exitCode=2
Jul 29 10:43:19.383342 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:19.383286    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9" event={"ID":"269bc236-82b6-4cb1-b594-13b1711573b0","Type":"ContainerDied","Data":"f77703714867b74f755655b3354f8d886e9f8dda9efd39c1919a66c0fd3a47eb"}
Jul 29 10:43:20.934053 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:20.933997    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x" podUID="c5a024eb-9344-4ae0-8a17-87ce32cfa61f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Jul 29 10:43:20.939405 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:20.939364    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x" podUID="c5a024eb-9344-4ae0-8a17-87ce32cfa61f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 29 10:43:22.101525 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:22.101500    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9"
Jul 29 10:43:22.157996 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:22.157962    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/269bc236-82b6-4cb1-b594-13b1711573b0-kserve-provision-location\") pod \"269bc236-82b6-4cb1-b594-13b1711573b0\" (UID: \"269bc236-82b6-4cb1-b594-13b1711573b0\") "
Jul 29 10:43:22.158191 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:22.158017    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/269bc236-82b6-4cb1-b594-13b1711573b0-proxy-tls\") pod \"269bc236-82b6-4cb1-b594-13b1711573b0\" (UID: \"269bc236-82b6-4cb1-b594-13b1711573b0\") "
Jul 29 10:43:22.158191 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:22.158051    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ghnx\" (UniqueName: \"kubernetes.io/projected/269bc236-82b6-4cb1-b594-13b1711573b0-kube-api-access-4ghnx\") pod \"269bc236-82b6-4cb1-b594-13b1711573b0\" (UID: \"269bc236-82b6-4cb1-b594-13b1711573b0\") "
Jul 29 10:43:22.158191 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:22.158099    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/269bc236-82b6-4cb1-b594-13b1711573b0-kube-rbac-proxy-sar-config\") pod \"269bc236-82b6-4cb1-b594-13b1711573b0\" (UID: \"269bc236-82b6-4cb1-b594-13b1711573b0\") "
Jul 29 10:43:22.158191 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:22.158165    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/269bc236-82b6-4cb1-b594-13b1711573b0-cabundle-cert\") pod \"269bc236-82b6-4cb1-b594-13b1711573b0\" (UID: \"269bc236-82b6-4cb1-b594-13b1711573b0\") "
Jul 29 10:43:22.158381 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:22.158316    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/269bc236-82b6-4cb1-b594-13b1711573b0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "269bc236-82b6-4cb1-b594-13b1711573b0" (UID: "269bc236-82b6-4cb1-b594-13b1711573b0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:43:22.158480 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:22.158464    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/269bc236-82b6-4cb1-b594-13b1711573b0-kserve-provision-location\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:43:22.158544 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:22.158522    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/269bc236-82b6-4cb1-b594-13b1711573b0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "269bc236-82b6-4cb1-b594-13b1711573b0" (UID: "269bc236-82b6-4cb1-b594-13b1711573b0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:43:22.158577 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:22.158521    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/269bc236-82b6-4cb1-b594-13b1711573b0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "269bc236-82b6-4cb1-b594-13b1711573b0" (UID: "269bc236-82b6-4cb1-b594-13b1711573b0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:43:22.160215 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:22.160190    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/269bc236-82b6-4cb1-b594-13b1711573b0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "269bc236-82b6-4cb1-b594-13b1711573b0" (UID: "269bc236-82b6-4cb1-b594-13b1711573b0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:43:22.160305 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:22.160254    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/269bc236-82b6-4cb1-b594-13b1711573b0-kube-api-access-4ghnx" (OuterVolumeSpecName: "kube-api-access-4ghnx") pod "269bc236-82b6-4cb1-b594-13b1711573b0" (UID: "269bc236-82b6-4cb1-b594-13b1711573b0"). InnerVolumeSpecName "kube-api-access-4ghnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:43:22.258941 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:22.258862    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/269bc236-82b6-4cb1-b594-13b1711573b0-cabundle-cert\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:43:22.258941 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:22.258892    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/269bc236-82b6-4cb1-b594-13b1711573b0-proxy-tls\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:43:22.258941 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:22.258904    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ghnx\" (UniqueName: \"kubernetes.io/projected/269bc236-82b6-4cb1-b594-13b1711573b0-kube-api-access-4ghnx\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:43:22.258941 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:22.258917    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/269bc236-82b6-4cb1-b594-13b1711573b0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:43:22.397934 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:22.397903    2569 generic.go:356] "Generic (PLEG): container finished" podID="e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01" containerID="1b427a8203bc27af0acee7c874a9bfdc4882273d11a1c242a99a317fec6b52d7" exitCode=0
Jul 29 10:43:22.398117 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:22.397982    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw" event={"ID":"e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01","Type":"ContainerDied","Data":"1b427a8203bc27af0acee7c874a9bfdc4882273d11a1c242a99a317fec6b52d7"}
Jul 29 10:43:22.399955 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:22.399933    2569 generic.go:356] "Generic (PLEG): container finished" podID="269bc236-82b6-4cb1-b594-13b1711573b0" containerID="7e2a6742565d7f6d4fb56bd17141e3f942b7b8a7e6678af97a79c354221b19e5" exitCode=0
Jul 29 10:43:22.400055 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:22.399980    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9"
Jul 29 10:43:22.400055 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:22.400013    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9" event={"ID":"269bc236-82b6-4cb1-b594-13b1711573b0","Type":"ContainerDied","Data":"7e2a6742565d7f6d4fb56bd17141e3f942b7b8a7e6678af97a79c354221b19e5"}
Jul 29 10:43:22.400152 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:22.400092    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9" event={"ID":"269bc236-82b6-4cb1-b594-13b1711573b0","Type":"ContainerDied","Data":"1b69c4e4460bce385add53d3729addafb495917fb654d02fbe8c77c8642c5097"}
Jul 29 10:43:22.400152 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:22.400103    2569 scope.go:117] "RemoveContainer" containerID="f77703714867b74f755655b3354f8d886e9f8dda9efd39c1919a66c0fd3a47eb"
Jul 29 10:43:22.409997 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:22.409979    2569 scope.go:117] "RemoveContainer" containerID="7e2a6742565d7f6d4fb56bd17141e3f942b7b8a7e6678af97a79c354221b19e5"
Jul 29 10:43:22.417904 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:22.417886    2569 scope.go:117] "RemoveContainer" containerID="9e33b9f1e06aa58e8f45bcd58007182e5e76d801d9f67dee934683de854da133"
Jul 29 10:43:22.429773 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:22.429750    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9"]
Jul 29 10:43:22.436189 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:22.436166    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-877ef-predictor-84748fccb5-6zjk9"]
Jul 29 10:43:22.436805 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:22.436675    2569 scope.go:117] "RemoveContainer" containerID="f77703714867b74f755655b3354f8d886e9f8dda9efd39c1919a66c0fd3a47eb"
Jul 29 10:43:22.437033 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:43:22.437010    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f77703714867b74f755655b3354f8d886e9f8dda9efd39c1919a66c0fd3a47eb\": container with ID starting with f77703714867b74f755655b3354f8d886e9f8dda9efd39c1919a66c0fd3a47eb not found: ID does not exist" containerID="f77703714867b74f755655b3354f8d886e9f8dda9efd39c1919a66c0fd3a47eb"
Jul 29 10:43:22.437123 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:22.437042    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f77703714867b74f755655b3354f8d886e9f8dda9efd39c1919a66c0fd3a47eb"} err="failed to get container status \"f77703714867b74f755655b3354f8d886e9f8dda9efd39c1919a66c0fd3a47eb\": rpc error: code = NotFound desc = could not find container \"f77703714867b74f755655b3354f8d886e9f8dda9efd39c1919a66c0fd3a47eb\": container with ID starting with f77703714867b74f755655b3354f8d886e9f8dda9efd39c1919a66c0fd3a47eb not found: ID does not exist"
Jul 29 10:43:22.437123 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:22.437060    2569 scope.go:117] "RemoveContainer" containerID="7e2a6742565d7f6d4fb56bd17141e3f942b7b8a7e6678af97a79c354221b19e5"
Jul 29 10:43:22.437358 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:43:22.437338    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e2a6742565d7f6d4fb56bd17141e3f942b7b8a7e6678af97a79c354221b19e5\": container with ID starting with 7e2a6742565d7f6d4fb56bd17141e3f942b7b8a7e6678af97a79c354221b19e5 not found: ID does not exist" containerID="7e2a6742565d7f6d4fb56bd17141e3f942b7b8a7e6678af97a79c354221b19e5"
Jul 29 10:43:22.437427 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:22.437362    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e2a6742565d7f6d4fb56bd17141e3f942b7b8a7e6678af97a79c354221b19e5"} err="failed to get container status \"7e2a6742565d7f6d4fb56bd17141e3f942b7b8a7e6678af97a79c354221b19e5\": rpc error: code = NotFound desc = could not find container \"7e2a6742565d7f6d4fb56bd17141e3f942b7b8a7e6678af97a79c354221b19e5\": container with ID starting with 7e2a6742565d7f6d4fb56bd17141e3f942b7b8a7e6678af97a79c354221b19e5 not found: ID does not exist"
Jul 29 10:43:22.437427 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:22.437376    2569 scope.go:117] "RemoveContainer" containerID="9e33b9f1e06aa58e8f45bcd58007182e5e76d801d9f67dee934683de854da133"
Jul 29 10:43:22.437634 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:43:22.437612    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e33b9f1e06aa58e8f45bcd58007182e5e76d801d9f67dee934683de854da133\": container with ID starting with 9e33b9f1e06aa58e8f45bcd58007182e5e76d801d9f67dee934683de854da133 not found: ID does not exist" containerID="9e33b9f1e06aa58e8f45bcd58007182e5e76d801d9f67dee934683de854da133"
Jul 29 10:43:22.437696 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:22.437641    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e33b9f1e06aa58e8f45bcd58007182e5e76d801d9f67dee934683de854da133"} err="failed to get container status \"9e33b9f1e06aa58e8f45bcd58007182e5e76d801d9f67dee934683de854da133\": rpc error: code = NotFound desc = could not find container \"9e33b9f1e06aa58e8f45bcd58007182e5e76d801d9f67dee934683de854da133\": container with ID starting with 9e33b9f1e06aa58e8f45bcd58007182e5e76d801d9f67dee934683de854da133 not found: ID does not exist"
Jul 29 10:43:23.000563 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.000542    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x"
Jul 29 10:43:23.066305 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.066257    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5a024eb-9344-4ae0-8a17-87ce32cfa61f-proxy-tls\") pod \"c5a024eb-9344-4ae0-8a17-87ce32cfa61f\" (UID: \"c5a024eb-9344-4ae0-8a17-87ce32cfa61f\") "
Jul 29 10:43:23.066478 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.066334    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dg445\" (UniqueName: \"kubernetes.io/projected/c5a024eb-9344-4ae0-8a17-87ce32cfa61f-kube-api-access-dg445\") pod \"c5a024eb-9344-4ae0-8a17-87ce32cfa61f\" (UID: \"c5a024eb-9344-4ae0-8a17-87ce32cfa61f\") "
Jul 29 10:43:23.066478 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.066378    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5a024eb-9344-4ae0-8a17-87ce32cfa61f-kube-rbac-proxy-sar-config\") pod \"c5a024eb-9344-4ae0-8a17-87ce32cfa61f\" (UID: \"c5a024eb-9344-4ae0-8a17-87ce32cfa61f\") "
Jul 29 10:43:23.066478 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.066405    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5a024eb-9344-4ae0-8a17-87ce32cfa61f-kserve-provision-location\") pod \"c5a024eb-9344-4ae0-8a17-87ce32cfa61f\" (UID: \"c5a024eb-9344-4ae0-8a17-87ce32cfa61f\") "
Jul 29 10:43:23.066478 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.066433    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5a024eb-9344-4ae0-8a17-87ce32cfa61f-cabundle-cert\") pod \"c5a024eb-9344-4ae0-8a17-87ce32cfa61f\" (UID: \"c5a024eb-9344-4ae0-8a17-87ce32cfa61f\") "
Jul 29 10:43:23.066770 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.066745    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a024eb-9344-4ae0-8a17-87ce32cfa61f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c5a024eb-9344-4ae0-8a17-87ce32cfa61f" (UID: "c5a024eb-9344-4ae0-8a17-87ce32cfa61f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:43:23.066833 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.066760    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a024eb-9344-4ae0-8a17-87ce32cfa61f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c5a024eb-9344-4ae0-8a17-87ce32cfa61f" (UID: "c5a024eb-9344-4ae0-8a17-87ce32cfa61f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:43:23.066833 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.066817    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a024eb-9344-4ae0-8a17-87ce32cfa61f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c5a024eb-9344-4ae0-8a17-87ce32cfa61f" (UID: "c5a024eb-9344-4ae0-8a17-87ce32cfa61f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:43:23.068437 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.068402    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5a024eb-9344-4ae0-8a17-87ce32cfa61f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c5a024eb-9344-4ae0-8a17-87ce32cfa61f" (UID: "c5a024eb-9344-4ae0-8a17-87ce32cfa61f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:43:23.068543 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.068459    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a024eb-9344-4ae0-8a17-87ce32cfa61f-kube-api-access-dg445" (OuterVolumeSpecName: "kube-api-access-dg445") pod "c5a024eb-9344-4ae0-8a17-87ce32cfa61f" (UID: "c5a024eb-9344-4ae0-8a17-87ce32cfa61f"). InnerVolumeSpecName "kube-api-access-dg445". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:43:23.167265 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.167226    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5a024eb-9344-4ae0-8a17-87ce32cfa61f-cabundle-cert\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:43:23.167265 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.167264    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5a024eb-9344-4ae0-8a17-87ce32cfa61f-proxy-tls\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:43:23.167265 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.167275    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg445\" (UniqueName: \"kubernetes.io/projected/c5a024eb-9344-4ae0-8a17-87ce32cfa61f-kube-api-access-dg445\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:43:23.167677 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.167284    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5a024eb-9344-4ae0-8a17-87ce32cfa61f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:43:23.167677 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.167295    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5a024eb-9344-4ae0-8a17-87ce32cfa61f-kserve-provision-location\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:43:23.405635 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.405601    2569 generic.go:356] "Generic (PLEG): container finished" podID="59ae549e-eeb1-49eb-8862-3cd5709a2efe" containerID="225b4026987ddfa6054123ad1ca0ada6821c27d3683e627cbb50c0f26d7a3d10" exitCode=0
Jul 29 10:43:23.405815 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.405668    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5" event={"ID":"59ae549e-eeb1-49eb-8862-3cd5709a2efe","Type":"ContainerDied","Data":"225b4026987ddfa6054123ad1ca0ada6821c27d3683e627cbb50c0f26d7a3d10"}
Jul 29 10:43:23.407858 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.407797    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw" event={"ID":"e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01","Type":"ContainerStarted","Data":"690b89cbacbde5793725496218906fd32c1c6a893244e8f02414af37dc1d99fd"}
Jul 29 10:43:23.407948 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.407866    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw" event={"ID":"e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01","Type":"ContainerStarted","Data":"4d77c906b1968974a05f53501996e0fe30f009e2c3dd140e99cb9186b6678e87"}
Jul 29 10:43:23.408201 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.408162    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw"
Jul 29 10:43:23.408370 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.408351    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw"
Jul 29 10:43:23.410060 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.410036    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw" podUID="e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 29 10:43:23.410922 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.410903    2569 generic.go:356] "Generic (PLEG): container finished" podID="c5a024eb-9344-4ae0-8a17-87ce32cfa61f" containerID="28261c143cd7d2b7047c82c089f5c9a28cafb1ab04cb9712dfaba99eb4ed7714" exitCode=0
Jul 29 10:43:23.410999 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.410949    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x" event={"ID":"c5a024eb-9344-4ae0-8a17-87ce32cfa61f","Type":"ContainerDied","Data":"28261c143cd7d2b7047c82c089f5c9a28cafb1ab04cb9712dfaba99eb4ed7714"}
Jul 29 10:43:23.410999 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.410970    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x"
Jul 29 10:43:23.411093 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.411009    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x" event={"ID":"c5a024eb-9344-4ae0-8a17-87ce32cfa61f","Type":"ContainerDied","Data":"8027951f324f6ff0a837efe433e3723f60e8dfb1a3baaa79ea82bd413ef22e6b"}
Jul 29 10:43:23.411093 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.411039    2569 scope.go:117] "RemoveContainer" containerID="6b114532d6755f5c6dc7be56a23ee31ed326e378579193cdd8c79445d5c44fbe"
Jul 29 10:43:23.420365 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.420345    2569 scope.go:117] "RemoveContainer" containerID="28261c143cd7d2b7047c82c089f5c9a28cafb1ab04cb9712dfaba99eb4ed7714"
Jul 29 10:43:23.428823 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.428798    2569 scope.go:117] "RemoveContainer" containerID="4da79cfe2ef4276b991fcc49a7935041578de7eef0dc4c86df8dc4b2e2ccdd05"
Jul 29 10:43:23.438186 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.438161    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x"]
Jul 29 10:43:23.440460 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.440367    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-877ef-predictor-5b5cd58955-kmw2x"]
Jul 29 10:43:23.446121 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.446097    2569 scope.go:117] "RemoveContainer" containerID="6b114532d6755f5c6dc7be56a23ee31ed326e378579193cdd8c79445d5c44fbe"
Jul 29 10:43:23.446465 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:43:23.446447    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b114532d6755f5c6dc7be56a23ee31ed326e378579193cdd8c79445d5c44fbe\": container with ID starting with 6b114532d6755f5c6dc7be56a23ee31ed326e378579193cdd8c79445d5c44fbe not found: ID does not exist" containerID="6b114532d6755f5c6dc7be56a23ee31ed326e378579193cdd8c79445d5c44fbe"
Jul 29 10:43:23.446530 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.446474    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b114532d6755f5c6dc7be56a23ee31ed326e378579193cdd8c79445d5c44fbe"} err="failed to get container status \"6b114532d6755f5c6dc7be56a23ee31ed326e378579193cdd8c79445d5c44fbe\": rpc error: code = NotFound desc = could not find container \"6b114532d6755f5c6dc7be56a23ee31ed326e378579193cdd8c79445d5c44fbe\": container with ID starting with 6b114532d6755f5c6dc7be56a23ee31ed326e378579193cdd8c79445d5c44fbe not found: ID does not exist"
Jul 29 10:43:23.446530 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.446493    2569 scope.go:117] "RemoveContainer" containerID="28261c143cd7d2b7047c82c089f5c9a28cafb1ab04cb9712dfaba99eb4ed7714"
Jul 29 10:43:23.446758 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:43:23.446739    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28261c143cd7d2b7047c82c089f5c9a28cafb1ab04cb9712dfaba99eb4ed7714\": container with ID starting with 28261c143cd7d2b7047c82c089f5c9a28cafb1ab04cb9712dfaba99eb4ed7714 not found: ID does not exist" containerID="28261c143cd7d2b7047c82c089f5c9a28cafb1ab04cb9712dfaba99eb4ed7714"
Jul 29 10:43:23.446820 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.446768    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28261c143cd7d2b7047c82c089f5c9a28cafb1ab04cb9712dfaba99eb4ed7714"} err="failed to get container status \"28261c143cd7d2b7047c82c089f5c9a28cafb1ab04cb9712dfaba99eb4ed7714\": rpc error: code = NotFound desc = could not find container \"28261c143cd7d2b7047c82c089f5c9a28cafb1ab04cb9712dfaba99eb4ed7714\": container with ID starting with 28261c143cd7d2b7047c82c089f5c9a28cafb1ab04cb9712dfaba99eb4ed7714 not found: ID does not exist"
Jul 29 10:43:23.446820 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.446793    2569 scope.go:117] "RemoveContainer" containerID="4da79cfe2ef4276b991fcc49a7935041578de7eef0dc4c86df8dc4b2e2ccdd05"
Jul 29 10:43:23.447102 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:43:23.447054    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4da79cfe2ef4276b991fcc49a7935041578de7eef0dc4c86df8dc4b2e2ccdd05\": container with ID starting with 4da79cfe2ef4276b991fcc49a7935041578de7eef0dc4c86df8dc4b2e2ccdd05 not found: ID does not exist" containerID="4da79cfe2ef4276b991fcc49a7935041578de7eef0dc4c86df8dc4b2e2ccdd05"
Jul 29 10:43:23.447168 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.447100    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4da79cfe2ef4276b991fcc49a7935041578de7eef0dc4c86df8dc4b2e2ccdd05"} err="failed to get container status \"4da79cfe2ef4276b991fcc49a7935041578de7eef0dc4c86df8dc4b2e2ccdd05\": rpc error: code = NotFound desc = could not find container \"4da79cfe2ef4276b991fcc49a7935041578de7eef0dc4c86df8dc4b2e2ccdd05\": container with ID starting with 4da79cfe2ef4276b991fcc49a7935041578de7eef0dc4c86df8dc4b2e2ccdd05 not found: ID does not exist"
Jul 29 10:43:23.455402 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.455358    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw" podStartSLOduration=1.77481321 podStartE2EDuration="5.455343876s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.680530666s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:43:23.454794884 +0000 UTC m=+880.179585557" watchObservedRunningTime="2026-07-29 10:43:23.455343876 +0000 UTC m=+880.180134548"
Jul 29 10:43:23.840256 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.840171    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="269bc236-82b6-4cb1-b594-13b1711573b0" path="/var/lib/kubelet/pods/269bc236-82b6-4cb1-b594-13b1711573b0/volumes"
Jul 29 10:43:23.840708 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:23.840691    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5a024eb-9344-4ae0-8a17-87ce32cfa61f" path="/var/lib/kubelet/pods/c5a024eb-9344-4ae0-8a17-87ce32cfa61f/volumes"
Jul 29 10:43:24.418240 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:24.418198    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5" event={"ID":"59ae549e-eeb1-49eb-8862-3cd5709a2efe","Type":"ContainerStarted","Data":"adbca8dfec2fc968016d41c024508284930bc108d6b509979d398292e671d572"}
Jul 29 10:43:24.418637 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:24.418274    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5" event={"ID":"59ae549e-eeb1-49eb-8862-3cd5709a2efe","Type":"ContainerStarted","Data":"f9d35c6cfcddcb98456d9d31a61a590fb7d936b60925515bd247e559e05e481f"}
Jul 29 10:43:24.418637 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:24.418448    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw" podUID="e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 29 10:43:24.418718 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:24.418682    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5"
Jul 29 10:43:24.418806 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:24.418786    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5"
Jul 29 10:43:24.420189 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:24.420165    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5" podUID="59ae549e-eeb1-49eb-8862-3cd5709a2efe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 29 10:43:24.437778 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:24.437724    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5" podStartSLOduration=2.7327296690000002 podStartE2EDuration="6.437710287s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.704980618s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:43:24.436133544 +0000 UTC m=+881.160924218" watchObservedRunningTime="2026-07-29 10:43:24.437710287 +0000 UTC m=+881.162500960"
Jul 29 10:43:25.422781 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:25.422738    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5" podUID="59ae549e-eeb1-49eb-8862-3cd5709a2efe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 29 10:43:29.423230 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:29.423194    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw"
Jul 29 10:43:29.423868 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:29.423829    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw" podUID="e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 29 10:43:30.428178 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:30.428147    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5"
Jul 29 10:43:30.428651 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:30.428625    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5" podUID="59ae549e-eeb1-49eb-8862-3cd5709a2efe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 29 10:43:39.423840 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:39.423755    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw" podUID="e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 29 10:43:40.429286 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:40.429248    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5" podUID="59ae549e-eeb1-49eb-8862-3cd5709a2efe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 29 10:43:43.754171 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:43.754142    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-pvdgw_afd0d65c-bbfc-49f0-81b6-e855fd81da7f/console-operator/1.log"
Jul 29 10:43:43.754657 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:43.754636    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-pvdgw_afd0d65c-bbfc-49f0-81b6-e855fd81da7f/console-operator/1.log"
Jul 29 10:43:49.424687 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:49.424642    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw" podUID="e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 29 10:43:50.428595 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:50.428554    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5" podUID="59ae549e-eeb1-49eb-8862-3cd5709a2efe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 29 10:43:59.423928 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:43:59.423885    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw" podUID="e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 29 10:44:00.428949 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:44:00.428903    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5" podUID="59ae549e-eeb1-49eb-8862-3cd5709a2efe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 29 10:44:09.423794 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:44:09.423750    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw" podUID="e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 29 10:44:10.429045 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:44:10.429003    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5" podUID="59ae549e-eeb1-49eb-8862-3cd5709a2efe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 29 10:44:19.424457 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:44:19.424409    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw" podUID="e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 29 10:44:20.429052 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:44:20.429012    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5" podUID="59ae549e-eeb1-49eb-8862-3cd5709a2efe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 29 10:44:29.424282 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:44:29.424246    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw"
Jul 29 10:44:30.428741 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:44:30.428701    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5" podUID="59ae549e-eeb1-49eb-8862-3cd5709a2efe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 29 10:44:40.429236 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:44:40.429206    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5"
Jul 29 10:44:58.153187 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:44:58.153149    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:44:59.604386 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:44:59.604349    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:45:08.205165 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.205066    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:45:08.336660 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.336623    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:45:08.348616 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.348586    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5"]
Jul 29 10:45:08.348891 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.348859    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5" podUID="59ae549e-eeb1-49eb-8862-3cd5709a2efe" containerName="kserve-container" containerID="cri-o://f9d35c6cfcddcb98456d9d31a61a590fb7d936b60925515bd247e559e05e481f" gracePeriod=30
Jul 29 10:45:08.348990 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.348890    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5" podUID="59ae549e-eeb1-49eb-8862-3cd5709a2efe" containerName="kube-rbac-proxy" containerID="cri-o://adbca8dfec2fc968016d41c024508284930bc108d6b509979d398292e671d572" gracePeriod=30
Jul 29 10:45:08.374174 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.374147    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4"]
Jul 29 10:45:08.374488 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.374475    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="269bc236-82b6-4cb1-b594-13b1711573b0" containerName="storage-initializer"
Jul 29 10:45:08.374531 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.374490    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="269bc236-82b6-4cb1-b594-13b1711573b0" containerName="storage-initializer"
Jul 29 10:45:08.374531 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.374501    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5a024eb-9344-4ae0-8a17-87ce32cfa61f" containerName="kube-rbac-proxy"
Jul 29 10:45:08.374531 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.374507    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5a024eb-9344-4ae0-8a17-87ce32cfa61f" containerName="kube-rbac-proxy"
Jul 29 10:45:08.374531 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.374518    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5a024eb-9344-4ae0-8a17-87ce32cfa61f" containerName="storage-initializer"
Jul 29 10:45:08.374531 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.374524    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5a024eb-9344-4ae0-8a17-87ce32cfa61f" containerName="storage-initializer"
Jul 29 10:45:08.374531 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.374530    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5a024eb-9344-4ae0-8a17-87ce32cfa61f" containerName="kserve-container"
Jul 29 10:45:08.374712 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.374535    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5a024eb-9344-4ae0-8a17-87ce32cfa61f" containerName="kserve-container"
Jul 29 10:45:08.374712 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.374546    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="269bc236-82b6-4cb1-b594-13b1711573b0" containerName="kserve-container"
Jul 29 10:45:08.374712 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.374551    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="269bc236-82b6-4cb1-b594-13b1711573b0" containerName="kserve-container"
Jul 29 10:45:08.374712 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.374560    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="269bc236-82b6-4cb1-b594-13b1711573b0" containerName="kube-rbac-proxy"
Jul 29 10:45:08.374712 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.374565    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="269bc236-82b6-4cb1-b594-13b1711573b0" containerName="kube-rbac-proxy"
Jul 29 10:45:08.374712 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.374615    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5a024eb-9344-4ae0-8a17-87ce32cfa61f" containerName="kserve-container"
Jul 29 10:45:08.374712 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.374624    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="269bc236-82b6-4cb1-b594-13b1711573b0" containerName="kube-rbac-proxy"
Jul 29 10:45:08.374712 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.374629    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5a024eb-9344-4ae0-8a17-87ce32cfa61f" containerName="kube-rbac-proxy"
Jul 29 10:45:08.374712 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.374634    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="269bc236-82b6-4cb1-b594-13b1711573b0" containerName="kserve-container"
Jul 29 10:45:08.377543 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.377526    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4"
Jul 29 10:45:08.379912 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.379893    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-66e3d-kube-rbac-proxy-sar-config\""
Jul 29 10:45:08.380011 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.379900    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-66e3d-predictor-serving-cert\""
Jul 29 10:45:08.387284 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.387262    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4"]
Jul 29 10:45:08.401372 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.401344    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:45:08.454383 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.454346    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01822d12-2967-48a8-bd3e-61a43409476b-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4\" (UID: \"01822d12-2967-48a8-bd3e-61a43409476b\") " pod="kserve-ci-e2e-test/message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4"
Jul 29 10:45:08.454575 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.454431    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01822d12-2967-48a8-bd3e-61a43409476b-proxy-tls\") pod \"message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4\" (UID: \"01822d12-2967-48a8-bd3e-61a43409476b\") " pod="kserve-ci-e2e-test/message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4"
Jul 29 10:45:08.454653 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.454638    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnhvp\" (UniqueName: \"kubernetes.io/projected/01822d12-2967-48a8-bd3e-61a43409476b-kube-api-access-bnhvp\") pod \"message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4\" (UID: \"01822d12-2967-48a8-bd3e-61a43409476b\") " pod="kserve-ci-e2e-test/message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4"
Jul 29 10:45:08.478233 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.478205    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw"]
Jul 29 10:45:08.478465 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.478439    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw" podUID="e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01" containerName="kserve-container" containerID="cri-o://4d77c906b1968974a05f53501996e0fe30f009e2c3dd140e99cb9186b6678e87" gracePeriod=30
Jul 29 10:45:08.478605 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.478476    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw" podUID="e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01" containerName="kube-rbac-proxy" containerID="cri-o://690b89cbacbde5793725496218906fd32c1c6a893244e8f02414af37dc1d99fd" gracePeriod=30
Jul 29 10:45:08.555168 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.555132    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01822d12-2967-48a8-bd3e-61a43409476b-proxy-tls\") pod \"message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4\" (UID: \"01822d12-2967-48a8-bd3e-61a43409476b\") " pod="kserve-ci-e2e-test/message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4"
Jul 29 10:45:08.555168 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.555175    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnhvp\" (UniqueName: \"kubernetes.io/projected/01822d12-2967-48a8-bd3e-61a43409476b-kube-api-access-bnhvp\") pod \"message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4\" (UID: \"01822d12-2967-48a8-bd3e-61a43409476b\") " pod="kserve-ci-e2e-test/message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4"
Jul 29 10:45:08.555425 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.555226    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01822d12-2967-48a8-bd3e-61a43409476b-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4\" (UID: \"01822d12-2967-48a8-bd3e-61a43409476b\") " pod="kserve-ci-e2e-test/message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4"
Jul 29 10:45:08.555874 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.555854    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01822d12-2967-48a8-bd3e-61a43409476b-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4\" (UID: \"01822d12-2967-48a8-bd3e-61a43409476b\") " pod="kserve-ci-e2e-test/message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4"
Jul 29 10:45:08.557582 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.557556    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01822d12-2967-48a8-bd3e-61a43409476b-proxy-tls\") pod \"message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4\" (UID: \"01822d12-2967-48a8-bd3e-61a43409476b\") " pod="kserve-ci-e2e-test/message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4"
Jul 29 10:45:08.563750 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.563730    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnhvp\" (UniqueName: \"kubernetes.io/projected/01822d12-2967-48a8-bd3e-61a43409476b-kube-api-access-bnhvp\") pod \"message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4\" (UID: \"01822d12-2967-48a8-bd3e-61a43409476b\") " pod="kserve-ci-e2e-test/message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4"
Jul 29 10:45:08.688322 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.688282    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4"
Jul 29 10:45:08.755490 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.755449    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:45:08.812330 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.812299    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:45:08.820893 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.820863    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4"]
Jul 29 10:45:08.823418 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:45:08.823392    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01822d12_2967_48a8_bd3e_61a43409476b.slice/crio-ba0914fd9fe64e29b061d9b80b28b087d615400119f1270128bd1758d59d144a WatchSource:0}: Error finding container ba0914fd9fe64e29b061d9b80b28b087d615400119f1270128bd1758d59d144a: Status 404 returned error can't find the container with id ba0914fd9fe64e29b061d9b80b28b087d615400119f1270128bd1758d59d144a
Jul 29 10:45:08.825274 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.825256    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 10:45:08.846097 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.846048    2569 generic.go:356] "Generic (PLEG): container finished" podID="59ae549e-eeb1-49eb-8862-3cd5709a2efe" containerID="adbca8dfec2fc968016d41c024508284930bc108d6b509979d398292e671d572" exitCode=2
Jul 29 10:45:08.846259 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.846117    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5" event={"ID":"59ae549e-eeb1-49eb-8862-3cd5709a2efe","Type":"ContainerDied","Data":"adbca8dfec2fc968016d41c024508284930bc108d6b509979d398292e671d572"}
Jul 29 10:45:08.847522 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.847498    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4" event={"ID":"01822d12-2967-48a8-bd3e-61a43409476b","Type":"ContainerStarted","Data":"ba0914fd9fe64e29b061d9b80b28b087d615400119f1270128bd1758d59d144a"}
Jul 29 10:45:08.849666 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.849645    2569 generic.go:356] "Generic (PLEG): container finished" podID="e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01" containerID="690b89cbacbde5793725496218906fd32c1c6a893244e8f02414af37dc1d99fd" exitCode=2
Jul 29 10:45:08.849767 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:08.849689    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw" event={"ID":"e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01","Type":"ContainerDied","Data":"690b89cbacbde5793725496218906fd32c1c6a893244e8f02414af37dc1d99fd"}
Jul 29 10:45:09.419346 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:09.419306    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw" podUID="e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Jul 29 10:45:09.424651 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:09.424626    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw" podUID="e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 29 10:45:10.423311 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:10.423269    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5" podUID="59ae549e-eeb1-49eb-8862-3cd5709a2efe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Jul 29 10:45:10.428863 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:10.428828    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5" podUID="59ae549e-eeb1-49eb-8862-3cd5709a2efe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 29 10:45:10.860190 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:10.860099    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4" event={"ID":"01822d12-2967-48a8-bd3e-61a43409476b","Type":"ContainerStarted","Data":"dc59abb121ae01fb5ac238c53dff4bf3515a308314a267c1c26cf7cc26c0f51f"}
Jul 29 10:45:10.860190 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:10.860158    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4" event={"ID":"01822d12-2967-48a8-bd3e-61a43409476b","Type":"ContainerStarted","Data":"0c69ad0e7e3ddba9d15107d540d79c546ef2a19551969e97ed17f9e27747c2fb"}
Jul 29 10:45:10.860378 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:10.860221    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4"
Jul 29 10:45:10.878129 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:10.878065    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4" podStartSLOduration=1.648198626 podStartE2EDuration="2.878048548s" totalImagesPullingTime="1.229849922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:45:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:45:10.877066365 +0000 UTC m=+987.601857038" watchObservedRunningTime="2026-07-29 10:45:10.878048548 +0000 UTC m=+987.602839221"
Jul 29 10:45:11.864569 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:11.864529    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4"
Jul 29 10:45:11.866233 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:11.866211    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4"
Jul 29 10:45:12.425949 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:12.425926    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw"
Jul 29 10:45:12.484619 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:12.484544    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01-kserve-provision-location\") pod \"e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01\" (UID: \"e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01\") "
Jul 29 10:45:12.484619 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:12.484593    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01-cabundle-cert\") pod \"e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01\" (UID: \"e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01\") "
Jul 29 10:45:12.484830 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:12.484623    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2kpc\" (UniqueName: \"kubernetes.io/projected/e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01-kube-api-access-k2kpc\") pod \"e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01\" (UID: \"e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01\") "
Jul 29 10:45:12.484830 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:12.484647    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01-proxy-tls\") pod \"e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01\" (UID: \"e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01\") "
Jul 29 10:45:12.484830 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:12.484692    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01-kube-rbac-proxy-sar-config\") pod \"e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01\" (UID: \"e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01\") "
Jul 29 10:45:12.484999 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:12.484915    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01" (UID: "e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:45:12.484999 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:12.484989    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01" (UID: "e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:45:12.485153 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:12.485127    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01" (UID: "e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:45:12.486735 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:12.486708    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01-kube-api-access-k2kpc" (OuterVolumeSpecName: "kube-api-access-k2kpc") pod "e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01" (UID: "e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01"). InnerVolumeSpecName "kube-api-access-k2kpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:45:12.487036 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:12.486757    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01" (UID: "e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:45:12.590357 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:12.585688    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:45:12.590357 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:12.585727    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01-kserve-provision-location\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:45:12.590357 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:12.585743    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01-cabundle-cert\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:45:12.590357 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:12.585758    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2kpc\" (UniqueName: \"kubernetes.io/projected/e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01-kube-api-access-k2kpc\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:45:12.590357 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:12.585781    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01-proxy-tls\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:45:12.869351 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:12.869265    2569 generic.go:356] "Generic (PLEG): container finished" podID="e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01" containerID="4d77c906b1968974a05f53501996e0fe30f009e2c3dd140e99cb9186b6678e87" exitCode=0
Jul 29 10:45:12.869351 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:12.869337    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw"
Jul 29 10:45:12.869804 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:12.869357    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw" event={"ID":"e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01","Type":"ContainerDied","Data":"4d77c906b1968974a05f53501996e0fe30f009e2c3dd140e99cb9186b6678e87"}
Jul 29 10:45:12.869804 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:12.869420    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw" event={"ID":"e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01","Type":"ContainerDied","Data":"f78668349de7ed64646da32093c97bddc38c346899ff0f6841ea8252eb63aa00"}
Jul 29 10:45:12.869804 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:12.869450    2569 scope.go:117] "RemoveContainer" containerID="690b89cbacbde5793725496218906fd32c1c6a893244e8f02414af37dc1d99fd"
Jul 29 10:45:12.878155 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:12.878138    2569 scope.go:117] "RemoveContainer" containerID="4d77c906b1968974a05f53501996e0fe30f009e2c3dd140e99cb9186b6678e87"
Jul 29 10:45:12.885527 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:12.885512    2569 scope.go:117] "RemoveContainer" containerID="1b427a8203bc27af0acee7c874a9bfdc4882273d11a1c242a99a317fec6b52d7"
Jul 29 10:45:12.891776 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:12.891754    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw"]
Jul 29 10:45:12.893305 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:12.893256    2569 scope.go:117] "RemoveContainer" containerID="690b89cbacbde5793725496218906fd32c1c6a893244e8f02414af37dc1d99fd"
Jul 29 10:45:12.893660 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:45:12.893610    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"690b89cbacbde5793725496218906fd32c1c6a893244e8f02414af37dc1d99fd\": container with ID starting with 690b89cbacbde5793725496218906fd32c1c6a893244e8f02414af37dc1d99fd not found: ID does not exist" containerID="690b89cbacbde5793725496218906fd32c1c6a893244e8f02414af37dc1d99fd"
Jul 29 10:45:12.893769 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:12.893654    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"690b89cbacbde5793725496218906fd32c1c6a893244e8f02414af37dc1d99fd"} err="failed to get container status \"690b89cbacbde5793725496218906fd32c1c6a893244e8f02414af37dc1d99fd\": rpc error: code = NotFound desc = could not find container \"690b89cbacbde5793725496218906fd32c1c6a893244e8f02414af37dc1d99fd\": container with ID starting with 690b89cbacbde5793725496218906fd32c1c6a893244e8f02414af37dc1d99fd not found: ID does not exist"
Jul 29 10:45:12.893769 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:12.893679    2569 scope.go:117] "RemoveContainer" containerID="4d77c906b1968974a05f53501996e0fe30f009e2c3dd140e99cb9186b6678e87"
Jul 29 10:45:12.894024 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:45:12.893994    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d77c906b1968974a05f53501996e0fe30f009e2c3dd140e99cb9186b6678e87\": container with ID starting with 4d77c906b1968974a05f53501996e0fe30f009e2c3dd140e99cb9186b6678e87 not found: ID does not exist" containerID="4d77c906b1968974a05f53501996e0fe30f009e2c3dd140e99cb9186b6678e87"
Jul 29 10:45:12.894149 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:12.894030    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d77c906b1968974a05f53501996e0fe30f009e2c3dd140e99cb9186b6678e87"} err="failed to get container status \"4d77c906b1968974a05f53501996e0fe30f009e2c3dd140e99cb9186b6678e87\": rpc error: code = NotFound desc = could not find container \"4d77c906b1968974a05f53501996e0fe30f009e2c3dd140e99cb9186b6678e87\": container with ID starting with 4d77c906b1968974a05f53501996e0fe30f009e2c3dd140e99cb9186b6678e87 not found: ID does not exist"
Jul 29 10:45:12.894149 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:12.894067    2569 scope.go:117] "RemoveContainer" containerID="1b427a8203bc27af0acee7c874a9bfdc4882273d11a1c242a99a317fec6b52d7"
Jul 29 10:45:12.894395 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:45:12.894376    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b427a8203bc27af0acee7c874a9bfdc4882273d11a1c242a99a317fec6b52d7\": container with ID starting with 1b427a8203bc27af0acee7c874a9bfdc4882273d11a1c242a99a317fec6b52d7 not found: ID does not exist" containerID="1b427a8203bc27af0acee7c874a9bfdc4882273d11a1c242a99a317fec6b52d7"
Jul 29 10:45:12.894461 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:12.894399    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b427a8203bc27af0acee7c874a9bfdc4882273d11a1c242a99a317fec6b52d7"} err="failed to get container status \"1b427a8203bc27af0acee7c874a9bfdc4882273d11a1c242a99a317fec6b52d7\": rpc error: code = NotFound desc = could not find container \"1b427a8203bc27af0acee7c874a9bfdc4882273d11a1c242a99a317fec6b52d7\": container with ID starting with 1b427a8203bc27af0acee7c874a9bfdc4882273d11a1c242a99a317fec6b52d7 not found: ID does not exist"
Jul 29 10:45:12.895534 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:12.895514    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2d850-predictor-db858f857-vsgkw"]
Jul 29 10:45:13.283944 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:13.283920    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5"
Jul 29 10:45:13.292253 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:13.292233    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59ae549e-eeb1-49eb-8862-3cd5709a2efe-kube-rbac-proxy-sar-config\") pod \"59ae549e-eeb1-49eb-8862-3cd5709a2efe\" (UID: \"59ae549e-eeb1-49eb-8862-3cd5709a2efe\") "
Jul 29 10:45:13.292315 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:13.292279    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59ae549e-eeb1-49eb-8862-3cd5709a2efe-kserve-provision-location\") pod \"59ae549e-eeb1-49eb-8862-3cd5709a2efe\" (UID: \"59ae549e-eeb1-49eb-8862-3cd5709a2efe\") "
Jul 29 10:45:13.292352 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:13.292310    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59ae549e-eeb1-49eb-8862-3cd5709a2efe-cabundle-cert\") pod \"59ae549e-eeb1-49eb-8862-3cd5709a2efe\" (UID: \"59ae549e-eeb1-49eb-8862-3cd5709a2efe\") "
Jul 29 10:45:13.292386 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:13.292371    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9q66h\" (UniqueName: \"kubernetes.io/projected/59ae549e-eeb1-49eb-8862-3cd5709a2efe-kube-api-access-9q66h\") pod \"59ae549e-eeb1-49eb-8862-3cd5709a2efe\" (UID: \"59ae549e-eeb1-49eb-8862-3cd5709a2efe\") "
Jul 29 10:45:13.292434 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:13.292420    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59ae549e-eeb1-49eb-8862-3cd5709a2efe-proxy-tls\") pod \"59ae549e-eeb1-49eb-8862-3cd5709a2efe\" (UID: \"59ae549e-eeb1-49eb-8862-3cd5709a2efe\") "
Jul 29 10:45:13.292609 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:13.292582    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ae549e-eeb1-49eb-8862-3cd5709a2efe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "59ae549e-eeb1-49eb-8862-3cd5709a2efe" (UID: "59ae549e-eeb1-49eb-8862-3cd5709a2efe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:45:13.292690 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:13.292609    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59ae549e-eeb1-49eb-8862-3cd5709a2efe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "59ae549e-eeb1-49eb-8862-3cd5709a2efe" (UID: "59ae549e-eeb1-49eb-8862-3cd5709a2efe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:45:13.292748 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:13.292700    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59ae549e-eeb1-49eb-8862-3cd5709a2efe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "59ae549e-eeb1-49eb-8862-3cd5709a2efe" (UID: "59ae549e-eeb1-49eb-8862-3cd5709a2efe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:45:13.294335 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:13.294304    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ae549e-eeb1-49eb-8862-3cd5709a2efe-kube-api-access-9q66h" (OuterVolumeSpecName: "kube-api-access-9q66h") pod "59ae549e-eeb1-49eb-8862-3cd5709a2efe" (UID: "59ae549e-eeb1-49eb-8862-3cd5709a2efe"). InnerVolumeSpecName "kube-api-access-9q66h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:45:13.294335 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:13.294328    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59ae549e-eeb1-49eb-8862-3cd5709a2efe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "59ae549e-eeb1-49eb-8862-3cd5709a2efe" (UID: "59ae549e-eeb1-49eb-8862-3cd5709a2efe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:45:13.393417 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:13.393338    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59ae549e-eeb1-49eb-8862-3cd5709a2efe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:45:13.393417 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:13.393367    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59ae549e-eeb1-49eb-8862-3cd5709a2efe-kserve-provision-location\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:45:13.393417 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:13.393377    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59ae549e-eeb1-49eb-8862-3cd5709a2efe-cabundle-cert\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:45:13.393417 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:13.393386    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q66h\" (UniqueName: \"kubernetes.io/projected/59ae549e-eeb1-49eb-8862-3cd5709a2efe-kube-api-access-9q66h\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:45:13.393417 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:13.393400    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59ae549e-eeb1-49eb-8862-3cd5709a2efe-proxy-tls\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:45:13.838564 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:13.838522    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01" path="/var/lib/kubelet/pods/e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01/volumes"
Jul 29 10:45:13.877635 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:13.877601    2569 generic.go:356] "Generic (PLEG): container finished" podID="59ae549e-eeb1-49eb-8862-3cd5709a2efe" containerID="f9d35c6cfcddcb98456d9d31a61a590fb7d936b60925515bd247e559e05e481f" exitCode=0
Jul 29 10:45:13.878048 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:13.877657    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5"
Jul 29 10:45:13.878048 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:13.877683    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5" event={"ID":"59ae549e-eeb1-49eb-8862-3cd5709a2efe","Type":"ContainerDied","Data":"f9d35c6cfcddcb98456d9d31a61a590fb7d936b60925515bd247e559e05e481f"}
Jul 29 10:45:13.878048 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:13.877743    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5" event={"ID":"59ae549e-eeb1-49eb-8862-3cd5709a2efe","Type":"ContainerDied","Data":"7b6c1109f3f40a67000c974086800c9f75547928d2d12687b54ec9976d113bc6"}
Jul 29 10:45:13.878048 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:13.877778    2569 scope.go:117] "RemoveContainer" containerID="adbca8dfec2fc968016d41c024508284930bc108d6b509979d398292e671d572"
Jul 29 10:45:13.886241 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:13.886218    2569 scope.go:117] "RemoveContainer" containerID="f9d35c6cfcddcb98456d9d31a61a590fb7d936b60925515bd247e559e05e481f"
Jul 29 10:45:13.893603 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:13.893582    2569 scope.go:117] "RemoveContainer" containerID="225b4026987ddfa6054123ad1ca0ada6821c27d3683e627cbb50c0f26d7a3d10"
Jul 29 10:45:13.897562 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:13.897539    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5"]
Jul 29 10:45:13.901291 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:13.901271    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2d850-predictor-bbb75dc6c-wmpv5"]
Jul 29 10:45:13.901563 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:13.901551    2569 scope.go:117] "RemoveContainer" containerID="adbca8dfec2fc968016d41c024508284930bc108d6b509979d398292e671d572"
Jul 29 10:45:13.901817 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:45:13.901802    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adbca8dfec2fc968016d41c024508284930bc108d6b509979d398292e671d572\": container with ID starting with adbca8dfec2fc968016d41c024508284930bc108d6b509979d398292e671d572 not found: ID does not exist" containerID="adbca8dfec2fc968016d41c024508284930bc108d6b509979d398292e671d572"
Jul 29 10:45:13.901869 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:13.901823    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adbca8dfec2fc968016d41c024508284930bc108d6b509979d398292e671d572"} err="failed to get container status \"adbca8dfec2fc968016d41c024508284930bc108d6b509979d398292e671d572\": rpc error: code = NotFound desc = could not find container \"adbca8dfec2fc968016d41c024508284930bc108d6b509979d398292e671d572\": container with ID starting with adbca8dfec2fc968016d41c024508284930bc108d6b509979d398292e671d572 not found: ID does not exist"
Jul 29 10:45:13.901869 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:13.901840    2569 scope.go:117] "RemoveContainer" containerID="f9d35c6cfcddcb98456d9d31a61a590fb7d936b60925515bd247e559e05e481f"
Jul 29 10:45:13.902051 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:45:13.902033    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9d35c6cfcddcb98456d9d31a61a590fb7d936b60925515bd247e559e05e481f\": container with ID starting with f9d35c6cfcddcb98456d9d31a61a590fb7d936b60925515bd247e559e05e481f not found: ID does not exist" containerID="f9d35c6cfcddcb98456d9d31a61a590fb7d936b60925515bd247e559e05e481f"
Jul 29 10:45:13.902110 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:13.902055    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9d35c6cfcddcb98456d9d31a61a590fb7d936b60925515bd247e559e05e481f"} err="failed to get container status \"f9d35c6cfcddcb98456d9d31a61a590fb7d936b60925515bd247e559e05e481f\": rpc error: code = NotFound desc = could not find container \"f9d35c6cfcddcb98456d9d31a61a590fb7d936b60925515bd247e559e05e481f\": container with ID starting with f9d35c6cfcddcb98456d9d31a61a590fb7d936b60925515bd247e559e05e481f not found: ID does not exist"
Jul 29 10:45:13.902110 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:13.902067    2569 scope.go:117] "RemoveContainer" containerID="225b4026987ddfa6054123ad1ca0ada6821c27d3683e627cbb50c0f26d7a3d10"
Jul 29 10:45:13.902330 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:45:13.902312    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"225b4026987ddfa6054123ad1ca0ada6821c27d3683e627cbb50c0f26d7a3d10\": container with ID starting with 225b4026987ddfa6054123ad1ca0ada6821c27d3683e627cbb50c0f26d7a3d10 not found: ID does not exist" containerID="225b4026987ddfa6054123ad1ca0ada6821c27d3683e627cbb50c0f26d7a3d10"
Jul 29 10:45:13.902378 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:13.902333    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"225b4026987ddfa6054123ad1ca0ada6821c27d3683e627cbb50c0f26d7a3d10"} err="failed to get container status \"225b4026987ddfa6054123ad1ca0ada6821c27d3683e627cbb50c0f26d7a3d10\": rpc error: code = NotFound desc = could not find container \"225b4026987ddfa6054123ad1ca0ada6821c27d3683e627cbb50c0f26d7a3d10\": container with ID starting with 225b4026987ddfa6054123ad1ca0ada6821c27d3683e627cbb50c0f26d7a3d10 not found: ID does not exist"
Jul 29 10:45:15.837524 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:15.837485    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59ae549e-eeb1-49eb-8862-3cd5709a2efe" path="/var/lib/kubelet/pods/59ae549e-eeb1-49eb-8862-3cd5709a2efe/volumes"
Jul 29 10:45:18.883479 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:18.883444    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4"
Jul 29 10:45:28.386868 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.386827    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:45:28.440613 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.440582    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx"]
Jul 29 10:45:28.440921 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.440907    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01" containerName="storage-initializer"
Jul 29 10:45:28.440966 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.440923    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01" containerName="storage-initializer"
Jul 29 10:45:28.440966 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.440934    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59ae549e-eeb1-49eb-8862-3cd5709a2efe" containerName="kube-rbac-proxy"
Jul 29 10:45:28.440966 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.440942    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="59ae549e-eeb1-49eb-8862-3cd5709a2efe" containerName="kube-rbac-proxy"
Jul 29 10:45:28.440966 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.440959    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01" containerName="kube-rbac-proxy"
Jul 29 10:45:28.440966 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.440965    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01" containerName="kube-rbac-proxy"
Jul 29 10:45:28.441134 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.440973    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01" containerName="kserve-container"
Jul 29 10:45:28.441134 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.440980    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01" containerName="kserve-container"
Jul 29 10:45:28.441134 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.440988    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59ae549e-eeb1-49eb-8862-3cd5709a2efe" containerName="storage-initializer"
Jul 29 10:45:28.441134 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.440993    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="59ae549e-eeb1-49eb-8862-3cd5709a2efe" containerName="storage-initializer"
Jul 29 10:45:28.441134 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.440998    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59ae549e-eeb1-49eb-8862-3cd5709a2efe" containerName="kserve-container"
Jul 29 10:45:28.441134 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.441005    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="59ae549e-eeb1-49eb-8862-3cd5709a2efe" containerName="kserve-container"
Jul 29 10:45:28.441134 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.441060    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="59ae549e-eeb1-49eb-8862-3cd5709a2efe" containerName="kserve-container"
Jul 29 10:45:28.441134 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.441080    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01" containerName="kserve-container"
Jul 29 10:45:28.441134 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.441087    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3ea49c6-c3ff-4bc8-8f72-d9ad35d3cc01" containerName="kube-rbac-proxy"
Jul 29 10:45:28.441134 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.441098    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="59ae549e-eeb1-49eb-8862-3cd5709a2efe" containerName="kube-rbac-proxy"
Jul 29 10:45:28.445727 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.445709    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx"
Jul 29 10:45:28.449086 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.449052    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 10:45:28.449872 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.449847    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-66e3d-predictor-serving-cert\""
Jul 29 10:45:28.450020 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.449874    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-66e3d-kube-rbac-proxy-sar-config\""
Jul 29 10:45:28.450020 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.449877    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 10:45:28.457153 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.457133    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx"]
Jul 29 10:45:28.516408 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.516372    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa-kserve-provision-location\") pod \"isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx\" (UID: \"fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa\") " pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx"
Jul 29 10:45:28.516550 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.516432    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa-proxy-tls\") pod \"isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx\" (UID: \"fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa\") " pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx"
Jul 29 10:45:28.516550 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.516465    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72blq\" (UniqueName: \"kubernetes.io/projected/fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa-kube-api-access-72blq\") pod \"isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx\" (UID: \"fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa\") " pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx"
Jul 29 10:45:28.516550 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.516512    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa-cabundle-cert\") pod \"isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx\" (UID: \"fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa\") " pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx"
Jul 29 10:45:28.516681 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.516557    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx\" (UID: \"fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa\") " pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx"
Jul 29 10:45:28.617637 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.617603    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa-cabundle-cert\") pod \"isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx\" (UID: \"fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa\") " pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx"
Jul 29 10:45:28.617637 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.617647    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx\" (UID: \"fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa\") " pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx"
Jul 29 10:45:28.617900 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.617687    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa-kserve-provision-location\") pod \"isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx\" (UID: \"fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa\") " pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx"
Jul 29 10:45:28.617900 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.617731    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa-proxy-tls\") pod \"isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx\" (UID: \"fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa\") " pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx"
Jul 29 10:45:28.617900 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.617852    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72blq\" (UniqueName: \"kubernetes.io/projected/fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa-kube-api-access-72blq\") pod \"isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx\" (UID: \"fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa\") " pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx"
Jul 29 10:45:28.618156 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.618136    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa-kserve-provision-location\") pod \"isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx\" (UID: \"fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa\") " pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx"
Jul 29 10:45:28.618461 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.618428    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx\" (UID: \"fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa\") " pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx"
Jul 29 10:45:28.618461 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.618434    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa-cabundle-cert\") pod \"isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx\" (UID: \"fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa\") " pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx"
Jul 29 10:45:28.620257 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.620240    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa-proxy-tls\") pod \"isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx\" (UID: \"fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa\") " pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx"
Jul 29 10:45:28.626783 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.626754    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72blq\" (UniqueName: \"kubernetes.io/projected/fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa-kube-api-access-72blq\") pod \"isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx\" (UID: \"fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa\") " pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx"
Jul 29 10:45:28.635745 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.635723    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:45:28.756264 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.756227    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx"
Jul 29 10:45:28.887877 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.887847    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx"]
Jul 29 10:45:28.891241 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:45:28.891214    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd2c5a5_64f6_4a6f_aa0e_99ad6d55d7fa.slice/crio-d1667c745afe465087fde86aff3692784c2f64e7297d184616f50b2d503f5e15 WatchSource:0}: Error finding container d1667c745afe465087fde86aff3692784c2f64e7297d184616f50b2d503f5e15: Status 404 returned error can't find the container with id d1667c745afe465087fde86aff3692784c2f64e7297d184616f50b2d503f5e15
Jul 29 10:45:28.939408 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:28.939379    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" event={"ID":"fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa","Type":"ContainerStarted","Data":"d1667c745afe465087fde86aff3692784c2f64e7297d184616f50b2d503f5e15"}
Jul 29 10:45:29.944931 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:29.944897    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" event={"ID":"fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa","Type":"ContainerStarted","Data":"ff8b188482912c59df78bc2f0dabc527b6a1d83f3370140ff4b3fd86e899e8f4"}
Jul 29 10:45:32.958925 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:32.958890    2569 generic.go:356] "Generic (PLEG): container finished" podID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerID="ff8b188482912c59df78bc2f0dabc527b6a1d83f3370140ff4b3fd86e899e8f4" exitCode=0
Jul 29 10:45:32.959309 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:32.958954    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" event={"ID":"fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa","Type":"ContainerDied","Data":"ff8b188482912c59df78bc2f0dabc527b6a1d83f3370140ff4b3fd86e899e8f4"}
Jul 29 10:45:33.965556 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:33.965516    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" event={"ID":"fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa","Type":"ContainerStarted","Data":"f61544f89776e75f0302bb108c6eb099ff9b71b9a35f2bdb2776c386523d74ed"}
Jul 29 10:45:33.965989 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:33.965589    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" event={"ID":"fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa","Type":"ContainerStarted","Data":"84ff74c0797ad5f78764d3f277645e457716989653766e3f550dbee6dab74592"}
Jul 29 10:45:33.965989 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:33.965618    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" event={"ID":"fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa","Type":"ContainerStarted","Data":"088d8b2c9e08d835aa73a2190fb6da074bb4e4ead48c972dc7d82dc849d4615b"}
Jul 29 10:45:33.965989 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:33.965822    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx"
Jul 29 10:45:33.988813 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:33.988764    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" podStartSLOduration=2.339216498 podStartE2EDuration="5.988749133s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.649532635s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:45:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:45:33.986218262 +0000 UTC m=+1010.711008936" watchObservedRunningTime="2026-07-29 10:45:33.988749133 +0000 UTC m=+1010.713539806"
Jul 29 10:45:34.970022 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:34.969986    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx"
Jul 29 10:45:34.970431 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:34.970061    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx"
Jul 29 10:45:34.971405 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:34.971369    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 29 10:45:34.972002 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:34.971980    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:45:35.973915 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:35.973873    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 29 10:45:35.974407 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:35.974250    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:45:40.978234 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:40.978201    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx"
Jul 29 10:45:40.978865 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:40.978829    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 29 10:45:40.979057 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:40.979032    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:45:50.979095 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:50.979030    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 29 10:45:50.979486 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:45:50.979441    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:46:00.978699 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:46:00.978650    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 29 10:46:00.979149 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:46:00.979102    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:46:09.409060 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:46:09.409023    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:46:10.979093 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:46:10.979030    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 29 10:46:10.979587 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:46:10.979452    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:46:20.979387 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:46:20.979336    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 29 10:46:20.979820 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:46:20.979794    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:46:30.979034 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:46:30.978985    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 29 10:46:30.979480 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:46:30.979398    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:46:40.455976 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:46:40.455928    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:46:40.979544 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:46:40.979495    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 29 10:46:40.979903 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:46:40.979879    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:46:44.832781 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:46:44.832731    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 29 10:46:44.833409 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:46:44.832992    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:46:54.833244 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:46:54.833206    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx"
Jul 29 10:46:54.833819 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:46:54.833721    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx"
Jul 29 10:47:03.440560 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:03.440529    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4_01822d12-2967-48a8-bd3e-61a43409476b/kserve-container/0.log"
Jul 29 10:47:03.649266 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:03.649234    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx"]
Jul 29 10:47:03.649553 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:03.649519    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="kserve-container" containerID="cri-o://088d8b2c9e08d835aa73a2190fb6da074bb4e4ead48c972dc7d82dc849d4615b" gracePeriod=30
Jul 29 10:47:03.649553 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:03.649546    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="kube-rbac-proxy" containerID="cri-o://84ff74c0797ad5f78764d3f277645e457716989653766e3f550dbee6dab74592" gracePeriod=30
Jul 29 10:47:03.649728 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:03.649540    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="agent" containerID="cri-o://f61544f89776e75f0302bb108c6eb099ff9b71b9a35f2bdb2776c386523d74ed" gracePeriod=30
Jul 29 10:47:03.664776 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:03.664737    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:47:03.725007 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:03.724919    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt"]
Jul 29 10:47:03.728368 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:03.728350    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt"
Jul 29 10:47:03.730941 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:03.730917    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-bf0a1-predictor-serving-cert\""
Jul 29 10:47:03.731062 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:03.731030    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-bf0a1-kube-rbac-proxy-sar-config\""
Jul 29 10:47:03.738284 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:03.738259    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt"]
Jul 29 10:47:03.767933 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:03.767904    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:47:03.775395 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:03.775364    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4"]
Jul 29 10:47:03.775653 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:03.775621    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4" podUID="01822d12-2967-48a8-bd3e-61a43409476b" containerName="kserve-container" containerID="cri-o://0c69ad0e7e3ddba9d15107d540d79c546ef2a19551969e97ed17f9e27747c2fb" gracePeriod=30
Jul 29 10:47:03.775966 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:03.775856    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4" podUID="01822d12-2967-48a8-bd3e-61a43409476b" containerName="kube-rbac-proxy" containerID="cri-o://dc59abb121ae01fb5ac238c53dff4bf3515a308314a267c1c26cf7cc26c0f51f" gracePeriod=30
Jul 29 10:47:03.833761 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:03.833717    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt\" (UID: \"2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt"
Jul 29 10:47:03.833947 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:03.833775    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p52t2\" (UniqueName: \"kubernetes.io/projected/2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade-kube-api-access-p52t2\") pod \"isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt\" (UID: \"2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt"
Jul 29 10:47:03.833947 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:03.833819    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt\" (UID: \"2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt"
Jul 29 10:47:03.833947 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:03.833846    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade-cabundle-cert\") pod \"isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt\" (UID: \"2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt"
Jul 29 10:47:03.834145 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:03.833945    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade-proxy-tls\") pod \"isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt\" (UID: \"2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt"
Jul 29 10:47:03.866473 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:03.866434    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:47:03.879500 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:03.879462    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4" podUID="01822d12-2967-48a8-bd3e-61a43409476b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 29 10:47:03.935234 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:03.935200    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p52t2\" (UniqueName: \"kubernetes.io/projected/2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade-kube-api-access-p52t2\") pod \"isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt\" (UID: \"2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt"
Jul 29 10:47:03.935435 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:03.935262    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt\" (UID: \"2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt"
Jul 29 10:47:03.935435 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:03.935308    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade-cabundle-cert\") pod \"isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt\" (UID: \"2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt"
Jul 29 10:47:03.935746 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:03.935720    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt\" (UID: \"2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt"
Jul 29 10:47:03.935746 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:03.935734    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade-proxy-tls\") pod \"isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt\" (UID: \"2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt"
Jul 29 10:47:03.935963 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:03.935812    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt\" (UID: \"2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt"
Jul 29 10:47:03.936175 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:47:03.936154    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-serving-cert: secret "isvc-sklearn-scale-raw-bf0a1-predictor-serving-cert" not found
Jul 29 10:47:03.936263 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:47:03.936225    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade-proxy-tls podName:2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade nodeName:}" failed. No retries permitted until 2026-07-29 10:47:04.436205984 +0000 UTC m=+1101.160996658 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade-proxy-tls") pod "isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt" (UID: "2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade") : secret "isvc-sklearn-scale-raw-bf0a1-predictor-serving-cert" not found
Jul 29 10:47:03.936330 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:03.936160    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade-cabundle-cert\") pod \"isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt\" (UID: \"2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt"
Jul 29 10:47:03.936647 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:03.936623    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt\" (UID: \"2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt"
Jul 29 10:47:03.947413 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:03.947384    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p52t2\" (UniqueName: \"kubernetes.io/projected/2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade-kube-api-access-p52t2\") pod \"isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt\" (UID: \"2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt"
Jul 29 10:47:04.014868 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.014837    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:47:04.023868 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.023847    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4"
Jul 29 10:47:04.137859 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.137821    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnhvp\" (UniqueName: \"kubernetes.io/projected/01822d12-2967-48a8-bd3e-61a43409476b-kube-api-access-bnhvp\") pod \"01822d12-2967-48a8-bd3e-61a43409476b\" (UID: \"01822d12-2967-48a8-bd3e-61a43409476b\") "
Jul 29 10:47:04.137859 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.137865    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01822d12-2967-48a8-bd3e-61a43409476b-kube-rbac-proxy-sar-config\") pod \"01822d12-2967-48a8-bd3e-61a43409476b\" (UID: \"01822d12-2967-48a8-bd3e-61a43409476b\") "
Jul 29 10:47:04.138167 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.137907    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01822d12-2967-48a8-bd3e-61a43409476b-proxy-tls\") pod \"01822d12-2967-48a8-bd3e-61a43409476b\" (UID: \"01822d12-2967-48a8-bd3e-61a43409476b\") "
Jul 29 10:47:04.138309 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.138285    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01822d12-2967-48a8-bd3e-61a43409476b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "01822d12-2967-48a8-bd3e-61a43409476b" (UID: "01822d12-2967-48a8-bd3e-61a43409476b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:47:04.140039 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.140015    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01822d12-2967-48a8-bd3e-61a43409476b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "01822d12-2967-48a8-bd3e-61a43409476b" (UID: "01822d12-2967-48a8-bd3e-61a43409476b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:47:04.140162 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.140099    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01822d12-2967-48a8-bd3e-61a43409476b-kube-api-access-bnhvp" (OuterVolumeSpecName: "kube-api-access-bnhvp") pod "01822d12-2967-48a8-bd3e-61a43409476b" (UID: "01822d12-2967-48a8-bd3e-61a43409476b"). InnerVolumeSpecName "kube-api-access-bnhvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:47:04.239039 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.238998    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnhvp\" (UniqueName: \"kubernetes.io/projected/01822d12-2967-48a8-bd3e-61a43409476b-kube-api-access-bnhvp\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:47:04.239039 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.239032    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01822d12-2967-48a8-bd3e-61a43409476b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:47:04.239039 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.239043    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01822d12-2967-48a8-bd3e-61a43409476b-proxy-tls\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:47:04.339446 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.339357    2569 generic.go:356] "Generic (PLEG): container finished" podID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerID="84ff74c0797ad5f78764d3f277645e457716989653766e3f550dbee6dab74592" exitCode=2
Jul 29 10:47:04.339594 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.339434    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" event={"ID":"fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa","Type":"ContainerDied","Data":"84ff74c0797ad5f78764d3f277645e457716989653766e3f550dbee6dab74592"}
Jul 29 10:47:04.341003 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.340980    2569 generic.go:356] "Generic (PLEG): container finished" podID="01822d12-2967-48a8-bd3e-61a43409476b" containerID="dc59abb121ae01fb5ac238c53dff4bf3515a308314a267c1c26cf7cc26c0f51f" exitCode=2
Jul 29 10:47:04.341003 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.341002    2569 generic.go:356] "Generic (PLEG): container finished" podID="01822d12-2967-48a8-bd3e-61a43409476b" containerID="0c69ad0e7e3ddba9d15107d540d79c546ef2a19551969e97ed17f9e27747c2fb" exitCode=2
Jul 29 10:47:04.341163 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.341027    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4" event={"ID":"01822d12-2967-48a8-bd3e-61a43409476b","Type":"ContainerDied","Data":"dc59abb121ae01fb5ac238c53dff4bf3515a308314a267c1c26cf7cc26c0f51f"}
Jul 29 10:47:04.341163 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.341034    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4"
Jul 29 10:47:04.341163 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.341066    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4" event={"ID":"01822d12-2967-48a8-bd3e-61a43409476b","Type":"ContainerDied","Data":"0c69ad0e7e3ddba9d15107d540d79c546ef2a19551969e97ed17f9e27747c2fb"}
Jul 29 10:47:04.341163 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.341104    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4" event={"ID":"01822d12-2967-48a8-bd3e-61a43409476b","Type":"ContainerDied","Data":"ba0914fd9fe64e29b061d9b80b28b087d615400119f1270128bd1758d59d144a"}
Jul 29 10:47:04.341163 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.341106    2569 scope.go:117] "RemoveContainer" containerID="dc59abb121ae01fb5ac238c53dff4bf3515a308314a267c1c26cf7cc26c0f51f"
Jul 29 10:47:04.349434 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.349415    2569 scope.go:117] "RemoveContainer" containerID="0c69ad0e7e3ddba9d15107d540d79c546ef2a19551969e97ed17f9e27747c2fb"
Jul 29 10:47:04.356738 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.356719    2569 scope.go:117] "RemoveContainer" containerID="dc59abb121ae01fb5ac238c53dff4bf3515a308314a267c1c26cf7cc26c0f51f"
Jul 29 10:47:04.356985 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:47:04.356963    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc59abb121ae01fb5ac238c53dff4bf3515a308314a267c1c26cf7cc26c0f51f\": container with ID starting with dc59abb121ae01fb5ac238c53dff4bf3515a308314a267c1c26cf7cc26c0f51f not found: ID does not exist" containerID="dc59abb121ae01fb5ac238c53dff4bf3515a308314a267c1c26cf7cc26c0f51f"
Jul 29 10:47:04.357029 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.356992    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc59abb121ae01fb5ac238c53dff4bf3515a308314a267c1c26cf7cc26c0f51f"} err="failed to get container status \"dc59abb121ae01fb5ac238c53dff4bf3515a308314a267c1c26cf7cc26c0f51f\": rpc error: code = NotFound desc = could not find container \"dc59abb121ae01fb5ac238c53dff4bf3515a308314a267c1c26cf7cc26c0f51f\": container with ID starting with dc59abb121ae01fb5ac238c53dff4bf3515a308314a267c1c26cf7cc26c0f51f not found: ID does not exist"
Jul 29 10:47:04.357029 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.357009    2569 scope.go:117] "RemoveContainer" containerID="0c69ad0e7e3ddba9d15107d540d79c546ef2a19551969e97ed17f9e27747c2fb"
Jul 29 10:47:04.357244 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:47:04.357229    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c69ad0e7e3ddba9d15107d540d79c546ef2a19551969e97ed17f9e27747c2fb\": container with ID starting with 0c69ad0e7e3ddba9d15107d540d79c546ef2a19551969e97ed17f9e27747c2fb not found: ID does not exist" containerID="0c69ad0e7e3ddba9d15107d540d79c546ef2a19551969e97ed17f9e27747c2fb"
Jul 29 10:47:04.357285 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.357248    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c69ad0e7e3ddba9d15107d540d79c546ef2a19551969e97ed17f9e27747c2fb"} err="failed to get container status \"0c69ad0e7e3ddba9d15107d540d79c546ef2a19551969e97ed17f9e27747c2fb\": rpc error: code = NotFound desc = could not find container \"0c69ad0e7e3ddba9d15107d540d79c546ef2a19551969e97ed17f9e27747c2fb\": container with ID starting with 0c69ad0e7e3ddba9d15107d540d79c546ef2a19551969e97ed17f9e27747c2fb not found: ID does not exist"
Jul 29 10:47:04.357285 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.357260    2569 scope.go:117] "RemoveContainer" containerID="dc59abb121ae01fb5ac238c53dff4bf3515a308314a267c1c26cf7cc26c0f51f"
Jul 29 10:47:04.357459 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.357445    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc59abb121ae01fb5ac238c53dff4bf3515a308314a267c1c26cf7cc26c0f51f"} err="failed to get container status \"dc59abb121ae01fb5ac238c53dff4bf3515a308314a267c1c26cf7cc26c0f51f\": rpc error: code = NotFound desc = could not find container \"dc59abb121ae01fb5ac238c53dff4bf3515a308314a267c1c26cf7cc26c0f51f\": container with ID starting with dc59abb121ae01fb5ac238c53dff4bf3515a308314a267c1c26cf7cc26c0f51f not found: ID does not exist"
Jul 29 10:47:04.357505 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.357459    2569 scope.go:117] "RemoveContainer" containerID="0c69ad0e7e3ddba9d15107d540d79c546ef2a19551969e97ed17f9e27747c2fb"
Jul 29 10:47:04.357662 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.357647    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c69ad0e7e3ddba9d15107d540d79c546ef2a19551969e97ed17f9e27747c2fb"} err="failed to get container status \"0c69ad0e7e3ddba9d15107d540d79c546ef2a19551969e97ed17f9e27747c2fb\": rpc error: code = NotFound desc = could not find container \"0c69ad0e7e3ddba9d15107d540d79c546ef2a19551969e97ed17f9e27747c2fb\": container with ID starting with 0c69ad0e7e3ddba9d15107d540d79c546ef2a19551969e97ed17f9e27747c2fb not found: ID does not exist"
Jul 29 10:47:04.362912 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.362891    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4"]
Jul 29 10:47:04.366167 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.366138    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-66e3d-predictor-86bbb6d7f6-bb9h4"]
Jul 29 10:47:04.440478 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.440434    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade-proxy-tls\") pod \"isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt\" (UID: \"2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt"
Jul 29 10:47:04.442865 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.442842    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade-proxy-tls\") pod \"isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt\" (UID: \"2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt"
Jul 29 10:47:04.639730 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.639643    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt"
Jul 29 10:47:04.769888 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.769855    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt"]
Jul 29 10:47:04.773919 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:47:04.773892    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fd52ae4_70dc_4cad_bd2c_b9fd73bb2ade.slice/crio-e418aa30439964727b24679ac91c5102dc01cddc3ddc3f64a1ac8f3ecb8b155f WatchSource:0}: Error finding container e418aa30439964727b24679ac91c5102dc01cddc3ddc3f64a1ac8f3ecb8b155f: Status 404 returned error can't find the container with id e418aa30439964727b24679ac91c5102dc01cddc3ddc3f64a1ac8f3ecb8b155f
Jul 29 10:47:04.832603 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.832565    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 29 10:47:04.834298 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:04.834264    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:47:05.348181 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:05.348141    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt" event={"ID":"2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade","Type":"ContainerStarted","Data":"1caf7c64849fec768561620bf251259bc2581446d33d5f3e3c3b07e2959b2250"}
Jul 29 10:47:05.348383 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:05.348206    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt" event={"ID":"2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade","Type":"ContainerStarted","Data":"e418aa30439964727b24679ac91c5102dc01cddc3ddc3f64a1ac8f3ecb8b155f"}
Jul 29 10:47:05.837236 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:05.837204    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01822d12-2967-48a8-bd3e-61a43409476b" path="/var/lib/kubelet/pods/01822d12-2967-48a8-bd3e-61a43409476b/volumes"
Jul 29 10:47:05.974980 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:05.974928    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Jul 29 10:47:09.367233 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:09.367199    2569 generic.go:356] "Generic (PLEG): container finished" podID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerID="088d8b2c9e08d835aa73a2190fb6da074bb4e4ead48c972dc7d82dc849d4615b" exitCode=0
Jul 29 10:47:09.367631 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:09.367275    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" event={"ID":"fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa","Type":"ContainerDied","Data":"088d8b2c9e08d835aa73a2190fb6da074bb4e4ead48c972dc7d82dc849d4615b"}
Jul 29 10:47:09.368670 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:09.368645    2569 generic.go:356] "Generic (PLEG): container finished" podID="2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" containerID="1caf7c64849fec768561620bf251259bc2581446d33d5f3e3c3b07e2959b2250" exitCode=0
Jul 29 10:47:09.368733 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:09.368715    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt" event={"ID":"2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade","Type":"ContainerDied","Data":"1caf7c64849fec768561620bf251259bc2581446d33d5f3e3c3b07e2959b2250"}
Jul 29 10:47:10.374243 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:10.374205    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt" event={"ID":"2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade","Type":"ContainerStarted","Data":"8c0f1f8ef17260aba04b8bd395b2b78497133d621e8344670ff1374841649557"}
Jul 29 10:47:10.374599 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:10.374271    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt" event={"ID":"2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade","Type":"ContainerStarted","Data":"d7b0d439a5541f61222321347c9dabc0d195d64203ad201d5b3a43c49b4ba3de"}
Jul 29 10:47:10.374599 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:10.374551    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt"
Jul 29 10:47:10.374710 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:10.374693    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt"
Jul 29 10:47:10.375906 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:10.375880    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt" podUID="2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 29 10:47:10.397864 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:10.397816    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt" podStartSLOduration=3.698250685 podStartE2EDuration="7.397800087s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.699549402s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:47:10.396767302 +0000 UTC m=+1107.121557976" watchObservedRunningTime="2026-07-29 10:47:10.397800087 +0000 UTC m=+1107.122590762"
Jul 29 10:47:10.974962 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:10.974918    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Jul 29 10:47:11.379136 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:11.379023    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt" podUID="2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 29 10:47:14.832654 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:14.832613    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 29 10:47:14.834337 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:14.834307    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:47:15.974464 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:15.974422    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Jul 29 10:47:15.974856 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:15.974558    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx"
Jul 29 10:47:16.383507 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:16.383422    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt"
Jul 29 10:47:16.383953 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:16.383919    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt" podUID="2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 29 10:47:20.974058 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:20.974014    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Jul 29 10:47:24.833033 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:24.832971    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 29 10:47:24.833477 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:24.833182    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx"
Jul 29 10:47:24.834006 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:24.833980    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:47:24.834135 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:24.834121    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx"
Jul 29 10:47:25.974225 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:25.974182    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Jul 29 10:47:26.384419 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:26.384321    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt" podUID="2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 29 10:47:30.974217 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:30.974172    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Jul 29 10:47:33.803878 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:33.803850    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx"
Jul 29 10:47:33.887987 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:33.887952    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa-cabundle-cert\") pod \"fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa\" (UID: \"fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa\") "
Jul 29 10:47:33.888169 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:33.888026    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa-kube-rbac-proxy-sar-config\") pod \"fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa\" (UID: \"fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa\") "
Jul 29 10:47:33.888169 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:33.888055    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa-kserve-provision-location\") pod \"fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa\" (UID: \"fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa\") "
Jul 29 10:47:33.888169 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:33.888100    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa-proxy-tls\") pod \"fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa\" (UID: \"fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa\") "
Jul 29 10:47:33.888343 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:33.888167    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72blq\" (UniqueName: \"kubernetes.io/projected/fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa-kube-api-access-72blq\") pod \"fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa\" (UID: \"fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa\") "
Jul 29 10:47:33.888400 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:33.888370    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" (UID: "fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:47:33.888454 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:33.888389    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" (UID: "fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:47:33.888454 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:33.888429    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" (UID: "fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:47:33.890317 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:33.890286    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa-kube-api-access-72blq" (OuterVolumeSpecName: "kube-api-access-72blq") pod "fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" (UID: "fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa"). InnerVolumeSpecName "kube-api-access-72blq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:47:33.890429 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:33.890329    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" (UID: "fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:47:33.989085 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:33.989034    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:47:33.989085 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:33.989063    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa-kserve-provision-location\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:47:33.989284 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:33.989095    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa-proxy-tls\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:47:33.989284 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:33.989105    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72blq\" (UniqueName: \"kubernetes.io/projected/fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa-kube-api-access-72blq\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:47:33.989284 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:33.989114    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa-cabundle-cert\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:47:34.472638 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:34.472601    2569 generic.go:356] "Generic (PLEG): container finished" podID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerID="f61544f89776e75f0302bb108c6eb099ff9b71b9a35f2bdb2776c386523d74ed" exitCode=0
Jul 29 10:47:34.472807 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:34.472671    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx"
Jul 29 10:47:34.472807 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:34.472686    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" event={"ID":"fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa","Type":"ContainerDied","Data":"f61544f89776e75f0302bb108c6eb099ff9b71b9a35f2bdb2776c386523d74ed"}
Jul 29 10:47:34.472807 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:34.472759    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx" event={"ID":"fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa","Type":"ContainerDied","Data":"d1667c745afe465087fde86aff3692784c2f64e7297d184616f50b2d503f5e15"}
Jul 29 10:47:34.472807 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:34.472793    2569 scope.go:117] "RemoveContainer" containerID="f61544f89776e75f0302bb108c6eb099ff9b71b9a35f2bdb2776c386523d74ed"
Jul 29 10:47:34.481503 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:34.481486    2569 scope.go:117] "RemoveContainer" containerID="84ff74c0797ad5f78764d3f277645e457716989653766e3f550dbee6dab74592"
Jul 29 10:47:34.488803 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:34.488784    2569 scope.go:117] "RemoveContainer" containerID="088d8b2c9e08d835aa73a2190fb6da074bb4e4ead48c972dc7d82dc849d4615b"
Jul 29 10:47:34.495364 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:34.495340    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx"]
Jul 29 10:47:34.496099 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:34.496061    2569 scope.go:117] "RemoveContainer" containerID="ff8b188482912c59df78bc2f0dabc527b6a1d83f3370140ff4b3fd86e899e8f4"
Jul 29 10:47:34.499765 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:34.499742    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-66e3d-predictor-67955bccd6-bt7vx"]
Jul 29 10:47:34.503823 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:34.503805    2569 scope.go:117] "RemoveContainer" containerID="f61544f89776e75f0302bb108c6eb099ff9b71b9a35f2bdb2776c386523d74ed"
Jul 29 10:47:34.504115 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:47:34.504093    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f61544f89776e75f0302bb108c6eb099ff9b71b9a35f2bdb2776c386523d74ed\": container with ID starting with f61544f89776e75f0302bb108c6eb099ff9b71b9a35f2bdb2776c386523d74ed not found: ID does not exist" containerID="f61544f89776e75f0302bb108c6eb099ff9b71b9a35f2bdb2776c386523d74ed"
Jul 29 10:47:34.504218 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:34.504121    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f61544f89776e75f0302bb108c6eb099ff9b71b9a35f2bdb2776c386523d74ed"} err="failed to get container status \"f61544f89776e75f0302bb108c6eb099ff9b71b9a35f2bdb2776c386523d74ed\": rpc error: code = NotFound desc = could not find container \"f61544f89776e75f0302bb108c6eb099ff9b71b9a35f2bdb2776c386523d74ed\": container with ID starting with f61544f89776e75f0302bb108c6eb099ff9b71b9a35f2bdb2776c386523d74ed not found: ID does not exist"
Jul 29 10:47:34.504218 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:34.504141    2569 scope.go:117] "RemoveContainer" containerID="84ff74c0797ad5f78764d3f277645e457716989653766e3f550dbee6dab74592"
Jul 29 10:47:34.504413 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:47:34.504392    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84ff74c0797ad5f78764d3f277645e457716989653766e3f550dbee6dab74592\": container with ID starting with 84ff74c0797ad5f78764d3f277645e457716989653766e3f550dbee6dab74592 not found: ID does not exist" containerID="84ff74c0797ad5f78764d3f277645e457716989653766e3f550dbee6dab74592"
Jul 29 10:47:34.504454 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:34.504423    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84ff74c0797ad5f78764d3f277645e457716989653766e3f550dbee6dab74592"} err="failed to get container status \"84ff74c0797ad5f78764d3f277645e457716989653766e3f550dbee6dab74592\": rpc error: code = NotFound desc = could not find container \"84ff74c0797ad5f78764d3f277645e457716989653766e3f550dbee6dab74592\": container with ID starting with 84ff74c0797ad5f78764d3f277645e457716989653766e3f550dbee6dab74592 not found: ID does not exist"
Jul 29 10:47:34.504454 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:34.504436    2569 scope.go:117] "RemoveContainer" containerID="088d8b2c9e08d835aa73a2190fb6da074bb4e4ead48c972dc7d82dc849d4615b"
Jul 29 10:47:34.504644 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:47:34.504628    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"088d8b2c9e08d835aa73a2190fb6da074bb4e4ead48c972dc7d82dc849d4615b\": container with ID starting with 088d8b2c9e08d835aa73a2190fb6da074bb4e4ead48c972dc7d82dc849d4615b not found: ID does not exist" containerID="088d8b2c9e08d835aa73a2190fb6da074bb4e4ead48c972dc7d82dc849d4615b"
Jul 29 10:47:34.504684 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:34.504650    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"088d8b2c9e08d835aa73a2190fb6da074bb4e4ead48c972dc7d82dc849d4615b"} err="failed to get container status \"088d8b2c9e08d835aa73a2190fb6da074bb4e4ead48c972dc7d82dc849d4615b\": rpc error: code = NotFound desc = could not find container \"088d8b2c9e08d835aa73a2190fb6da074bb4e4ead48c972dc7d82dc849d4615b\": container with ID starting with 088d8b2c9e08d835aa73a2190fb6da074bb4e4ead48c972dc7d82dc849d4615b not found: ID does not exist"
Jul 29 10:47:34.504684 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:34.504668    2569 scope.go:117] "RemoveContainer" containerID="ff8b188482912c59df78bc2f0dabc527b6a1d83f3370140ff4b3fd86e899e8f4"
Jul 29 10:47:34.504870 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:47:34.504855    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff8b188482912c59df78bc2f0dabc527b6a1d83f3370140ff4b3fd86e899e8f4\": container with ID starting with ff8b188482912c59df78bc2f0dabc527b6a1d83f3370140ff4b3fd86e899e8f4 not found: ID does not exist" containerID="ff8b188482912c59df78bc2f0dabc527b6a1d83f3370140ff4b3fd86e899e8f4"
Jul 29 10:47:34.504933 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:34.504874    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff8b188482912c59df78bc2f0dabc527b6a1d83f3370140ff4b3fd86e899e8f4"} err="failed to get container status \"ff8b188482912c59df78bc2f0dabc527b6a1d83f3370140ff4b3fd86e899e8f4\": rpc error: code = NotFound desc = could not find container \"ff8b188482912c59df78bc2f0dabc527b6a1d83f3370140ff4b3fd86e899e8f4\": container with ID starting with ff8b188482912c59df78bc2f0dabc527b6a1d83f3370140ff4b3fd86e899e8f4 not found: ID does not exist"
Jul 29 10:47:35.837626 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:35.837583    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" path="/var/lib/kubelet/pods/fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa/volumes"
Jul 29 10:47:36.384151 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:36.384115    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt" podUID="2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 29 10:47:46.384408 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:46.384366    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt" podUID="2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 29 10:47:56.384815 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:47:56.384774    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt" podUID="2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 29 10:48:06.384588 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:48:06.384503    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt" podUID="2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 29 10:48:16.384718 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:48:16.384672    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt" podUID="2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 29 10:48:26.384651 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:48:26.384611    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt" podUID="2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 29 10:48:36.384135 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:48:36.384060    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt" podUID="2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 29 10:48:43.777182 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:48:43.777146    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-pvdgw_afd0d65c-bbfc-49f0-81b6-e855fd81da7f/console-operator/1.log"
Jul 29 10:48:43.778714 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:48:43.778684    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-pvdgw_afd0d65c-bbfc-49f0-81b6-e855fd81da7f/console-operator/1.log"
Jul 29 10:48:45.832947 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:48:45.832899    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt" podUID="2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 29 10:48:55.833167 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:48:55.833115    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt" podUID="2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 29 10:49:05.832849 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:05.832802    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt" podUID="2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 29 10:49:15.833152 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:15.833100    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt" podUID="2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 29 10:49:25.833684 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:25.833638    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt" podUID="2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 29 10:49:35.833186 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:35.833142    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt" podUID="2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 29 10:49:45.841102 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:45.841058    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt"
Jul 29 10:49:53.869634 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:53.869601    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt"]
Jul 29 10:49:53.870044 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:53.869864    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt" podUID="2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" containerName="kserve-container" containerID="cri-o://d7b0d439a5541f61222321347c9dabc0d195d64203ad201d5b3a43c49b4ba3de" gracePeriod=30
Jul 29 10:49:53.870044 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:53.869889    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt" podUID="2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" containerName="kube-rbac-proxy" containerID="cri-o://8c0f1f8ef17260aba04b8bd395b2b78497133d621e8344670ff1374841649557" gracePeriod=30
Jul 29 10:49:53.909231 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:53.909202    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:49:54.022732 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:54.022701    2569 generic.go:356] "Generic (PLEG): container finished" podID="2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" containerID="8c0f1f8ef17260aba04b8bd395b2b78497133d621e8344670ff1374841649557" exitCode=2
Jul 29 10:49:54.022869 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:54.022763    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt" event={"ID":"2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade","Type":"ContainerDied","Data":"8c0f1f8ef17260aba04b8bd395b2b78497133d621e8344670ff1374841649557"}
Jul 29 10:49:55.833898 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:55.833856    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt" podUID="2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 29 10:49:56.379643 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:56.379595    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt" podUID="2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Jul 29 10:49:58.975465 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:58.975425    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:49:59.027895 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.027867    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc"]
Jul 29 10:49:59.028196 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.028182    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="agent"
Jul 29 10:49:59.028196 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.028198    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="agent"
Jul 29 10:49:59.028289 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.028211    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01822d12-2967-48a8-bd3e-61a43409476b" containerName="kserve-container"
Jul 29 10:49:59.028289 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.028217    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="01822d12-2967-48a8-bd3e-61a43409476b" containerName="kserve-container"
Jul 29 10:49:59.028289 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.028231    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="storage-initializer"
Jul 29 10:49:59.028289 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.028236    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="storage-initializer"
Jul 29 10:49:59.028289 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.028243    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="kube-rbac-proxy"
Jul 29 10:49:59.028289 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.028248    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="kube-rbac-proxy"
Jul 29 10:49:59.028289 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.028257    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01822d12-2967-48a8-bd3e-61a43409476b" containerName="kube-rbac-proxy"
Jul 29 10:49:59.028289 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.028263    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="01822d12-2967-48a8-bd3e-61a43409476b" containerName="kube-rbac-proxy"
Jul 29 10:49:59.028289 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.028269    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="kserve-container"
Jul 29 10:49:59.028289 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.028274    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="kserve-container"
Jul 29 10:49:59.028576 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.028325    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="01822d12-2967-48a8-bd3e-61a43409476b" containerName="kube-rbac-proxy"
Jul 29 10:49:59.028576 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.028333    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="kube-rbac-proxy"
Jul 29 10:49:59.028576 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.028338    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="agent"
Jul 29 10:49:59.028576 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.028344    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="01822d12-2967-48a8-bd3e-61a43409476b" containerName="kserve-container"
Jul 29 10:49:59.028576 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.028350    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbd2c5a5-64f6-4a6f-aa0e-99ad6d55d7fa" containerName="kserve-container"
Jul 29 10:49:59.031613 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.031588    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc"
Jul 29 10:49:59.033890 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.033864    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Jul 29 10:49:59.033890 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.033881    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Jul 29 10:49:59.040437 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.040403    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2"]
Jul 29 10:49:59.044285 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.044266    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc"]
Jul 29 10:49:59.044376 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.044357    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2"
Jul 29 10:49:59.046818 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.046796    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Jul 29 10:49:59.053304 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.053267    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:49:59.057856 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.057836    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2"]
Jul 29 10:49:59.108771 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.108743    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:49:59.162265 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.162221    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bf40351-c332-4a46-89d1-a2cc403df496-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-75f789cb99-txdp2\" (UID: \"2bf40351-c332-4a46-89d1-a2cc403df496\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2"
Jul 29 10:49:59.162426 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.162282    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c22299df-4d45-4433-9be5-d9c5ef45a27f-proxy-tls\") pod \"isvc-canary-create-predictor-5d59fd866b-f5lfc\" (UID: \"c22299df-4d45-4433-9be5-d9c5ef45a27f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc"
Jul 29 10:49:59.162426 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.162315    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pnbk\" (UniqueName: \"kubernetes.io/projected/c22299df-4d45-4433-9be5-d9c5ef45a27f-kube-api-access-5pnbk\") pod \"isvc-canary-create-predictor-5d59fd866b-f5lfc\" (UID: \"c22299df-4d45-4433-9be5-d9c5ef45a27f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc"
Jul 29 10:49:59.162426 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.162342    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgj9d\" (UniqueName: \"kubernetes.io/projected/2bf40351-c332-4a46-89d1-a2cc403df496-kube-api-access-fgj9d\") pod \"isvc-canary-create-v2-predictor-75f789cb99-txdp2\" (UID: \"2bf40351-c332-4a46-89d1-a2cc403df496\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2"
Jul 29 10:49:59.162426 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.162370    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c22299df-4d45-4433-9be5-d9c5ef45a27f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5d59fd866b-f5lfc\" (UID: \"c22299df-4d45-4433-9be5-d9c5ef45a27f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc"
Jul 29 10:49:59.162426 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.162398    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c22299df-4d45-4433-9be5-d9c5ef45a27f-cabundle-cert\") pod \"isvc-canary-create-predictor-5d59fd866b-f5lfc\" (UID: \"c22299df-4d45-4433-9be5-d9c5ef45a27f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc"
Jul 29 10:49:59.162426 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.162423    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c22299df-4d45-4433-9be5-d9c5ef45a27f-kserve-provision-location\") pod \"isvc-canary-create-predictor-5d59fd866b-f5lfc\" (UID: \"c22299df-4d45-4433-9be5-d9c5ef45a27f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc"
Jul 29 10:49:59.162625 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.162496    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bf40351-c332-4a46-89d1-a2cc403df496-proxy-tls\") pod \"isvc-canary-create-v2-predictor-75f789cb99-txdp2\" (UID: \"2bf40351-c332-4a46-89d1-a2cc403df496\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2"
Jul 29 10:49:59.162625 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.162529    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2bf40351-c332-4a46-89d1-a2cc403df496-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-75f789cb99-txdp2\" (UID: \"2bf40351-c332-4a46-89d1-a2cc403df496\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2"
Jul 29 10:49:59.162625 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.162573    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bf40351-c332-4a46-89d1-a2cc403df496-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-75f789cb99-txdp2\" (UID: \"2bf40351-c332-4a46-89d1-a2cc403df496\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2"
Jul 29 10:49:59.263091 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.262977    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bf40351-c332-4a46-89d1-a2cc403df496-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-75f789cb99-txdp2\" (UID: \"2bf40351-c332-4a46-89d1-a2cc403df496\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2"
Jul 29 10:49:59.263091 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.263026    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bf40351-c332-4a46-89d1-a2cc403df496-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-75f789cb99-txdp2\" (UID: \"2bf40351-c332-4a46-89d1-a2cc403df496\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2"
Jul 29 10:49:59.263091 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.263051    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c22299df-4d45-4433-9be5-d9c5ef45a27f-proxy-tls\") pod \"isvc-canary-create-predictor-5d59fd866b-f5lfc\" (UID: \"c22299df-4d45-4433-9be5-d9c5ef45a27f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc"
Jul 29 10:49:59.263385 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.263102    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pnbk\" (UniqueName: \"kubernetes.io/projected/c22299df-4d45-4433-9be5-d9c5ef45a27f-kube-api-access-5pnbk\") pod \"isvc-canary-create-predictor-5d59fd866b-f5lfc\" (UID: \"c22299df-4d45-4433-9be5-d9c5ef45a27f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc"
Jul 29 10:49:59.263385 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.263129    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgj9d\" (UniqueName: \"kubernetes.io/projected/2bf40351-c332-4a46-89d1-a2cc403df496-kube-api-access-fgj9d\") pod \"isvc-canary-create-v2-predictor-75f789cb99-txdp2\" (UID: \"2bf40351-c332-4a46-89d1-a2cc403df496\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2"
Jul 29 10:49:59.263385 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.263162    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c22299df-4d45-4433-9be5-d9c5ef45a27f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5d59fd866b-f5lfc\" (UID: \"c22299df-4d45-4433-9be5-d9c5ef45a27f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc"
Jul 29 10:49:59.263385 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.263190    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c22299df-4d45-4433-9be5-d9c5ef45a27f-cabundle-cert\") pod \"isvc-canary-create-predictor-5d59fd866b-f5lfc\" (UID: \"c22299df-4d45-4433-9be5-d9c5ef45a27f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc"
Jul 29 10:49:59.263385 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.263218    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c22299df-4d45-4433-9be5-d9c5ef45a27f-kserve-provision-location\") pod \"isvc-canary-create-predictor-5d59fd866b-f5lfc\" (UID: \"c22299df-4d45-4433-9be5-d9c5ef45a27f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc"
Jul 29 10:49:59.263385 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:49:59.263226    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Jul 29 10:49:59.263385 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.263264    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bf40351-c332-4a46-89d1-a2cc403df496-proxy-tls\") pod \"isvc-canary-create-v2-predictor-75f789cb99-txdp2\" (UID: \"2bf40351-c332-4a46-89d1-a2cc403df496\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2"
Jul 29 10:49:59.263385 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:49:59.263301    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c22299df-4d45-4433-9be5-d9c5ef45a27f-proxy-tls podName:c22299df-4d45-4433-9be5-d9c5ef45a27f nodeName:}" failed. No retries permitted until 2026-07-29 10:49:59.763279583 +0000 UTC m=+1276.488070252 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c22299df-4d45-4433-9be5-d9c5ef45a27f-proxy-tls") pod "isvc-canary-create-predictor-5d59fd866b-f5lfc" (UID: "c22299df-4d45-4433-9be5-d9c5ef45a27f") : secret "isvc-canary-create-predictor-serving-cert" not found
Jul 29 10:49:59.263385 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.263326    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2bf40351-c332-4a46-89d1-a2cc403df496-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-75f789cb99-txdp2\" (UID: \"2bf40351-c332-4a46-89d1-a2cc403df496\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2"
Jul 29 10:49:59.263863 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.263528    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bf40351-c332-4a46-89d1-a2cc403df496-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-75f789cb99-txdp2\" (UID: \"2bf40351-c332-4a46-89d1-a2cc403df496\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2"
Jul 29 10:49:59.263863 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.263799    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c22299df-4d45-4433-9be5-d9c5ef45a27f-kserve-provision-location\") pod \"isvc-canary-create-predictor-5d59fd866b-f5lfc\" (UID: \"c22299df-4d45-4433-9be5-d9c5ef45a27f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc"
Jul 29 10:49:59.264003 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.263981    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c22299df-4d45-4433-9be5-d9c5ef45a27f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5d59fd866b-f5lfc\" (UID: \"c22299df-4d45-4433-9be5-d9c5ef45a27f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc"
Jul 29 10:49:59.264061 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.264017    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2bf40351-c332-4a46-89d1-a2cc403df496-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-75f789cb99-txdp2\" (UID: \"2bf40351-c332-4a46-89d1-a2cc403df496\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2"
Jul 29 10:49:59.264150 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.264133    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c22299df-4d45-4433-9be5-d9c5ef45a27f-cabundle-cert\") pod \"isvc-canary-create-predictor-5d59fd866b-f5lfc\" (UID: \"c22299df-4d45-4433-9be5-d9c5ef45a27f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc"
Jul 29 10:49:59.264441 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.264423    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bf40351-c332-4a46-89d1-a2cc403df496-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-75f789cb99-txdp2\" (UID: \"2bf40351-c332-4a46-89d1-a2cc403df496\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2"
Jul 29 10:49:59.265740 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.265721    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bf40351-c332-4a46-89d1-a2cc403df496-proxy-tls\") pod \"isvc-canary-create-v2-predictor-75f789cb99-txdp2\" (UID: \"2bf40351-c332-4a46-89d1-a2cc403df496\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2"
Jul 29 10:49:59.271878 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.271859    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pnbk\" (UniqueName: \"kubernetes.io/projected/c22299df-4d45-4433-9be5-d9c5ef45a27f-kube-api-access-5pnbk\") pod \"isvc-canary-create-predictor-5d59fd866b-f5lfc\" (UID: \"c22299df-4d45-4433-9be5-d9c5ef45a27f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc"
Jul 29 10:49:59.271878 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.271870    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgj9d\" (UniqueName: \"kubernetes.io/projected/2bf40351-c332-4a46-89d1-a2cc403df496-kube-api-access-fgj9d\") pod \"isvc-canary-create-v2-predictor-75f789cb99-txdp2\" (UID: \"2bf40351-c332-4a46-89d1-a2cc403df496\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2"
Jul 29 10:49:59.353937 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.353896    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2"
Jul 29 10:49:59.493986 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.493958    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2"]
Jul 29 10:49:59.496490 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:49:59.496461    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bf40351_c332_4a46_89d1_a2cc403df496.slice/crio-284f6cb5893d4bcb8c0b93a097b898f4cee5a07b3aad6e89f23c247a9dc8a05a WatchSource:0}: Error finding container 284f6cb5893d4bcb8c0b93a097b898f4cee5a07b3aad6e89f23c247a9dc8a05a: Status 404 returned error can't find the container with id 284f6cb5893d4bcb8c0b93a097b898f4cee5a07b3aad6e89f23c247a9dc8a05a
Jul 29 10:49:59.581692 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.581654    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:49:59.767902 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.767862    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c22299df-4d45-4433-9be5-d9c5ef45a27f-proxy-tls\") pod \"isvc-canary-create-predictor-5d59fd866b-f5lfc\" (UID: \"c22299df-4d45-4433-9be5-d9c5ef45a27f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc"
Jul 29 10:49:59.770432 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.770403    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c22299df-4d45-4433-9be5-d9c5ef45a27f-proxy-tls\") pod \"isvc-canary-create-predictor-5d59fd866b-f5lfc\" (UID: \"c22299df-4d45-4433-9be5-d9c5ef45a27f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc"
Jul 29 10:49:59.942867 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:49:59.942839    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc"
Jul 29 10:50:00.050194 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:00.050152    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2" event={"ID":"2bf40351-c332-4a46-89d1-a2cc403df496","Type":"ContainerStarted","Data":"947440a831eae4f7beceec56ab3c26623cb27e45fa640a3b10d6cbdbe0a85f35"}
Jul 29 10:50:00.050607 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:00.050230    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2" event={"ID":"2bf40351-c332-4a46-89d1-a2cc403df496","Type":"ContainerStarted","Data":"284f6cb5893d4bcb8c0b93a097b898f4cee5a07b3aad6e89f23c247a9dc8a05a"}
Jul 29 10:50:00.080252 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:00.080221    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc"]
Jul 29 10:50:00.083954 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:50:00.083927    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc22299df_4d45_4433_9be5_d9c5ef45a27f.slice/crio-de7ee5337ec192d2ab6565244cf89068870cbea9656dde782010641180177052 WatchSource:0}: Error finding container de7ee5337ec192d2ab6565244cf89068870cbea9656dde782010641180177052: Status 404 returned error can't find the container with id de7ee5337ec192d2ab6565244cf89068870cbea9656dde782010641180177052
Jul 29 10:50:01.055325 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:01.055289    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc" event={"ID":"c22299df-4d45-4433-9be5-d9c5ef45a27f","Type":"ContainerStarted","Data":"03626066e82f217fe18f2220c2067c02326197776dd92837dc75ea2846d74ed4"}
Jul 29 10:50:01.055779 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:01.055361    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc" event={"ID":"c22299df-4d45-4433-9be5-d9c5ef45a27f","Type":"ContainerStarted","Data":"de7ee5337ec192d2ab6565244cf89068870cbea9656dde782010641180177052"}
Jul 29 10:50:01.379855 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:01.379756    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt" podUID="2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Jul 29 10:50:04.068858 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:04.068822    2569 generic.go:356] "Generic (PLEG): container finished" podID="2bf40351-c332-4a46-89d1-a2cc403df496" containerID="947440a831eae4f7beceec56ab3c26623cb27e45fa640a3b10d6cbdbe0a85f35" exitCode=0
Jul 29 10:50:04.069251 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:04.068891    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2" event={"ID":"2bf40351-c332-4a46-89d1-a2cc403df496","Type":"ContainerDied","Data":"947440a831eae4f7beceec56ab3c26623cb27e45fa640a3b10d6cbdbe0a85f35"}
Jul 29 10:50:04.070412 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:04.070392    2569 generic.go:356] "Generic (PLEG): container finished" podID="c22299df-4d45-4433-9be5-d9c5ef45a27f" containerID="03626066e82f217fe18f2220c2067c02326197776dd92837dc75ea2846d74ed4" exitCode=0
Jul 29 10:50:04.070491 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:04.070451    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc" event={"ID":"c22299df-4d45-4433-9be5-d9c5ef45a27f","Type":"ContainerDied","Data":"03626066e82f217fe18f2220c2067c02326197776dd92837dc75ea2846d74ed4"}
Jul 29 10:50:04.228756 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:04.228733    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt"
Jul 29 10:50:04.302392 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:04.302301    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade-kube-rbac-proxy-sar-config\") pod \"2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade\" (UID: \"2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade\") "
Jul 29 10:50:04.302392 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:04.302336    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade-cabundle-cert\") pod \"2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade\" (UID: \"2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade\") "
Jul 29 10:50:04.302392 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:04.302365    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p52t2\" (UniqueName: \"kubernetes.io/projected/2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade-kube-api-access-p52t2\") pod \"2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade\" (UID: \"2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade\") "
Jul 29 10:50:04.302392 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:04.302383    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade-proxy-tls\") pod \"2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade\" (UID: \"2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade\") "
Jul 29 10:50:04.302702 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:04.302431    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade-kserve-provision-location\") pod \"2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade\" (UID: \"2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade\") "
Jul 29 10:50:04.302822 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:04.302792    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" (UID: "2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:50:04.302822 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:04.302807    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" (UID: "2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:50:04.302917 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:04.302825    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" (UID: "2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:50:04.304511 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:04.304481    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade-kube-api-access-p52t2" (OuterVolumeSpecName: "kube-api-access-p52t2") pod "2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" (UID: "2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade"). InnerVolumeSpecName "kube-api-access-p52t2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:50:04.304618 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:04.304552    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" (UID: "2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:50:04.403230 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:04.403189    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p52t2\" (UniqueName: \"kubernetes.io/projected/2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade-kube-api-access-p52t2\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:50:04.403230 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:04.403222    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade-proxy-tls\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:50:04.403230 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:04.403232    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade-kserve-provision-location\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:50:04.403230 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:04.403242    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:50:04.403518 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:04.403253    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade-cabundle-cert\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:50:05.076882 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:05.076841    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2" event={"ID":"2bf40351-c332-4a46-89d1-a2cc403df496","Type":"ContainerStarted","Data":"3289e24c2684d198a79ce7a1ede8f0884d7c626ead34e68410a8963abe09780b"}
Jul 29 10:50:05.077396 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:05.076929    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2" event={"ID":"2bf40351-c332-4a46-89d1-a2cc403df496","Type":"ContainerStarted","Data":"7f800a6e2d51c74b7b8f83e63e637895f22c144fa89c7efa9bf420151da0c047"}
Jul 29 10:50:05.077396 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:05.077179    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2"
Jul 29 10:50:05.078905 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:05.078878    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc" event={"ID":"c22299df-4d45-4433-9be5-d9c5ef45a27f","Type":"ContainerStarted","Data":"3678a759ff3ca8635c654d58d2058b3b01672f6d9109d4bc3e673a5e083a40ea"}
Jul 29 10:50:05.079042 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:05.078933    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc" event={"ID":"c22299df-4d45-4433-9be5-d9c5ef45a27f","Type":"ContainerStarted","Data":"e7983744bc64b1807e727d346631031b8c3e86f863728938bf4e1469dffd6131"}
Jul 29 10:50:05.079257 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:05.079241    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc"
Jul 29 10:50:05.079306 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:05.079290    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc"
Jul 29 10:50:05.080730 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:05.080707    2569 generic.go:356] "Generic (PLEG): container finished" podID="2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" containerID="d7b0d439a5541f61222321347c9dabc0d195d64203ad201d5b3a43c49b4ba3de" exitCode=0
Jul 29 10:50:05.080839 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:05.080756    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt"
Jul 29 10:50:05.080839 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:05.080789    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc" podUID="c22299df-4d45-4433-9be5-d9c5ef45a27f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 29 10:50:05.080839 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:05.080789    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt" event={"ID":"2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade","Type":"ContainerDied","Data":"d7b0d439a5541f61222321347c9dabc0d195d64203ad201d5b3a43c49b4ba3de"}
Jul 29 10:50:05.080996 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:05.080869    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt" event={"ID":"2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade","Type":"ContainerDied","Data":"e418aa30439964727b24679ac91c5102dc01cddc3ddc3f64a1ac8f3ecb8b155f"}
Jul 29 10:50:05.080996 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:05.080895    2569 scope.go:117] "RemoveContainer" containerID="8c0f1f8ef17260aba04b8bd395b2b78497133d621e8344670ff1374841649557"
Jul 29 10:50:05.089755 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:05.089739    2569 scope.go:117] "RemoveContainer" containerID="d7b0d439a5541f61222321347c9dabc0d195d64203ad201d5b3a43c49b4ba3de"
Jul 29 10:50:05.097636 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:05.097618    2569 scope.go:117] "RemoveContainer" containerID="1caf7c64849fec768561620bf251259bc2581446d33d5f3e3c3b07e2959b2250"
Jul 29 10:50:05.100062 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:05.100016    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2" podStartSLOduration=2.234022269 podStartE2EDuration="6.100001253s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.865978984s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:49:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:50:05.098239302 +0000 UTC m=+1281.823030000" watchObservedRunningTime="2026-07-29 10:50:05.100001253 +0000 UTC m=+1281.824791926"
Jul 29 10:50:05.105778 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:05.105758    2569 scope.go:117] "RemoveContainer" containerID="8c0f1f8ef17260aba04b8bd395b2b78497133d621e8344670ff1374841649557"
Jul 29 10:50:05.106038 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:50:05.106018    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c0f1f8ef17260aba04b8bd395b2b78497133d621e8344670ff1374841649557\": container with ID starting with 8c0f1f8ef17260aba04b8bd395b2b78497133d621e8344670ff1374841649557 not found: ID does not exist" containerID="8c0f1f8ef17260aba04b8bd395b2b78497133d621e8344670ff1374841649557"
Jul 29 10:50:05.106135 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:05.106051    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c0f1f8ef17260aba04b8bd395b2b78497133d621e8344670ff1374841649557"} err="failed to get container status \"8c0f1f8ef17260aba04b8bd395b2b78497133d621e8344670ff1374841649557\": rpc error: code = NotFound desc = could not find container \"8c0f1f8ef17260aba04b8bd395b2b78497133d621e8344670ff1374841649557\": container with ID starting with 8c0f1f8ef17260aba04b8bd395b2b78497133d621e8344670ff1374841649557 not found: ID does not exist"
Jul 29 10:50:05.106135 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:05.106083    2569 scope.go:117] "RemoveContainer" containerID="d7b0d439a5541f61222321347c9dabc0d195d64203ad201d5b3a43c49b4ba3de"
Jul 29 10:50:05.106351 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:50:05.106331    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7b0d439a5541f61222321347c9dabc0d195d64203ad201d5b3a43c49b4ba3de\": container with ID starting with d7b0d439a5541f61222321347c9dabc0d195d64203ad201d5b3a43c49b4ba3de not found: ID does not exist" containerID="d7b0d439a5541f61222321347c9dabc0d195d64203ad201d5b3a43c49b4ba3de"
Jul 29 10:50:05.106391 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:05.106358    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7b0d439a5541f61222321347c9dabc0d195d64203ad201d5b3a43c49b4ba3de"} err="failed to get container status \"d7b0d439a5541f61222321347c9dabc0d195d64203ad201d5b3a43c49b4ba3de\": rpc error: code = NotFound desc = could not find container \"d7b0d439a5541f61222321347c9dabc0d195d64203ad201d5b3a43c49b4ba3de\": container with ID starting with d7b0d439a5541f61222321347c9dabc0d195d64203ad201d5b3a43c49b4ba3de not found: ID does not exist"
Jul 29 10:50:05.106391 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:05.106372    2569 scope.go:117] "RemoveContainer" containerID="1caf7c64849fec768561620bf251259bc2581446d33d5f3e3c3b07e2959b2250"
Jul 29 10:50:05.106597 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:50:05.106579    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1caf7c64849fec768561620bf251259bc2581446d33d5f3e3c3b07e2959b2250\": container with ID starting with 1caf7c64849fec768561620bf251259bc2581446d33d5f3e3c3b07e2959b2250 not found: ID does not exist" containerID="1caf7c64849fec768561620bf251259bc2581446d33d5f3e3c3b07e2959b2250"
Jul 29 10:50:05.106642 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:05.106601    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1caf7c64849fec768561620bf251259bc2581446d33d5f3e3c3b07e2959b2250"} err="failed to get container status \"1caf7c64849fec768561620bf251259bc2581446d33d5f3e3c3b07e2959b2250\": rpc error: code = NotFound desc = could not find container \"1caf7c64849fec768561620bf251259bc2581446d33d5f3e3c3b07e2959b2250\": container with ID starting with 1caf7c64849fec768561620bf251259bc2581446d33d5f3e3c3b07e2959b2250 not found: ID does not exist"
Jul 29 10:50:05.116866 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:05.116819    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc" podStartSLOduration=2.264118648 podStartE2EDuration="6.116802809s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.852684161s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:49:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:50:05.116434142 +0000 UTC m=+1281.841224827" watchObservedRunningTime="2026-07-29 10:50:05.116802809 +0000 UTC m=+1281.841593484"
Jul 29 10:50:05.130047 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:05.130016    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt"]
Jul 29 10:50:05.133888 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:05.133855    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-bf0a1-predictor-5d8cb546c7-lm6jt"]
Jul 29 10:50:05.838028 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:05.837991    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" path="/var/lib/kubelet/pods/2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade/volumes"
Jul 29 10:50:06.088565 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:06.088205    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc" podUID="c22299df-4d45-4433-9be5-d9c5ef45a27f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 29 10:50:06.088565 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:06.088353    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2"
Jul 29 10:50:06.089506 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:06.089480    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2" podUID="2bf40351-c332-4a46-89d1-a2cc403df496" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 29 10:50:07.092256 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:07.092219    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2" podUID="2bf40351-c332-4a46-89d1-a2cc403df496" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 29 10:50:11.092633 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:11.092605    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc"
Jul 29 10:50:11.093111 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:11.093065    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc" podUID="c22299df-4d45-4433-9be5-d9c5ef45a27f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 29 10:50:12.097341 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:12.097315    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2"
Jul 29 10:50:12.097775 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:12.097751    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2" podUID="2bf40351-c332-4a46-89d1-a2cc403df496" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 29 10:50:21.093026 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:21.092981    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc" podUID="c22299df-4d45-4433-9be5-d9c5ef45a27f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 29 10:50:22.098454 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:22.098416    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2" podUID="2bf40351-c332-4a46-89d1-a2cc403df496" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 29 10:50:31.093547 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:31.093507    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc" podUID="c22299df-4d45-4433-9be5-d9c5ef45a27f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 29 10:50:32.097863 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:32.097815    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2" podUID="2bf40351-c332-4a46-89d1-a2cc403df496" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 29 10:50:41.093251 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:41.093206    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc" podUID="c22299df-4d45-4433-9be5-d9c5ef45a27f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 29 10:50:42.098427 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:42.098389    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2" podUID="2bf40351-c332-4a46-89d1-a2cc403df496" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 29 10:50:51.093191 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:51.093148    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc" podUID="c22299df-4d45-4433-9be5-d9c5ef45a27f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 29 10:50:52.098436 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:52.098400    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2" podUID="2bf40351-c332-4a46-89d1-a2cc403df496" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 29 10:50:59.018657 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:59.018625    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:50:59.074946 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:50:59.074913    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:51:01.093855 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:01.093818    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc" podUID="c22299df-4d45-4433-9be5-d9c5ef45a27f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 29 10:51:02.098116 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:02.098061    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2" podUID="2bf40351-c332-4a46-89d1-a2cc403df496" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 29 10:51:11.093433 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:11.093346    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc" podUID="c22299df-4d45-4433-9be5-d9c5ef45a27f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 29 10:51:12.097945 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:12.097910    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2" podUID="2bf40351-c332-4a46-89d1-a2cc403df496" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 29 10:51:21.094262 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:21.094224    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc"
Jul 29 10:51:22.098221 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:22.098188    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2"
Jul 29 10:51:29.116884 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:29.116854    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2"]
Jul 29 10:51:29.117351 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:29.117136    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2" podUID="2bf40351-c332-4a46-89d1-a2cc403df496" containerName="kserve-container" containerID="cri-o://7f800a6e2d51c74b7b8f83e63e637895f22c144fa89c7efa9bf420151da0c047" gracePeriod=30
Jul 29 10:51:29.117351 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:29.117156    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2" podUID="2bf40351-c332-4a46-89d1-a2cc403df496" containerName="kube-rbac-proxy" containerID="cri-o://3289e24c2684d198a79ce7a1ede8f0884d7c626ead34e68410a8963abe09780b" gracePeriod=30
Jul 29 10:51:29.122667 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:29.122643    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc"]
Jul 29 10:51:29.122874 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:29.122843    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc" podUID="c22299df-4d45-4433-9be5-d9c5ef45a27f" containerName="kserve-container" containerID="cri-o://e7983744bc64b1807e727d346631031b8c3e86f863728938bf4e1469dffd6131" gracePeriod=30
Jul 29 10:51:29.122993 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:29.122952    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc" podUID="c22299df-4d45-4433-9be5-d9c5ef45a27f" containerName="kube-rbac-proxy" containerID="cri-o://3678a759ff3ca8635c654d58d2058b3b01672f6d9109d4bc3e673a5e083a40ea" gracePeriod=30
Jul 29 10:51:29.129687 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:29.129659    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:51:29.183057 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:29.183025    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:51:29.414455 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:29.414424    2569 generic.go:356] "Generic (PLEG): container finished" podID="2bf40351-c332-4a46-89d1-a2cc403df496" containerID="3289e24c2684d198a79ce7a1ede8f0884d7c626ead34e68410a8963abe09780b" exitCode=2
Jul 29 10:51:29.414617 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:29.414501    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2" event={"ID":"2bf40351-c332-4a46-89d1-a2cc403df496","Type":"ContainerDied","Data":"3289e24c2684d198a79ce7a1ede8f0884d7c626ead34e68410a8963abe09780b"}
Jul 29 10:51:29.416501 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:29.416474    2569 generic.go:356] "Generic (PLEG): container finished" podID="c22299df-4d45-4433-9be5-d9c5ef45a27f" containerID="3678a759ff3ca8635c654d58d2058b3b01672f6d9109d4bc3e673a5e083a40ea" exitCode=2
Jul 29 10:51:29.416627 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:29.416535    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc" event={"ID":"c22299df-4d45-4433-9be5-d9c5ef45a27f","Type":"ContainerDied","Data":"3678a759ff3ca8635c654d58d2058b3b01672f6d9109d4bc3e673a5e083a40ea"}
Jul 29 10:51:31.089089 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:31.089039    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc" podUID="c22299df-4d45-4433-9be5-d9c5ef45a27f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Jul 29 10:51:31.093684 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:31.093657    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc" podUID="c22299df-4d45-4433-9be5-d9c5ef45a27f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 29 10:51:32.093061 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:32.093018    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2" podUID="2bf40351-c332-4a46-89d1-a2cc403df496" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Jul 29 10:51:32.097918 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:32.097886    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2" podUID="2bf40351-c332-4a46-89d1-a2cc403df496" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 29 10:51:34.084786 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.084762    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2"
Jul 29 10:51:34.087505 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.087488    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc"
Jul 29 10:51:34.146774 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.146697    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c22299df-4d45-4433-9be5-d9c5ef45a27f-kube-rbac-proxy-sar-config\") pod \"c22299df-4d45-4433-9be5-d9c5ef45a27f\" (UID: \"c22299df-4d45-4433-9be5-d9c5ef45a27f\") "
Jul 29 10:51:34.146774 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.146734    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgj9d\" (UniqueName: \"kubernetes.io/projected/2bf40351-c332-4a46-89d1-a2cc403df496-kube-api-access-fgj9d\") pod \"2bf40351-c332-4a46-89d1-a2cc403df496\" (UID: \"2bf40351-c332-4a46-89d1-a2cc403df496\") "
Jul 29 10:51:34.146945 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.146775    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2bf40351-c332-4a46-89d1-a2cc403df496-kube-rbac-proxy-sar-config\") pod \"2bf40351-c332-4a46-89d1-a2cc403df496\" (UID: \"2bf40351-c332-4a46-89d1-a2cc403df496\") "
Jul 29 10:51:34.146982 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.146936    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bf40351-c332-4a46-89d1-a2cc403df496-proxy-tls\") pod \"2bf40351-c332-4a46-89d1-a2cc403df496\" (UID: \"2bf40351-c332-4a46-89d1-a2cc403df496\") "
Jul 29 10:51:34.146982 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.146969    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c22299df-4d45-4433-9be5-d9c5ef45a27f-cabundle-cert\") pod \"c22299df-4d45-4433-9be5-d9c5ef45a27f\" (UID: \"c22299df-4d45-4433-9be5-d9c5ef45a27f\") "
Jul 29 10:51:34.147057 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.147013    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bf40351-c332-4a46-89d1-a2cc403df496-kserve-provision-location\") pod \"2bf40351-c332-4a46-89d1-a2cc403df496\" (UID: \"2bf40351-c332-4a46-89d1-a2cc403df496\") "
Jul 29 10:51:34.147057 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.147052    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c22299df-4d45-4433-9be5-d9c5ef45a27f-proxy-tls\") pod \"c22299df-4d45-4433-9be5-d9c5ef45a27f\" (UID: \"c22299df-4d45-4433-9be5-d9c5ef45a27f\") "
Jul 29 10:51:34.147265 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.147120    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pnbk\" (UniqueName: \"kubernetes.io/projected/c22299df-4d45-4433-9be5-d9c5ef45a27f-kube-api-access-5pnbk\") pod \"c22299df-4d45-4433-9be5-d9c5ef45a27f\" (UID: \"c22299df-4d45-4433-9be5-d9c5ef45a27f\") "
Jul 29 10:51:34.147265 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.147117    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c22299df-4d45-4433-9be5-d9c5ef45a27f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c22299df-4d45-4433-9be5-d9c5ef45a27f" (UID: "c22299df-4d45-4433-9be5-d9c5ef45a27f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:51:34.147265 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.147122    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bf40351-c332-4a46-89d1-a2cc403df496-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2bf40351-c332-4a46-89d1-a2cc403df496" (UID: "2bf40351-c332-4a46-89d1-a2cc403df496"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:51:34.147265 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.147152    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c22299df-4d45-4433-9be5-d9c5ef45a27f-kserve-provision-location\") pod \"c22299df-4d45-4433-9be5-d9c5ef45a27f\" (UID: \"c22299df-4d45-4433-9be5-d9c5ef45a27f\") "
Jul 29 10:51:34.147265 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.147252    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bf40351-c332-4a46-89d1-a2cc403df496-cabundle-cert\") pod \"2bf40351-c332-4a46-89d1-a2cc403df496\" (UID: \"2bf40351-c332-4a46-89d1-a2cc403df496\") "
Jul 29 10:51:34.147521 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.147403    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c22299df-4d45-4433-9be5-d9c5ef45a27f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c22299df-4d45-4433-9be5-d9c5ef45a27f" (UID: "c22299df-4d45-4433-9be5-d9c5ef45a27f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:51:34.147521 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.147426    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c22299df-4d45-4433-9be5-d9c5ef45a27f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c22299df-4d45-4433-9be5-d9c5ef45a27f" (UID: "c22299df-4d45-4433-9be5-d9c5ef45a27f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:51:34.147623 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.147542    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2bf40351-c332-4a46-89d1-a2cc403df496-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:51:34.147695 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.147671    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c22299df-4d45-4433-9be5-d9c5ef45a27f-cabundle-cert\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:51:34.147695 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.147691    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c22299df-4d45-4433-9be5-d9c5ef45a27f-kserve-provision-location\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:51:34.147782 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.147706    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c22299df-4d45-4433-9be5-d9c5ef45a27f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:51:34.147782 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.147712    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bf40351-c332-4a46-89d1-a2cc403df496-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2bf40351-c332-4a46-89d1-a2cc403df496" (UID: "2bf40351-c332-4a46-89d1-a2cc403df496"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:51:34.147782 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.147726    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bf40351-c332-4a46-89d1-a2cc403df496-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2bf40351-c332-4a46-89d1-a2cc403df496" (UID: "2bf40351-c332-4a46-89d1-a2cc403df496"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:51:34.149450 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.149424    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bf40351-c332-4a46-89d1-a2cc403df496-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2bf40351-c332-4a46-89d1-a2cc403df496" (UID: "2bf40351-c332-4a46-89d1-a2cc403df496"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:51:34.149450 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.149434    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf40351-c332-4a46-89d1-a2cc403df496-kube-api-access-fgj9d" (OuterVolumeSpecName: "kube-api-access-fgj9d") pod "2bf40351-c332-4a46-89d1-a2cc403df496" (UID: "2bf40351-c332-4a46-89d1-a2cc403df496"). InnerVolumeSpecName "kube-api-access-fgj9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:51:34.149659 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.149637    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c22299df-4d45-4433-9be5-d9c5ef45a27f-kube-api-access-5pnbk" (OuterVolumeSpecName: "kube-api-access-5pnbk") pod "c22299df-4d45-4433-9be5-d9c5ef45a27f" (UID: "c22299df-4d45-4433-9be5-d9c5ef45a27f"). InnerVolumeSpecName "kube-api-access-5pnbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:51:34.149659 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.149650    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c22299df-4d45-4433-9be5-d9c5ef45a27f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c22299df-4d45-4433-9be5-d9c5ef45a27f" (UID: "c22299df-4d45-4433-9be5-d9c5ef45a27f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:51:34.229101 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.229057    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:51:34.248276 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.248242    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bf40351-c332-4a46-89d1-a2cc403df496-kserve-provision-location\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:51:34.248276 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.248268    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c22299df-4d45-4433-9be5-d9c5ef45a27f-proxy-tls\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:51:34.248276 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.248279    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pnbk\" (UniqueName: \"kubernetes.io/projected/c22299df-4d45-4433-9be5-d9c5ef45a27f-kube-api-access-5pnbk\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:51:34.248535 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.248288    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bf40351-c332-4a46-89d1-a2cc403df496-cabundle-cert\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:51:34.248535 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.248296    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgj9d\" (UniqueName: \"kubernetes.io/projected/2bf40351-c332-4a46-89d1-a2cc403df496-kube-api-access-fgj9d\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:51:34.248535 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.248304    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bf40351-c332-4a46-89d1-a2cc403df496-proxy-tls\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:51:34.268574 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.268539    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf"]
Jul 29 10:51:34.268884 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.268869    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bf40351-c332-4a46-89d1-a2cc403df496" containerName="kube-rbac-proxy"
Jul 29 10:51:34.268967 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.268885    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bf40351-c332-4a46-89d1-a2cc403df496" containerName="kube-rbac-proxy"
Jul 29 10:51:34.268967 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.268900    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" containerName="kube-rbac-proxy"
Jul 29 10:51:34.268967 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.268912    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" containerName="kube-rbac-proxy"
Jul 29 10:51:34.268967 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.268924    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" containerName="kserve-container"
Jul 29 10:51:34.268967 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.268930    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" containerName="kserve-container"
Jul 29 10:51:34.268967 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.268937    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c22299df-4d45-4433-9be5-d9c5ef45a27f" containerName="kserve-container"
Jul 29 10:51:34.268967 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.268943    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c22299df-4d45-4433-9be5-d9c5ef45a27f" containerName="kserve-container"
Jul 29 10:51:34.268967 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.268949    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c22299df-4d45-4433-9be5-d9c5ef45a27f" containerName="storage-initializer"
Jul 29 10:51:34.268967 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.268961    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c22299df-4d45-4433-9be5-d9c5ef45a27f" containerName="storage-initializer"
Jul 29 10:51:34.268967 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.268969    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bf40351-c332-4a46-89d1-a2cc403df496" containerName="kserve-container"
Jul 29 10:51:34.269486 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.268975    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bf40351-c332-4a46-89d1-a2cc403df496" containerName="kserve-container"
Jul 29 10:51:34.269486 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.268982    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" containerName="storage-initializer"
Jul 29 10:51:34.269486 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.268989    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" containerName="storage-initializer"
Jul 29 10:51:34.269486 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.268998    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c22299df-4d45-4433-9be5-d9c5ef45a27f" containerName="kube-rbac-proxy"
Jul 29 10:51:34.269486 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.269004    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c22299df-4d45-4433-9be5-d9c5ef45a27f" containerName="kube-rbac-proxy"
Jul 29 10:51:34.269486 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.269010    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bf40351-c332-4a46-89d1-a2cc403df496" containerName="storage-initializer"
Jul 29 10:51:34.269486 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.269015    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bf40351-c332-4a46-89d1-a2cc403df496" containerName="storage-initializer"
Jul 29 10:51:34.269486 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.269062    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c22299df-4d45-4433-9be5-d9c5ef45a27f" containerName="kserve-container"
Jul 29 10:51:34.269486 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.269085    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bf40351-c332-4a46-89d1-a2cc403df496" containerName="kube-rbac-proxy"
Jul 29 10:51:34.269486 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.269094    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" containerName="kserve-container"
Jul 29 10:51:34.269486 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.269103    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fd52ae4-70dc-4cad-bd2c-b9fd73bb2ade" containerName="kube-rbac-proxy"
Jul 29 10:51:34.269486 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.269109    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bf40351-c332-4a46-89d1-a2cc403df496" containerName="kserve-container"
Jul 29 10:51:34.269486 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.269115    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c22299df-4d45-4433-9be5-d9c5ef45a27f" containerName="kube-rbac-proxy"
Jul 29 10:51:34.273860 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.273835    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf"
Jul 29 10:51:34.275783 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.275763    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn"]
Jul 29 10:51:34.276393 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.276371    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 29 10:51:34.276508 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.276376    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Jul 29 10:51:34.279948 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.279930    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn"
Jul 29 10:51:34.282122 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.282102    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 29 10:51:34.284349 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.284326    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf"]
Jul 29 10:51:34.290481 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.290457    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn"]
Jul 29 10:51:34.325552 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.325517    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:51:34.348661 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.348635    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d0dcf35-fd34-436e-bf26-55bd01b530e0-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn\" (UID: \"7d0dcf35-fd34-436e-bf26-55bd01b530e0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn"
Jul 29 10:51:34.348784 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.348675    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a549df1-665c-4085-9011-272965be30a5-proxy-tls\") pod \"isvc-canary-promote-predictor-69b9b94c74-xs4xf\" (UID: \"6a549df1-665c-4085-9011-272965be30a5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf"
Jul 29 10:51:34.348784 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.348756    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d0dcf35-fd34-436e-bf26-55bd01b530e0-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn\" (UID: \"7d0dcf35-fd34-436e-bf26-55bd01b530e0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn"
Jul 29 10:51:34.348857 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.348809    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dk6m\" (UniqueName: \"kubernetes.io/projected/7d0dcf35-fd34-436e-bf26-55bd01b530e0-kube-api-access-7dk6m\") pod \"isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn\" (UID: \"7d0dcf35-fd34-436e-bf26-55bd01b530e0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn"
Jul 29 10:51:34.348857 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.348844    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d0dcf35-fd34-436e-bf26-55bd01b530e0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn\" (UID: \"7d0dcf35-fd34-436e-bf26-55bd01b530e0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn"
Jul 29 10:51:34.348935 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.348916    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a549df1-665c-4085-9011-272965be30a5-cabundle-cert\") pod \"isvc-canary-promote-predictor-69b9b94c74-xs4xf\" (UID: \"6a549df1-665c-4085-9011-272965be30a5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf"
Jul 29 10:51:34.348974 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.348958    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a549df1-665c-4085-9011-272965be30a5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-69b9b94c74-xs4xf\" (UID: \"6a549df1-665c-4085-9011-272965be30a5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf"
Jul 29 10:51:34.349009 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.348984    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a549df1-665c-4085-9011-272965be30a5-kserve-provision-location\") pod \"isvc-canary-promote-predictor-69b9b94c74-xs4xf\" (UID: \"6a549df1-665c-4085-9011-272965be30a5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf"
Jul 29 10:51:34.349043 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.349026    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d0dcf35-fd34-436e-bf26-55bd01b530e0-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn\" (UID: \"7d0dcf35-fd34-436e-bf26-55bd01b530e0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn"
Jul 29 10:51:34.349101 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.349051    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7656f\" (UniqueName: \"kubernetes.io/projected/6a549df1-665c-4085-9011-272965be30a5-kube-api-access-7656f\") pod \"isvc-canary-promote-predictor-69b9b94c74-xs4xf\" (UID: \"6a549df1-665c-4085-9011-272965be30a5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf"
Jul 29 10:51:34.440493 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.440450    2569 generic.go:356] "Generic (PLEG): container finished" podID="c22299df-4d45-4433-9be5-d9c5ef45a27f" containerID="e7983744bc64b1807e727d346631031b8c3e86f863728938bf4e1469dffd6131" exitCode=0
Jul 29 10:51:34.440682 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.440529    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc"
Jul 29 10:51:34.440682 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.440537    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc" event={"ID":"c22299df-4d45-4433-9be5-d9c5ef45a27f","Type":"ContainerDied","Data":"e7983744bc64b1807e727d346631031b8c3e86f863728938bf4e1469dffd6131"}
Jul 29 10:51:34.440682 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.440608    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc" event={"ID":"c22299df-4d45-4433-9be5-d9c5ef45a27f","Type":"ContainerDied","Data":"de7ee5337ec192d2ab6565244cf89068870cbea9656dde782010641180177052"}
Jul 29 10:51:34.440682 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.440647    2569 scope.go:117] "RemoveContainer" containerID="3678a759ff3ca8635c654d58d2058b3b01672f6d9109d4bc3e673a5e083a40ea"
Jul 29 10:51:34.442833 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.442805    2569 generic.go:356] "Generic (PLEG): container finished" podID="2bf40351-c332-4a46-89d1-a2cc403df496" containerID="7f800a6e2d51c74b7b8f83e63e637895f22c144fa89c7efa9bf420151da0c047" exitCode=0
Jul 29 10:51:34.442952 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.442872    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2" event={"ID":"2bf40351-c332-4a46-89d1-a2cc403df496","Type":"ContainerDied","Data":"7f800a6e2d51c74b7b8f83e63e637895f22c144fa89c7efa9bf420151da0c047"}
Jul 29 10:51:34.442952 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.442892    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2"
Jul 29 10:51:34.442952 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.442946    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2" event={"ID":"2bf40351-c332-4a46-89d1-a2cc403df496","Type":"ContainerDied","Data":"284f6cb5893d4bcb8c0b93a097b898f4cee5a07b3aad6e89f23c247a9dc8a05a"}
Jul 29 10:51:34.449630 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.449604    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a549df1-665c-4085-9011-272965be30a5-cabundle-cert\") pod \"isvc-canary-promote-predictor-69b9b94c74-xs4xf\" (UID: \"6a549df1-665c-4085-9011-272965be30a5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf"
Jul 29 10:51:34.449757 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.449648    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a549df1-665c-4085-9011-272965be30a5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-69b9b94c74-xs4xf\" (UID: \"6a549df1-665c-4085-9011-272965be30a5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf"
Jul 29 10:51:34.449757 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.449675    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a549df1-665c-4085-9011-272965be30a5-kserve-provision-location\") pod \"isvc-canary-promote-predictor-69b9b94c74-xs4xf\" (UID: \"6a549df1-665c-4085-9011-272965be30a5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf"
Jul 29 10:51:34.449757 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.449707    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d0dcf35-fd34-436e-bf26-55bd01b530e0-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn\" (UID: \"7d0dcf35-fd34-436e-bf26-55bd01b530e0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn"
Jul 29 10:51:34.449757 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.449732    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7656f\" (UniqueName: \"kubernetes.io/projected/6a549df1-665c-4085-9011-272965be30a5-kube-api-access-7656f\") pod \"isvc-canary-promote-predictor-69b9b94c74-xs4xf\" (UID: \"6a549df1-665c-4085-9011-272965be30a5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf"
Jul 29 10:51:34.449980 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.449775    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d0dcf35-fd34-436e-bf26-55bd01b530e0-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn\" (UID: \"7d0dcf35-fd34-436e-bf26-55bd01b530e0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn"
Jul 29 10:51:34.450261 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.450141    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a549df1-665c-4085-9011-272965be30a5-kserve-provision-location\") pod \"isvc-canary-promote-predictor-69b9b94c74-xs4xf\" (UID: \"6a549df1-665c-4085-9011-272965be30a5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf"
Jul 29 10:51:34.450261 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.450194    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a549df1-665c-4085-9011-272965be30a5-proxy-tls\") pod \"isvc-canary-promote-predictor-69b9b94c74-xs4xf\" (UID: \"6a549df1-665c-4085-9011-272965be30a5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf"
Jul 29 10:51:34.450261 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.450258    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d0dcf35-fd34-436e-bf26-55bd01b530e0-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn\" (UID: \"7d0dcf35-fd34-436e-bf26-55bd01b530e0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn"
Jul 29 10:51:34.450450 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.450288    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dk6m\" (UniqueName: \"kubernetes.io/projected/7d0dcf35-fd34-436e-bf26-55bd01b530e0-kube-api-access-7dk6m\") pod \"isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn\" (UID: \"7d0dcf35-fd34-436e-bf26-55bd01b530e0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn"
Jul 29 10:51:34.450450 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.450303    2569 scope.go:117] "RemoveContainer" containerID="e7983744bc64b1807e727d346631031b8c3e86f863728938bf4e1469dffd6131"
Jul 29 10:51:34.450450 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.450315    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d0dcf35-fd34-436e-bf26-55bd01b530e0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn\" (UID: \"7d0dcf35-fd34-436e-bf26-55bd01b530e0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn"
Jul 29 10:51:34.450612 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:51:34.450532    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Jul 29 10:51:34.450612 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:51:34.450602    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d0dcf35-fd34-436e-bf26-55bd01b530e0-proxy-tls podName:7d0dcf35-fd34-436e-bf26-55bd01b530e0 nodeName:}" failed. No retries permitted until 2026-07-29 10:51:34.950581929 +0000 UTC m=+1371.675372583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7d0dcf35-fd34-436e-bf26-55bd01b530e0-proxy-tls") pod "isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn" (UID: "7d0dcf35-fd34-436e-bf26-55bd01b530e0") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Jul 29 10:51:34.450888 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.450682    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a549df1-665c-4085-9011-272965be30a5-cabundle-cert\") pod \"isvc-canary-promote-predictor-69b9b94c74-xs4xf\" (UID: \"6a549df1-665c-4085-9011-272965be30a5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf"
Jul 29 10:51:34.450888 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:51:34.450848    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Jul 29 10:51:34.451217 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:51:34.450895    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a549df1-665c-4085-9011-272965be30a5-proxy-tls podName:6a549df1-665c-4085-9011-272965be30a5 nodeName:}" failed. No retries permitted until 2026-07-29 10:51:34.950881415 +0000 UTC m=+1371.675672073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6a549df1-665c-4085-9011-272965be30a5-proxy-tls") pod "isvc-canary-promote-predictor-69b9b94c74-xs4xf" (UID: "6a549df1-665c-4085-9011-272965be30a5") : secret "isvc-canary-promote-predictor-serving-cert" not found
Jul 29 10:51:34.451290 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.451214    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a549df1-665c-4085-9011-272965be30a5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-69b9b94c74-xs4xf\" (UID: \"6a549df1-665c-4085-9011-272965be30a5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf"
Jul 29 10:51:34.451723 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.451696    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d0dcf35-fd34-436e-bf26-55bd01b530e0-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn\" (UID: \"7d0dcf35-fd34-436e-bf26-55bd01b530e0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn"
Jul 29 10:51:34.451905 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.451823    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d0dcf35-fd34-436e-bf26-55bd01b530e0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn\" (UID: \"7d0dcf35-fd34-436e-bf26-55bd01b530e0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn"
Jul 29 10:51:34.451905 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.451819    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d0dcf35-fd34-436e-bf26-55bd01b530e0-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn\" (UID: \"7d0dcf35-fd34-436e-bf26-55bd01b530e0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn"
Jul 29 10:51:34.459331 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.459303    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dk6m\" (UniqueName: \"kubernetes.io/projected/7d0dcf35-fd34-436e-bf26-55bd01b530e0-kube-api-access-7dk6m\") pod \"isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn\" (UID: \"7d0dcf35-fd34-436e-bf26-55bd01b530e0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn"
Jul 29 10:51:34.459793 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.459774    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7656f\" (UniqueName: \"kubernetes.io/projected/6a549df1-665c-4085-9011-272965be30a5-kube-api-access-7656f\") pod \"isvc-canary-promote-predictor-69b9b94c74-xs4xf\" (UID: \"6a549df1-665c-4085-9011-272965be30a5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf"
Jul 29 10:51:34.459942 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.459927    2569 scope.go:117] "RemoveContainer" containerID="03626066e82f217fe18f2220c2067c02326197776dd92837dc75ea2846d74ed4"
Jul 29 10:51:34.466387 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.466366    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc"]
Jul 29 10:51:34.468274 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.468259    2569 scope.go:117] "RemoveContainer" containerID="3678a759ff3ca8635c654d58d2058b3b01672f6d9109d4bc3e673a5e083a40ea"
Jul 29 10:51:34.468540 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:51:34.468521    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3678a759ff3ca8635c654d58d2058b3b01672f6d9109d4bc3e673a5e083a40ea\": container with ID starting with 3678a759ff3ca8635c654d58d2058b3b01672f6d9109d4bc3e673a5e083a40ea not found: ID does not exist" containerID="3678a759ff3ca8635c654d58d2058b3b01672f6d9109d4bc3e673a5e083a40ea"
Jul 29 10:51:34.468619 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.468551    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3678a759ff3ca8635c654d58d2058b3b01672f6d9109d4bc3e673a5e083a40ea"} err="failed to get container status \"3678a759ff3ca8635c654d58d2058b3b01672f6d9109d4bc3e673a5e083a40ea\": rpc error: code = NotFound desc = could not find container \"3678a759ff3ca8635c654d58d2058b3b01672f6d9109d4bc3e673a5e083a40ea\": container with ID starting with 3678a759ff3ca8635c654d58d2058b3b01672f6d9109d4bc3e673a5e083a40ea not found: ID does not exist"
Jul 29 10:51:34.468619 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.468572    2569 scope.go:117] "RemoveContainer" containerID="e7983744bc64b1807e727d346631031b8c3e86f863728938bf4e1469dffd6131"
Jul 29 10:51:34.468804 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:51:34.468789    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7983744bc64b1807e727d346631031b8c3e86f863728938bf4e1469dffd6131\": container with ID starting with e7983744bc64b1807e727d346631031b8c3e86f863728938bf4e1469dffd6131 not found: ID does not exist" containerID="e7983744bc64b1807e727d346631031b8c3e86f863728938bf4e1469dffd6131"
Jul 29 10:51:34.468866 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.468811    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7983744bc64b1807e727d346631031b8c3e86f863728938bf4e1469dffd6131"} err="failed to get container status \"e7983744bc64b1807e727d346631031b8c3e86f863728938bf4e1469dffd6131\": rpc error: code = NotFound desc = could not find container \"e7983744bc64b1807e727d346631031b8c3e86f863728938bf4e1469dffd6131\": container with ID starting with e7983744bc64b1807e727d346631031b8c3e86f863728938bf4e1469dffd6131 not found: ID does not exist"
Jul 29 10:51:34.468866 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.468831    2569 scope.go:117] "RemoveContainer" containerID="03626066e82f217fe18f2220c2067c02326197776dd92837dc75ea2846d74ed4"
Jul 29 10:51:34.469119 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:51:34.469098    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03626066e82f217fe18f2220c2067c02326197776dd92837dc75ea2846d74ed4\": container with ID starting with 03626066e82f217fe18f2220c2067c02326197776dd92837dc75ea2846d74ed4 not found: ID does not exist" containerID="03626066e82f217fe18f2220c2067c02326197776dd92837dc75ea2846d74ed4"
Jul 29 10:51:34.469175 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.469126    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03626066e82f217fe18f2220c2067c02326197776dd92837dc75ea2846d74ed4"} err="failed to get container status \"03626066e82f217fe18f2220c2067c02326197776dd92837dc75ea2846d74ed4\": rpc error: code = NotFound desc = could not find container \"03626066e82f217fe18f2220c2067c02326197776dd92837dc75ea2846d74ed4\": container with ID starting with 03626066e82f217fe18f2220c2067c02326197776dd92837dc75ea2846d74ed4 not found: ID does not exist"
Jul 29 10:51:34.469175 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.469146    2569 scope.go:117] "RemoveContainer" containerID="3289e24c2684d198a79ce7a1ede8f0884d7c626ead34e68410a8963abe09780b"
Jul 29 10:51:34.472299 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.472277    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-f5lfc"]
Jul 29 10:51:34.476482 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.476461    2569 scope.go:117] "RemoveContainer" containerID="7f800a6e2d51c74b7b8f83e63e637895f22c144fa89c7efa9bf420151da0c047"
Jul 29 10:51:34.484085 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.484047    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2"]
Jul 29 10:51:34.485029 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.485009    2569 scope.go:117] "RemoveContainer" containerID="947440a831eae4f7beceec56ab3c26623cb27e45fa640a3b10d6cbdbe0a85f35"
Jul 29 10:51:34.488557 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.488533    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-txdp2"]
Jul 29 10:51:34.494295 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.494275    2569 scope.go:117] "RemoveContainer" containerID="3289e24c2684d198a79ce7a1ede8f0884d7c626ead34e68410a8963abe09780b"
Jul 29 10:51:34.494589 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:51:34.494569    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3289e24c2684d198a79ce7a1ede8f0884d7c626ead34e68410a8963abe09780b\": container with ID starting with 3289e24c2684d198a79ce7a1ede8f0884d7c626ead34e68410a8963abe09780b not found: ID does not exist" containerID="3289e24c2684d198a79ce7a1ede8f0884d7c626ead34e68410a8963abe09780b"
Jul 29 10:51:34.494653 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.494595    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3289e24c2684d198a79ce7a1ede8f0884d7c626ead34e68410a8963abe09780b"} err="failed to get container status \"3289e24c2684d198a79ce7a1ede8f0884d7c626ead34e68410a8963abe09780b\": rpc error: code = NotFound desc = could not find container \"3289e24c2684d198a79ce7a1ede8f0884d7c626ead34e68410a8963abe09780b\": container with ID starting with 3289e24c2684d198a79ce7a1ede8f0884d7c626ead34e68410a8963abe09780b not found: ID does not exist"
Jul 29 10:51:34.494653 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.494612    2569 scope.go:117] "RemoveContainer" containerID="7f800a6e2d51c74b7b8f83e63e637895f22c144fa89c7efa9bf420151da0c047"
Jul 29 10:51:34.494886 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:51:34.494872    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f800a6e2d51c74b7b8f83e63e637895f22c144fa89c7efa9bf420151da0c047\": container with ID starting with 7f800a6e2d51c74b7b8f83e63e637895f22c144fa89c7efa9bf420151da0c047 not found: ID does not exist" containerID="7f800a6e2d51c74b7b8f83e63e637895f22c144fa89c7efa9bf420151da0c047"
Jul 29 10:51:34.494950 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.494890    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f800a6e2d51c74b7b8f83e63e637895f22c144fa89c7efa9bf420151da0c047"} err="failed to get container status \"7f800a6e2d51c74b7b8f83e63e637895f22c144fa89c7efa9bf420151da0c047\": rpc error: code = NotFound desc = could not find container \"7f800a6e2d51c74b7b8f83e63e637895f22c144fa89c7efa9bf420151da0c047\": container with ID starting with 7f800a6e2d51c74b7b8f83e63e637895f22c144fa89c7efa9bf420151da0c047 not found: ID does not exist"
Jul 29 10:51:34.494950 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.494902    2569 scope.go:117] "RemoveContainer" containerID="947440a831eae4f7beceec56ab3c26623cb27e45fa640a3b10d6cbdbe0a85f35"
Jul 29 10:51:34.495154 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:51:34.495137    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"947440a831eae4f7beceec56ab3c26623cb27e45fa640a3b10d6cbdbe0a85f35\": container with ID starting with 947440a831eae4f7beceec56ab3c26623cb27e45fa640a3b10d6cbdbe0a85f35 not found: ID does not exist" containerID="947440a831eae4f7beceec56ab3c26623cb27e45fa640a3b10d6cbdbe0a85f35"
Jul 29 10:51:34.495220 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.495158    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"947440a831eae4f7beceec56ab3c26623cb27e45fa640a3b10d6cbdbe0a85f35"} err="failed to get container status \"947440a831eae4f7beceec56ab3c26623cb27e45fa640a3b10d6cbdbe0a85f35\": rpc error: code = NotFound desc = could not find container \"947440a831eae4f7beceec56ab3c26623cb27e45fa640a3b10d6cbdbe0a85f35\": container with ID starting with 947440a831eae4f7beceec56ab3c26623cb27e45fa640a3b10d6cbdbe0a85f35 not found: ID does not exist"
Jul 29 10:51:34.610762 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.610725    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:51:34.908173 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.908144    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:51:34.954975 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.954945    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a549df1-665c-4085-9011-272965be30a5-proxy-tls\") pod \"isvc-canary-promote-predictor-69b9b94c74-xs4xf\" (UID: \"6a549df1-665c-4085-9011-272965be30a5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf"
Jul 29 10:51:34.955117 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.955042    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d0dcf35-fd34-436e-bf26-55bd01b530e0-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn\" (UID: \"7d0dcf35-fd34-436e-bf26-55bd01b530e0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn"
Jul 29 10:51:34.957430 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.957405    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a549df1-665c-4085-9011-272965be30a5-proxy-tls\") pod \"isvc-canary-promote-predictor-69b9b94c74-xs4xf\" (UID: \"6a549df1-665c-4085-9011-272965be30a5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf"
Jul 29 10:51:34.957552 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:34.957410    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d0dcf35-fd34-436e-bf26-55bd01b530e0-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn\" (UID: \"7d0dcf35-fd34-436e-bf26-55bd01b530e0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn"
Jul 29 10:51:35.184943 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:35.184904    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf"
Jul 29 10:51:35.196654 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:35.196635    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn"
Jul 29 10:51:35.325580 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:35.325554    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf"]
Jul 29 10:51:35.328321 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:51:35.328285    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a549df1_665c_4085_9011_272965be30a5.slice/crio-2cba685f8a29d6117193ddd54ea3e9bfe2fca23f11c32073d0c4bc562ce25aa1 WatchSource:0}: Error finding container 2cba685f8a29d6117193ddd54ea3e9bfe2fca23f11c32073d0c4bc562ce25aa1: Status 404 returned error can't find the container with id 2cba685f8a29d6117193ddd54ea3e9bfe2fca23f11c32073d0c4bc562ce25aa1
Jul 29 10:51:35.345032 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:35.344848    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn"]
Jul 29 10:51:35.346865 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:51:35.346842    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d0dcf35_fd34_436e_bf26_55bd01b530e0.slice/crio-2f6950b8cd6fa4660664f180fbc51be8fb9e647fa5da754635197140ba3b3c18 WatchSource:0}: Error finding container 2f6950b8cd6fa4660664f180fbc51be8fb9e647fa5da754635197140ba3b3c18: Status 404 returned error can't find the container with id 2f6950b8cd6fa4660664f180fbc51be8fb9e647fa5da754635197140ba3b3c18
Jul 29 10:51:35.455022 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:35.454912    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn" event={"ID":"7d0dcf35-fd34-436e-bf26-55bd01b530e0","Type":"ContainerStarted","Data":"b00dd025f8a0db2204492a0687ab5fcfcd94f7cc16fdf083116affacc75be833"}
Jul 29 10:51:35.455022 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:35.454996    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn" event={"ID":"7d0dcf35-fd34-436e-bf26-55bd01b530e0","Type":"ContainerStarted","Data":"2f6950b8cd6fa4660664f180fbc51be8fb9e647fa5da754635197140ba3b3c18"}
Jul 29 10:51:35.456590 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:35.456557    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf" event={"ID":"6a549df1-665c-4085-9011-272965be30a5","Type":"ContainerStarted","Data":"143f23f0634c47f870b81b6410c0188e9f92c04d66f7e517d20bcd09f588d208"}
Jul 29 10:51:35.456688 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:35.456629    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf" event={"ID":"6a549df1-665c-4085-9011-272965be30a5","Type":"ContainerStarted","Data":"2cba685f8a29d6117193ddd54ea3e9bfe2fca23f11c32073d0c4bc562ce25aa1"}
Jul 29 10:51:35.838537 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:35.838444    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bf40351-c332-4a46-89d1-a2cc403df496" path="/var/lib/kubelet/pods/2bf40351-c332-4a46-89d1-a2cc403df496/volumes"
Jul 29 10:51:35.839090 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:35.839052    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c22299df-4d45-4433-9be5-d9c5ef45a27f" path="/var/lib/kubelet/pods/c22299df-4d45-4433-9be5-d9c5ef45a27f/volumes"
Jul 29 10:51:39.474357 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:39.474265    2569 generic.go:356] "Generic (PLEG): container finished" podID="7d0dcf35-fd34-436e-bf26-55bd01b530e0" containerID="b00dd025f8a0db2204492a0687ab5fcfcd94f7cc16fdf083116affacc75be833" exitCode=0
Jul 29 10:51:39.474357 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:39.474338    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn" event={"ID":"7d0dcf35-fd34-436e-bf26-55bd01b530e0","Type":"ContainerDied","Data":"b00dd025f8a0db2204492a0687ab5fcfcd94f7cc16fdf083116affacc75be833"}
Jul 29 10:51:39.475922 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:39.475901    2569 generic.go:356] "Generic (PLEG): container finished" podID="6a549df1-665c-4085-9011-272965be30a5" containerID="143f23f0634c47f870b81b6410c0188e9f92c04d66f7e517d20bcd09f588d208" exitCode=0
Jul 29 10:51:39.475998 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:39.475976    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf" event={"ID":"6a549df1-665c-4085-9011-272965be30a5","Type":"ContainerDied","Data":"143f23f0634c47f870b81b6410c0188e9f92c04d66f7e517d20bcd09f588d208"}
Jul 29 10:51:40.482511 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:40.482472    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn" event={"ID":"7d0dcf35-fd34-436e-bf26-55bd01b530e0","Type":"ContainerStarted","Data":"777bef0517e6b6e442e8499771e450e6c68f71db360300bf51d598726fe2f2c6"}
Jul 29 10:51:40.482945 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:40.482545    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn" event={"ID":"7d0dcf35-fd34-436e-bf26-55bd01b530e0","Type":"ContainerStarted","Data":"3817f42e0ef03b6b4ca65f7616aa4a0eb5c10291b4a362282d473da137276c25"}
Jul 29 10:51:40.482945 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:40.482858    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn"
Jul 29 10:51:40.483049 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:40.483005    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn"
Jul 29 10:51:40.484540 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:40.484504    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn" podUID="7d0dcf35-fd34-436e-bf26-55bd01b530e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 29 10:51:40.484705 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:40.484685    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf" event={"ID":"6a549df1-665c-4085-9011-272965be30a5","Type":"ContainerStarted","Data":"743c7c71d4c64e1324ba3f3150563764c5ba879c57364ac2fff5f35503861e7c"}
Jul 29 10:51:40.484764 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:40.484734    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf" event={"ID":"6a549df1-665c-4085-9011-272965be30a5","Type":"ContainerStarted","Data":"f34d9037781203f013c4350b86e0bab74951ee4010fbc7e83a1db2a56ecd1047"}
Jul 29 10:51:40.485039 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:40.485022    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf"
Jul 29 10:51:40.485132 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:40.485099    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf"
Jul 29 10:51:40.486027 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:40.486004    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf" podUID="6a549df1-665c-4085-9011-272965be30a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 29 10:51:40.501217 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:40.501171    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn" podStartSLOduration=2.705632914 podStartE2EDuration="6.50115853s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.795525616s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:51:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:51:40.500409262 +0000 UTC m=+1377.225199935" watchObservedRunningTime="2026-07-29 10:51:40.50115853 +0000 UTC m=+1377.225949202"
Jul 29 10:51:40.518645 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:40.518592    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf" podStartSLOduration=2.669735256 podStartE2EDuration="6.518576775s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.848841519s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:51:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:51:40.51789261 +0000 UTC m=+1377.242683282" watchObservedRunningTime="2026-07-29 10:51:40.518576775 +0000 UTC m=+1377.243367502"
Jul 29 10:51:41.493787 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:41.493737    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf" podUID="6a549df1-665c-4085-9011-272965be30a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 29 10:51:41.493787 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:41.493748    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn" podUID="7d0dcf35-fd34-436e-bf26-55bd01b530e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 29 10:51:46.497906 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:46.497876    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf"
Jul 29 10:51:46.498334 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:46.497961    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn"
Jul 29 10:51:46.498391 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:46.498345    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf" podUID="6a549df1-665c-4085-9011-272965be30a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 29 10:51:46.498391 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:46.498351    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn" podUID="7d0dcf35-fd34-436e-bf26-55bd01b530e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 29 10:51:56.499152 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:56.499112    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf" podUID="6a549df1-665c-4085-9011-272965be30a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 29 10:51:56.499575 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:51:56.499114    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn" podUID="7d0dcf35-fd34-436e-bf26-55bd01b530e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 29 10:52:06.498419 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:52:06.498377    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn" podUID="7d0dcf35-fd34-436e-bf26-55bd01b530e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 29 10:52:06.498832 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:52:06.498377    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf" podUID="6a549df1-665c-4085-9011-272965be30a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 29 10:52:16.498374 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:52:16.498333    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf" podUID="6a549df1-665c-4085-9011-272965be30a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 29 10:52:16.498826 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:52:16.498333    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn" podUID="7d0dcf35-fd34-436e-bf26-55bd01b530e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 29 10:52:26.502067 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:52:26.501985    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf" podUID="6a549df1-665c-4085-9011-272965be30a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 29 10:52:26.502641 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:52:26.502066    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn" podUID="7d0dcf35-fd34-436e-bf26-55bd01b530e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 29 10:52:36.498870 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:52:36.498782    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf" podUID="6a549df1-665c-4085-9011-272965be30a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 29 10:52:36.499266 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:52:36.498782    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn" podUID="7d0dcf35-fd34-436e-bf26-55bd01b530e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 29 10:52:46.498315 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:52:46.498276    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf" podUID="6a549df1-665c-4085-9011-272965be30a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 29 10:52:46.498704 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:52:46.498278    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn" podUID="7d0dcf35-fd34-436e-bf26-55bd01b530e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 29 10:52:53.835830 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:52:53.835781    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn" podUID="7d0dcf35-fd34-436e-bf26-55bd01b530e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 29 10:52:56.499688 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:52:56.499651    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf"
Jul 29 10:53:03.837167 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:03.837138    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn"
Jul 29 10:53:04.388340 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:04.388307    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf"]
Jul 29 10:53:04.388560 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:04.388524    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf" podUID="6a549df1-665c-4085-9011-272965be30a5" containerName="kserve-container" containerID="cri-o://f34d9037781203f013c4350b86e0bab74951ee4010fbc7e83a1db2a56ecd1047" gracePeriod=30
Jul 29 10:53:04.388698 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:04.388591    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf" podUID="6a549df1-665c-4085-9011-272965be30a5" containerName="kube-rbac-proxy" containerID="cri-o://743c7c71d4c64e1324ba3f3150563764c5ba879c57364ac2fff5f35503861e7c" gracePeriod=30
Jul 29 10:53:04.405779 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:04.405752    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:53:04.827038 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:04.827006    2569 generic.go:356] "Generic (PLEG): container finished" podID="6a549df1-665c-4085-9011-272965be30a5" containerID="743c7c71d4c64e1324ba3f3150563764c5ba879c57364ac2fff5f35503861e7c" exitCode=2
Jul 29 10:53:04.827231 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:04.827058    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf" event={"ID":"6a549df1-665c-4085-9011-272965be30a5","Type":"ContainerDied","Data":"743c7c71d4c64e1324ba3f3150563764c5ba879c57364ac2fff5f35503861e7c"}
Jul 29 10:53:06.493817 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:06.493762    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf" podUID="6a549df1-665c-4085-9011-272965be30a5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Jul 29 10:53:06.498984 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:06.498954    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf" podUID="6a549df1-665c-4085-9011-272965be30a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 29 10:53:09.228853 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:09.228828    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf"
Jul 29 10:53:09.333131 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:09.333027    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a549df1-665c-4085-9011-272965be30a5-cabundle-cert\") pod \"6a549df1-665c-4085-9011-272965be30a5\" (UID: \"6a549df1-665c-4085-9011-272965be30a5\") "
Jul 29 10:53:09.333131 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:09.333066    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a549df1-665c-4085-9011-272965be30a5-kube-rbac-proxy-sar-config\") pod \"6a549df1-665c-4085-9011-272965be30a5\" (UID: \"6a549df1-665c-4085-9011-272965be30a5\") "
Jul 29 10:53:09.333367 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:09.333139    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a549df1-665c-4085-9011-272965be30a5-proxy-tls\") pod \"6a549df1-665c-4085-9011-272965be30a5\" (UID: \"6a549df1-665c-4085-9011-272965be30a5\") "
Jul 29 10:53:09.333367 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:09.333168    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7656f\" (UniqueName: \"kubernetes.io/projected/6a549df1-665c-4085-9011-272965be30a5-kube-api-access-7656f\") pod \"6a549df1-665c-4085-9011-272965be30a5\" (UID: \"6a549df1-665c-4085-9011-272965be30a5\") "
Jul 29 10:53:09.333367 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:09.333199    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a549df1-665c-4085-9011-272965be30a5-kserve-provision-location\") pod \"6a549df1-665c-4085-9011-272965be30a5\" (UID: \"6a549df1-665c-4085-9011-272965be30a5\") "
Jul 29 10:53:09.333537 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:09.333469    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a549df1-665c-4085-9011-272965be30a5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6a549df1-665c-4085-9011-272965be30a5" (UID: "6a549df1-665c-4085-9011-272965be30a5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:53:09.333594 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:09.333530    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a549df1-665c-4085-9011-272965be30a5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6a549df1-665c-4085-9011-272965be30a5" (UID: "6a549df1-665c-4085-9011-272965be30a5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:53:09.333594 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:09.333583    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a549df1-665c-4085-9011-272965be30a5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6a549df1-665c-4085-9011-272965be30a5" (UID: "6a549df1-665c-4085-9011-272965be30a5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:53:09.335354 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:09.335330    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a549df1-665c-4085-9011-272965be30a5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6a549df1-665c-4085-9011-272965be30a5" (UID: "6a549df1-665c-4085-9011-272965be30a5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:53:09.335458 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:09.335380    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a549df1-665c-4085-9011-272965be30a5-kube-api-access-7656f" (OuterVolumeSpecName: "kube-api-access-7656f") pod "6a549df1-665c-4085-9011-272965be30a5" (UID: "6a549df1-665c-4085-9011-272965be30a5"). InnerVolumeSpecName "kube-api-access-7656f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:53:09.434360 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:09.434327    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a549df1-665c-4085-9011-272965be30a5-proxy-tls\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:53:09.434360 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:09.434355    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7656f\" (UniqueName: \"kubernetes.io/projected/6a549df1-665c-4085-9011-272965be30a5-kube-api-access-7656f\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:53:09.434558 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:09.434369    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a549df1-665c-4085-9011-272965be30a5-kserve-provision-location\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:53:09.434558 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:09.434382    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a549df1-665c-4085-9011-272965be30a5-cabundle-cert\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:53:09.434558 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:09.434395    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a549df1-665c-4085-9011-272965be30a5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:53:09.848514 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:09.848478    2569 generic.go:356] "Generic (PLEG): container finished" podID="6a549df1-665c-4085-9011-272965be30a5" containerID="f34d9037781203f013c4350b86e0bab74951ee4010fbc7e83a1db2a56ecd1047" exitCode=0
Jul 29 10:53:09.848622 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:09.848532    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf"
Jul 29 10:53:09.848622 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:09.848556    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf" event={"ID":"6a549df1-665c-4085-9011-272965be30a5","Type":"ContainerDied","Data":"f34d9037781203f013c4350b86e0bab74951ee4010fbc7e83a1db2a56ecd1047"}
Jul 29 10:53:09.848622 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:09.848616    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf" event={"ID":"6a549df1-665c-4085-9011-272965be30a5","Type":"ContainerDied","Data":"2cba685f8a29d6117193ddd54ea3e9bfe2fca23f11c32073d0c4bc562ce25aa1"}
Jul 29 10:53:09.848737 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:09.848642    2569 scope.go:117] "RemoveContainer" containerID="743c7c71d4c64e1324ba3f3150563764c5ba879c57364ac2fff5f35503861e7c"
Jul 29 10:53:09.857040 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:09.857022    2569 scope.go:117] "RemoveContainer" containerID="f34d9037781203f013c4350b86e0bab74951ee4010fbc7e83a1db2a56ecd1047"
Jul 29 10:53:09.864346 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:09.864326    2569 scope.go:117] "RemoveContainer" containerID="143f23f0634c47f870b81b6410c0188e9f92c04d66f7e517d20bcd09f588d208"
Jul 29 10:53:09.867511 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:09.867490    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf"]
Jul 29 10:53:09.871766 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:09.871676    2569 scope.go:117] "RemoveContainer" containerID="743c7c71d4c64e1324ba3f3150563764c5ba879c57364ac2fff5f35503861e7c"
Jul 29 10:53:09.872263 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:53:09.872242    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"743c7c71d4c64e1324ba3f3150563764c5ba879c57364ac2fff5f35503861e7c\": container with ID starting with 743c7c71d4c64e1324ba3f3150563764c5ba879c57364ac2fff5f35503861e7c not found: ID does not exist" containerID="743c7c71d4c64e1324ba3f3150563764c5ba879c57364ac2fff5f35503861e7c"
Jul 29 10:53:09.872346 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:09.872272    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"743c7c71d4c64e1324ba3f3150563764c5ba879c57364ac2fff5f35503861e7c"} err="failed to get container status \"743c7c71d4c64e1324ba3f3150563764c5ba879c57364ac2fff5f35503861e7c\": rpc error: code = NotFound desc = could not find container \"743c7c71d4c64e1324ba3f3150563764c5ba879c57364ac2fff5f35503861e7c\": container with ID starting with 743c7c71d4c64e1324ba3f3150563764c5ba879c57364ac2fff5f35503861e7c not found: ID does not exist"
Jul 29 10:53:09.872346 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:09.872289    2569 scope.go:117] "RemoveContainer" containerID="f34d9037781203f013c4350b86e0bab74951ee4010fbc7e83a1db2a56ecd1047"
Jul 29 10:53:09.872579 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:53:09.872555    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f34d9037781203f013c4350b86e0bab74951ee4010fbc7e83a1db2a56ecd1047\": container with ID starting with f34d9037781203f013c4350b86e0bab74951ee4010fbc7e83a1db2a56ecd1047 not found: ID does not exist" containerID="f34d9037781203f013c4350b86e0bab74951ee4010fbc7e83a1db2a56ecd1047"
Jul 29 10:53:09.872634 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:09.872587    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f34d9037781203f013c4350b86e0bab74951ee4010fbc7e83a1db2a56ecd1047"} err="failed to get container status \"f34d9037781203f013c4350b86e0bab74951ee4010fbc7e83a1db2a56ecd1047\": rpc error: code = NotFound desc = could not find container \"f34d9037781203f013c4350b86e0bab74951ee4010fbc7e83a1db2a56ecd1047\": container with ID starting with f34d9037781203f013c4350b86e0bab74951ee4010fbc7e83a1db2a56ecd1047 not found: ID does not exist"
Jul 29 10:53:09.872634 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:09.872609    2569 scope.go:117] "RemoveContainer" containerID="143f23f0634c47f870b81b6410c0188e9f92c04d66f7e517d20bcd09f588d208"
Jul 29 10:53:09.872878 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:53:09.872861    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"143f23f0634c47f870b81b6410c0188e9f92c04d66f7e517d20bcd09f588d208\": container with ID starting with 143f23f0634c47f870b81b6410c0188e9f92c04d66f7e517d20bcd09f588d208 not found: ID does not exist" containerID="143f23f0634c47f870b81b6410c0188e9f92c04d66f7e517d20bcd09f588d208"
Jul 29 10:53:09.872928 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:09.872882    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"143f23f0634c47f870b81b6410c0188e9f92c04d66f7e517d20bcd09f588d208"} err="failed to get container status \"143f23f0634c47f870b81b6410c0188e9f92c04d66f7e517d20bcd09f588d208\": rpc error: code = NotFound desc = could not find container \"143f23f0634c47f870b81b6410c0188e9f92c04d66f7e517d20bcd09f588d208\": container with ID starting with 143f23f0634c47f870b81b6410c0188e9f92c04d66f7e517d20bcd09f588d208 not found: ID does not exist"
Jul 29 10:53:09.873365 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:09.873347    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-xs4xf"]
Jul 29 10:53:11.837680 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:11.837648    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a549df1-665c-4085-9011-272965be30a5" path="/var/lib/kubelet/pods/6a549df1-665c-4085-9011-272965be30a5/volumes"
Jul 29 10:53:43.800718 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:43.800688    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-pvdgw_afd0d65c-bbfc-49f0-81b6-e855fd81da7f/console-operator/1.log"
Jul 29 10:53:43.802799 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:43.802773    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-pvdgw_afd0d65c-bbfc-49f0-81b6-e855fd81da7f/console-operator/1.log"
Jul 29 10:53:55.504212 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:55.504161    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:53:55.556368 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:53:55.556336    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:54:54.599819 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:54.599778    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn"]
Jul 29 10:54:54.601883 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:54.600051    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn" podUID="7d0dcf35-fd34-436e-bf26-55bd01b530e0" containerName="kserve-container" containerID="cri-o://3817f42e0ef03b6b4ca65f7616aa4a0eb5c10291b4a362282d473da137276c25" gracePeriod=30
Jul 29 10:54:54.601883 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:54.600118    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn" podUID="7d0dcf35-fd34-436e-bf26-55bd01b530e0" containerName="kube-rbac-proxy" containerID="cri-o://777bef0517e6b6e442e8499771e450e6c68f71db360300bf51d598726fe2f2c6" gracePeriod=30
Jul 29 10:54:55.144419 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:55.144384    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:54:55.202608 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:55.202574    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:54:55.251596 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:55.251567    2569 generic.go:356] "Generic (PLEG): container finished" podID="7d0dcf35-fd34-436e-bf26-55bd01b530e0" containerID="777bef0517e6b6e442e8499771e450e6c68f71db360300bf51d598726fe2f2c6" exitCode=2
Jul 29 10:54:55.251747 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:55.251644    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn" event={"ID":"7d0dcf35-fd34-436e-bf26-55bd01b530e0","Type":"ContainerDied","Data":"777bef0517e6b6e442e8499771e450e6c68f71db360300bf51d598726fe2f2c6"}
Jul 29 10:54:56.494718 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:56.494670    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn" podUID="7d0dcf35-fd34-436e-bf26-55bd01b530e0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Jul 29 10:54:59.269632 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.269552    2569 generic.go:356] "Generic (PLEG): container finished" podID="7d0dcf35-fd34-436e-bf26-55bd01b530e0" containerID="3817f42e0ef03b6b4ca65f7616aa4a0eb5c10291b4a362282d473da137276c25" exitCode=0
Jul 29 10:54:59.269632 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.269601    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn" event={"ID":"7d0dcf35-fd34-436e-bf26-55bd01b530e0","Type":"ContainerDied","Data":"3817f42e0ef03b6b4ca65f7616aa4a0eb5c10291b4a362282d473da137276c25"}
Jul 29 10:54:59.351539 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.351515    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn"
Jul 29 10:54:59.426396 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.426355    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d0dcf35-fd34-436e-bf26-55bd01b530e0-kserve-provision-location\") pod \"7d0dcf35-fd34-436e-bf26-55bd01b530e0\" (UID: \"7d0dcf35-fd34-436e-bf26-55bd01b530e0\") "
Jul 29 10:54:59.426603 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.426446    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d0dcf35-fd34-436e-bf26-55bd01b530e0-proxy-tls\") pod \"7d0dcf35-fd34-436e-bf26-55bd01b530e0\" (UID: \"7d0dcf35-fd34-436e-bf26-55bd01b530e0\") "
Jul 29 10:54:59.426603 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.426494    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d0dcf35-fd34-436e-bf26-55bd01b530e0-kube-rbac-proxy-sar-config\") pod \"7d0dcf35-fd34-436e-bf26-55bd01b530e0\" (UID: \"7d0dcf35-fd34-436e-bf26-55bd01b530e0\") "
Jul 29 10:54:59.426603 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.426527    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d0dcf35-fd34-436e-bf26-55bd01b530e0-cabundle-cert\") pod \"7d0dcf35-fd34-436e-bf26-55bd01b530e0\" (UID: \"7d0dcf35-fd34-436e-bf26-55bd01b530e0\") "
Jul 29 10:54:59.426603 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.426568    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dk6m\" (UniqueName: \"kubernetes.io/projected/7d0dcf35-fd34-436e-bf26-55bd01b530e0-kube-api-access-7dk6m\") pod \"7d0dcf35-fd34-436e-bf26-55bd01b530e0\" (UID: \"7d0dcf35-fd34-436e-bf26-55bd01b530e0\") "
Jul 29 10:54:59.426812 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.426776    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d0dcf35-fd34-436e-bf26-55bd01b530e0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7d0dcf35-fd34-436e-bf26-55bd01b530e0" (UID: "7d0dcf35-fd34-436e-bf26-55bd01b530e0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:54:59.426884 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.426855    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d0dcf35-fd34-436e-bf26-55bd01b530e0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7d0dcf35-fd34-436e-bf26-55bd01b530e0" (UID: "7d0dcf35-fd34-436e-bf26-55bd01b530e0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:54:59.426884 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.426868    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d0dcf35-fd34-436e-bf26-55bd01b530e0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7d0dcf35-fd34-436e-bf26-55bd01b530e0" (UID: "7d0dcf35-fd34-436e-bf26-55bd01b530e0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:54:59.428618 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.428595    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d0dcf35-fd34-436e-bf26-55bd01b530e0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7d0dcf35-fd34-436e-bf26-55bd01b530e0" (UID: "7d0dcf35-fd34-436e-bf26-55bd01b530e0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:54:59.428690 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.428660    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d0dcf35-fd34-436e-bf26-55bd01b530e0-kube-api-access-7dk6m" (OuterVolumeSpecName: "kube-api-access-7dk6m") pod "7d0dcf35-fd34-436e-bf26-55bd01b530e0" (UID: "7d0dcf35-fd34-436e-bf26-55bd01b530e0"). InnerVolumeSpecName "kube-api-access-7dk6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:54:59.527858 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.527815    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d0dcf35-fd34-436e-bf26-55bd01b530e0-cabundle-cert\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:54:59.527858 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.527848    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dk6m\" (UniqueName: \"kubernetes.io/projected/7d0dcf35-fd34-436e-bf26-55bd01b530e0-kube-api-access-7dk6m\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:54:59.527858 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.527861    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d0dcf35-fd34-436e-bf26-55bd01b530e0-kserve-provision-location\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:54:59.527858 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.527870    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d0dcf35-fd34-436e-bf26-55bd01b530e0-proxy-tls\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:54:59.528157 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.527879    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d0dcf35-fd34-436e-bf26-55bd01b530e0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:54:59.725213 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.725120    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:54:59.778247 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.778205    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg"]
Jul 29 10:54:59.778558 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.778542    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d0dcf35-fd34-436e-bf26-55bd01b530e0" containerName="storage-initializer"
Jul 29 10:54:59.778623 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.778561    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d0dcf35-fd34-436e-bf26-55bd01b530e0" containerName="storage-initializer"
Jul 29 10:54:59.778623 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.778577    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a549df1-665c-4085-9011-272965be30a5" containerName="storage-initializer"
Jul 29 10:54:59.778623 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.778583    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a549df1-665c-4085-9011-272965be30a5" containerName="storage-initializer"
Jul 29 10:54:59.778623 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.778591    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d0dcf35-fd34-436e-bf26-55bd01b530e0" containerName="kserve-container"
Jul 29 10:54:59.778623 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.778596    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d0dcf35-fd34-436e-bf26-55bd01b530e0" containerName="kserve-container"
Jul 29 10:54:59.778623 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.778609    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a549df1-665c-4085-9011-272965be30a5" containerName="kserve-container"
Jul 29 10:54:59.778623 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.778616    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a549df1-665c-4085-9011-272965be30a5" containerName="kserve-container"
Jul 29 10:54:59.778976 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.778631    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d0dcf35-fd34-436e-bf26-55bd01b530e0" containerName="kube-rbac-proxy"
Jul 29 10:54:59.778976 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.778640    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d0dcf35-fd34-436e-bf26-55bd01b530e0" containerName="kube-rbac-proxy"
Jul 29 10:54:59.778976 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.778648    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a549df1-665c-4085-9011-272965be30a5" containerName="kube-rbac-proxy"
Jul 29 10:54:59.778976 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.778654    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a549df1-665c-4085-9011-272965be30a5" containerName="kube-rbac-proxy"
Jul 29 10:54:59.778976 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.778701    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a549df1-665c-4085-9011-272965be30a5" containerName="kube-rbac-proxy"
Jul 29 10:54:59.778976 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.778709    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a549df1-665c-4085-9011-272965be30a5" containerName="kserve-container"
Jul 29 10:54:59.778976 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.778722    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d0dcf35-fd34-436e-bf26-55bd01b530e0" containerName="kserve-container"
Jul 29 10:54:59.778976 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.778731    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d0dcf35-fd34-436e-bf26-55bd01b530e0" containerName="kube-rbac-proxy"
Jul 29 10:54:59.782053 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.782034    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg"
Jul 29 10:54:59.784506 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.784479    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Jul 29 10:54:59.784506 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.784497    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 29 10:54:59.794055 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.794032    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg"]
Jul 29 10:54:59.809944 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.809910    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:54:59.829655 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.829609    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89522978-782d-4988-ba3e-e7585b3799ce-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg\" (UID: \"89522978-782d-4988-ba3e-e7585b3799ce\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg"
Jul 29 10:54:59.829847 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.829665    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7czz\" (UniqueName: \"kubernetes.io/projected/89522978-782d-4988-ba3e-e7585b3799ce-kube-api-access-r7czz\") pod \"isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg\" (UID: \"89522978-782d-4988-ba3e-e7585b3799ce\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg"
Jul 29 10:54:59.829847 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.829711    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89522978-782d-4988-ba3e-e7585b3799ce-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg\" (UID: \"89522978-782d-4988-ba3e-e7585b3799ce\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg"
Jul 29 10:54:59.829847 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.829739    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89522978-782d-4988-ba3e-e7585b3799ce-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg\" (UID: \"89522978-782d-4988-ba3e-e7585b3799ce\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg"
Jul 29 10:54:59.829847 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.829810    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89522978-782d-4988-ba3e-e7585b3799ce-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg\" (UID: \"89522978-782d-4988-ba3e-e7585b3799ce\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg"
Jul 29 10:54:59.913728 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.913693    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:54:59.930919 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.930878    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89522978-782d-4988-ba3e-e7585b3799ce-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg\" (UID: \"89522978-782d-4988-ba3e-e7585b3799ce\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg"
Jul 29 10:54:59.931104 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.930927    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7czz\" (UniqueName: \"kubernetes.io/projected/89522978-782d-4988-ba3e-e7585b3799ce-kube-api-access-r7czz\") pod \"isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg\" (UID: \"89522978-782d-4988-ba3e-e7585b3799ce\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg"
Jul 29 10:54:59.931104 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.930975    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89522978-782d-4988-ba3e-e7585b3799ce-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg\" (UID: \"89522978-782d-4988-ba3e-e7585b3799ce\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg"
Jul 29 10:54:59.931104 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.931005    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89522978-782d-4988-ba3e-e7585b3799ce-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg\" (UID: \"89522978-782d-4988-ba3e-e7585b3799ce\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg"
Jul 29 10:54:59.931104 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:54:59.931035    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Jul 29 10:54:59.931320 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.931097    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89522978-782d-4988-ba3e-e7585b3799ce-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg\" (UID: \"89522978-782d-4988-ba3e-e7585b3799ce\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg"
Jul 29 10:54:59.931320 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:54:59.931118    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89522978-782d-4988-ba3e-e7585b3799ce-proxy-tls podName:89522978-782d-4988-ba3e-e7585b3799ce nodeName:}" failed. No retries permitted until 2026-07-29 10:55:00.431095614 +0000 UTC m=+1577.155886278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/89522978-782d-4988-ba3e-e7585b3799ce-proxy-tls") pod "isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg" (UID: "89522978-782d-4988-ba3e-e7585b3799ce") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Jul 29 10:54:59.931548 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.931527    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89522978-782d-4988-ba3e-e7585b3799ce-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg\" (UID: \"89522978-782d-4988-ba3e-e7585b3799ce\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg"
Jul 29 10:54:59.931766 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.931750    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89522978-782d-4988-ba3e-e7585b3799ce-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg\" (UID: \"89522978-782d-4988-ba3e-e7585b3799ce\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg"
Jul 29 10:54:59.931887 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.931865    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89522978-782d-4988-ba3e-e7585b3799ce-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg\" (UID: \"89522978-782d-4988-ba3e-e7585b3799ce\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg"
Jul 29 10:54:59.939407 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:54:59.939384    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7czz\" (UniqueName: \"kubernetes.io/projected/89522978-782d-4988-ba3e-e7585b3799ce-kube-api-access-r7czz\") pod \"isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg\" (UID: \"89522978-782d-4988-ba3e-e7585b3799ce\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg"
Jul 29 10:55:00.016998 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:55:00.016913    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:55:00.275839 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:55:00.275759    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn"
Jul 29 10:55:00.275839 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:55:00.275783    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn" event={"ID":"7d0dcf35-fd34-436e-bf26-55bd01b530e0","Type":"ContainerDied","Data":"2f6950b8cd6fa4660664f180fbc51be8fb9e647fa5da754635197140ba3b3c18"}
Jul 29 10:55:00.276275 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:55:00.275859    2569 scope.go:117] "RemoveContainer" containerID="777bef0517e6b6e442e8499771e450e6c68f71db360300bf51d598726fe2f2c6"
Jul 29 10:55:00.283971 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:55:00.283957    2569 scope.go:117] "RemoveContainer" containerID="3817f42e0ef03b6b4ca65f7616aa4a0eb5c10291b4a362282d473da137276c25"
Jul 29 10:55:00.293053 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:55:00.293034    2569 scope.go:117] "RemoveContainer" containerID="b00dd025f8a0db2204492a0687ab5fcfcd94f7cc16fdf083116affacc75be833"
Jul 29 10:55:00.293922 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:55:00.293902    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn"]
Jul 29 10:55:00.296248 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:55:00.296227    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-xmlqn"]
Jul 29 10:55:00.435757 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:55:00.435714    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89522978-782d-4988-ba3e-e7585b3799ce-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg\" (UID: \"89522978-782d-4988-ba3e-e7585b3799ce\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg"
Jul 29 10:55:00.438220 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:55:00.438193    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89522978-782d-4988-ba3e-e7585b3799ce-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg\" (UID: \"89522978-782d-4988-ba3e-e7585b3799ce\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg"
Jul 29 10:55:00.692134 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:55:00.692097    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg"
Jul 29 10:55:00.824297 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:55:00.824262    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg"]
Jul 29 10:55:00.827239 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:55:00.827210    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89522978_782d_4988_ba3e_e7585b3799ce.slice/crio-13ebcbdbe1e94b3d96f1eb3b81ea19692c2404464130d6c461eba2ba71c3a8f8 WatchSource:0}: Error finding container 13ebcbdbe1e94b3d96f1eb3b81ea19692c2404464130d6c461eba2ba71c3a8f8: Status 404 returned error can't find the container with id 13ebcbdbe1e94b3d96f1eb3b81ea19692c2404464130d6c461eba2ba71c3a8f8
Jul 29 10:55:01.288606 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:55:01.288565    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg" event={"ID":"89522978-782d-4988-ba3e-e7585b3799ce","Type":"ContainerStarted","Data":"b9c8bc1e25f719e56f56a8526e409e9636187712e8b824b8663f00b59429f619"}
Jul 29 10:55:01.289126 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:55:01.288651    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg" event={"ID":"89522978-782d-4988-ba3e-e7585b3799ce","Type":"ContainerStarted","Data":"13ebcbdbe1e94b3d96f1eb3b81ea19692c2404464130d6c461eba2ba71c3a8f8"}
Jul 29 10:55:01.837592 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:55:01.837560    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d0dcf35-fd34-436e-bf26-55bd01b530e0" path="/var/lib/kubelet/pods/7d0dcf35-fd34-436e-bf26-55bd01b530e0/volumes"
Jul 29 10:55:05.305355 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:55:05.305319    2569 generic.go:356] "Generic (PLEG): container finished" podID="89522978-782d-4988-ba3e-e7585b3799ce" containerID="b9c8bc1e25f719e56f56a8526e409e9636187712e8b824b8663f00b59429f619" exitCode=0
Jul 29 10:55:05.305723 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:55:05.305377    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg" event={"ID":"89522978-782d-4988-ba3e-e7585b3799ce","Type":"ContainerDied","Data":"b9c8bc1e25f719e56f56a8526e409e9636187712e8b824b8663f00b59429f619"}
Jul 29 10:55:06.311915 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:55:06.311876    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg" event={"ID":"89522978-782d-4988-ba3e-e7585b3799ce","Type":"ContainerStarted","Data":"dd3db18747b892595523a07af137fb90b50b67e13f1b62bbecf469712e637392"}
Jul 29 10:55:06.312374 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:55:06.311952    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg" event={"ID":"89522978-782d-4988-ba3e-e7585b3799ce","Type":"ContainerStarted","Data":"9006505b2533b5704a279202a5a40108a006996a88e49cae0b52270d7f282709"}
Jul 29 10:55:06.312374 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:55:06.312207    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg"
Jul 29 10:55:06.332351 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:55:06.332298    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg" podStartSLOduration=3.4657547109999998 podStartE2EDuration="7.332282399s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.866527688s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:55:06.33031857 +0000 UTC m=+1583.055109242" watchObservedRunningTime="2026-07-29 10:55:06.332282399 +0000 UTC m=+1583.057073071"
Jul 29 10:55:07.321946 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:55:07.321908    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg"
Jul 29 10:55:07.323223 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:55:07.323196    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg" podUID="89522978-782d-4988-ba3e-e7585b3799ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 29 10:55:08.325381 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:55:08.325341    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg" podUID="89522978-782d-4988-ba3e-e7585b3799ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 29 10:55:13.329221 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:55:13.329191    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg"
Jul 29 10:55:13.329650 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:55:13.329623    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg" podUID="89522978-782d-4988-ba3e-e7585b3799ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 29 10:55:23.329666 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:55:23.329625    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg" podUID="89522978-782d-4988-ba3e-e7585b3799ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 29 10:55:33.330194 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:55:33.330149    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg" podUID="89522978-782d-4988-ba3e-e7585b3799ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 29 10:55:43.329705 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:55:43.329621    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg" podUID="89522978-782d-4988-ba3e-e7585b3799ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 29 10:55:53.329624 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:55:53.329581    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg" podUID="89522978-782d-4988-ba3e-e7585b3799ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 29 10:56:03.330480 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:03.330428    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg" podUID="89522978-782d-4988-ba3e-e7585b3799ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 29 10:56:12.413611 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:12.413578    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:56:13.330537 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:13.330501    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg" podUID="89522978-782d-4988-ba3e-e7585b3799ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 29 10:56:23.330239 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:23.330207    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg"
Jul 29 10:56:29.826011 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:29.825977    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg"]
Jul 29 10:56:29.826549 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:29.826229    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg" podUID="89522978-782d-4988-ba3e-e7585b3799ce" containerName="kserve-container" containerID="cri-o://9006505b2533b5704a279202a5a40108a006996a88e49cae0b52270d7f282709" gracePeriod=30
Jul 29 10:56:29.826549 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:29.826272    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg" podUID="89522978-782d-4988-ba3e-e7585b3799ce" containerName="kube-rbac-proxy" containerID="cri-o://dd3db18747b892595523a07af137fb90b50b67e13f1b62bbecf469712e637392" gracePeriod=30
Jul 29 10:56:29.847042 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:29.847014    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:56:29.892570 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:29.892540    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v"]
Jul 29 10:56:29.896009 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:29.895992    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v"
Jul 29 10:56:29.898608 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:29.898591    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 29 10:56:29.906636 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:29.906617    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v"]
Jul 29 10:56:29.987360 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:29.987318    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e1f84a9-442e-4699-b03d-3cc8c472c022-proxy-tls\") pod \"isvc-canary-rollback-predictor-79b59b5966-h679v\" (UID: \"2e1f84a9-442e-4699-b03d-3cc8c472c022\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v"
Jul 29 10:56:29.987512 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:29.987368    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e1f84a9-442e-4699-b03d-3cc8c472c022-cabundle-cert\") pod \"isvc-canary-rollback-predictor-79b59b5966-h679v\" (UID: \"2e1f84a9-442e-4699-b03d-3cc8c472c022\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v"
Jul 29 10:56:29.987512 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:29.987440    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb9gs\" (UniqueName: \"kubernetes.io/projected/2e1f84a9-442e-4699-b03d-3cc8c472c022-kube-api-access-zb9gs\") pod \"isvc-canary-rollback-predictor-79b59b5966-h679v\" (UID: \"2e1f84a9-442e-4699-b03d-3cc8c472c022\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v"
Jul 29 10:56:29.987585 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:29.987510    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e1f84a9-442e-4699-b03d-3cc8c472c022-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-79b59b5966-h679v\" (UID: \"2e1f84a9-442e-4699-b03d-3cc8c472c022\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v"
Jul 29 10:56:29.987585 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:29.987574    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e1f84a9-442e-4699-b03d-3cc8c472c022-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-79b59b5966-h679v\" (UID: \"2e1f84a9-442e-4699-b03d-3cc8c472c022\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v"
Jul 29 10:56:30.088332 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:30.088234    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e1f84a9-442e-4699-b03d-3cc8c472c022-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-79b59b5966-h679v\" (UID: \"2e1f84a9-442e-4699-b03d-3cc8c472c022\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v"
Jul 29 10:56:30.088332 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:30.088293    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e1f84a9-442e-4699-b03d-3cc8c472c022-proxy-tls\") pod \"isvc-canary-rollback-predictor-79b59b5966-h679v\" (UID: \"2e1f84a9-442e-4699-b03d-3cc8c472c022\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v"
Jul 29 10:56:30.088332 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:30.088314    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e1f84a9-442e-4699-b03d-3cc8c472c022-cabundle-cert\") pod \"isvc-canary-rollback-predictor-79b59b5966-h679v\" (UID: \"2e1f84a9-442e-4699-b03d-3cc8c472c022\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v"
Jul 29 10:56:30.088636 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:30.088337    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb9gs\" (UniqueName: \"kubernetes.io/projected/2e1f84a9-442e-4699-b03d-3cc8c472c022-kube-api-access-zb9gs\") pod \"isvc-canary-rollback-predictor-79b59b5966-h679v\" (UID: \"2e1f84a9-442e-4699-b03d-3cc8c472c022\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v"
Jul 29 10:56:30.088636 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:30.088375    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e1f84a9-442e-4699-b03d-3cc8c472c022-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-79b59b5966-h679v\" (UID: \"2e1f84a9-442e-4699-b03d-3cc8c472c022\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v"
Jul 29 10:56:30.088800 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:30.088775    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e1f84a9-442e-4699-b03d-3cc8c472c022-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-79b59b5966-h679v\" (UID: \"2e1f84a9-442e-4699-b03d-3cc8c472c022\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v"
Jul 29 10:56:30.088998 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:30.088974    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e1f84a9-442e-4699-b03d-3cc8c472c022-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-79b59b5966-h679v\" (UID: \"2e1f84a9-442e-4699-b03d-3cc8c472c022\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v"
Jul 29 10:56:30.089057 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:30.088979    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e1f84a9-442e-4699-b03d-3cc8c472c022-cabundle-cert\") pod \"isvc-canary-rollback-predictor-79b59b5966-h679v\" (UID: \"2e1f84a9-442e-4699-b03d-3cc8c472c022\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v"
Jul 29 10:56:30.090872 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:30.090853    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e1f84a9-442e-4699-b03d-3cc8c472c022-proxy-tls\") pod \"isvc-canary-rollback-predictor-79b59b5966-h679v\" (UID: \"2e1f84a9-442e-4699-b03d-3cc8c472c022\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v"
Jul 29 10:56:30.096931 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:30.096895    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb9gs\" (UniqueName: \"kubernetes.io/projected/2e1f84a9-442e-4699-b03d-3cc8c472c022-kube-api-access-zb9gs\") pod \"isvc-canary-rollback-predictor-79b59b5966-h679v\" (UID: \"2e1f84a9-442e-4699-b03d-3cc8c472c022\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v"
Jul 29 10:56:30.205929 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:30.205888    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v"
Jul 29 10:56:30.334126 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:30.334092    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v"]
Jul 29 10:56:30.337820 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:56:30.337791    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e1f84a9_442e_4699_b03d_3cc8c472c022.slice/crio-bf1889d85a47720eee04b024ca6b01ce7d48634736f8ebdf52ee4625f1bbe495 WatchSource:0}: Error finding container bf1889d85a47720eee04b024ca6b01ce7d48634736f8ebdf52ee4625f1bbe495: Status 404 returned error can't find the container with id bf1889d85a47720eee04b024ca6b01ce7d48634736f8ebdf52ee4625f1bbe495
Jul 29 10:56:30.649025 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:30.648908    2569 generic.go:356] "Generic (PLEG): container finished" podID="89522978-782d-4988-ba3e-e7585b3799ce" containerID="dd3db18747b892595523a07af137fb90b50b67e13f1b62bbecf469712e637392" exitCode=2
Jul 29 10:56:30.649025 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:30.648977    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg" event={"ID":"89522978-782d-4988-ba3e-e7585b3799ce","Type":"ContainerDied","Data":"dd3db18747b892595523a07af137fb90b50b67e13f1b62bbecf469712e637392"}
Jul 29 10:56:30.650492 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:30.650462    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v" event={"ID":"2e1f84a9-442e-4699-b03d-3cc8c472c022","Type":"ContainerStarted","Data":"b766c7a6963e1f0d77a1478be86eb005771f17209dcf342b43586b292d71b1a7"}
Jul 29 10:56:30.650633 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:30.650521    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v" event={"ID":"2e1f84a9-442e-4699-b03d-3cc8c472c022","Type":"ContainerStarted","Data":"bf1889d85a47720eee04b024ca6b01ce7d48634736f8ebdf52ee4625f1bbe495"}
Jul 29 10:56:33.325882 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:33.325841    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg" podUID="89522978-782d-4988-ba3e-e7585b3799ce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Jul 29 10:56:33.330180 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:33.330147    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg" podUID="89522978-782d-4988-ba3e-e7585b3799ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 29 10:56:34.666577 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:34.666548    2569 generic.go:356] "Generic (PLEG): container finished" podID="89522978-782d-4988-ba3e-e7585b3799ce" containerID="9006505b2533b5704a279202a5a40108a006996a88e49cae0b52270d7f282709" exitCode=0
Jul 29 10:56:34.666932 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:34.666621    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg" event={"ID":"89522978-782d-4988-ba3e-e7585b3799ce","Type":"ContainerDied","Data":"9006505b2533b5704a279202a5a40108a006996a88e49cae0b52270d7f282709"}
Jul 29 10:56:34.666932 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:34.666692    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg" event={"ID":"89522978-782d-4988-ba3e-e7585b3799ce","Type":"ContainerDied","Data":"13ebcbdbe1e94b3d96f1eb3b81ea19692c2404464130d6c461eba2ba71c3a8f8"}
Jul 29 10:56:34.666932 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:34.666725    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13ebcbdbe1e94b3d96f1eb3b81ea19692c2404464130d6c461eba2ba71c3a8f8"
Jul 29 10:56:34.668118 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:34.668098    2569 generic.go:356] "Generic (PLEG): container finished" podID="2e1f84a9-442e-4699-b03d-3cc8c472c022" containerID="b766c7a6963e1f0d77a1478be86eb005771f17209dcf342b43586b292d71b1a7" exitCode=0
Jul 29 10:56:34.668220 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:34.668173    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v" event={"ID":"2e1f84a9-442e-4699-b03d-3cc8c472c022","Type":"ContainerDied","Data":"b766c7a6963e1f0d77a1478be86eb005771f17209dcf342b43586b292d71b1a7"}
Jul 29 10:56:34.672885 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:34.672869    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg"
Jul 29 10:56:34.726720 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:34.726694    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89522978-782d-4988-ba3e-e7585b3799ce-proxy-tls\") pod \"89522978-782d-4988-ba3e-e7585b3799ce\" (UID: \"89522978-782d-4988-ba3e-e7585b3799ce\") "
Jul 29 10:56:34.726813 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:34.726795    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89522978-782d-4988-ba3e-e7585b3799ce-kube-rbac-proxy-sar-config\") pod \"89522978-782d-4988-ba3e-e7585b3799ce\" (UID: \"89522978-782d-4988-ba3e-e7585b3799ce\") "
Jul 29 10:56:34.727222 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:34.727189    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89522978-782d-4988-ba3e-e7585b3799ce-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "89522978-782d-4988-ba3e-e7585b3799ce" (UID: "89522978-782d-4988-ba3e-e7585b3799ce"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:56:34.728923 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:34.728904    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89522978-782d-4988-ba3e-e7585b3799ce-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "89522978-782d-4988-ba3e-e7585b3799ce" (UID: "89522978-782d-4988-ba3e-e7585b3799ce"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:56:34.827509 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:34.827451    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7czz\" (UniqueName: \"kubernetes.io/projected/89522978-782d-4988-ba3e-e7585b3799ce-kube-api-access-r7czz\") pod \"89522978-782d-4988-ba3e-e7585b3799ce\" (UID: \"89522978-782d-4988-ba3e-e7585b3799ce\") "
Jul 29 10:56:34.827509 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:34.827483    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89522978-782d-4988-ba3e-e7585b3799ce-cabundle-cert\") pod \"89522978-782d-4988-ba3e-e7585b3799ce\" (UID: \"89522978-782d-4988-ba3e-e7585b3799ce\") "
Jul 29 10:56:34.827658 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:34.827523    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89522978-782d-4988-ba3e-e7585b3799ce-kserve-provision-location\") pod \"89522978-782d-4988-ba3e-e7585b3799ce\" (UID: \"89522978-782d-4988-ba3e-e7585b3799ce\") "
Jul 29 10:56:34.827704 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:34.827683    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89522978-782d-4988-ba3e-e7585b3799ce-proxy-tls\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:56:34.827704 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:34.827693    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89522978-782d-4988-ba3e-e7585b3799ce-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:56:34.827808 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:34.827787    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89522978-782d-4988-ba3e-e7585b3799ce-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "89522978-782d-4988-ba3e-e7585b3799ce" (UID: "89522978-782d-4988-ba3e-e7585b3799ce"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:56:34.827853 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:34.827810    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89522978-782d-4988-ba3e-e7585b3799ce-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "89522978-782d-4988-ba3e-e7585b3799ce" (UID: "89522978-782d-4988-ba3e-e7585b3799ce"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:56:34.829339 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:34.829313    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89522978-782d-4988-ba3e-e7585b3799ce-kube-api-access-r7czz" (OuterVolumeSpecName: "kube-api-access-r7czz") pod "89522978-782d-4988-ba3e-e7585b3799ce" (UID: "89522978-782d-4988-ba3e-e7585b3799ce"). InnerVolumeSpecName "kube-api-access-r7czz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:56:34.928103 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:34.928047    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7czz\" (UniqueName: \"kubernetes.io/projected/89522978-782d-4988-ba3e-e7585b3799ce-kube-api-access-r7czz\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:56:34.928103 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:34.928101    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89522978-782d-4988-ba3e-e7585b3799ce-cabundle-cert\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:56:34.928103 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:34.928114    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89522978-782d-4988-ba3e-e7585b3799ce-kserve-provision-location\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:56:35.674306 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:35.674271    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v" event={"ID":"2e1f84a9-442e-4699-b03d-3cc8c472c022","Type":"ContainerStarted","Data":"3bb01307bb38e2a6182e6d856ea01017136a0f20b1aa13f068ebd0f5d4b282a6"}
Jul 29 10:56:35.674306 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:35.674307    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg"
Jul 29 10:56:35.674766 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:35.674335    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v" event={"ID":"2e1f84a9-442e-4699-b03d-3cc8c472c022","Type":"ContainerStarted","Data":"b3379fce7f44ba55289e2d66ec9dddafde5a3a99e99c2f500227272163a4c610"}
Jul 29 10:56:35.674766 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:35.674707    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v"
Jul 29 10:56:35.674766 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:35.674765    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v"
Jul 29 10:56:35.676367 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:35.676344    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v" podUID="2e1f84a9-442e-4699-b03d-3cc8c472c022" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 29 10:56:35.697294 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:35.697251    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v" podStartSLOduration=2.834354875 podStartE2EDuration="6.6972365s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.862881625s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:56:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:56:35.696314504 +0000 UTC m=+1672.421105177" watchObservedRunningTime="2026-07-29 10:56:35.6972365 +0000 UTC m=+1672.422027251"
Jul 29 10:56:35.709014 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:35.708990    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg"]
Jul 29 10:56:35.712874 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:35.712850    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-gjvjg"]
Jul 29 10:56:35.838252 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:35.838215    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89522978-782d-4988-ba3e-e7585b3799ce" path="/var/lib/kubelet/pods/89522978-782d-4988-ba3e-e7585b3799ce/volumes"
Jul 29 10:56:36.679558 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:36.679509    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v" podUID="2e1f84a9-442e-4699-b03d-3cc8c472c022" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 29 10:56:41.684457 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:41.684427    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v"
Jul 29 10:56:41.684937 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:41.684911    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v" podUID="2e1f84a9-442e-4699-b03d-3cc8c472c022" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 29 10:56:51.685832 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:56:51.685789    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v" podUID="2e1f84a9-442e-4699-b03d-3cc8c472c022" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 29 10:57:01.685342 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:57:01.685303    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v" podUID="2e1f84a9-442e-4699-b03d-3cc8c472c022" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 29 10:57:11.685709 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:57:11.685623    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v" podUID="2e1f84a9-442e-4699-b03d-3cc8c472c022" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 29 10:57:21.685289 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:57:21.685251    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v" podUID="2e1f84a9-442e-4699-b03d-3cc8c472c022" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 29 10:57:31.685354 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:57:31.685311    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v" podUID="2e1f84a9-442e-4699-b03d-3cc8c472c022" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 29 10:57:41.685593 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:57:41.685549    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v" podUID="2e1f84a9-442e-4699-b03d-3cc8c472c022" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 29 10:57:51.685764 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:57:51.685728    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v"
Jul 29 10:58:00.063042 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:00.063006    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v"]
Jul 29 10:58:00.063485 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:00.063252    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v" podUID="2e1f84a9-442e-4699-b03d-3cc8c472c022" containerName="kserve-container" containerID="cri-o://b3379fce7f44ba55289e2d66ec9dddafde5a3a99e99c2f500227272163a4c610" gracePeriod=30
Jul 29 10:58:00.063485 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:00.063361    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v" podUID="2e1f84a9-442e-4699-b03d-3cc8c472c022" containerName="kube-rbac-proxy" containerID="cri-o://3bb01307bb38e2a6182e6d856ea01017136a0f20b1aa13f068ebd0f5d4b282a6" gracePeriod=30
Jul 29 10:58:00.128510 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:00.128453    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:58:01.011932 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:01.011893    2569 generic.go:356] "Generic (PLEG): container finished" podID="2e1f84a9-442e-4699-b03d-3cc8c472c022" containerID="3bb01307bb38e2a6182e6d856ea01017136a0f20b1aa13f068ebd0f5d4b282a6" exitCode=2
Jul 29 10:58:01.012122 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:01.011961    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v" event={"ID":"2e1f84a9-442e-4699-b03d-3cc8c472c022","Type":"ContainerDied","Data":"3bb01307bb38e2a6182e6d856ea01017136a0f20b1aa13f068ebd0f5d4b282a6"}
Jul 29 10:58:01.680342 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:01.680302    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v" podUID="2e1f84a9-442e-4699-b03d-3cc8c472c022" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Jul 29 10:58:01.684931 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:01.684898    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v" podUID="2e1f84a9-442e-4699-b03d-3cc8c472c022" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 29 10:58:04.905439 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:04.905416    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v"
Jul 29 10:58:04.978564 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:04.978535    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e1f84a9-442e-4699-b03d-3cc8c472c022-cabundle-cert\") pod \"2e1f84a9-442e-4699-b03d-3cc8c472c022\" (UID: \"2e1f84a9-442e-4699-b03d-3cc8c472c022\") "
Jul 29 10:58:04.978769 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:04.978606    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zb9gs\" (UniqueName: \"kubernetes.io/projected/2e1f84a9-442e-4699-b03d-3cc8c472c022-kube-api-access-zb9gs\") pod \"2e1f84a9-442e-4699-b03d-3cc8c472c022\" (UID: \"2e1f84a9-442e-4699-b03d-3cc8c472c022\") "
Jul 29 10:58:04.978769 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:04.978646    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e1f84a9-442e-4699-b03d-3cc8c472c022-kserve-provision-location\") pod \"2e1f84a9-442e-4699-b03d-3cc8c472c022\" (UID: \"2e1f84a9-442e-4699-b03d-3cc8c472c022\") "
Jul 29 10:58:04.978769 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:04.978673    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e1f84a9-442e-4699-b03d-3cc8c472c022-proxy-tls\") pod \"2e1f84a9-442e-4699-b03d-3cc8c472c022\" (UID: \"2e1f84a9-442e-4699-b03d-3cc8c472c022\") "
Jul 29 10:58:04.978769 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:04.978734    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e1f84a9-442e-4699-b03d-3cc8c472c022-kube-rbac-proxy-sar-config\") pod \"2e1f84a9-442e-4699-b03d-3cc8c472c022\" (UID: \"2e1f84a9-442e-4699-b03d-3cc8c472c022\") "
Jul 29 10:58:04.978972 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:04.978922    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e1f84a9-442e-4699-b03d-3cc8c472c022-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2e1f84a9-442e-4699-b03d-3cc8c472c022" (UID: "2e1f84a9-442e-4699-b03d-3cc8c472c022"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:58:04.978972 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:04.978950    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e1f84a9-442e-4699-b03d-3cc8c472c022-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2e1f84a9-442e-4699-b03d-3cc8c472c022" (UID: "2e1f84a9-442e-4699-b03d-3cc8c472c022"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:58:04.979162 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:04.979137    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e1f84a9-442e-4699-b03d-3cc8c472c022-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2e1f84a9-442e-4699-b03d-3cc8c472c022" (UID: "2e1f84a9-442e-4699-b03d-3cc8c472c022"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:58:04.980755 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:04.980736    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e1f84a9-442e-4699-b03d-3cc8c472c022-kube-api-access-zb9gs" (OuterVolumeSpecName: "kube-api-access-zb9gs") pod "2e1f84a9-442e-4699-b03d-3cc8c472c022" (UID: "2e1f84a9-442e-4699-b03d-3cc8c472c022"). InnerVolumeSpecName "kube-api-access-zb9gs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:58:04.980851 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:04.980776    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e1f84a9-442e-4699-b03d-3cc8c472c022-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2e1f84a9-442e-4699-b03d-3cc8c472c022" (UID: "2e1f84a9-442e-4699-b03d-3cc8c472c022"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:58:05.034099 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.034003    2569 generic.go:356] "Generic (PLEG): container finished" podID="2e1f84a9-442e-4699-b03d-3cc8c472c022" containerID="b3379fce7f44ba55289e2d66ec9dddafde5a3a99e99c2f500227272163a4c610" exitCode=0
Jul 29 10:58:05.034232 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.034095    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v"
Jul 29 10:58:05.034232 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.034094    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v" event={"ID":"2e1f84a9-442e-4699-b03d-3cc8c472c022","Type":"ContainerDied","Data":"b3379fce7f44ba55289e2d66ec9dddafde5a3a99e99c2f500227272163a4c610"}
Jul 29 10:58:05.034232 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.034194    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v" event={"ID":"2e1f84a9-442e-4699-b03d-3cc8c472c022","Type":"ContainerDied","Data":"bf1889d85a47720eee04b024ca6b01ce7d48634736f8ebdf52ee4625f1bbe495"}
Jul 29 10:58:05.034361 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.034233    2569 scope.go:117] "RemoveContainer" containerID="3bb01307bb38e2a6182e6d856ea01017136a0f20b1aa13f068ebd0f5d4b282a6"
Jul 29 10:58:05.042564 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.042532    2569 scope.go:117] "RemoveContainer" containerID="b3379fce7f44ba55289e2d66ec9dddafde5a3a99e99c2f500227272163a4c610"
Jul 29 10:58:05.049810 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.049790    2569 scope.go:117] "RemoveContainer" containerID="b766c7a6963e1f0d77a1478be86eb005771f17209dcf342b43586b292d71b1a7"
Jul 29 10:58:05.056466 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.056446    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v"]
Jul 29 10:58:05.056782 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.056766    2569 scope.go:117] "RemoveContainer" containerID="3bb01307bb38e2a6182e6d856ea01017136a0f20b1aa13f068ebd0f5d4b282a6"
Jul 29 10:58:05.057064 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:58:05.057043    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bb01307bb38e2a6182e6d856ea01017136a0f20b1aa13f068ebd0f5d4b282a6\": container with ID starting with 3bb01307bb38e2a6182e6d856ea01017136a0f20b1aa13f068ebd0f5d4b282a6 not found: ID does not exist" containerID="3bb01307bb38e2a6182e6d856ea01017136a0f20b1aa13f068ebd0f5d4b282a6"
Jul 29 10:58:05.057212 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.057090    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb01307bb38e2a6182e6d856ea01017136a0f20b1aa13f068ebd0f5d4b282a6"} err="failed to get container status \"3bb01307bb38e2a6182e6d856ea01017136a0f20b1aa13f068ebd0f5d4b282a6\": rpc error: code = NotFound desc = could not find container \"3bb01307bb38e2a6182e6d856ea01017136a0f20b1aa13f068ebd0f5d4b282a6\": container with ID starting with 3bb01307bb38e2a6182e6d856ea01017136a0f20b1aa13f068ebd0f5d4b282a6 not found: ID does not exist"
Jul 29 10:58:05.057212 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.057108    2569 scope.go:117] "RemoveContainer" containerID="b3379fce7f44ba55289e2d66ec9dddafde5a3a99e99c2f500227272163a4c610"
Jul 29 10:58:05.057331 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:58:05.057314    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3379fce7f44ba55289e2d66ec9dddafde5a3a99e99c2f500227272163a4c610\": container with ID starting with b3379fce7f44ba55289e2d66ec9dddafde5a3a99e99c2f500227272163a4c610 not found: ID does not exist" containerID="b3379fce7f44ba55289e2d66ec9dddafde5a3a99e99c2f500227272163a4c610"
Jul 29 10:58:05.057370 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.057338    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3379fce7f44ba55289e2d66ec9dddafde5a3a99e99c2f500227272163a4c610"} err="failed to get container status \"b3379fce7f44ba55289e2d66ec9dddafde5a3a99e99c2f500227272163a4c610\": rpc error: code = NotFound desc = could not find container \"b3379fce7f44ba55289e2d66ec9dddafde5a3a99e99c2f500227272163a4c610\": container with ID starting with b3379fce7f44ba55289e2d66ec9dddafde5a3a99e99c2f500227272163a4c610 not found: ID does not exist"
Jul 29 10:58:05.057370 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.057352    2569 scope.go:117] "RemoveContainer" containerID="b766c7a6963e1f0d77a1478be86eb005771f17209dcf342b43586b292d71b1a7"
Jul 29 10:58:05.057551 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:58:05.057532    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b766c7a6963e1f0d77a1478be86eb005771f17209dcf342b43586b292d71b1a7\": container with ID starting with b766c7a6963e1f0d77a1478be86eb005771f17209dcf342b43586b292d71b1a7 not found: ID does not exist" containerID="b766c7a6963e1f0d77a1478be86eb005771f17209dcf342b43586b292d71b1a7"
Jul 29 10:58:05.057616 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.057556    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b766c7a6963e1f0d77a1478be86eb005771f17209dcf342b43586b292d71b1a7"} err="failed to get container status \"b766c7a6963e1f0d77a1478be86eb005771f17209dcf342b43586b292d71b1a7\": rpc error: code = NotFound desc = could not find container \"b766c7a6963e1f0d77a1478be86eb005771f17209dcf342b43586b292d71b1a7\": container with ID starting with b766c7a6963e1f0d77a1478be86eb005771f17209dcf342b43586b292d71b1a7 not found: ID does not exist"
Jul 29 10:58:05.063731 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.063708    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-h679v"]
Jul 29 10:58:05.079541 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.079520    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e1f84a9-442e-4699-b03d-3cc8c472c022-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:58:05.079541 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.079542    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e1f84a9-442e-4699-b03d-3cc8c472c022-cabundle-cert\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:58:05.079676 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.079553    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb9gs\" (UniqueName: \"kubernetes.io/projected/2e1f84a9-442e-4699-b03d-3cc8c472c022-kube-api-access-zb9gs\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:58:05.079676 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.079562    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e1f84a9-442e-4699-b03d-3cc8c472c022-kserve-provision-location\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:58:05.079676 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.079572    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e1f84a9-442e-4699-b03d-3cc8c472c022-proxy-tls\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:58:05.107160 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.107135    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:58:05.168577 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.168548    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:58:05.172239 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.172216    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx"]
Jul 29 10:58:05.172498 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.172485    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e1f84a9-442e-4699-b03d-3cc8c472c022" containerName="storage-initializer"
Jul 29 10:58:05.172551 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.172499    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e1f84a9-442e-4699-b03d-3cc8c472c022" containerName="storage-initializer"
Jul 29 10:58:05.172551 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.172514    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89522978-782d-4988-ba3e-e7585b3799ce" containerName="storage-initializer"
Jul 29 10:58:05.172551 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.172520    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="89522978-782d-4988-ba3e-e7585b3799ce" containerName="storage-initializer"
Jul 29 10:58:05.172551 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.172527    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89522978-782d-4988-ba3e-e7585b3799ce" containerName="kube-rbac-proxy"
Jul 29 10:58:05.172551 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.172532    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="89522978-782d-4988-ba3e-e7585b3799ce" containerName="kube-rbac-proxy"
Jul 29 10:58:05.172551 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.172538    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e1f84a9-442e-4699-b03d-3cc8c472c022" containerName="kube-rbac-proxy"
Jul 29 10:58:05.172551 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.172544    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e1f84a9-442e-4699-b03d-3cc8c472c022" containerName="kube-rbac-proxy"
Jul 29 10:58:05.172551 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.172554    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e1f84a9-442e-4699-b03d-3cc8c472c022" containerName="kserve-container"
Jul 29 10:58:05.172789 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.172560    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e1f84a9-442e-4699-b03d-3cc8c472c022" containerName="kserve-container"
Jul 29 10:58:05.172789 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.172568    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89522978-782d-4988-ba3e-e7585b3799ce" containerName="kserve-container"
Jul 29 10:58:05.172789 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.172573    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="89522978-782d-4988-ba3e-e7585b3799ce" containerName="kserve-container"
Jul 29 10:58:05.172789 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.172623    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="89522978-782d-4988-ba3e-e7585b3799ce" containerName="kserve-container"
Jul 29 10:58:05.172789 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.172630    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="89522978-782d-4988-ba3e-e7585b3799ce" containerName="kube-rbac-proxy"
Jul 29 10:58:05.172789 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.172636    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e1f84a9-442e-4699-b03d-3cc8c472c022" containerName="kserve-container"
Jul 29 10:58:05.172789 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.172641    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e1f84a9-442e-4699-b03d-3cc8c472c022" containerName="kube-rbac-proxy"
Jul 29 10:58:05.177151 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.177133    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx"
Jul 29 10:58:05.180031 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.180010    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6nnxs\""
Jul 29 10:58:05.180140 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.180097    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 10:58:05.180276 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.180259    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Jul 29 10:58:05.180543 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.180528    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 10:58:05.180624 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.180609    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 10:58:05.180682 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.180624    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Jul 29 10:58:05.180738 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.180723    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 10:58:05.187719 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.187698    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx"]
Jul 29 10:58:05.220466 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.220444    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:58:05.280964 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.280928    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c404ff9c-cffc-4ab0-b890-987ee38f152f-proxy-tls\") pod \"isvc-canary-stop-predictor-75dfd479cd-dlskx\" (UID: \"c404ff9c-cffc-4ab0-b890-987ee38f152f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx"
Jul 29 10:58:05.280964 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.280966    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c404ff9c-cffc-4ab0-b890-987ee38f152f-cabundle-cert\") pod \"isvc-canary-stop-predictor-75dfd479cd-dlskx\" (UID: \"c404ff9c-cffc-4ab0-b890-987ee38f152f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx"
Jul 29 10:58:05.281196 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.281031    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbx4d\" (UniqueName: \"kubernetes.io/projected/c404ff9c-cffc-4ab0-b890-987ee38f152f-kube-api-access-xbx4d\") pod \"isvc-canary-stop-predictor-75dfd479cd-dlskx\" (UID: \"c404ff9c-cffc-4ab0-b890-987ee38f152f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx"
Jul 29 10:58:05.281196 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.281117    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c404ff9c-cffc-4ab0-b890-987ee38f152f-kserve-provision-location\") pod \"isvc-canary-stop-predictor-75dfd479cd-dlskx\" (UID: \"c404ff9c-cffc-4ab0-b890-987ee38f152f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx"
Jul 29 10:58:05.281196 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.281177    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c404ff9c-cffc-4ab0-b890-987ee38f152f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-75dfd479cd-dlskx\" (UID: \"c404ff9c-cffc-4ab0-b890-987ee38f152f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx"
Jul 29 10:58:05.345219 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.345131    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:58:05.382305 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.382270    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c404ff9c-cffc-4ab0-b890-987ee38f152f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-75dfd479cd-dlskx\" (UID: \"c404ff9c-cffc-4ab0-b890-987ee38f152f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx"
Jul 29 10:58:05.382456 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.382329    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c404ff9c-cffc-4ab0-b890-987ee38f152f-proxy-tls\") pod \"isvc-canary-stop-predictor-75dfd479cd-dlskx\" (UID: \"c404ff9c-cffc-4ab0-b890-987ee38f152f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx"
Jul 29 10:58:05.382456 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.382348    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c404ff9c-cffc-4ab0-b890-987ee38f152f-cabundle-cert\") pod \"isvc-canary-stop-predictor-75dfd479cd-dlskx\" (UID: \"c404ff9c-cffc-4ab0-b890-987ee38f152f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx"
Jul 29 10:58:05.382456 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.382381    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbx4d\" (UniqueName: \"kubernetes.io/projected/c404ff9c-cffc-4ab0-b890-987ee38f152f-kube-api-access-xbx4d\") pod \"isvc-canary-stop-predictor-75dfd479cd-dlskx\" (UID: \"c404ff9c-cffc-4ab0-b890-987ee38f152f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx"
Jul 29 10:58:05.382456 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.382409    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c404ff9c-cffc-4ab0-b890-987ee38f152f-kserve-provision-location\") pod \"isvc-canary-stop-predictor-75dfd479cd-dlskx\" (UID: \"c404ff9c-cffc-4ab0-b890-987ee38f152f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx"
Jul 29 10:58:05.382861 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.382837    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c404ff9c-cffc-4ab0-b890-987ee38f152f-kserve-provision-location\") pod \"isvc-canary-stop-predictor-75dfd479cd-dlskx\" (UID: \"c404ff9c-cffc-4ab0-b890-987ee38f152f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx"
Jul 29 10:58:05.383127 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.383065    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c404ff9c-cffc-4ab0-b890-987ee38f152f-cabundle-cert\") pod \"isvc-canary-stop-predictor-75dfd479cd-dlskx\" (UID: \"c404ff9c-cffc-4ab0-b890-987ee38f152f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx"
Jul 29 10:58:05.383127 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.383062    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c404ff9c-cffc-4ab0-b890-987ee38f152f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-75dfd479cd-dlskx\" (UID: \"c404ff9c-cffc-4ab0-b890-987ee38f152f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx"
Jul 29 10:58:05.384803 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.384787    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c404ff9c-cffc-4ab0-b890-987ee38f152f-proxy-tls\") pod \"isvc-canary-stop-predictor-75dfd479cd-dlskx\" (UID: \"c404ff9c-cffc-4ab0-b890-987ee38f152f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx"
Jul 29 10:58:05.391567 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.391549    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbx4d\" (UniqueName: \"kubernetes.io/projected/c404ff9c-cffc-4ab0-b890-987ee38f152f-kube-api-access-xbx4d\") pod \"isvc-canary-stop-predictor-75dfd479cd-dlskx\" (UID: \"c404ff9c-cffc-4ab0-b890-987ee38f152f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx"
Jul 29 10:58:05.489985 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.489958    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx"
Jul 29 10:58:05.619661 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.619573    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx"]
Jul 29 10:58:05.622689 ip-10-0-137-111 kubenswrapper[2569]: W0729 10:58:05.622666    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc404ff9c_cffc_4ab0_b890_987ee38f152f.slice/crio-aa1f81696f317c1bfe23206f824658d1a7ae4b27198579cbd5727dbcdc8542c1 WatchSource:0}: Error finding container aa1f81696f317c1bfe23206f824658d1a7ae4b27198579cbd5727dbcdc8542c1: Status 404 returned error can't find the container with id aa1f81696f317c1bfe23206f824658d1a7ae4b27198579cbd5727dbcdc8542c1
Jul 29 10:58:05.837609 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:05.837575    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e1f84a9-442e-4699-b03d-3cc8c472c022" path="/var/lib/kubelet/pods/2e1f84a9-442e-4699-b03d-3cc8c472c022/volumes"
Jul 29 10:58:06.039567 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:06.039525    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx" event={"ID":"c404ff9c-cffc-4ab0-b890-987ee38f152f","Type":"ContainerStarted","Data":"f2a58b222a8d28b9a236aeb1deb064ecc4c76013455738e5949103c2ba8b8afb"}
Jul 29 10:58:06.039995 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:06.039600    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx" event={"ID":"c404ff9c-cffc-4ab0-b890-987ee38f152f","Type":"ContainerStarted","Data":"aa1f81696f317c1bfe23206f824658d1a7ae4b27198579cbd5727dbcdc8542c1"}
Jul 29 10:58:10.057543 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:10.057510    2569 generic.go:356] "Generic (PLEG): container finished" podID="c404ff9c-cffc-4ab0-b890-987ee38f152f" containerID="f2a58b222a8d28b9a236aeb1deb064ecc4c76013455738e5949103c2ba8b8afb" exitCode=0
Jul 29 10:58:10.057891 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:10.057584    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx" event={"ID":"c404ff9c-cffc-4ab0-b890-987ee38f152f","Type":"ContainerDied","Data":"f2a58b222a8d28b9a236aeb1deb064ecc4c76013455738e5949103c2ba8b8afb"}
Jul 29 10:58:11.064709 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:11.064670    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx" event={"ID":"c404ff9c-cffc-4ab0-b890-987ee38f152f","Type":"ContainerStarted","Data":"2999476daf190eba33ea61e8a63061e47ad368edf95735c2d87ee57e674c670a"}
Jul 29 10:58:11.065132 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:11.064734    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx" event={"ID":"c404ff9c-cffc-4ab0-b890-987ee38f152f","Type":"ContainerStarted","Data":"ef22acd159e25dc747d11353bf09ba6322dbd61b16ca4a82a910bc0ce3fd7547"}
Jul 29 10:58:11.065132 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:11.064931    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx"
Jul 29 10:58:11.083949 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:11.083900    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx" podStartSLOduration=2.358662832 podStartE2EDuration="6.083884993s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.725222161s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:58:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:58:11.082594822 +0000 UTC m=+1767.807385497" watchObservedRunningTime="2026-07-29 10:58:11.083884993 +0000 UTC m=+1767.808675665"
Jul 29 10:58:12.069473 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:12.069444    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx"
Jul 29 10:58:12.070655 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:12.070629    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx" podUID="c404ff9c-cffc-4ab0-b890-987ee38f152f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 29 10:58:13.073703 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:13.073659    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx" podUID="c404ff9c-cffc-4ab0-b890-987ee38f152f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 29 10:58:18.078431 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:18.078400    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx"
Jul 29 10:58:18.078855 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:18.078829    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx" podUID="c404ff9c-cffc-4ab0-b890-987ee38f152f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 29 10:58:19.566831 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:19.566789    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:58:28.078847 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:28.078795    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx" podUID="c404ff9c-cffc-4ab0-b890-987ee38f152f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 29 10:58:38.079632 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:38.079546    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx" podUID="c404ff9c-cffc-4ab0-b890-987ee38f152f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 29 10:58:43.825963 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:43.825936    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-pvdgw_afd0d65c-bbfc-49f0-81b6-e855fd81da7f/console-operator/1.log"
Jul 29 10:58:43.828715 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:43.828684    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-pvdgw_afd0d65c-bbfc-49f0-81b6-e855fd81da7f/console-operator/1.log"
Jul 29 10:58:48.079015 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:48.078976    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx" podUID="c404ff9c-cffc-4ab0-b890-987ee38f152f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 29 10:58:58.079465 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:58:58.079427    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx" podUID="c404ff9c-cffc-4ab0-b890-987ee38f152f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 29 10:59:08.079575 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:08.079534    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx" podUID="c404ff9c-cffc-4ab0-b890-987ee38f152f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 29 10:59:18.078913 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:18.078871    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx" podUID="c404ff9c-cffc-4ab0-b890-987ee38f152f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 29 10:59:28.079780 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:28.079737    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx"
Jul 29 10:59:35.194012 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:35.193967    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx"]
Jul 29 10:59:35.194504 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:35.194281    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx" podUID="c404ff9c-cffc-4ab0-b890-987ee38f152f" containerName="kserve-container" containerID="cri-o://ef22acd159e25dc747d11353bf09ba6322dbd61b16ca4a82a910bc0ce3fd7547" gracePeriod=30
Jul 29 10:59:35.194504 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:35.194333    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx" podUID="c404ff9c-cffc-4ab0-b890-987ee38f152f" containerName="kube-rbac-proxy" containerID="cri-o://2999476daf190eba33ea61e8a63061e47ad368edf95735c2d87ee57e674c670a" gracePeriod=30
Jul 29 10:59:35.198331 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:35.198296    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:59:35.260916 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:35.260874    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:59:35.404296 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:35.404265    2569 generic.go:356] "Generic (PLEG): container finished" podID="c404ff9c-cffc-4ab0-b890-987ee38f152f" containerID="2999476daf190eba33ea61e8a63061e47ad368edf95735c2d87ee57e674c670a" exitCode=2
Jul 29 10:59:35.404499 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:35.404323    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx" event={"ID":"c404ff9c-cffc-4ab0-b890-987ee38f152f","Type":"ContainerDied","Data":"2999476daf190eba33ea61e8a63061e47ad368edf95735c2d87ee57e674c670a"}
Jul 29 10:59:38.074790 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:38.074745    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx" podUID="c404ff9c-cffc-4ab0-b890-987ee38f152f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Jul 29 10:59:38.079220 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:38.079178    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx" podUID="c404ff9c-cffc-4ab0-b890-987ee38f152f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 29 10:59:40.041122 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:40.041097    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx"
Jul 29 10:59:40.063138 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:40.063108    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c404ff9c-cffc-4ab0-b890-987ee38f152f-proxy-tls\") pod \"c404ff9c-cffc-4ab0-b890-987ee38f152f\" (UID: \"c404ff9c-cffc-4ab0-b890-987ee38f152f\") "
Jul 29 10:59:40.063311 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:40.063154    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbx4d\" (UniqueName: \"kubernetes.io/projected/c404ff9c-cffc-4ab0-b890-987ee38f152f-kube-api-access-xbx4d\") pod \"c404ff9c-cffc-4ab0-b890-987ee38f152f\" (UID: \"c404ff9c-cffc-4ab0-b890-987ee38f152f\") "
Jul 29 10:59:40.063311 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:40.063214    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c404ff9c-cffc-4ab0-b890-987ee38f152f-kserve-provision-location\") pod \"c404ff9c-cffc-4ab0-b890-987ee38f152f\" (UID: \"c404ff9c-cffc-4ab0-b890-987ee38f152f\") "
Jul 29 10:59:40.063311 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:40.063276    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c404ff9c-cffc-4ab0-b890-987ee38f152f-kube-rbac-proxy-sar-config\") pod \"c404ff9c-cffc-4ab0-b890-987ee38f152f\" (UID: \"c404ff9c-cffc-4ab0-b890-987ee38f152f\") "
Jul 29 10:59:40.063311 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:40.063306    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c404ff9c-cffc-4ab0-b890-987ee38f152f-cabundle-cert\") pod \"c404ff9c-cffc-4ab0-b890-987ee38f152f\" (UID: \"c404ff9c-cffc-4ab0-b890-987ee38f152f\") "
Jul 29 10:59:40.063676 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:40.063584    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c404ff9c-cffc-4ab0-b890-987ee38f152f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c404ff9c-cffc-4ab0-b890-987ee38f152f" (UID: "c404ff9c-cffc-4ab0-b890-987ee38f152f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:59:40.063784 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:40.063724    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c404ff9c-cffc-4ab0-b890-987ee38f152f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c404ff9c-cffc-4ab0-b890-987ee38f152f" (UID: "c404ff9c-cffc-4ab0-b890-987ee38f152f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:59:40.063849 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:40.063798    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c404ff9c-cffc-4ab0-b890-987ee38f152f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c404ff9c-cffc-4ab0-b890-987ee38f152f" (UID: "c404ff9c-cffc-4ab0-b890-987ee38f152f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:59:40.065393 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:40.065368    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c404ff9c-cffc-4ab0-b890-987ee38f152f-kube-api-access-xbx4d" (OuterVolumeSpecName: "kube-api-access-xbx4d") pod "c404ff9c-cffc-4ab0-b890-987ee38f152f" (UID: "c404ff9c-cffc-4ab0-b890-987ee38f152f"). InnerVolumeSpecName "kube-api-access-xbx4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:59:40.065502 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:40.065486    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c404ff9c-cffc-4ab0-b890-987ee38f152f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c404ff9c-cffc-4ab0-b890-987ee38f152f" (UID: "c404ff9c-cffc-4ab0-b890-987ee38f152f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:59:40.164789 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:40.164754    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c404ff9c-cffc-4ab0-b890-987ee38f152f-kserve-provision-location\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:59:40.164789 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:40.164789    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c404ff9c-cffc-4ab0-b890-987ee38f152f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:59:40.165002 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:40.164802    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c404ff9c-cffc-4ab0-b890-987ee38f152f-cabundle-cert\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:59:40.165002 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:40.164813    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c404ff9c-cffc-4ab0-b890-987ee38f152f-proxy-tls\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:59:40.165002 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:40.164821    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbx4d\" (UniqueName: \"kubernetes.io/projected/c404ff9c-cffc-4ab0-b890-987ee38f152f-kube-api-access-xbx4d\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 10:59:40.425515 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:40.425428    2569 generic.go:356] "Generic (PLEG): container finished" podID="c404ff9c-cffc-4ab0-b890-987ee38f152f" containerID="ef22acd159e25dc747d11353bf09ba6322dbd61b16ca4a82a910bc0ce3fd7547" exitCode=0
Jul 29 10:59:40.425515 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:40.425483    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx"
Jul 29 10:59:40.425692 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:40.425485    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx" event={"ID":"c404ff9c-cffc-4ab0-b890-987ee38f152f","Type":"ContainerDied","Data":"ef22acd159e25dc747d11353bf09ba6322dbd61b16ca4a82a910bc0ce3fd7547"}
Jul 29 10:59:40.425692 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:40.425624    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx" event={"ID":"c404ff9c-cffc-4ab0-b890-987ee38f152f","Type":"ContainerDied","Data":"aa1f81696f317c1bfe23206f824658d1a7ae4b27198579cbd5727dbcdc8542c1"}
Jul 29 10:59:40.425692 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:40.425667    2569 scope.go:117] "RemoveContainer" containerID="2999476daf190eba33ea61e8a63061e47ad368edf95735c2d87ee57e674c670a"
Jul 29 10:59:40.434190 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:40.434172    2569 scope.go:117] "RemoveContainer" containerID="ef22acd159e25dc747d11353bf09ba6322dbd61b16ca4a82a910bc0ce3fd7547"
Jul 29 10:59:40.441321 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:40.441306    2569 scope.go:117] "RemoveContainer" containerID="f2a58b222a8d28b9a236aeb1deb064ecc4c76013455738e5949103c2ba8b8afb"
Jul 29 10:59:40.447423 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:40.447400    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx"]
Jul 29 10:59:40.448683 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:40.448666    2569 scope.go:117] "RemoveContainer" containerID="2999476daf190eba33ea61e8a63061e47ad368edf95735c2d87ee57e674c670a"
Jul 29 10:59:40.448937 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:59:40.448916    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2999476daf190eba33ea61e8a63061e47ad368edf95735c2d87ee57e674c670a\": container with ID starting with 2999476daf190eba33ea61e8a63061e47ad368edf95735c2d87ee57e674c670a not found: ID does not exist" containerID="2999476daf190eba33ea61e8a63061e47ad368edf95735c2d87ee57e674c670a"
Jul 29 10:59:40.448998 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:40.448943    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2999476daf190eba33ea61e8a63061e47ad368edf95735c2d87ee57e674c670a"} err="failed to get container status \"2999476daf190eba33ea61e8a63061e47ad368edf95735c2d87ee57e674c670a\": rpc error: code = NotFound desc = could not find container \"2999476daf190eba33ea61e8a63061e47ad368edf95735c2d87ee57e674c670a\": container with ID starting with 2999476daf190eba33ea61e8a63061e47ad368edf95735c2d87ee57e674c670a not found: ID does not exist"
Jul 29 10:59:40.448998 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:40.448959    2569 scope.go:117] "RemoveContainer" containerID="ef22acd159e25dc747d11353bf09ba6322dbd61b16ca4a82a910bc0ce3fd7547"
Jul 29 10:59:40.449244 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:59:40.449228    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef22acd159e25dc747d11353bf09ba6322dbd61b16ca4a82a910bc0ce3fd7547\": container with ID starting with ef22acd159e25dc747d11353bf09ba6322dbd61b16ca4a82a910bc0ce3fd7547 not found: ID does not exist" containerID="ef22acd159e25dc747d11353bf09ba6322dbd61b16ca4a82a910bc0ce3fd7547"
Jul 29 10:59:40.449300 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:40.449251    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef22acd159e25dc747d11353bf09ba6322dbd61b16ca4a82a910bc0ce3fd7547"} err="failed to get container status \"ef22acd159e25dc747d11353bf09ba6322dbd61b16ca4a82a910bc0ce3fd7547\": rpc error: code = NotFound desc = could not find container \"ef22acd159e25dc747d11353bf09ba6322dbd61b16ca4a82a910bc0ce3fd7547\": container with ID starting with ef22acd159e25dc747d11353bf09ba6322dbd61b16ca4a82a910bc0ce3fd7547 not found: ID does not exist"
Jul 29 10:59:40.449300 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:40.449269    2569 scope.go:117] "RemoveContainer" containerID="f2a58b222a8d28b9a236aeb1deb064ecc4c76013455738e5949103c2ba8b8afb"
Jul 29 10:59:40.449737 ip-10-0-137-111 kubenswrapper[2569]: E0729 10:59:40.449716    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2a58b222a8d28b9a236aeb1deb064ecc4c76013455738e5949103c2ba8b8afb\": container with ID starting with f2a58b222a8d28b9a236aeb1deb064ecc4c76013455738e5949103c2ba8b8afb not found: ID does not exist" containerID="f2a58b222a8d28b9a236aeb1deb064ecc4c76013455738e5949103c2ba8b8afb"
Jul 29 10:59:40.449814 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:40.449741    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2a58b222a8d28b9a236aeb1deb064ecc4c76013455738e5949103c2ba8b8afb"} err="failed to get container status \"f2a58b222a8d28b9a236aeb1deb064ecc4c76013455738e5949103c2ba8b8afb\": rpc error: code = NotFound desc = could not find container \"f2a58b222a8d28b9a236aeb1deb064ecc4c76013455738e5949103c2ba8b8afb\": container with ID starting with f2a58b222a8d28b9a236aeb1deb064ecc4c76013455738e5949103c2ba8b8afb not found: ID does not exist"
Jul 29 10:59:40.451148 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:40.451130    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-dlskx"]
Jul 29 10:59:41.837413 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:41.837380    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c404ff9c-cffc-4ab0-b890-987ee38f152f" path="/var/lib/kubelet/pods/c404ff9c-cffc-4ab0-b890-987ee38f152f/volumes"
Jul 29 10:59:42.529667 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:42.529632    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:59:45.414013 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:45.413975    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 10:59:45.528695 ip-10-0-137-111 kubenswrapper[2569]: I0729 10:59:45.528664    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 11:01:30.664000 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:30.663963    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 11:01:30.676792 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:30.676759    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a23c96-predictor-87474ddf-ww49p"]
Jul 29 11:01:30.677180 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:30.677162    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c404ff9c-cffc-4ab0-b890-987ee38f152f" containerName="kserve-container"
Jul 29 11:01:30.677274 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:30.677182    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c404ff9c-cffc-4ab0-b890-987ee38f152f" containerName="kserve-container"
Jul 29 11:01:30.677274 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:30.677218    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c404ff9c-cffc-4ab0-b890-987ee38f152f" containerName="kube-rbac-proxy"
Jul 29 11:01:30.677274 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:30.677228    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c404ff9c-cffc-4ab0-b890-987ee38f152f" containerName="kube-rbac-proxy"
Jul 29 11:01:30.677274 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:30.677240    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c404ff9c-cffc-4ab0-b890-987ee38f152f" containerName="storage-initializer"
Jul 29 11:01:30.677274 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:30.677250    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c404ff9c-cffc-4ab0-b890-987ee38f152f" containerName="storage-initializer"
Jul 29 11:01:30.677537 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:30.677325    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c404ff9c-cffc-4ab0-b890-987ee38f152f" containerName="kserve-container"
Jul 29 11:01:30.677537 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:30.677338    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c404ff9c-cffc-4ab0-b890-987ee38f152f" containerName="kube-rbac-proxy"
Jul 29 11:01:30.680358 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:30.680338    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a23c96-predictor-87474ddf-ww49p"
Jul 29 11:01:30.683606 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:30.683588    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-a23c96-dockercfg-24jgf\""
Jul 29 11:01:30.683895 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:30.683878    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-a23c96-kube-rbac-proxy-sar-config\""
Jul 29 11:01:30.683952 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:30.683892    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 11:01:30.684691 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:30.684672    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-a23c96\""
Jul 29 11:01:30.684779 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:30.684670    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:01:30.684779 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:30.684720    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 11:01:30.684779 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:30.684734    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-a23c96-predictor-serving-cert\""
Jul 29 11:01:30.690831 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:30.690810    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a23c96-predictor-87474ddf-ww49p"]
Jul 29 11:01:30.771595 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:30.771561    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2e7c7dc-9da4-4c4b-9ac9-2283614a8637-kserve-provision-location\") pod \"isvc-secondary-a23c96-predictor-87474ddf-ww49p\" (UID: \"b2e7c7dc-9da4-4c4b-9ac9-2283614a8637\") " pod="kserve-ci-e2e-test/isvc-secondary-a23c96-predictor-87474ddf-ww49p"
Jul 29 11:01:30.771754 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:30.771603    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2e7c7dc-9da4-4c4b-9ac9-2283614a8637-proxy-tls\") pod \"isvc-secondary-a23c96-predictor-87474ddf-ww49p\" (UID: \"b2e7c7dc-9da4-4c4b-9ac9-2283614a8637\") " pod="kserve-ci-e2e-test/isvc-secondary-a23c96-predictor-87474ddf-ww49p"
Jul 29 11:01:30.771754 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:30.771627    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2e7c7dc-9da4-4c4b-9ac9-2283614a8637-cabundle-cert\") pod \"isvc-secondary-a23c96-predictor-87474ddf-ww49p\" (UID: \"b2e7c7dc-9da4-4c4b-9ac9-2283614a8637\") " pod="kserve-ci-e2e-test/isvc-secondary-a23c96-predictor-87474ddf-ww49p"
Jul 29 11:01:30.771754 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:30.771649    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2e7c7dc-9da4-4c4b-9ac9-2283614a8637-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-a23c96-predictor-87474ddf-ww49p\" (UID: \"b2e7c7dc-9da4-4c4b-9ac9-2283614a8637\") " pod="kserve-ci-e2e-test/isvc-secondary-a23c96-predictor-87474ddf-ww49p"
Jul 29 11:01:30.771754 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:30.771710    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gbbh\" (UniqueName: \"kubernetes.io/projected/b2e7c7dc-9da4-4c4b-9ac9-2283614a8637-kube-api-access-7gbbh\") pod \"isvc-secondary-a23c96-predictor-87474ddf-ww49p\" (UID: \"b2e7c7dc-9da4-4c4b-9ac9-2283614a8637\") " pod="kserve-ci-e2e-test/isvc-secondary-a23c96-predictor-87474ddf-ww49p"
Jul 29 11:01:30.872429 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:30.872399    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gbbh\" (UniqueName: \"kubernetes.io/projected/b2e7c7dc-9da4-4c4b-9ac9-2283614a8637-kube-api-access-7gbbh\") pod \"isvc-secondary-a23c96-predictor-87474ddf-ww49p\" (UID: \"b2e7c7dc-9da4-4c4b-9ac9-2283614a8637\") " pod="kserve-ci-e2e-test/isvc-secondary-a23c96-predictor-87474ddf-ww49p"
Jul 29 11:01:30.872429 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:30.872434    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2e7c7dc-9da4-4c4b-9ac9-2283614a8637-kserve-provision-location\") pod \"isvc-secondary-a23c96-predictor-87474ddf-ww49p\" (UID: \"b2e7c7dc-9da4-4c4b-9ac9-2283614a8637\") " pod="kserve-ci-e2e-test/isvc-secondary-a23c96-predictor-87474ddf-ww49p"
Jul 29 11:01:30.872632 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:30.872541    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2e7c7dc-9da4-4c4b-9ac9-2283614a8637-proxy-tls\") pod \"isvc-secondary-a23c96-predictor-87474ddf-ww49p\" (UID: \"b2e7c7dc-9da4-4c4b-9ac9-2283614a8637\") " pod="kserve-ci-e2e-test/isvc-secondary-a23c96-predictor-87474ddf-ww49p"
Jul 29 11:01:30.872632 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:30.872563    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2e7c7dc-9da4-4c4b-9ac9-2283614a8637-cabundle-cert\") pod \"isvc-secondary-a23c96-predictor-87474ddf-ww49p\" (UID: \"b2e7c7dc-9da4-4c4b-9ac9-2283614a8637\") " pod="kserve-ci-e2e-test/isvc-secondary-a23c96-predictor-87474ddf-ww49p"
Jul 29 11:01:30.872632 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:30.872595    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2e7c7dc-9da4-4c4b-9ac9-2283614a8637-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-a23c96-predictor-87474ddf-ww49p\" (UID: \"b2e7c7dc-9da4-4c4b-9ac9-2283614a8637\") " pod="kserve-ci-e2e-test/isvc-secondary-a23c96-predictor-87474ddf-ww49p"
Jul 29 11:01:30.872975 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:30.872946    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2e7c7dc-9da4-4c4b-9ac9-2283614a8637-kserve-provision-location\") pod \"isvc-secondary-a23c96-predictor-87474ddf-ww49p\" (UID: \"b2e7c7dc-9da4-4c4b-9ac9-2283614a8637\") " pod="kserve-ci-e2e-test/isvc-secondary-a23c96-predictor-87474ddf-ww49p"
Jul 29 11:01:30.873221 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:30.873200    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2e7c7dc-9da4-4c4b-9ac9-2283614a8637-cabundle-cert\") pod \"isvc-secondary-a23c96-predictor-87474ddf-ww49p\" (UID: \"b2e7c7dc-9da4-4c4b-9ac9-2283614a8637\") " pod="kserve-ci-e2e-test/isvc-secondary-a23c96-predictor-87474ddf-ww49p"
Jul 29 11:01:30.873322 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:30.873298    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2e7c7dc-9da4-4c4b-9ac9-2283614a8637-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-a23c96-predictor-87474ddf-ww49p\" (UID: \"b2e7c7dc-9da4-4c4b-9ac9-2283614a8637\") " pod="kserve-ci-e2e-test/isvc-secondary-a23c96-predictor-87474ddf-ww49p"
Jul 29 11:01:30.875005 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:30.874988    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2e7c7dc-9da4-4c4b-9ac9-2283614a8637-proxy-tls\") pod \"isvc-secondary-a23c96-predictor-87474ddf-ww49p\" (UID: \"b2e7c7dc-9da4-4c4b-9ac9-2283614a8637\") " pod="kserve-ci-e2e-test/isvc-secondary-a23c96-predictor-87474ddf-ww49p"
Jul 29 11:01:30.880783 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:30.880761    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gbbh\" (UniqueName: \"kubernetes.io/projected/b2e7c7dc-9da4-4c4b-9ac9-2283614a8637-kube-api-access-7gbbh\") pod \"isvc-secondary-a23c96-predictor-87474ddf-ww49p\" (UID: \"b2e7c7dc-9da4-4c4b-9ac9-2283614a8637\") " pod="kserve-ci-e2e-test/isvc-secondary-a23c96-predictor-87474ddf-ww49p"
Jul 29 11:01:30.991116 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:30.991003    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a23c96-predictor-87474ddf-ww49p"
Jul 29 11:01:31.125685 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:31.125650    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a23c96-predictor-87474ddf-ww49p"]
Jul 29 11:01:31.128254 ip-10-0-137-111 kubenswrapper[2569]: W0729 11:01:31.128228    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e7c7dc_9da4_4c4b_9ac9_2283614a8637.slice/crio-b90e478465958d773b9661c321be9e9c3635960df2f94a91f917399461ddd88b WatchSource:0}: Error finding container b90e478465958d773b9661c321be9e9c3635960df2f94a91f917399461ddd88b: Status 404 returned error can't find the container with id b90e478465958d773b9661c321be9e9c3635960df2f94a91f917399461ddd88b
Jul 29 11:01:31.855214 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:31.855179    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a23c96-predictor-87474ddf-ww49p" event={"ID":"b2e7c7dc-9da4-4c4b-9ac9-2283614a8637","Type":"ContainerStarted","Data":"5615add369f0f6a122c2d56e55719fa907e0f813ac7af355ad277f861be401f0"}
Jul 29 11:01:31.855617 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:31.855238    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a23c96-predictor-87474ddf-ww49p" event={"ID":"b2e7c7dc-9da4-4c4b-9ac9-2283614a8637","Type":"ContainerStarted","Data":"b90e478465958d773b9661c321be9e9c3635960df2f94a91f917399461ddd88b"}
Jul 29 11:01:32.461584 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:32.461533    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 11:01:32.860502 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:32.860469    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a23c96-predictor-87474ddf-ww49p_b2e7c7dc-9da4-4c4b-9ac9-2283614a8637/storage-initializer/0.log"
Jul 29 11:01:32.860982 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:32.860515    2569 generic.go:356] "Generic (PLEG): container finished" podID="b2e7c7dc-9da4-4c4b-9ac9-2283614a8637" containerID="5615add369f0f6a122c2d56e55719fa907e0f813ac7af355ad277f861be401f0" exitCode=1
Jul 29 11:01:32.860982 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:32.860558    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a23c96-predictor-87474ddf-ww49p" event={"ID":"b2e7c7dc-9da4-4c4b-9ac9-2283614a8637","Type":"ContainerDied","Data":"5615add369f0f6a122c2d56e55719fa907e0f813ac7af355ad277f861be401f0"}
Jul 29 11:01:33.866061 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:33.866029    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a23c96-predictor-87474ddf-ww49p_b2e7c7dc-9da4-4c4b-9ac9-2283614a8637/storage-initializer/0.log"
Jul 29 11:01:33.866446 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:33.866093    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a23c96-predictor-87474ddf-ww49p" event={"ID":"b2e7c7dc-9da4-4c4b-9ac9-2283614a8637","Type":"ContainerStarted","Data":"c4c2c3ed817caf224ab9ce6c6c5bcf06068283bd34615fcaf48061c671417bf5"}
Jul 29 11:01:40.896317 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:40.896288    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a23c96-predictor-87474ddf-ww49p_b2e7c7dc-9da4-4c4b-9ac9-2283614a8637/storage-initializer/1.log"
Jul 29 11:01:40.896730 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:40.896705    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a23c96-predictor-87474ddf-ww49p_b2e7c7dc-9da4-4c4b-9ac9-2283614a8637/storage-initializer/0.log"
Jul 29 11:01:40.896777 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:40.896737    2569 generic.go:356] "Generic (PLEG): container finished" podID="b2e7c7dc-9da4-4c4b-9ac9-2283614a8637" containerID="c4c2c3ed817caf224ab9ce6c6c5bcf06068283bd34615fcaf48061c671417bf5" exitCode=1
Jul 29 11:01:40.896817 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:40.896770    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a23c96-predictor-87474ddf-ww49p" event={"ID":"b2e7c7dc-9da4-4c4b-9ac9-2283614a8637","Type":"ContainerDied","Data":"c4c2c3ed817caf224ab9ce6c6c5bcf06068283bd34615fcaf48061c671417bf5"}
Jul 29 11:01:40.896851 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:40.896826    2569 scope.go:117] "RemoveContainer" containerID="5615add369f0f6a122c2d56e55719fa907e0f813ac7af355ad277f861be401f0"
Jul 29 11:01:40.897203 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:40.897184    2569 scope.go:117] "RemoveContainer" containerID="5615add369f0f6a122c2d56e55719fa907e0f813ac7af355ad277f861be401f0"
Jul 29 11:01:40.907298 ip-10-0-137-111 kubenswrapper[2569]: E0729 11:01:40.907259    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-a23c96-predictor-87474ddf-ww49p_kserve-ci-e2e-test(b2e7c7dc-9da4-4c4b-9ac9-2283614a8637)\"" pod="kserve-ci-e2e-test/isvc-secondary-a23c96-predictor-87474ddf-ww49p" podUID="b2e7c7dc-9da4-4c4b-9ac9-2283614a8637"
Jul 29 11:01:40.907543 ip-10-0-137-111 kubenswrapper[2569]: E0729 11:01:40.907515    2569 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-a23c96-predictor-87474ddf-ww49p_kserve-ci-e2e-test_b2e7c7dc-9da4-4c4b-9ac9-2283614a8637_0 in pod sandbox b90e478465958d773b9661c321be9e9c3635960df2f94a91f917399461ddd88b from index: no such id: '5615add369f0f6a122c2d56e55719fa907e0f813ac7af355ad277f861be401f0'" containerID="5615add369f0f6a122c2d56e55719fa907e0f813ac7af355ad277f861be401f0"
Jul 29 11:01:40.907618 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:40.907556    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5615add369f0f6a122c2d56e55719fa907e0f813ac7af355ad277f861be401f0"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-a23c96-predictor-87474ddf-ww49p_kserve-ci-e2e-test_b2e7c7dc-9da4-4c4b-9ac9-2283614a8637_0 in pod sandbox b90e478465958d773b9661c321be9e9c3635960df2f94a91f917399461ddd88b from index: no such id: '5615add369f0f6a122c2d56e55719fa907e0f813ac7af355ad277f861be401f0'"
Jul 29 11:01:41.902252 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:41.902224    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a23c96-predictor-87474ddf-ww49p_b2e7c7dc-9da4-4c4b-9ac9-2283614a8637/storage-initializer/1.log"
Jul 29 11:01:43.945631 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:43.945602    2569 scope.go:117] "RemoveContainer" containerID="dd3db18747b892595523a07af137fb90b50b67e13f1b62bbecf469712e637392"
Jul 29 11:01:43.953252 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:43.953233    2569 scope.go:117] "RemoveContainer" containerID="9006505b2533b5704a279202a5a40108a006996a88e49cae0b52270d7f282709"
Jul 29 11:01:43.960333 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:43.960312    2569 scope.go:117] "RemoveContainer" containerID="b9c8bc1e25f719e56f56a8526e409e9636187712e8b824b8663f00b59429f619"
Jul 29 11:01:44.717683 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:44.717635    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a23c96-predictor-87474ddf-ww49p"]
Jul 29 11:01:44.737812 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:44.737783    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 11:01:44.799937 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:44.799898    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 11:01:44.854050 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:44.854030    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a23c96-predictor-87474ddf-ww49p_b2e7c7dc-9da4-4c4b-9ac9-2283614a8637/storage-initializer/1.log"
Jul 29 11:01:44.857824 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:44.857807    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a23c96-predictor-87474ddf-ww49p"
Jul 29 11:01:44.914468 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:44.914440    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a23c96-predictor-87474ddf-ww49p_b2e7c7dc-9da4-4c4b-9ac9-2283614a8637/storage-initializer/1.log"
Jul 29 11:01:44.914622 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:44.914527    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a23c96-predictor-87474ddf-ww49p"
Jul 29 11:01:44.914622 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:44.914530    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a23c96-predictor-87474ddf-ww49p" event={"ID":"b2e7c7dc-9da4-4c4b-9ac9-2283614a8637","Type":"ContainerDied","Data":"b90e478465958d773b9661c321be9e9c3635960df2f94a91f917399461ddd88b"}
Jul 29 11:01:44.914622 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:44.914601    2569 scope.go:117] "RemoveContainer" containerID="c4c2c3ed817caf224ab9ce6c6c5bcf06068283bd34615fcaf48061c671417bf5"
Jul 29 11:01:44.991481 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:44.991393    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gbbh\" (UniqueName: \"kubernetes.io/projected/b2e7c7dc-9da4-4c4b-9ac9-2283614a8637-kube-api-access-7gbbh\") pod \"b2e7c7dc-9da4-4c4b-9ac9-2283614a8637\" (UID: \"b2e7c7dc-9da4-4c4b-9ac9-2283614a8637\") "
Jul 29 11:01:44.991481 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:44.991433    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2e7c7dc-9da4-4c4b-9ac9-2283614a8637-cabundle-cert\") pod \"b2e7c7dc-9da4-4c4b-9ac9-2283614a8637\" (UID: \"b2e7c7dc-9da4-4c4b-9ac9-2283614a8637\") "
Jul 29 11:01:44.991958 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:44.991489    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2e7c7dc-9da4-4c4b-9ac9-2283614a8637-kube-rbac-proxy-sar-config\") pod \"b2e7c7dc-9da4-4c4b-9ac9-2283614a8637\" (UID: \"b2e7c7dc-9da4-4c4b-9ac9-2283614a8637\") "
Jul 29 11:01:44.991958 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:44.991541    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2e7c7dc-9da4-4c4b-9ac9-2283614a8637-kserve-provision-location\") pod \"b2e7c7dc-9da4-4c4b-9ac9-2283614a8637\" (UID: \"b2e7c7dc-9da4-4c4b-9ac9-2283614a8637\") "
Jul 29 11:01:44.991958 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:44.991559    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2e7c7dc-9da4-4c4b-9ac9-2283614a8637-proxy-tls\") pod \"b2e7c7dc-9da4-4c4b-9ac9-2283614a8637\" (UID: \"b2e7c7dc-9da4-4c4b-9ac9-2283614a8637\") "
Jul 29 11:01:44.991958 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:44.991805    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e7c7dc-9da4-4c4b-9ac9-2283614a8637-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b2e7c7dc-9da4-4c4b-9ac9-2283614a8637" (UID: "b2e7c7dc-9da4-4c4b-9ac9-2283614a8637"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:01:44.991958 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:44.991889    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2e7c7dc-9da4-4c4b-9ac9-2283614a8637-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b2e7c7dc-9da4-4c4b-9ac9-2283614a8637" (UID: "b2e7c7dc-9da4-4c4b-9ac9-2283614a8637"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:01:44.991958 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:44.991893    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2e7c7dc-9da4-4c4b-9ac9-2283614a8637-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b2e7c7dc-9da4-4c4b-9ac9-2283614a8637" (UID: "b2e7c7dc-9da4-4c4b-9ac9-2283614a8637"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:01:44.993621 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:44.993598    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2e7c7dc-9da4-4c4b-9ac9-2283614a8637-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b2e7c7dc-9da4-4c4b-9ac9-2283614a8637" (UID: "b2e7c7dc-9da4-4c4b-9ac9-2283614a8637"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:01:44.993621 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:44.993602    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e7c7dc-9da4-4c4b-9ac9-2283614a8637-kube-api-access-7gbbh" (OuterVolumeSpecName: "kube-api-access-7gbbh") pod "b2e7c7dc-9da4-4c4b-9ac9-2283614a8637" (UID: "b2e7c7dc-9da4-4c4b-9ac9-2283614a8637"). InnerVolumeSpecName "kube-api-access-7gbbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:01:45.093027 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:45.092988    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2e7c7dc-9da4-4c4b-9ac9-2283614a8637-kserve-provision-location\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 11:01:45.093027 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:45.093023    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2e7c7dc-9da4-4c4b-9ac9-2283614a8637-proxy-tls\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 11:01:45.093027 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:45.093037    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gbbh\" (UniqueName: \"kubernetes.io/projected/b2e7c7dc-9da4-4c4b-9ac9-2283614a8637-kube-api-access-7gbbh\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 11:01:45.093310 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:45.093050    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2e7c7dc-9da4-4c4b-9ac9-2283614a8637-cabundle-cert\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 11:01:45.093310 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:45.093062    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2e7c7dc-9da4-4c4b-9ac9-2283614a8637-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 11:01:45.249163 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:45.249066    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a23c96-predictor-87474ddf-ww49p"]
Jul 29 11:01:45.254923 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:45.254894    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a23c96-predictor-87474ddf-ww49p"]
Jul 29 11:01:45.837489 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:45.837457    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2e7c7dc-9da4-4c4b-9ac9-2283614a8637" path="/var/lib/kubelet/pods/b2e7c7dc-9da4-4c4b-9ac9-2283614a8637/volumes"
Jul 29 11:01:49.892285 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:49.892251    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 11:01:49.916508 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:49.916473    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r"]
Jul 29 11:01:49.916890 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:49.916868    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2e7c7dc-9da4-4c4b-9ac9-2283614a8637" containerName="storage-initializer"
Jul 29 11:01:49.916935 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:49.916894    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2e7c7dc-9da4-4c4b-9ac9-2283614a8637" containerName="storage-initializer"
Jul 29 11:01:49.916999 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:49.916987    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2e7c7dc-9da4-4c4b-9ac9-2283614a8637" containerName="storage-initializer"
Jul 29 11:01:49.917035 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:49.917004    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2e7c7dc-9da4-4c4b-9ac9-2283614a8637" containerName="storage-initializer"
Jul 29 11:01:49.917120 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:49.917108    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2e7c7dc-9da4-4c4b-9ac9-2283614a8637" containerName="storage-initializer"
Jul 29 11:01:49.917153 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:49.917122    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2e7c7dc-9da4-4c4b-9ac9-2283614a8637" containerName="storage-initializer"
Jul 29 11:01:49.920426 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:49.920408    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r"
Jul 29 11:01:49.923669 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:49.923531    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 11:01:49.923669 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:49.923556    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-096ae9-predictor-serving-cert\""
Jul 29 11:01:49.923669 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:49.923575    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 11:01:49.923669 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:49.923603    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:01:49.923669 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:49.923665    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-096ae9-dockercfg-sml8p\""
Jul 29 11:01:49.924047 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:49.923751    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-096ae9-kube-rbac-proxy-sar-config\""
Jul 29 11:01:49.924605 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:49.924587    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-096ae9\""
Jul 29 11:01:49.934243 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:49.934214    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r"]
Jul 29 11:01:50.019848 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:50.019810    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 11:01:50.033884 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:50.033851    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z7qn\" (UniqueName: \"kubernetes.io/projected/20ed41cd-4c0f-4036-bcf8-3151fc89c835-kube-api-access-2z7qn\") pod \"isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r\" (UID: \"20ed41cd-4c0f-4036-bcf8-3151fc89c835\") " pod="kserve-ci-e2e-test/isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r"
Jul 29 11:01:50.034050 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:50.033966    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20ed41cd-4c0f-4036-bcf8-3151fc89c835-kserve-provision-location\") pod \"isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r\" (UID: \"20ed41cd-4c0f-4036-bcf8-3151fc89c835\") " pod="kserve-ci-e2e-test/isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r"
Jul 29 11:01:50.034050 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:50.034007    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20ed41cd-4c0f-4036-bcf8-3151fc89c835-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r\" (UID: \"20ed41cd-4c0f-4036-bcf8-3151fc89c835\") " pod="kserve-ci-e2e-test/isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r"
Jul 29 11:01:50.034050 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:50.034039    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20ed41cd-4c0f-4036-bcf8-3151fc89c835-proxy-tls\") pod \"isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r\" (UID: \"20ed41cd-4c0f-4036-bcf8-3151fc89c835\") " pod="kserve-ci-e2e-test/isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r"
Jul 29 11:01:50.034299 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:50.034176    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20ed41cd-4c0f-4036-bcf8-3151fc89c835-cabundle-cert\") pod \"isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r\" (UID: \"20ed41cd-4c0f-4036-bcf8-3151fc89c835\") " pod="kserve-ci-e2e-test/isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r"
Jul 29 11:01:50.135121 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:50.135066    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20ed41cd-4c0f-4036-bcf8-3151fc89c835-cabundle-cert\") pod \"isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r\" (UID: \"20ed41cd-4c0f-4036-bcf8-3151fc89c835\") " pod="kserve-ci-e2e-test/isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r"
Jul 29 11:01:50.135315 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:50.135133    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z7qn\" (UniqueName: \"kubernetes.io/projected/20ed41cd-4c0f-4036-bcf8-3151fc89c835-kube-api-access-2z7qn\") pod \"isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r\" (UID: \"20ed41cd-4c0f-4036-bcf8-3151fc89c835\") " pod="kserve-ci-e2e-test/isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r"
Jul 29 11:01:50.135315 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:50.135212    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20ed41cd-4c0f-4036-bcf8-3151fc89c835-kserve-provision-location\") pod \"isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r\" (UID: \"20ed41cd-4c0f-4036-bcf8-3151fc89c835\") " pod="kserve-ci-e2e-test/isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r"
Jul 29 11:01:50.135315 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:50.135239    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20ed41cd-4c0f-4036-bcf8-3151fc89c835-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r\" (UID: \"20ed41cd-4c0f-4036-bcf8-3151fc89c835\") " pod="kserve-ci-e2e-test/isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r"
Jul 29 11:01:50.135315 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:50.135263    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20ed41cd-4c0f-4036-bcf8-3151fc89c835-proxy-tls\") pod \"isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r\" (UID: \"20ed41cd-4c0f-4036-bcf8-3151fc89c835\") " pod="kserve-ci-e2e-test/isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r"
Jul 29 11:01:50.135626 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:50.135600    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20ed41cd-4c0f-4036-bcf8-3151fc89c835-kserve-provision-location\") pod \"isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r\" (UID: \"20ed41cd-4c0f-4036-bcf8-3151fc89c835\") " pod="kserve-ci-e2e-test/isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r"
Jul 29 11:01:50.135880 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:50.135857    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20ed41cd-4c0f-4036-bcf8-3151fc89c835-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r\" (UID: \"20ed41cd-4c0f-4036-bcf8-3151fc89c835\") " pod="kserve-ci-e2e-test/isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r"
Jul 29 11:01:50.135927 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:50.135863    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20ed41cd-4c0f-4036-bcf8-3151fc89c835-cabundle-cert\") pod \"isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r\" (UID: \"20ed41cd-4c0f-4036-bcf8-3151fc89c835\") " pod="kserve-ci-e2e-test/isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r"
Jul 29 11:01:50.137710 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:50.137687    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20ed41cd-4c0f-4036-bcf8-3151fc89c835-proxy-tls\") pod \"isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r\" (UID: \"20ed41cd-4c0f-4036-bcf8-3151fc89c835\") " pod="kserve-ci-e2e-test/isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r"
Jul 29 11:01:50.164432 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:50.164405    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z7qn\" (UniqueName: \"kubernetes.io/projected/20ed41cd-4c0f-4036-bcf8-3151fc89c835-kube-api-access-2z7qn\") pod \"isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r\" (UID: \"20ed41cd-4c0f-4036-bcf8-3151fc89c835\") " pod="kserve-ci-e2e-test/isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r"
Jul 29 11:01:50.230614 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:50.230576    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r"
Jul 29 11:01:50.361485 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:50.361452    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r"]
Jul 29 11:01:50.364593 ip-10-0-137-111 kubenswrapper[2569]: W0729 11:01:50.364565    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20ed41cd_4c0f_4036_bcf8_3151fc89c835.slice/crio-eb31021b1db162e9d86d7b6b13345fdeeff3a83a37f7a4d6d4e75f14aa7d8354 WatchSource:0}: Error finding container eb31021b1db162e9d86d7b6b13345fdeeff3a83a37f7a4d6d4e75f14aa7d8354: Status 404 returned error can't find the container with id eb31021b1db162e9d86d7b6b13345fdeeff3a83a37f7a4d6d4e75f14aa7d8354
Jul 29 11:01:50.940678 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:50.940641    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r" event={"ID":"20ed41cd-4c0f-4036-bcf8-3151fc89c835","Type":"ContainerStarted","Data":"bea3cfcd855648e53fb5e64f803c0008ab43b7a33462e98c21ba4f77c4765bf4"}
Jul 29 11:01:50.941055 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:50.940714    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r" event={"ID":"20ed41cd-4c0f-4036-bcf8-3151fc89c835","Type":"ContainerStarted","Data":"eb31021b1db162e9d86d7b6b13345fdeeff3a83a37f7a4d6d4e75f14aa7d8354"}
Jul 29 11:01:54.958275 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:54.958243    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r_20ed41cd-4c0f-4036-bcf8-3151fc89c835/storage-initializer/0.log"
Jul 29 11:01:54.958644 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:54.958287    2569 generic.go:356] "Generic (PLEG): container finished" podID="20ed41cd-4c0f-4036-bcf8-3151fc89c835" containerID="bea3cfcd855648e53fb5e64f803c0008ab43b7a33462e98c21ba4f77c4765bf4" exitCode=1
Jul 29 11:01:54.958644 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:54.958346    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r" event={"ID":"20ed41cd-4c0f-4036-bcf8-3151fc89c835","Type":"ContainerDied","Data":"bea3cfcd855648e53fb5e64f803c0008ab43b7a33462e98c21ba4f77c4765bf4"}
Jul 29 11:01:55.964187 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:55.964160    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r_20ed41cd-4c0f-4036-bcf8-3151fc89c835/storage-initializer/0.log"
Jul 29 11:01:55.964555 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:55.964252    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r" event={"ID":"20ed41cd-4c0f-4036-bcf8-3151fc89c835","Type":"ContainerStarted","Data":"ffa1ffa15a676ac6ce484828d3936dbeaad973c1c127f1237a8408b71f874e8f"}
Jul 29 11:01:59.925263 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:59.925225    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r"]
Jul 29 11:01:59.925742 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:59.925438    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r" podUID="20ed41cd-4c0f-4036-bcf8-3151fc89c835" containerName="storage-initializer" containerID="cri-o://ffa1ffa15a676ac6ce484828d3936dbeaad973c1c127f1237a8408b71f874e8f" gracePeriod=30
Jul 29 11:01:59.946455 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:01:59.946419    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 11:02:01.768167 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:02:01.768143    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r_20ed41cd-4c0f-4036-bcf8-3151fc89c835/storage-initializer/1.log"
Jul 29 11:02:01.768540 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:02:01.768521    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r_20ed41cd-4c0f-4036-bcf8-3151fc89c835/storage-initializer/0.log"
Jul 29 11:02:01.770891 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:02:01.770876    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r"
Jul 29 11:02:01.838811 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:02:01.838711    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2z7qn\" (UniqueName: \"kubernetes.io/projected/20ed41cd-4c0f-4036-bcf8-3151fc89c835-kube-api-access-2z7qn\") pod \"20ed41cd-4c0f-4036-bcf8-3151fc89c835\" (UID: \"20ed41cd-4c0f-4036-bcf8-3151fc89c835\") "
Jul 29 11:02:01.838811 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:02:01.838776    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20ed41cd-4c0f-4036-bcf8-3151fc89c835-proxy-tls\") pod \"20ed41cd-4c0f-4036-bcf8-3151fc89c835\" (UID: \"20ed41cd-4c0f-4036-bcf8-3151fc89c835\") "
Jul 29 11:02:01.839095 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:02:01.838894    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20ed41cd-4c0f-4036-bcf8-3151fc89c835-kube-rbac-proxy-sar-config\") pod \"20ed41cd-4c0f-4036-bcf8-3151fc89c835\" (UID: \"20ed41cd-4c0f-4036-bcf8-3151fc89c835\") "
Jul 29 11:02:01.839095 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:02:01.838951    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20ed41cd-4c0f-4036-bcf8-3151fc89c835-cabundle-cert\") pod \"20ed41cd-4c0f-4036-bcf8-3151fc89c835\" (UID: \"20ed41cd-4c0f-4036-bcf8-3151fc89c835\") "
Jul 29 11:02:01.839095 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:02:01.839009    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20ed41cd-4c0f-4036-bcf8-3151fc89c835-kserve-provision-location\") pod \"20ed41cd-4c0f-4036-bcf8-3151fc89c835\" (UID: \"20ed41cd-4c0f-4036-bcf8-3151fc89c835\") "
Jul 29 11:02:01.839412 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:02:01.839376    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ed41cd-4c0f-4036-bcf8-3151fc89c835-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "20ed41cd-4c0f-4036-bcf8-3151fc89c835" (UID: "20ed41cd-4c0f-4036-bcf8-3151fc89c835"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:02:01.839412 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:02:01.839376    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20ed41cd-4c0f-4036-bcf8-3151fc89c835-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "20ed41cd-4c0f-4036-bcf8-3151fc89c835" (UID: "20ed41cd-4c0f-4036-bcf8-3151fc89c835"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:02:01.839618 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:02:01.839600    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20ed41cd-4c0f-4036-bcf8-3151fc89c835-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "20ed41cd-4c0f-4036-bcf8-3151fc89c835" (UID: "20ed41cd-4c0f-4036-bcf8-3151fc89c835"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:02:01.841416 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:02:01.841387    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20ed41cd-4c0f-4036-bcf8-3151fc89c835-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "20ed41cd-4c0f-4036-bcf8-3151fc89c835" (UID: "20ed41cd-4c0f-4036-bcf8-3151fc89c835"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:02:01.841416 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:02:01.841393    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20ed41cd-4c0f-4036-bcf8-3151fc89c835-kube-api-access-2z7qn" (OuterVolumeSpecName: "kube-api-access-2z7qn") pod "20ed41cd-4c0f-4036-bcf8-3151fc89c835" (UID: "20ed41cd-4c0f-4036-bcf8-3151fc89c835"). InnerVolumeSpecName "kube-api-access-2z7qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:02:01.939769 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:02:01.939730    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20ed41cd-4c0f-4036-bcf8-3151fc89c835-kserve-provision-location\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 11:02:01.939769 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:02:01.939763    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z7qn\" (UniqueName: \"kubernetes.io/projected/20ed41cd-4c0f-4036-bcf8-3151fc89c835-kube-api-access-2z7qn\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 11:02:01.939769 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:02:01.939773    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20ed41cd-4c0f-4036-bcf8-3151fc89c835-proxy-tls\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 11:02:01.939998 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:02:01.939782    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20ed41cd-4c0f-4036-bcf8-3151fc89c835-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 11:02:01.939998 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:02:01.939791    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20ed41cd-4c0f-4036-bcf8-3151fc89c835-cabundle-cert\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 11:02:01.987996 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:02:01.987972    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r_20ed41cd-4c0f-4036-bcf8-3151fc89c835/storage-initializer/1.log"
Jul 29 11:02:01.988425 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:02:01.988407    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r_20ed41cd-4c0f-4036-bcf8-3151fc89c835/storage-initializer/0.log"
Jul 29 11:02:01.988518 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:02:01.988459    2569 generic.go:356] "Generic (PLEG): container finished" podID="20ed41cd-4c0f-4036-bcf8-3151fc89c835" containerID="ffa1ffa15a676ac6ce484828d3936dbeaad973c1c127f1237a8408b71f874e8f" exitCode=1
Jul 29 11:02:01.988577 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:02:01.988526    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r" event={"ID":"20ed41cd-4c0f-4036-bcf8-3151fc89c835","Type":"ContainerDied","Data":"ffa1ffa15a676ac6ce484828d3936dbeaad973c1c127f1237a8408b71f874e8f"}
Jul 29 11:02:01.988577 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:02:01.988540    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r"
Jul 29 11:02:01.988679 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:02:01.988581    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r" event={"ID":"20ed41cd-4c0f-4036-bcf8-3151fc89c835","Type":"ContainerDied","Data":"eb31021b1db162e9d86d7b6b13345fdeeff3a83a37f7a4d6d4e75f14aa7d8354"}
Jul 29 11:02:01.988679 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:02:01.988611    2569 scope.go:117] "RemoveContainer" containerID="ffa1ffa15a676ac6ce484828d3936dbeaad973c1c127f1237a8408b71f874e8f"
Jul 29 11:02:02.000409 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:02:02.000373    2569 scope.go:117] "RemoveContainer" containerID="bea3cfcd855648e53fb5e64f803c0008ab43b7a33462e98c21ba4f77c4765bf4"
Jul 29 11:02:02.007694 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:02:02.007677    2569 scope.go:117] "RemoveContainer" containerID="ffa1ffa15a676ac6ce484828d3936dbeaad973c1c127f1237a8408b71f874e8f"
Jul 29 11:02:02.007947 ip-10-0-137-111 kubenswrapper[2569]: E0729 11:02:02.007925    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffa1ffa15a676ac6ce484828d3936dbeaad973c1c127f1237a8408b71f874e8f\": container with ID starting with ffa1ffa15a676ac6ce484828d3936dbeaad973c1c127f1237a8408b71f874e8f not found: ID does not exist" containerID="ffa1ffa15a676ac6ce484828d3936dbeaad973c1c127f1237a8408b71f874e8f"
Jul 29 11:02:02.008001 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:02:02.007955    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffa1ffa15a676ac6ce484828d3936dbeaad973c1c127f1237a8408b71f874e8f"} err="failed to get container status \"ffa1ffa15a676ac6ce484828d3936dbeaad973c1c127f1237a8408b71f874e8f\": rpc error: code = NotFound desc = could not find container \"ffa1ffa15a676ac6ce484828d3936dbeaad973c1c127f1237a8408b71f874e8f\": container with ID starting with ffa1ffa15a676ac6ce484828d3936dbeaad973c1c127f1237a8408b71f874e8f not found: ID does not exist"
Jul 29 11:02:02.008001 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:02:02.007972    2569 scope.go:117] "RemoveContainer" containerID="bea3cfcd855648e53fb5e64f803c0008ab43b7a33462e98c21ba4f77c4765bf4"
Jul 29 11:02:02.008216 ip-10-0-137-111 kubenswrapper[2569]: E0729 11:02:02.008198    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bea3cfcd855648e53fb5e64f803c0008ab43b7a33462e98c21ba4f77c4765bf4\": container with ID starting with bea3cfcd855648e53fb5e64f803c0008ab43b7a33462e98c21ba4f77c4765bf4 not found: ID does not exist" containerID="bea3cfcd855648e53fb5e64f803c0008ab43b7a33462e98c21ba4f77c4765bf4"
Jul 29 11:02:02.008272 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:02:02.008223    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bea3cfcd855648e53fb5e64f803c0008ab43b7a33462e98c21ba4f77c4765bf4"} err="failed to get container status \"bea3cfcd855648e53fb5e64f803c0008ab43b7a33462e98c21ba4f77c4765bf4\": rpc error: code = NotFound desc = could not find container \"bea3cfcd855648e53fb5e64f803c0008ab43b7a33462e98c21ba4f77c4765bf4\": container with ID starting with bea3cfcd855648e53fb5e64f803c0008ab43b7a33462e98c21ba4f77c4765bf4 not found: ID does not exist"
Jul 29 11:02:02.027297 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:02:02.027265    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r"]
Jul 29 11:02:02.031541 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:02:02.031500    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-096ae9-predictor-77d8d5bb75-br52r"]
Jul 29 11:02:03.839033 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:02:03.838999    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20ed41cd-4c0f-4036-bcf8-3151fc89c835" path="/var/lib/kubelet/pods/20ed41cd-4c0f-4036-bcf8-3151fc89c835/volumes"
Jul 29 11:02:05.098018 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:02:05.097978    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 11:02:05.599410 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:02:05.599376    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 11:03:03.203732 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:03.203689    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 11:03:35.285163 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:35.285129    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 11:03:40.420401 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:40.420356    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 11:03:40.479433 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:40.479395    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw"]
Jul 29 11:03:40.479732 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:40.479719    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20ed41cd-4c0f-4036-bcf8-3151fc89c835" containerName="storage-initializer"
Jul 29 11:03:40.479794 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:40.479733    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="20ed41cd-4c0f-4036-bcf8-3151fc89c835" containerName="storage-initializer"
Jul 29 11:03:40.479794 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:40.479741    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20ed41cd-4c0f-4036-bcf8-3151fc89c835" containerName="storage-initializer"
Jul 29 11:03:40.479794 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:40.479747    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="20ed41cd-4c0f-4036-bcf8-3151fc89c835" containerName="storage-initializer"
Jul 29 11:03:40.479947 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:40.479804    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="20ed41cd-4c0f-4036-bcf8-3151fc89c835" containerName="storage-initializer"
Jul 29 11:03:40.479947 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:40.479912    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="20ed41cd-4c0f-4036-bcf8-3151fc89c835" containerName="storage-initializer"
Jul 29 11:03:40.482794 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:40.482777    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw"
Jul 29 11:03:40.485782 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:40.485750    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-1999c-predictor-serving-cert\""
Jul 29 11:03:40.485968 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:40.485798    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 11:03:40.485968 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:40.485803    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:03:40.485968 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:40.485858    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6nnxs\""
Jul 29 11:03:40.485968 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:40.485750    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 11:03:40.485968 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:40.485862    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 11:03:40.486261 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:40.486133    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-1999c-kube-rbac-proxy-sar-config\""
Jul 29 11:03:40.494244 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:40.494222    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw"]
Jul 29 11:03:40.526876 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:40.526846    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebc59f98-4ca1-4c6e-8028-4694a9e9dd97-kserve-provision-location\") pod \"raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw\" (UID: \"ebc59f98-4ca1-4c6e-8028-4694a9e9dd97\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw"
Jul 29 11:03:40.527014 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:40.526879    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebc59f98-4ca1-4c6e-8028-4694a9e9dd97-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw\" (UID: \"ebc59f98-4ca1-4c6e-8028-4694a9e9dd97\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw"
Jul 29 11:03:40.527014 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:40.526904    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g42wl\" (UniqueName: \"kubernetes.io/projected/ebc59f98-4ca1-4c6e-8028-4694a9e9dd97-kube-api-access-g42wl\") pod \"raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw\" (UID: \"ebc59f98-4ca1-4c6e-8028-4694a9e9dd97\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw"
Jul 29 11:03:40.527014 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:40.526976    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebc59f98-4ca1-4c6e-8028-4694a9e9dd97-cabundle-cert\") pod \"raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw\" (UID: \"ebc59f98-4ca1-4c6e-8028-4694a9e9dd97\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw"
Jul 29 11:03:40.527266 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:40.527042    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebc59f98-4ca1-4c6e-8028-4694a9e9dd97-proxy-tls\") pod \"raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw\" (UID: \"ebc59f98-4ca1-4c6e-8028-4694a9e9dd97\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw"
Jul 29 11:03:40.627628 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:40.627583    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g42wl\" (UniqueName: \"kubernetes.io/projected/ebc59f98-4ca1-4c6e-8028-4694a9e9dd97-kube-api-access-g42wl\") pod \"raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw\" (UID: \"ebc59f98-4ca1-4c6e-8028-4694a9e9dd97\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw"
Jul 29 11:03:40.627628 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:40.627620    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebc59f98-4ca1-4c6e-8028-4694a9e9dd97-cabundle-cert\") pod \"raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw\" (UID: \"ebc59f98-4ca1-4c6e-8028-4694a9e9dd97\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw"
Jul 29 11:03:40.627825 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:40.627748    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebc59f98-4ca1-4c6e-8028-4694a9e9dd97-proxy-tls\") pod \"raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw\" (UID: \"ebc59f98-4ca1-4c6e-8028-4694a9e9dd97\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw"
Jul 29 11:03:40.627866 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:40.627829    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebc59f98-4ca1-4c6e-8028-4694a9e9dd97-kserve-provision-location\") pod \"raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw\" (UID: \"ebc59f98-4ca1-4c6e-8028-4694a9e9dd97\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw"
Jul 29 11:03:40.627866 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:40.627849    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebc59f98-4ca1-4c6e-8028-4694a9e9dd97-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw\" (UID: \"ebc59f98-4ca1-4c6e-8028-4694a9e9dd97\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw"
Jul 29 11:03:40.628239 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:40.628219    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebc59f98-4ca1-4c6e-8028-4694a9e9dd97-kserve-provision-location\") pod \"raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw\" (UID: \"ebc59f98-4ca1-4c6e-8028-4694a9e9dd97\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw"
Jul 29 11:03:40.628349 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:40.628329    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebc59f98-4ca1-4c6e-8028-4694a9e9dd97-cabundle-cert\") pod \"raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw\" (UID: \"ebc59f98-4ca1-4c6e-8028-4694a9e9dd97\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw"
Jul 29 11:03:40.628440 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:40.628423    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebc59f98-4ca1-4c6e-8028-4694a9e9dd97-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw\" (UID: \"ebc59f98-4ca1-4c6e-8028-4694a9e9dd97\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw"
Jul 29 11:03:40.630266 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:40.630246    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebc59f98-4ca1-4c6e-8028-4694a9e9dd97-proxy-tls\") pod \"raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw\" (UID: \"ebc59f98-4ca1-4c6e-8028-4694a9e9dd97\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw"
Jul 29 11:03:40.640135 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:40.640113    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g42wl\" (UniqueName: \"kubernetes.io/projected/ebc59f98-4ca1-4c6e-8028-4694a9e9dd97-kube-api-access-g42wl\") pod \"raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw\" (UID: \"ebc59f98-4ca1-4c6e-8028-4694a9e9dd97\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw"
Jul 29 11:03:40.688790 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:40.688700    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 11:03:40.794088 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:40.794026    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw"
Jul 29 11:03:40.922919 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:40.922895    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw"]
Jul 29 11:03:40.924954 ip-10-0-137-111 kubenswrapper[2569]: W0729 11:03:40.924915    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc59f98_4ca1_4c6e_8028_4694a9e9dd97.slice/crio-049736c69f15f1883548790eed2c9c23e23b264dbb61bd497e9c3029e2a7e073 WatchSource:0}: Error finding container 049736c69f15f1883548790eed2c9c23e23b264dbb61bd497e9c3029e2a7e073: Status 404 returned error can't find the container with id 049736c69f15f1883548790eed2c9c23e23b264dbb61bd497e9c3029e2a7e073
Jul 29 11:03:41.366109 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:41.366011    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw" event={"ID":"ebc59f98-4ca1-4c6e-8028-4694a9e9dd97","Type":"ContainerStarted","Data":"0dfe2e6153c9520db94dfe9f11533c51c4c23a6c4a9ef765c6a85dbd15dd8ee3"}
Jul 29 11:03:41.366109 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:41.366083    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw" event={"ID":"ebc59f98-4ca1-4c6e-8028-4694a9e9dd97","Type":"ContainerStarted","Data":"049736c69f15f1883548790eed2c9c23e23b264dbb61bd497e9c3029e2a7e073"}
Jul 29 11:03:43.849361 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:43.849332    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-pvdgw_afd0d65c-bbfc-49f0-81b6-e855fd81da7f/console-operator/1.log"
Jul 29 11:03:43.853879 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:43.853855    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-pvdgw_afd0d65c-bbfc-49f0-81b6-e855fd81da7f/console-operator/1.log"
Jul 29 11:03:45.381849 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:45.381812    2569 generic.go:356] "Generic (PLEG): container finished" podID="ebc59f98-4ca1-4c6e-8028-4694a9e9dd97" containerID="0dfe2e6153c9520db94dfe9f11533c51c4c23a6c4a9ef765c6a85dbd15dd8ee3" exitCode=0
Jul 29 11:03:45.382304 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:45.381884    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw" event={"ID":"ebc59f98-4ca1-4c6e-8028-4694a9e9dd97","Type":"ContainerDied","Data":"0dfe2e6153c9520db94dfe9f11533c51c4c23a6c4a9ef765c6a85dbd15dd8ee3"}
Jul 29 11:03:46.388066 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:46.388031    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw" event={"ID":"ebc59f98-4ca1-4c6e-8028-4694a9e9dd97","Type":"ContainerStarted","Data":"c2104f519e1c64f257e544d7c95c16b88364305984401becd69951199f4a9df2"}
Jul 29 11:03:46.388461 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:46.388110    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw" event={"ID":"ebc59f98-4ca1-4c6e-8028-4694a9e9dd97","Type":"ContainerStarted","Data":"5711abc24e048ac38a16577d4efde11da0e44f85b45dfc20314f5c3fe7ad9b82"}
Jul 29 11:03:46.388461 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:46.388295    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw"
Jul 29 11:03:46.408057 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:46.408012    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw" podStartSLOduration=2.812961393 podStartE2EDuration="6.407996886s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.595035493s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:03:46.40661029 +0000 UTC m=+2103.131400964" watchObservedRunningTime="2026-07-29 11:03:46.407996886 +0000 UTC m=+2103.132787559"
Jul 29 11:03:47.391933 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:47.391890    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw"
Jul 29 11:03:47.393146 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:47.393115    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw" podUID="ebc59f98-4ca1-4c6e-8028-4694a9e9dd97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 29 11:03:48.396353 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:48.396312    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw" podUID="ebc59f98-4ca1-4c6e-8028-4694a9e9dd97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 29 11:03:53.400643 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:53.400610    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw"
Jul 29 11:03:53.401144 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:03:53.401113    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw" podUID="ebc59f98-4ca1-4c6e-8028-4694a9e9dd97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 29 11:04:03.402043 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:04:03.401999    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw" podUID="ebc59f98-4ca1-4c6e-8028-4694a9e9dd97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 29 11:04:13.401229 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:04:13.401186    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw" podUID="ebc59f98-4ca1-4c6e-8028-4694a9e9dd97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 29 11:04:23.401695 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:04:23.401654    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw" podUID="ebc59f98-4ca1-4c6e-8028-4694a9e9dd97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 29 11:04:33.401399 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:04:33.401353    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw" podUID="ebc59f98-4ca1-4c6e-8028-4694a9e9dd97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 29 11:04:43.401268 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:04:43.401224    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw" podUID="ebc59f98-4ca1-4c6e-8028-4694a9e9dd97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 29 11:04:53.401886 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:04:53.401844    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw" podUID="ebc59f98-4ca1-4c6e-8028-4694a9e9dd97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 29 11:05:03.401945 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:03.401910    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw"
Jul 29 11:05:10.594580 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:10.594541    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw"]
Jul 29 11:05:10.595139 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:10.594767    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw" podUID="ebc59f98-4ca1-4c6e-8028-4694a9e9dd97" containerName="kserve-container" containerID="cri-o://5711abc24e048ac38a16577d4efde11da0e44f85b45dfc20314f5c3fe7ad9b82" gracePeriod=30
Jul 29 11:05:10.595139 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:10.594816    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw" podUID="ebc59f98-4ca1-4c6e-8028-4694a9e9dd97" containerName="kube-rbac-proxy" containerID="cri-o://c2104f519e1c64f257e544d7c95c16b88364305984401becd69951199f4a9df2" gracePeriod=30
Jul 29 11:05:10.601783 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:10.601752    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 11:05:10.731643 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:10.731614    2569 generic.go:356] "Generic (PLEG): container finished" podID="ebc59f98-4ca1-4c6e-8028-4694a9e9dd97" containerID="c2104f519e1c64f257e544d7c95c16b88364305984401becd69951199f4a9df2" exitCode=2
Jul 29 11:05:10.731770 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:10.731675    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw" event={"ID":"ebc59f98-4ca1-4c6e-8028-4694a9e9dd97","Type":"ContainerDied","Data":"c2104f519e1c64f257e544d7c95c16b88364305984401becd69951199f4a9df2"}
Jul 29 11:05:13.396700 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:13.396656    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw" podUID="ebc59f98-4ca1-4c6e-8028-4694a9e9dd97" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Jul 29 11:05:13.401912 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:13.401884    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw" podUID="ebc59f98-4ca1-4c6e-8028-4694a9e9dd97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 29 11:05:15.435042 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:15.435021    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw"
Jul 29 11:05:15.507297 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:15.507224    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebc59f98-4ca1-4c6e-8028-4694a9e9dd97-kserve-provision-location\") pod \"ebc59f98-4ca1-4c6e-8028-4694a9e9dd97\" (UID: \"ebc59f98-4ca1-4c6e-8028-4694a9e9dd97\") "
Jul 29 11:05:15.507297 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:15.507261    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebc59f98-4ca1-4c6e-8028-4694a9e9dd97-kube-rbac-proxy-sar-config\") pod \"ebc59f98-4ca1-4c6e-8028-4694a9e9dd97\" (UID: \"ebc59f98-4ca1-4c6e-8028-4694a9e9dd97\") "
Jul 29 11:05:15.507297 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:15.507281    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebc59f98-4ca1-4c6e-8028-4694a9e9dd97-cabundle-cert\") pod \"ebc59f98-4ca1-4c6e-8028-4694a9e9dd97\" (UID: \"ebc59f98-4ca1-4c6e-8028-4694a9e9dd97\") "
Jul 29 11:05:15.507512 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:15.507322    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g42wl\" (UniqueName: \"kubernetes.io/projected/ebc59f98-4ca1-4c6e-8028-4694a9e9dd97-kube-api-access-g42wl\") pod \"ebc59f98-4ca1-4c6e-8028-4694a9e9dd97\" (UID: \"ebc59f98-4ca1-4c6e-8028-4694a9e9dd97\") "
Jul 29 11:05:15.507512 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:15.507388    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebc59f98-4ca1-4c6e-8028-4694a9e9dd97-proxy-tls\") pod \"ebc59f98-4ca1-4c6e-8028-4694a9e9dd97\" (UID: \"ebc59f98-4ca1-4c6e-8028-4694a9e9dd97\") "
Jul 29 11:05:15.507613 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:15.507593    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebc59f98-4ca1-4c6e-8028-4694a9e9dd97-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ebc59f98-4ca1-4c6e-8028-4694a9e9dd97" (UID: "ebc59f98-4ca1-4c6e-8028-4694a9e9dd97"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:05:15.507661 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:15.507633    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebc59f98-4ca1-4c6e-8028-4694a9e9dd97-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ebc59f98-4ca1-4c6e-8028-4694a9e9dd97" (UID: "ebc59f98-4ca1-4c6e-8028-4694a9e9dd97"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:05:15.507736 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:15.507708    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebc59f98-4ca1-4c6e-8028-4694a9e9dd97-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ebc59f98-4ca1-4c6e-8028-4694a9e9dd97" (UID: "ebc59f98-4ca1-4c6e-8028-4694a9e9dd97"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:05:15.509439 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:15.509414    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebc59f98-4ca1-4c6e-8028-4694a9e9dd97-kube-api-access-g42wl" (OuterVolumeSpecName: "kube-api-access-g42wl") pod "ebc59f98-4ca1-4c6e-8028-4694a9e9dd97" (UID: "ebc59f98-4ca1-4c6e-8028-4694a9e9dd97"). InnerVolumeSpecName "kube-api-access-g42wl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:05:15.509519 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:15.509459    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebc59f98-4ca1-4c6e-8028-4694a9e9dd97-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ebc59f98-4ca1-4c6e-8028-4694a9e9dd97" (UID: "ebc59f98-4ca1-4c6e-8028-4694a9e9dd97"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:05:15.608758 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:15.608723    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebc59f98-4ca1-4c6e-8028-4694a9e9dd97-proxy-tls\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 11:05:15.608758 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:15.608752    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebc59f98-4ca1-4c6e-8028-4694a9e9dd97-kserve-provision-location\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 11:05:15.608758 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:15.608762    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebc59f98-4ca1-4c6e-8028-4694a9e9dd97-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 11:05:15.608994 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:15.608771    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebc59f98-4ca1-4c6e-8028-4694a9e9dd97-cabundle-cert\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 11:05:15.608994 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:15.608780    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g42wl\" (UniqueName: \"kubernetes.io/projected/ebc59f98-4ca1-4c6e-8028-4694a9e9dd97-kube-api-access-g42wl\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 11:05:15.756141 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:15.756059    2569 generic.go:356] "Generic (PLEG): container finished" podID="ebc59f98-4ca1-4c6e-8028-4694a9e9dd97" containerID="5711abc24e048ac38a16577d4efde11da0e44f85b45dfc20314f5c3fe7ad9b82" exitCode=0
Jul 29 11:05:15.756141 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:15.756121    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw" event={"ID":"ebc59f98-4ca1-4c6e-8028-4694a9e9dd97","Type":"ContainerDied","Data":"5711abc24e048ac38a16577d4efde11da0e44f85b45dfc20314f5c3fe7ad9b82"}
Jul 29 11:05:15.756141 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:15.756144    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw"
Jul 29 11:05:15.756488 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:15.756196    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw" event={"ID":"ebc59f98-4ca1-4c6e-8028-4694a9e9dd97","Type":"ContainerDied","Data":"049736c69f15f1883548790eed2c9c23e23b264dbb61bd497e9c3029e2a7e073"}
Jul 29 11:05:15.756488 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:15.756226    2569 scope.go:117] "RemoveContainer" containerID="c2104f519e1c64f257e544d7c95c16b88364305984401becd69951199f4a9df2"
Jul 29 11:05:15.764905 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:15.764888    2569 scope.go:117] "RemoveContainer" containerID="5711abc24e048ac38a16577d4efde11da0e44f85b45dfc20314f5c3fe7ad9b82"
Jul 29 11:05:15.772058 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:15.772039    2569 scope.go:117] "RemoveContainer" containerID="0dfe2e6153c9520db94dfe9f11533c51c4c23a6c4a9ef765c6a85dbd15dd8ee3"
Jul 29 11:05:15.781817 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:15.778185    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw"]
Jul 29 11:05:15.781817 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:15.780736    2569 scope.go:117] "RemoveContainer" containerID="c2104f519e1c64f257e544d7c95c16b88364305984401becd69951199f4a9df2"
Jul 29 11:05:15.783170 ip-10-0-137-111 kubenswrapper[2569]: E0729 11:05:15.782829    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2104f519e1c64f257e544d7c95c16b88364305984401becd69951199f4a9df2\": container with ID starting with c2104f519e1c64f257e544d7c95c16b88364305984401becd69951199f4a9df2 not found: ID does not exist" containerID="c2104f519e1c64f257e544d7c95c16b88364305984401becd69951199f4a9df2"
Jul 29 11:05:15.783170 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:15.782866    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2104f519e1c64f257e544d7c95c16b88364305984401becd69951199f4a9df2"} err="failed to get container status \"c2104f519e1c64f257e544d7c95c16b88364305984401becd69951199f4a9df2\": rpc error: code = NotFound desc = could not find container \"c2104f519e1c64f257e544d7c95c16b88364305984401becd69951199f4a9df2\": container with ID starting with c2104f519e1c64f257e544d7c95c16b88364305984401becd69951199f4a9df2 not found: ID does not exist"
Jul 29 11:05:15.783170 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:15.782898    2569 scope.go:117] "RemoveContainer" containerID="5711abc24e048ac38a16577d4efde11da0e44f85b45dfc20314f5c3fe7ad9b82"
Jul 29 11:05:15.783662 ip-10-0-137-111 kubenswrapper[2569]: E0729 11:05:15.783633    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5711abc24e048ac38a16577d4efde11da0e44f85b45dfc20314f5c3fe7ad9b82\": container with ID starting with 5711abc24e048ac38a16577d4efde11da0e44f85b45dfc20314f5c3fe7ad9b82 not found: ID does not exist" containerID="5711abc24e048ac38a16577d4efde11da0e44f85b45dfc20314f5c3fe7ad9b82"
Jul 29 11:05:15.783894 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:15.783848    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5711abc24e048ac38a16577d4efde11da0e44f85b45dfc20314f5c3fe7ad9b82"} err="failed to get container status \"5711abc24e048ac38a16577d4efde11da0e44f85b45dfc20314f5c3fe7ad9b82\": rpc error: code = NotFound desc = could not find container \"5711abc24e048ac38a16577d4efde11da0e44f85b45dfc20314f5c3fe7ad9b82\": container with ID starting with 5711abc24e048ac38a16577d4efde11da0e44f85b45dfc20314f5c3fe7ad9b82 not found: ID does not exist"
Jul 29 11:05:15.783954 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:15.783900    2569 scope.go:117] "RemoveContainer" containerID="0dfe2e6153c9520db94dfe9f11533c51c4c23a6c4a9ef765c6a85dbd15dd8ee3"
Jul 29 11:05:15.784395 ip-10-0-137-111 kubenswrapper[2569]: E0729 11:05:15.784367    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dfe2e6153c9520db94dfe9f11533c51c4c23a6c4a9ef765c6a85dbd15dd8ee3\": container with ID starting with 0dfe2e6153c9520db94dfe9f11533c51c4c23a6c4a9ef765c6a85dbd15dd8ee3 not found: ID does not exist" containerID="0dfe2e6153c9520db94dfe9f11533c51c4c23a6c4a9ef765c6a85dbd15dd8ee3"
Jul 29 11:05:15.784489 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:15.784402    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dfe2e6153c9520db94dfe9f11533c51c4c23a6c4a9ef765c6a85dbd15dd8ee3"} err="failed to get container status \"0dfe2e6153c9520db94dfe9f11533c51c4c23a6c4a9ef765c6a85dbd15dd8ee3\": rpc error: code = NotFound desc = could not find container \"0dfe2e6153c9520db94dfe9f11533c51c4c23a6c4a9ef765c6a85dbd15dd8ee3\": container with ID starting with 0dfe2e6153c9520db94dfe9f11533c51c4c23a6c4a9ef765c6a85dbd15dd8ee3 not found: ID does not exist"
Jul 29 11:05:15.785218 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:15.785197    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-1999c-predictor-55bb7799cf-ds5rw"]
Jul 29 11:05:15.837398 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:15.837354    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebc59f98-4ca1-4c6e-8028-4694a9e9dd97" path="/var/lib/kubelet/pods/ebc59f98-4ca1-4c6e-8028-4694a9e9dd97/volumes"
Jul 29 11:05:16.877777 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:16.877742    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dwfs9/must-gather-pcwm6"]
Jul 29 11:05:16.878189 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:16.878024    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebc59f98-4ca1-4c6e-8028-4694a9e9dd97" containerName="storage-initializer"
Jul 29 11:05:16.878189 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:16.878034    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebc59f98-4ca1-4c6e-8028-4694a9e9dd97" containerName="storage-initializer"
Jul 29 11:05:16.878189 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:16.878042    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebc59f98-4ca1-4c6e-8028-4694a9e9dd97" containerName="kserve-container"
Jul 29 11:05:16.878189 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:16.878048    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebc59f98-4ca1-4c6e-8028-4694a9e9dd97" containerName="kserve-container"
Jul 29 11:05:16.878189 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:16.878067    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebc59f98-4ca1-4c6e-8028-4694a9e9dd97" containerName="kube-rbac-proxy"
Jul 29 11:05:16.878189 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:16.878090    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebc59f98-4ca1-4c6e-8028-4694a9e9dd97" containerName="kube-rbac-proxy"
Jul 29 11:05:16.878189 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:16.878148    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebc59f98-4ca1-4c6e-8028-4694a9e9dd97" containerName="kube-rbac-proxy"
Jul 29 11:05:16.878189 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:16.878158    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebc59f98-4ca1-4c6e-8028-4694a9e9dd97" containerName="kserve-container"
Jul 29 11:05:16.882546 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:16.882524    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dwfs9/must-gather-pcwm6"
Jul 29 11:05:16.885411 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:16.885376    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 11:05:16.885812 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:16.885791    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dwfs9\"/\"openshift-service-ca.crt\""
Jul 29 11:05:16.885903 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:16.885828    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dwfs9\"/\"kube-root-ca.crt\""
Jul 29 11:05:16.886806 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:16.886787    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dwfs9\"/\"default-dockercfg-2g9g4\""
Jul 29 11:05:16.890348 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:16.890329    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dwfs9/must-gather-pcwm6"]
Jul 29 11:05:16.918107 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:16.918054    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2j52\" (UniqueName: \"kubernetes.io/projected/8880dbdc-ab0c-4d1e-892e-9b7276377a34-kube-api-access-k2j52\") pod \"must-gather-pcwm6\" (UID: \"8880dbdc-ab0c-4d1e-892e-9b7276377a34\") " pod="openshift-must-gather-dwfs9/must-gather-pcwm6"
Jul 29 11:05:16.918209 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:16.918120    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8880dbdc-ab0c-4d1e-892e-9b7276377a34-must-gather-output\") pod \"must-gather-pcwm6\" (UID: \"8880dbdc-ab0c-4d1e-892e-9b7276377a34\") " pod="openshift-must-gather-dwfs9/must-gather-pcwm6"
Jul 29 11:05:16.934677 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:16.934653    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 11:05:17.019334 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:17.019302    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2j52\" (UniqueName: \"kubernetes.io/projected/8880dbdc-ab0c-4d1e-892e-9b7276377a34-kube-api-access-k2j52\") pod \"must-gather-pcwm6\" (UID: \"8880dbdc-ab0c-4d1e-892e-9b7276377a34\") " pod="openshift-must-gather-dwfs9/must-gather-pcwm6"
Jul 29 11:05:17.019507 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:17.019348    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8880dbdc-ab0c-4d1e-892e-9b7276377a34-must-gather-output\") pod \"must-gather-pcwm6\" (UID: \"8880dbdc-ab0c-4d1e-892e-9b7276377a34\") " pod="openshift-must-gather-dwfs9/must-gather-pcwm6"
Jul 29 11:05:17.019661 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:17.019637    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8880dbdc-ab0c-4d1e-892e-9b7276377a34-must-gather-output\") pod \"must-gather-pcwm6\" (UID: \"8880dbdc-ab0c-4d1e-892e-9b7276377a34\") " pod="openshift-must-gather-dwfs9/must-gather-pcwm6"
Jul 29 11:05:17.028808 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:17.028786    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2j52\" (UniqueName: \"kubernetes.io/projected/8880dbdc-ab0c-4d1e-892e-9b7276377a34-kube-api-access-k2j52\") pod \"must-gather-pcwm6\" (UID: \"8880dbdc-ab0c-4d1e-892e-9b7276377a34\") " pod="openshift-must-gather-dwfs9/must-gather-pcwm6"
Jul 29 11:05:17.206600 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:17.206559    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dwfs9/must-gather-pcwm6"
Jul 29 11:05:17.335198 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:17.335083    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dwfs9/must-gather-pcwm6"]
Jul 29 11:05:17.337756 ip-10-0-137-111 kubenswrapper[2569]: W0729 11:05:17.337728    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8880dbdc_ab0c_4d1e_892e_9b7276377a34.slice/crio-e82770d29bc668bd88ecb1511dd7e721fb38facda82d5a93299bf7ba5310f749 WatchSource:0}: Error finding container e82770d29bc668bd88ecb1511dd7e721fb38facda82d5a93299bf7ba5310f749: Status 404 returned error can't find the container with id e82770d29bc668bd88ecb1511dd7e721fb38facda82d5a93299bf7ba5310f749
Jul 29 11:05:17.339651 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:17.339635    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 11:05:17.765744 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:17.765709    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dwfs9/must-gather-pcwm6" event={"ID":"8880dbdc-ab0c-4d1e-892e-9b7276377a34","Type":"ContainerStarted","Data":"e82770d29bc668bd88ecb1511dd7e721fb38facda82d5a93299bf7ba5310f749"}
Jul 29 11:05:23.798439 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:23.798407    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dwfs9/must-gather-pcwm6" event={"ID":"8880dbdc-ab0c-4d1e-892e-9b7276377a34","Type":"ContainerStarted","Data":"7710f4f2be27e2a0699bb591413fa91b3c7301c48abe921a522e0adb52ba7b4e"}
Jul 29 11:05:24.805255 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:24.805214    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dwfs9/must-gather-pcwm6" event={"ID":"8880dbdc-ab0c-4d1e-892e-9b7276377a34","Type":"ContainerStarted","Data":"544ffa438874af94043e2ee282065c8f2b0ae62483f413eea15cff150c8fa1a9"}
Jul 29 11:05:24.822200 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:24.822143    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dwfs9/must-gather-pcwm6" podStartSLOduration=2.4867374079999998 podStartE2EDuration="8.822121958s" totalImagesPullingTime="6.33538455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:05:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:05:24.820714249 +0000 UTC m=+2201.545504923" watchObservedRunningTime="2026-07-29 11:05:24.822121958 +0000 UTC m=+2201.546912632"
Jul 29 11:05:45.892707 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:45.892613    2569 generic.go:356] "Generic (PLEG): container finished" podID="8880dbdc-ab0c-4d1e-892e-9b7276377a34" containerID="7710f4f2be27e2a0699bb591413fa91b3c7301c48abe921a522e0adb52ba7b4e" exitCode=0
Jul 29 11:05:45.892707 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:45.892686    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dwfs9/must-gather-pcwm6" event={"ID":"8880dbdc-ab0c-4d1e-892e-9b7276377a34","Type":"ContainerDied","Data":"7710f4f2be27e2a0699bb591413fa91b3c7301c48abe921a522e0adb52ba7b4e"}
Jul 29 11:05:45.893191 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:45.893019    2569 scope.go:117] "RemoveContainer" containerID="7710f4f2be27e2a0699bb591413fa91b3c7301c48abe921a522e0adb52ba7b4e"
Jul 29 11:05:46.501223 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:46.501196    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-dwfs9_must-gather-pcwm6_8880dbdc-ab0c-4d1e-892e-9b7276377a34/gather/0.log"
Jul 29 11:05:46.900780 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:46.900678    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 11:05:47.044562 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:47.044529    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mxlpr/must-gather-xm89r"]
Jul 29 11:05:47.046911 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:47.046895    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mxlpr/must-gather-xm89r"
Jul 29 11:05:47.050504 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:47.050481    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mxlpr\"/\"kube-root-ca.crt\""
Jul 29 11:05:47.050504 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:47.050485    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mxlpr\"/\"default-dockercfg-dc47v\""
Jul 29 11:05:47.050692 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:47.050542    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mxlpr\"/\"openshift-service-ca.crt\""
Jul 29 11:05:47.058326 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:47.058305    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mxlpr/must-gather-xm89r"]
Jul 29 11:05:47.060900 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:47.060867    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 11:05:47.155921 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:47.155805    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 11:05:47.187061 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:47.187028    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcw9f\" (UniqueName: \"kubernetes.io/projected/6fb8a3e0-6cdf-4eb8-bc03-8a446848032f-kube-api-access-jcw9f\") pod \"must-gather-xm89r\" (UID: \"6fb8a3e0-6cdf-4eb8-bc03-8a446848032f\") " pod="openshift-must-gather-mxlpr/must-gather-xm89r"
Jul 29 11:05:47.187247 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:47.187142    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6fb8a3e0-6cdf-4eb8-bc03-8a446848032f-must-gather-output\") pod \"must-gather-xm89r\" (UID: \"6fb8a3e0-6cdf-4eb8-bc03-8a446848032f\") " pod="openshift-must-gather-mxlpr/must-gather-xm89r"
Jul 29 11:05:47.287740 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:47.287701    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6fb8a3e0-6cdf-4eb8-bc03-8a446848032f-must-gather-output\") pod \"must-gather-xm89r\" (UID: \"6fb8a3e0-6cdf-4eb8-bc03-8a446848032f\") " pod="openshift-must-gather-mxlpr/must-gather-xm89r"
Jul 29 11:05:47.287740 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:47.287744    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcw9f\" (UniqueName: \"kubernetes.io/projected/6fb8a3e0-6cdf-4eb8-bc03-8a446848032f-kube-api-access-jcw9f\") pod \"must-gather-xm89r\" (UID: \"6fb8a3e0-6cdf-4eb8-bc03-8a446848032f\") " pod="openshift-must-gather-mxlpr/must-gather-xm89r"
Jul 29 11:05:47.288145 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:47.288120    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6fb8a3e0-6cdf-4eb8-bc03-8a446848032f-must-gather-output\") pod \"must-gather-xm89r\" (UID: \"6fb8a3e0-6cdf-4eb8-bc03-8a446848032f\") " pod="openshift-must-gather-mxlpr/must-gather-xm89r"
Jul 29 11:05:47.296031 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:47.296010    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcw9f\" (UniqueName: \"kubernetes.io/projected/6fb8a3e0-6cdf-4eb8-bc03-8a446848032f-kube-api-access-jcw9f\") pod \"must-gather-xm89r\" (UID: \"6fb8a3e0-6cdf-4eb8-bc03-8a446848032f\") " pod="openshift-must-gather-mxlpr/must-gather-xm89r"
Jul 29 11:05:47.356112 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:47.356060    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mxlpr/must-gather-xm89r"
Jul 29 11:05:47.487739 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:47.487713    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mxlpr/must-gather-xm89r"]
Jul 29 11:05:47.490038 ip-10-0-137-111 kubenswrapper[2569]: W0729 11:05:47.490005    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6fb8a3e0_6cdf_4eb8_bc03_8a446848032f.slice/crio-d05c44334cdfa75b607966e48c5fc6096c1d4bcafa7e384df8fbbba694958802 WatchSource:0}: Error finding container d05c44334cdfa75b607966e48c5fc6096c1d4bcafa7e384df8fbbba694958802: Status 404 returned error can't find the container with id d05c44334cdfa75b607966e48c5fc6096c1d4bcafa7e384df8fbbba694958802
Jul 29 11:05:47.901493 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:47.901457    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mxlpr/must-gather-xm89r" event={"ID":"6fb8a3e0-6cdf-4eb8-bc03-8a446848032f","Type":"ContainerStarted","Data":"d05c44334cdfa75b607966e48c5fc6096c1d4bcafa7e384df8fbbba694958802"}
Jul 29 11:05:48.908661 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:48.908623    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mxlpr/must-gather-xm89r" event={"ID":"6fb8a3e0-6cdf-4eb8-bc03-8a446848032f","Type":"ContainerStarted","Data":"e399638919c9f5ef370aac4d941661f249492c0a9b000158634168f10dc9c03d"}
Jul 29 11:05:48.909362 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:48.909229    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mxlpr/must-gather-xm89r" event={"ID":"6fb8a3e0-6cdf-4eb8-bc03-8a446848032f","Type":"ContainerStarted","Data":"ee90fc31fa5141e74a4b0c4f3d4322d7965ad37f6c83d6ce4ded04d1e33a966d"}
Jul 29 11:05:48.926145 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:48.926058    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mxlpr/must-gather-xm89r" podStartSLOduration=1.23049038 podStartE2EDuration="1.926036188s" totalImagesPullingTime="695.545808ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:05:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:05:48.924747901 +0000 UTC m=+2225.649538575" watchObservedRunningTime="2026-07-29 11:05:48.926036188 +0000 UTC m=+2225.650826861"
Jul 29 11:05:49.513272 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:49.513170    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-p8p95_27e6f470-ad48-45f1-b171-ae66fac19573/global-pull-secret-syncer/0.log"
Jul 29 11:05:49.613879 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:49.613842    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log"
Jul 29 11:05:49.730353 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:49.730321    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-111.ec2.internal_e7e9628ca209a194bcb37f10a7461829/haproxy/0.log"
Jul 29 11:05:51.927253 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:51.926650    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-dwfs9/must-gather-pcwm6"]
Jul 29 11:05:51.927253 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:51.926869    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-dwfs9/must-gather-pcwm6" podUID="8880dbdc-ab0c-4d1e-892e-9b7276377a34" containerName="copy" containerID="cri-o://544ffa438874af94043e2ee282065c8f2b0ae62483f413eea15cff150c8fa1a9" gracePeriod=2
Jul 29 11:05:51.932121 ip-10-0-137-111 kubenswrapper[2569]: E0729 11:05:51.931686    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-pcwm6\" is forbidden: User \"system:node:ip-10-0-137-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-dwfs9\": no relationship found between node 'ip-10-0-137-111.ec2.internal' and this object" podUID="8880dbdc-ab0c-4d1e-892e-9b7276377a34" pod="openshift-must-gather-dwfs9/must-gather-pcwm6"
Jul 29 11:05:51.933901 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:51.933878    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-dwfs9/must-gather-pcwm6"]
Jul 29 11:05:52.289988 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:52.289727    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-dwfs9_must-gather-pcwm6_8880dbdc-ab0c-4d1e-892e-9b7276377a34/copy/0.log"
Jul 29 11:05:52.293205 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:52.292928    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dwfs9/must-gather-pcwm6"
Jul 29 11:05:52.295973 ip-10-0-137-111 kubenswrapper[2569]: E0729 11:05:52.295126    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-pcwm6\" is forbidden: User \"system:node:ip-10-0-137-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-dwfs9\": no relationship found between node 'ip-10-0-137-111.ec2.internal' and this object" podUID="8880dbdc-ab0c-4d1e-892e-9b7276377a34" pod="openshift-must-gather-dwfs9/must-gather-pcwm6"
Jul 29 11:05:52.445553 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:52.445510    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8880dbdc-ab0c-4d1e-892e-9b7276377a34-must-gather-output\") pod \"8880dbdc-ab0c-4d1e-892e-9b7276377a34\" (UID: \"8880dbdc-ab0c-4d1e-892e-9b7276377a34\") "
Jul 29 11:05:52.445740 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:52.445616    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2j52\" (UniqueName: \"kubernetes.io/projected/8880dbdc-ab0c-4d1e-892e-9b7276377a34-kube-api-access-k2j52\") pod \"8880dbdc-ab0c-4d1e-892e-9b7276377a34\" (UID: \"8880dbdc-ab0c-4d1e-892e-9b7276377a34\") "
Jul 29 11:05:52.447592 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:52.447554    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8880dbdc-ab0c-4d1e-892e-9b7276377a34-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "8880dbdc-ab0c-4d1e-892e-9b7276377a34" (UID: "8880dbdc-ab0c-4d1e-892e-9b7276377a34"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:05:52.448956 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:52.448909    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8880dbdc-ab0c-4d1e-892e-9b7276377a34-kube-api-access-k2j52" (OuterVolumeSpecName: "kube-api-access-k2j52") pod "8880dbdc-ab0c-4d1e-892e-9b7276377a34" (UID: "8880dbdc-ab0c-4d1e-892e-9b7276377a34"). InnerVolumeSpecName "kube-api-access-k2j52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:05:52.546826 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:52.546694    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2j52\" (UniqueName: \"kubernetes.io/projected/8880dbdc-ab0c-4d1e-892e-9b7276377a34-kube-api-access-k2j52\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 11:05:52.546826 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:52.546739    2569 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8880dbdc-ab0c-4d1e-892e-9b7276377a34-must-gather-output\") on node \"ip-10-0-137-111.ec2.internal\" DevicePath \"\""
Jul 29 11:05:52.704912 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:52.704760    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 11:05:52.806730 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:52.806638    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6c2hd_512bc3c2-b150-4eb6-ba9f-b02810556544/konnectivity-agent/0.log" limit=1024
Jul 29 11:05:52.933644 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:52.933364    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-dwfs9_must-gather-pcwm6_8880dbdc-ab0c-4d1e-892e-9b7276377a34/copy/0.log"
Jul 29 11:05:52.934217 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:52.933842    2569 generic.go:356] "Generic (PLEG): container finished" podID="8880dbdc-ab0c-4d1e-892e-9b7276377a34" containerID="544ffa438874af94043e2ee282065c8f2b0ae62483f413eea15cff150c8fa1a9" exitCode=143
Jul 29 11:05:52.934217 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:52.933967    2569 scope.go:117] "RemoveContainer" containerID="544ffa438874af94043e2ee282065c8f2b0ae62483f413eea15cff150c8fa1a9"
Jul 29 11:05:52.934217 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:52.934145    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dwfs9/must-gather-pcwm6"
Jul 29 11:05:52.936670 ip-10-0-137-111 kubenswrapper[2569]: E0729 11:05:52.936629    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-pcwm6\" is forbidden: User \"system:node:ip-10-0-137-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-dwfs9\": no relationship found between node 'ip-10-0-137-111.ec2.internal' and this object" podUID="8880dbdc-ab0c-4d1e-892e-9b7276377a34" pod="openshift-must-gather-dwfs9/must-gather-pcwm6"
Jul 29 11:05:52.948337 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:52.947499    2569 scope.go:117] "RemoveContainer" containerID="7710f4f2be27e2a0699bb591413fa91b3c7301c48abe921a522e0adb52ba7b4e"
Jul 29 11:05:52.980132 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:52.978257    2569 scope.go:117] "RemoveContainer" containerID="544ffa438874af94043e2ee282065c8f2b0ae62483f413eea15cff150c8fa1a9"
Jul 29 11:05:52.984172 ip-10-0-137-111 kubenswrapper[2569]: E0729 11:05:52.980512    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-pcwm6\" is forbidden: User \"system:node:ip-10-0-137-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-dwfs9\": no relationship found between node 'ip-10-0-137-111.ec2.internal' and this object" podUID="8880dbdc-ab0c-4d1e-892e-9b7276377a34" pod="openshift-must-gather-dwfs9/must-gather-pcwm6"
Jul 29 11:05:52.986740 ip-10-0-137-111 kubenswrapper[2569]: E0729 11:05:52.986570    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"544ffa438874af94043e2ee282065c8f2b0ae62483f413eea15cff150c8fa1a9\": container with ID starting with 544ffa438874af94043e2ee282065c8f2b0ae62483f413eea15cff150c8fa1a9 not found: ID does not exist" containerID="544ffa438874af94043e2ee282065c8f2b0ae62483f413eea15cff150c8fa1a9"
Jul 29 11:05:52.986740 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:52.986614    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"544ffa438874af94043e2ee282065c8f2b0ae62483f413eea15cff150c8fa1a9"} err="failed to get container status \"544ffa438874af94043e2ee282065c8f2b0ae62483f413eea15cff150c8fa1a9\": rpc error: code = NotFound desc = could not find container \"544ffa438874af94043e2ee282065c8f2b0ae62483f413eea15cff150c8fa1a9\": container with ID starting with 544ffa438874af94043e2ee282065c8f2b0ae62483f413eea15cff150c8fa1a9 not found: ID does not exist"
Jul 29 11:05:52.986740 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:52.986641    2569 scope.go:117] "RemoveContainer" containerID="7710f4f2be27e2a0699bb591413fa91b3c7301c48abe921a522e0adb52ba7b4e"
Jul 29 11:05:52.987288 ip-10-0-137-111 kubenswrapper[2569]: E0729 11:05:52.987226    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7710f4f2be27e2a0699bb591413fa91b3c7301c48abe921a522e0adb52ba7b4e\": container with ID starting with 7710f4f2be27e2a0699bb591413fa91b3c7301c48abe921a522e0adb52ba7b4e not found: ID does not exist" containerID="7710f4f2be27e2a0699bb591413fa91b3c7301c48abe921a522e0adb52ba7b4e"
Jul 29 11:05:52.987288 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:52.987258    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7710f4f2be27e2a0699bb591413fa91b3c7301c48abe921a522e0adb52ba7b4e"} err="failed to get container status \"7710f4f2be27e2a0699bb591413fa91b3c7301c48abe921a522e0adb52ba7b4e\": rpc error: code = NotFound desc = could not find container \"7710f4f2be27e2a0699bb591413fa91b3c7301c48abe921a522e0adb52ba7b4e\": container with ID starting with 7710f4f2be27e2a0699bb591413fa91b3c7301c48abe921a522e0adb52ba7b4e not found: ID does not exist"
Jul 29 11:05:53.838449 ip-10-0-137-111 kubenswrapper[2569]: E0729 11:05:53.838384    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-pcwm6\" is forbidden: User \"system:node:ip-10-0-137-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-dwfs9\": no relationship found between node 'ip-10-0-137-111.ec2.internal' and this object" podUID="8880dbdc-ab0c-4d1e-892e-9b7276377a34" pod="openshift-must-gather-dwfs9/must-gather-pcwm6"
Jul 29 11:05:53.840056 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:53.840016    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8880dbdc-ab0c-4d1e-892e-9b7276377a34" path="/var/lib/kubelet/pods/8880dbdc-ab0c-4d1e-892e-9b7276377a34/volumes"
Jul 29 11:05:54.401844 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:54.401808    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-9dd45858d-4m6ts_c0e19b31-af9a-4a7b-8dcb-b4c76bd6d3be/metrics-server/0.log"
Jul 29 11:05:54.426063 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:54.426029    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-q7dn5_7e7009ad-7d80-45c3-aff0-22e228496341/monitoring-plugin/0.log"
Jul 29 11:05:54.519972 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:54.519889    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dn2fp_683dda88-94e8-4960-be54-7e26765bd876/node-exporter/0.log"
Jul 29 11:05:54.548583 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:54.548543    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dn2fp_683dda88-94e8-4960-be54-7e26765bd876/kube-rbac-proxy/0.log"
Jul 29 11:05:54.569787 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:54.569757    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dn2fp_683dda88-94e8-4960-be54-7e26765bd876/init-textfile/0.log"
Jul 29 11:05:54.768995 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:54.768956    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_672f11b9-45a4-4641-be41-e585b857b1ac/prometheus/0.log"
Jul 29 11:05:54.796115 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:54.795974    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_672f11b9-45a4-4641-be41-e585b857b1ac/config-reloader/0.log"
Jul 29 11:05:54.816272 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:54.816239    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_672f11b9-45a4-4641-be41-e585b857b1ac/thanos-sidecar/0.log"
Jul 29 11:05:54.834498 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:54.834467    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_672f11b9-45a4-4641-be41-e585b857b1ac/kube-rbac-proxy-web/0.log"
Jul 29 11:05:54.853114 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:54.853059    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_672f11b9-45a4-4641-be41-e585b857b1ac/kube-rbac-proxy/0.log"
Jul 29 11:05:54.873941 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:54.873889    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_672f11b9-45a4-4641-be41-e585b857b1ac/kube-rbac-proxy-thanos/0.log"
Jul 29 11:05:54.891608 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:54.891576    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_672f11b9-45a4-4641-be41-e585b857b1ac/init-config-reloader/0.log"
Jul 29 11:05:54.919213 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:54.919181    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-qzzx2_70bc0d3f-ea52-4c82-b033-961c7fd5fc8c/prometheus-operator/0.log"
Jul 29 11:05:54.939589 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:54.939560    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-qzzx2_70bc0d3f-ea52-4c82-b033-961c7fd5fc8c/kube-rbac-proxy/0.log"
Jul 29 11:05:56.910535 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:56.910501    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-pvdgw_afd0d65c-bbfc-49f0-81b6-e855fd81da7f/console-operator/1.log"
Jul 29 11:05:56.919997 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:56.919949    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-pvdgw_afd0d65c-bbfc-49f0-81b6-e855fd81da7f/console-operator/2.log"
Jul 29 11:05:57.351798 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:57.351767    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-7gk4j_f0278c1e-7a6a-472f-aaba-09c9a22c9fa9/download-server/0.log"
Jul 29 11:05:58.356473 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:58.356438    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mxlpr/perf-node-gather-daemonset-7nnfv"]
Jul 29 11:05:58.356836 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:58.356745    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8880dbdc-ab0c-4d1e-892e-9b7276377a34" containerName="gather"
Jul 29 11:05:58.356836 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:58.356756    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8880dbdc-ab0c-4d1e-892e-9b7276377a34" containerName="gather"
Jul 29 11:05:58.356836 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:58.356768    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8880dbdc-ab0c-4d1e-892e-9b7276377a34" containerName="copy"
Jul 29 11:05:58.356836 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:58.356774    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8880dbdc-ab0c-4d1e-892e-9b7276377a34" containerName="copy"
Jul 29 11:05:58.356836 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:58.356830    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8880dbdc-ab0c-4d1e-892e-9b7276377a34" containerName="copy"
Jul 29 11:05:58.356836 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:58.356839    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8880dbdc-ab0c-4d1e-892e-9b7276377a34" containerName="gather"
Jul 29 11:05:58.360647 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:58.360619    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mxlpr/perf-node-gather-daemonset-7nnfv"
Jul 29 11:05:58.373848 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:58.373824    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mxlpr/perf-node-gather-daemonset-7nnfv"]
Jul 29 11:05:58.502000 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:58.501955    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/39999d64-e1a8-436b-87cc-308d5e540a1d-proc\") pod \"perf-node-gather-daemonset-7nnfv\" (UID: \"39999d64-e1a8-436b-87cc-308d5e540a1d\") " pod="openshift-must-gather-mxlpr/perf-node-gather-daemonset-7nnfv"
Jul 29 11:05:58.502000 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:58.502012    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlfvp\" (UniqueName: \"kubernetes.io/projected/39999d64-e1a8-436b-87cc-308d5e540a1d-kube-api-access-vlfvp\") pod \"perf-node-gather-daemonset-7nnfv\" (UID: \"39999d64-e1a8-436b-87cc-308d5e540a1d\") " pod="openshift-must-gather-mxlpr/perf-node-gather-daemonset-7nnfv"
Jul 29 11:05:58.502236 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:58.502105    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/39999d64-e1a8-436b-87cc-308d5e540a1d-lib-modules\") pod \"perf-node-gather-daemonset-7nnfv\" (UID: \"39999d64-e1a8-436b-87cc-308d5e540a1d\") " pod="openshift-must-gather-mxlpr/perf-node-gather-daemonset-7nnfv"
Jul 29 11:05:58.502236 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:58.502124    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/39999d64-e1a8-436b-87cc-308d5e540a1d-podres\") pod \"perf-node-gather-daemonset-7nnfv\" (UID: \"39999d64-e1a8-436b-87cc-308d5e540a1d\") " pod="openshift-must-gather-mxlpr/perf-node-gather-daemonset-7nnfv"
Jul 29 11:05:58.502236 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:58.502158    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39999d64-e1a8-436b-87cc-308d5e540a1d-sys\") pod \"perf-node-gather-daemonset-7nnfv\" (UID: \"39999d64-e1a8-436b-87cc-308d5e540a1d\") " pod="openshift-must-gather-mxlpr/perf-node-gather-daemonset-7nnfv"
Jul 29 11:05:58.603446 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:58.603385    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/39999d64-e1a8-436b-87cc-308d5e540a1d-proc\") pod \"perf-node-gather-daemonset-7nnfv\" (UID: \"39999d64-e1a8-436b-87cc-308d5e540a1d\") " pod="openshift-must-gather-mxlpr/perf-node-gather-daemonset-7nnfv"
Jul 29 11:05:58.603630 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:58.603504    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlfvp\" (UniqueName: \"kubernetes.io/projected/39999d64-e1a8-436b-87cc-308d5e540a1d-kube-api-access-vlfvp\") pod \"perf-node-gather-daemonset-7nnfv\" (UID: \"39999d64-e1a8-436b-87cc-308d5e540a1d\") " pod="openshift-must-gather-mxlpr/perf-node-gather-daemonset-7nnfv"
Jul 29 11:05:58.603630 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:58.603543    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/39999d64-e1a8-436b-87cc-308d5e540a1d-proc\") pod \"perf-node-gather-daemonset-7nnfv\" (UID: \"39999d64-e1a8-436b-87cc-308d5e540a1d\") " pod="openshift-must-gather-mxlpr/perf-node-gather-daemonset-7nnfv"
Jul 29 11:05:58.604309 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:58.604246    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/39999d64-e1a8-436b-87cc-308d5e540a1d-lib-modules\") pod \"perf-node-gather-daemonset-7nnfv\" (UID: \"39999d64-e1a8-436b-87cc-308d5e540a1d\") " pod="openshift-must-gather-mxlpr/perf-node-gather-daemonset-7nnfv"
Jul 29 11:05:58.604449 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:58.604310    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/39999d64-e1a8-436b-87cc-308d5e540a1d-lib-modules\") pod \"perf-node-gather-daemonset-7nnfv\" (UID: \"39999d64-e1a8-436b-87cc-308d5e540a1d\") " pod="openshift-must-gather-mxlpr/perf-node-gather-daemonset-7nnfv"
Jul 29 11:05:58.604449 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:58.604348    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/39999d64-e1a8-436b-87cc-308d5e540a1d-podres\") pod \"perf-node-gather-daemonset-7nnfv\" (UID: \"39999d64-e1a8-436b-87cc-308d5e540a1d\") " pod="openshift-must-gather-mxlpr/perf-node-gather-daemonset-7nnfv"
Jul 29 11:05:58.604449 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:58.604421    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39999d64-e1a8-436b-87cc-308d5e540a1d-sys\") pod \"perf-node-gather-daemonset-7nnfv\" (UID: \"39999d64-e1a8-436b-87cc-308d5e540a1d\") " pod="openshift-must-gather-mxlpr/perf-node-gather-daemonset-7nnfv"
Jul 29 11:05:58.604615 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:58.604515    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/39999d64-e1a8-436b-87cc-308d5e540a1d-podres\") pod \"perf-node-gather-daemonset-7nnfv\" (UID: \"39999d64-e1a8-436b-87cc-308d5e540a1d\") " pod="openshift-must-gather-mxlpr/perf-node-gather-daemonset-7nnfv"
Jul 29 11:05:58.604615 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:58.604544    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39999d64-e1a8-436b-87cc-308d5e540a1d-sys\") pod \"perf-node-gather-daemonset-7nnfv\" (UID: \"39999d64-e1a8-436b-87cc-308d5e540a1d\") " pod="openshift-must-gather-mxlpr/perf-node-gather-daemonset-7nnfv"
Jul 29 11:05:58.611959 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:58.611901    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlfvp\" (UniqueName: \"kubernetes.io/projected/39999d64-e1a8-436b-87cc-308d5e540a1d-kube-api-access-vlfvp\") pod \"perf-node-gather-daemonset-7nnfv\" (UID: \"39999d64-e1a8-436b-87cc-308d5e540a1d\") " pod="openshift-must-gather-mxlpr/perf-node-gather-daemonset-7nnfv"
Jul 29 11:05:58.673688 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:58.673657    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mxlpr/perf-node-gather-daemonset-7nnfv"
Jul 29 11:05:58.802905 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:58.802869    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ljsqp_5edb4b57-220e-4f39-b481-164e813bbf3c/dns/0.log"
Jul 29 11:05:58.818590 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:58.818563    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mxlpr/perf-node-gather-daemonset-7nnfv"]
Jul 29 11:05:58.820944 ip-10-0-137-111 kubenswrapper[2569]: W0729 11:05:58.820915    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod39999d64_e1a8_436b_87cc_308d5e540a1d.slice/crio-d311d960dcb4a007f6db453b1c4139a61e9bbd702308b5c26020fc021f277b12 WatchSource:0}: Error finding container d311d960dcb4a007f6db453b1c4139a61e9bbd702308b5c26020fc021f277b12: Status 404 returned error can't find the container with id d311d960dcb4a007f6db453b1c4139a61e9bbd702308b5c26020fc021f277b12
Jul 29 11:05:58.821980 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:58.821959    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ljsqp_5edb4b57-220e-4f39-b481-164e813bbf3c/kube-rbac-proxy/0.log"
Jul 29 11:05:58.860412 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:58.860384    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hpmhx_faeca467-a198-4dd6-9da8-e4f72ac68c77/dns-node-resolver/0.log"
Jul 29 11:05:58.968562 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:58.968526    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mxlpr/perf-node-gather-daemonset-7nnfv" event={"ID":"39999d64-e1a8-436b-87cc-308d5e540a1d","Type":"ContainerStarted","Data":"7ed2af645953a359f53ed2e5383e0946d90957654f1b1c9278ac5d22743c7433"}
Jul 29 11:05:58.968751 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:58.968581    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mxlpr/perf-node-gather-daemonset-7nnfv" event={"ID":"39999d64-e1a8-436b-87cc-308d5e540a1d","Type":"ContainerStarted","Data":"d311d960dcb4a007f6db453b1c4139a61e9bbd702308b5c26020fc021f277b12"}
Jul 29 11:05:58.968751 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:58.968639    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mxlpr/perf-node-gather-daemonset-7nnfv"
Jul 29 11:05:58.985510 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:58.985461    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mxlpr/perf-node-gather-daemonset-7nnfv" podStartSLOduration=0.985444608 podStartE2EDuration="985.444608ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 11:05:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:05:58.983407258 +0000 UTC m=+2235.708197931" watchObservedRunningTime="2026-07-29 11:05:58.985444608 +0000 UTC m=+2235.710235280"
Jul 29 11:05:59.317181 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:05:59.317152    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-b6v6k_bd747eb7-faeb-48ab-804a-96355d0b79c4/node-ca/0.log"
Jul 29 11:06:00.343393 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:06:00.343366    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jvdpp_f3e9971d-59b4-4ea2-a9f3-5792b9e0b804/serve-healthcheck-canary/0.log"
Jul 29 11:06:00.751024 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:06:00.750996    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-llm74_7687cc83-d880-467f-a270-32bba1bba683/kube-rbac-proxy/0.log"
Jul 29 11:06:00.768830 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:06:00.768802    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-llm74_7687cc83-d880-467f-a270-32bba1bba683/exporter/0.log"
Jul 29 11:06:00.786523 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:06:00.786485    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-llm74_7687cc83-d880-467f-a270-32bba1bba683/extractor/0.log"
Jul 29 11:06:03.031478 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:06:03.031452    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-pz52z_7a2002a3-039c-4654-b534-0dff05f3c704/server/0.log"
Jul 29 11:06:03.170187 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:06:03.170154    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-ttxsr_f0c13552-5b54-4754-b970-65a8307a4698/s3-init/0.log"
Jul 29 11:06:03.183875 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:06:03.183847    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-ttxsr_f0c13552-5b54-4754-b970-65a8307a4698/download-hf-model/0.log"
Jul 29 11:06:04.984827 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:06:04.984798    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mxlpr/perf-node-gather-daemonset-7nnfv"
Jul 29 11:06:06.777145 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:06:06.777020    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-mhnng_192911bd-5538-4220-8429-4e435554e1ef/migrator/0.log"
Jul 29 11:06:06.799701 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:06:06.799672    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-mhnng_192911bd-5538-4220-8429-4e435554e1ef/graceful-termination/0.log"
Jul 29 11:06:08.333833 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:06:08.333759    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r7xrd_2e6f3639-fa84-4599-8a55-a6d7242cea3a/kube-multus-additional-cni-plugins/0.log"
Jul 29 11:06:08.352452 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:06:08.352420    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r7xrd_2e6f3639-fa84-4599-8a55-a6d7242cea3a/egress-router-binary-copy/0.log"
Jul 29 11:06:08.370057 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:06:08.370030    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r7xrd_2e6f3639-fa84-4599-8a55-a6d7242cea3a/cni-plugins/0.log"
Jul 29 11:06:08.390128 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:06:08.390106    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r7xrd_2e6f3639-fa84-4599-8a55-a6d7242cea3a/bond-cni-plugin/0.log"
Jul 29 11:06:08.407193 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:06:08.407172    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r7xrd_2e6f3639-fa84-4599-8a55-a6d7242cea3a/routeoverride-cni/0.log"
Jul 29 11:06:08.423705 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:06:08.423687    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r7xrd_2e6f3639-fa84-4599-8a55-a6d7242cea3a/whereabouts-cni-bincopy/0.log"
Jul 29 11:06:08.440121 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:06:08.440104    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r7xrd_2e6f3639-fa84-4599-8a55-a6d7242cea3a/whereabouts-cni/0.log"
Jul 29 11:06:08.649831 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:06:08.649742    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-wkrbj_cffc10c9-fe6c-420b-b994-869307922201/kube-multus/0.log"
Jul 29 11:06:08.704849 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:06:08.704809    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-szr8n_51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c/network-metrics-daemon/0.log"
Jul 29 11:06:08.718200 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:06:08.718178    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-szr8n_51dcc151-74b3-46ad-9ef5-5d1e5acc9f4c/kube-rbac-proxy/0.log"
Jul 29 11:06:09.547304 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:06:09.547268    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6jzgn_90f1789e-de1e-4cc2-a128-ee7621bf80ca/ovn-controller/0.log"
Jul 29 11:06:09.564443 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:06:09.564416    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6jzgn_90f1789e-de1e-4cc2-a128-ee7621bf80ca/ovn-acl-logging/0.log"
Jul 29 11:06:09.592089 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:06:09.592049    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6jzgn_90f1789e-de1e-4cc2-a128-ee7621bf80ca/kube-rbac-proxy-node/0.log"
Jul 29 11:06:09.614419 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:06:09.614389    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6jzgn_90f1789e-de1e-4cc2-a128-ee7621bf80ca/kube-rbac-proxy-ovn-metrics/0.log"
Jul 29 11:06:09.628482 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:06:09.628441    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6jzgn_90f1789e-de1e-4cc2-a128-ee7621bf80ca/northd/0.log"
Jul 29 11:06:09.645552 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:06:09.645526    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6jzgn_90f1789e-de1e-4cc2-a128-ee7621bf80ca/nbdb/0.log"
Jul 29 11:06:09.663859 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:06:09.663826    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6jzgn_90f1789e-de1e-4cc2-a128-ee7621bf80ca/sbdb/0.log"
Jul 29 11:06:09.726058 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:06:09.726019    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6jzgn_90f1789e-de1e-4cc2-a128-ee7621bf80ca/ovnkube-controller/0.log"
Jul 29 11:06:10.980444 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:06:10.980417    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7d88cb55d8-fgw4v_223649ac-cf58-4cbe-827d-9b16b0ee0359/check-endpoints/0.log"
Jul 29 11:06:11.021134 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:06:11.021107    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-mdqnc_f7b4bba0-77fe-4240-9858-433b0f431c59/network-check-target-container/0.log"
Jul 29 11:06:12.053307 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:06:12.053279    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-n5pfx_770e3b65-661a-4dad-aaa9-5fbb68ae8dea/iptables-alerter/0.log"
Jul 29 11:06:12.703380 ip-10-0-137-111 kubenswrapper[2569]: I0729 11:06:12.703349    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-str4m_b9d340b7-82a4-43bd-96b8-e19532b16b4b/tuned/0.log"