Jul 15 10:46:21.063093 ip-10-0-131-34 systemd[1]: Starting Kubernetes Kubelet...
Jul 15 10:46:21.423192 ip-10-0-131-34 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 15 10:46:21.423192 ip-10-0-131-34 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 15 10:46:21.423192 ip-10-0-131-34 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 15 10:46:21.423192 ip-10-0-131-34 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 15 10:46:21.423192 ip-10-0-131-34 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 15 10:46:21.424060 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.423966    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 15 10:46:21.429009 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.428990    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 15 10:46:21.429009 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429005    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 15 10:46:21.429009 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429009    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 15 10:46:21.429009 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429012    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 15 10:46:21.429009 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429015    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 15 10:46:21.429205 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429019    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 15 10:46:21.429205 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429022    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 15 10:46:21.429205 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429025    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 15 10:46:21.429205 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429028    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 15 10:46:21.429205 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429030    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 15 10:46:21.429205 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429033    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 15 10:46:21.429205 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429037    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 15 10:46:21.429205 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429040    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 15 10:46:21.429205 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429043    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 15 10:46:21.429205 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429045    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 15 10:46:21.429205 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429048    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 15 10:46:21.429205 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429051    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 15 10:46:21.429205 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429054    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 15 10:46:21.429205 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429057    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 15 10:46:21.429205 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429059    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 15 10:46:21.429205 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429062    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 15 10:46:21.429205 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429065    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 15 10:46:21.429205 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429068    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 15 10:46:21.429205 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429070    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 15 10:46:21.429205 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429072    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 15 10:46:21.429753 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429075    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 15 10:46:21.429753 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429077    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 15 10:46:21.429753 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429080    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 15 10:46:21.429753 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429083    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 15 10:46:21.429753 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429085    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 15 10:46:21.429753 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429088    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 15 10:46:21.429753 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429091    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 15 10:46:21.429753 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429094    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 15 10:46:21.429753 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429097    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 15 10:46:21.429753 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429099    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 15 10:46:21.429753 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429102    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 15 10:46:21.429753 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429104    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 15 10:46:21.429753 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429108    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 15 10:46:21.429753 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429111    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 15 10:46:21.429753 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429114    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 15 10:46:21.429753 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429117    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 15 10:46:21.429753 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429119    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 15 10:46:21.429753 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429122    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 15 10:46:21.429753 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429125    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 15 10:46:21.429753 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429127    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 15 10:46:21.430253 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429131    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 15 10:46:21.430253 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429136    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 15 10:46:21.430253 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429154    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 15 10:46:21.430253 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429157    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 15 10:46:21.430253 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429161    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 15 10:46:21.430253 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429164    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 15 10:46:21.430253 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429167    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 15 10:46:21.430253 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429170    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 15 10:46:21.430253 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429172    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 15 10:46:21.430253 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429175    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 15 10:46:21.430253 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429178    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 15 10:46:21.430253 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429182    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 15 10:46:21.430253 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429186    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 15 10:46:21.430253 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429188    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 15 10:46:21.430253 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429191    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 15 10:46:21.430253 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429195    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 15 10:46:21.430253 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429197    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 15 10:46:21.430253 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429200    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 15 10:46:21.430253 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429203    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 15 10:46:21.430751 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429206    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 15 10:46:21.430751 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429209    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 15 10:46:21.430751 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429212    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 15 10:46:21.430751 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429215    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 15 10:46:21.430751 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429218    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 15 10:46:21.430751 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429220    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 15 10:46:21.430751 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429224    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 15 10:46:21.430751 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429227    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 15 10:46:21.430751 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429230    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 15 10:46:21.430751 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429233    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 15 10:46:21.430751 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429237    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 15 10:46:21.430751 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429240    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 15 10:46:21.430751 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429243    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 15 10:46:21.430751 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429245    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 15 10:46:21.430751 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429248    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 15 10:46:21.430751 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429250    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 15 10:46:21.430751 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429253    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 15 10:46:21.430751 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429255    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 15 10:46:21.430751 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429258    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 15 10:46:21.431217 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429261    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 15 10:46:21.431217 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429265    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 15 10:46:21.431217 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429268    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 15 10:46:21.431217 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429271    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 15 10:46:21.431217 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429274    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 15 10:46:21.431217 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429277    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 15 10:46:21.431217 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429279    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 15 10:46:21.431217 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429282    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 15 10:46:21.431217 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429284    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 15 10:46:21.431217 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429287    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 15 10:46:21.431217 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429290    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 15 10:46:21.431217 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429293    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 15 10:46:21.431217 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429295    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 15 10:46:21.431217 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429298    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 15 10:46:21.431217 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429301    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 15 10:46:21.431217 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.429304    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 15 10:46:21.431217 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430599    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 15 10:46:21.431217 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430607    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 15 10:46:21.431217 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430611    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 15 10:46:21.431217 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430614    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 15 10:46:21.431731 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430618    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 15 10:46:21.431731 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430621    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 15 10:46:21.431731 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430624    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 15 10:46:21.431731 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430626    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 15 10:46:21.431731 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430629    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 15 10:46:21.431731 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430631    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 15 10:46:21.431731 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430634    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 15 10:46:21.431731 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430637    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 15 10:46:21.431731 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430639    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 15 10:46:21.431731 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430642    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 15 10:46:21.431731 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430644    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 15 10:46:21.431731 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430647    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 15 10:46:21.431731 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430650    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 15 10:46:21.431731 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430652    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 15 10:46:21.431731 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430655    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 15 10:46:21.431731 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430657    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 15 10:46:21.431731 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430660    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 15 10:46:21.431731 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430663    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 15 10:46:21.431731 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430666    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 15 10:46:21.431731 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430668    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 15 10:46:21.432274 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430671    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 15 10:46:21.432274 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430673    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 15 10:46:21.432274 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430676    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 15 10:46:21.432274 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430679    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 15 10:46:21.432274 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430682    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 15 10:46:21.432274 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430685    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 15 10:46:21.432274 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430687    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 15 10:46:21.432274 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430691    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 15 10:46:21.432274 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430693    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 15 10:46:21.432274 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430696    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 15 10:46:21.432274 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430698    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 15 10:46:21.432274 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430702    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 15 10:46:21.432274 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430706    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 15 10:46:21.432274 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430709    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 15 10:46:21.432274 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430712    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 15 10:46:21.432274 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430715    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 15 10:46:21.432274 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430718    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 15 10:46:21.432274 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430721    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 15 10:46:21.432274 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430724    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 15 10:46:21.432736 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430726    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 15 10:46:21.432736 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430729    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 15 10:46:21.432736 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430732    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 15 10:46:21.432736 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430735    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 15 10:46:21.432736 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430737    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 15 10:46:21.432736 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430740    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 15 10:46:21.432736 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430743    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 15 10:46:21.432736 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430745    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 15 10:46:21.432736 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430748    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 15 10:46:21.432736 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430750    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 15 10:46:21.432736 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430753    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 15 10:46:21.432736 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430756    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 15 10:46:21.432736 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430759    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 15 10:46:21.432736 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430762    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 15 10:46:21.432736 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430764    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 15 10:46:21.432736 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430767    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 15 10:46:21.432736 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430770    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 15 10:46:21.432736 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430773    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 15 10:46:21.432736 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430775    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 15 10:46:21.432736 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430778    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 15 10:46:21.433248 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430781    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 15 10:46:21.433248 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430784    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 15 10:46:21.433248 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430788    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 15 10:46:21.433248 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430791    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 15 10:46:21.433248 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430793    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 15 10:46:21.433248 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430796    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 15 10:46:21.433248 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430798    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 15 10:46:21.433248 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430801    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 15 10:46:21.433248 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430804    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 15 10:46:21.433248 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430808    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 15 10:46:21.433248 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430811    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 15 10:46:21.433248 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430814    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 15 10:46:21.433248 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430818    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 15 10:46:21.433248 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430820    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 15 10:46:21.433248 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430823    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 15 10:46:21.433248 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430826    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 15 10:46:21.433248 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430829    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 15 10:46:21.433248 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430832    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 15 10:46:21.433248 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430835    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 15 10:46:21.433248 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430838    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 15 10:46:21.433720 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430841    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 15 10:46:21.433720 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430843    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 15 10:46:21.433720 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430846    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 15 10:46:21.433720 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430850    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 15 10:46:21.433720 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430853    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 15 10:46:21.433720 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430855    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 15 10:46:21.433720 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430858    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 15 10:46:21.433720 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430860    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 15 10:46:21.433720 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430863    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 15 10:46:21.433720 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430866    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 15 10:46:21.433720 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430868    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 15 10:46:21.433720 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430871    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 15 10:46:21.433720 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430875    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 15 10:46:21.433720 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430878    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 15 10:46:21.433720 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430881    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 15 10:46:21.433720 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.430884    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 15 10:46:21.433720 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431294    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 15 10:46:21.433720 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431301    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 15 10:46:21.433720 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431305    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 15 10:46:21.434202 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431308    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 15 10:46:21.434202 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431311    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 15 10:46:21.434202 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431314    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 15 10:46:21.434202 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431317    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 15 10:46:21.434202 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431319    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 15 10:46:21.434202 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431322    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 15 10:46:21.434202 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431325    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 15 10:46:21.434202 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431328    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 15 10:46:21.434202 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431331    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 15 10:46:21.434202 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431333    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 15 10:46:21.434202 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431336    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 15 10:46:21.434202 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431338    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 15 10:46:21.434202 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431341    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 15 10:46:21.434202 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431344    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 15 10:46:21.434202 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431346    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 15 10:46:21.434202 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431349    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 15 10:46:21.434202 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431352    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 15 10:46:21.434202 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431354    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 15 10:46:21.434202 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431357    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 15 10:46:21.434202 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431360    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 15 10:46:21.434699 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431362    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 15 10:46:21.434699 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431365    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 15 10:46:21.434699 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431367    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 15 10:46:21.434699 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431370    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 15 10:46:21.434699 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431372    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 15 10:46:21.434699 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431375    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 15 10:46:21.434699 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431377    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 15 10:46:21.434699 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431380    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 15 10:46:21.434699 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431384    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 15 10:46:21.434699 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431389    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 15 10:46:21.434699 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431392    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 15 10:46:21.434699 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431394    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 15 10:46:21.434699 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431398    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 15 10:46:21.434699 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431401    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 15 10:46:21.434699 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431404    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 15 10:46:21.434699 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431407    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 15 10:46:21.434699 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431410    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 15 10:46:21.434699 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431413    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 15 10:46:21.434699 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431416    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 15 10:46:21.435171 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431419    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 15 10:46:21.435171 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431421    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 15 10:46:21.435171 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431423    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 15 10:46:21.435171 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431426    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 15 10:46:21.435171 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431428    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 15 10:46:21.435171 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431431    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 15 10:46:21.435171 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431434    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 15 10:46:21.435171 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431436    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 15 10:46:21.435171 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431439    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 15 10:46:21.435171 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431441    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 15 10:46:21.435171 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431445    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 15 10:46:21.435171 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431448    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 15 10:46:21.435171 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431450    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 15 10:46:21.435171 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431452    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 15 10:46:21.435171 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431455    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 15 10:46:21.435171 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431457    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 15 10:46:21.435171 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431460    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 15 10:46:21.435171 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431462    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 15 10:46:21.435171 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431465    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 15 10:46:21.435171 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431467    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 15 10:46:21.435646 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431470    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 15 10:46:21.435646 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431473    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 15 10:46:21.435646 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431475    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 15 10:46:21.435646 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431478    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 15 10:46:21.435646 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431481    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 15 10:46:21.435646 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431484    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 15 10:46:21.435646 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431486    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 15 10:46:21.435646 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431489    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 15 10:46:21.435646 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431491    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 15 10:46:21.435646 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431495    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 15 10:46:21.435646 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431498    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 15 10:46:21.435646 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431501    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 15 10:46:21.435646 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431503    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 15 10:46:21.435646 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431506    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 15 10:46:21.435646 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431508    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 15 10:46:21.435646 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431511    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 15 10:46:21.435646 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431513    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 15 10:46:21.435646 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431516    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 15 10:46:21.435646 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431518    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 15 10:46:21.436103 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431521    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 15 10:46:21.436103 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431523    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 15 10:46:21.436103 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431526    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 15 10:46:21.436103 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431528    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 15 10:46:21.436103 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431531    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 15 10:46:21.436103 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431533    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 15 10:46:21.436103 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431536    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 15 10:46:21.436103 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431539    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 15 10:46:21.436103 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431541    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 15 10:46:21.436103 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431544    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 15 10:46:21.436103 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431547    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 15 10:46:21.436103 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431549    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 15 10:46:21.436103 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431552    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 15 10:46:21.436103 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431554    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 15 10:46:21.436103 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431557    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 15 10:46:21.436103 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431560    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 15 10:46:21.436103 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431562    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 15 10:46:21.436103 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.431565    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 15 10:46:21.436103 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431638    2563 flags.go:64] FLAG: --address="0.0.0.0"
Jul 15 10:46:21.436103 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431645    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 15 10:46:21.436605 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431654    2563 flags.go:64] FLAG: --anonymous-auth="true"
Jul 15 10:46:21.436605 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431661    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 15 10:46:21.436605 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431667    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 15 10:46:21.436605 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431672    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 15 10:46:21.436605 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431678    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 15 10:46:21.436605 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431685    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 15 10:46:21.436605 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431689    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 15 10:46:21.436605 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431692    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 15 10:46:21.436605 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431696    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 15 10:46:21.436605 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431699    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 15 10:46:21.436605 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431702    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 15 10:46:21.436605 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431705    2563 flags.go:64] FLAG: --cgroup-root=""
Jul 15 10:46:21.436605 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431708    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 15 10:46:21.436605 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431711    2563 flags.go:64] FLAG: --client-ca-file=""
Jul 15 10:46:21.436605 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431714    2563 flags.go:64] FLAG: --cloud-provider="external"
Jul 15 10:46:21.436605 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431717    2563 flags.go:64] FLAG: --cluster-dns="[]"
Jul 15 10:46:21.436605 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431722    2563 flags.go:64] FLAG: --cluster-domain=""
Jul 15 10:46:21.436605 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431725    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 15 10:46:21.436605 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431728    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 15 10:46:21.436605 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431731    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 15 10:46:21.436605 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431734    2563 flags.go:64] FLAG: --container-log-max-files="5"
Jul 15 10:46:21.436605 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431739    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 15 10:46:21.436605 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431742    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 15 10:46:21.437195 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431745    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 15 10:46:21.437195 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431748    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 15 10:46:21.437195 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431751    2563 flags.go:64] FLAG: --contention-profiling="false"
Jul 15 10:46:21.437195 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431754    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 15 10:46:21.437195 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431758    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 15 10:46:21.437195 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431770    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 15 10:46:21.437195 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431773    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 15 10:46:21.437195 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431782    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 15 10:46:21.437195 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431785    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 15 10:46:21.437195 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431788    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 15 10:46:21.437195 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431791    2563 flags.go:64] FLAG: --enable-load-reader="false"
Jul 15 10:46:21.437195 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431794    2563 flags.go:64] FLAG: --enable-server="true"
Jul 15 10:46:21.437195 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431797    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 15 10:46:21.437195 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431801    2563 flags.go:64] FLAG: --event-burst="100"
Jul 15 10:46:21.437195 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431804    2563 flags.go:64] FLAG: --event-qps="50"
Jul 15 10:46:21.437195 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431807    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 15 10:46:21.437195 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431810    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 15 10:46:21.437195 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431813    2563 flags.go:64] FLAG: --eviction-hard=""
Jul 15 10:46:21.437195 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431817    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 15 10:46:21.437195 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431820    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 15 10:46:21.437195 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431823    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 15 10:46:21.437195 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431826    2563 flags.go:64] FLAG: --eviction-soft=""
Jul 15 10:46:21.437195 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431829    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 15 10:46:21.437195 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431832    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 15 10:46:21.437195 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431835    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 15 10:46:21.437838 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431838    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 15 10:46:21.437838 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431841    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 15 10:46:21.437838 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431844    2563 flags.go:64] FLAG: --fail-swap-on="true"
Jul 15 10:46:21.437838 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431847    2563 flags.go:64] FLAG: --feature-gates=""
Jul 15 10:46:21.437838 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431851    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 15 10:46:21.437838 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431854    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 15 10:46:21.437838 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431857    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 15 10:46:21.437838 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431860    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 15 10:46:21.437838 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431863    2563 flags.go:64] FLAG: --healthz-port="10248"
Jul 15 10:46:21.437838 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431866    2563 flags.go:64] FLAG: --help="false"
Jul 15 10:46:21.437838 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431869    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-131-34.ec2.internal"
Jul 15 10:46:21.437838 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431872    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 15 10:46:21.437838 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431878    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 15 10:46:21.437838 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431882    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 15 10:46:21.437838 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431885    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 15 10:46:21.437838 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431889    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 15 10:46:21.437838 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431892    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 15 10:46:21.437838 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431895    2563 flags.go:64] FLAG: --image-service-endpoint=""
Jul 15 10:46:21.437838 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431897    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 15 10:46:21.437838 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431900    2563 flags.go:64] FLAG: --kube-api-burst="100"
Jul 15 10:46:21.437838 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431903    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 15 10:46:21.437838 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431906    2563 flags.go:64] FLAG: --kube-api-qps="50"
Jul 15 10:46:21.437838 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431910    2563 flags.go:64] FLAG: --kube-reserved=""
Jul 15 10:46:21.437838 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431912    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 15 10:46:21.438478 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431915    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 15 10:46:21.438478 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431918    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 15 10:46:21.438478 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431922    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 15 10:46:21.438478 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431925    2563 flags.go:64] FLAG: --lock-file=""
Jul 15 10:46:21.438478 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431928    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 15 10:46:21.438478 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431931    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 15 10:46:21.438478 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431934    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 15 10:46:21.438478 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431939    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 15 10:46:21.438478 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431942    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 15 10:46:21.438478 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431945    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 15 10:46:21.438478 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431948    2563 flags.go:64] FLAG: --logging-format="text"
Jul 15 10:46:21.438478 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431950    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 15 10:46:21.438478 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431954    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 15 10:46:21.438478 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431957    2563 flags.go:64] FLAG: --manifest-url=""
Jul 15 10:46:21.438478 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431960    2563 flags.go:64] FLAG: --manifest-url-header=""
Jul 15 10:46:21.438478 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431964    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 15 10:46:21.438478 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431967    2563 flags.go:64] FLAG: --max-open-files="1000000"
Jul 15 10:46:21.438478 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431972    2563 flags.go:64] FLAG: --max-pods="110"
Jul 15 10:46:21.438478 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431975    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 15 10:46:21.438478 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431978    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 15 10:46:21.438478 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431982    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 15 10:46:21.438478 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431985    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 15 10:46:21.438478 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431988    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 15 10:46:21.438478 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431992    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 15 10:46:21.438478 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.431995    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 15 10:46:21.439112 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432002    2563 flags.go:64] FLAG: --node-status-max-images="50"
Jul 15 10:46:21.439112 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432005    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 15 10:46:21.439112 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432008    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 15 10:46:21.439112 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432011    2563 flags.go:64] FLAG: --pod-cidr=""
Jul 15 10:46:21.439112 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432014    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 15 10:46:21.439112 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432019    2563 flags.go:64] FLAG: --pod-manifest-path=""
Jul 15 10:46:21.439112 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432022    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 15 10:46:21.439112 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432025    2563 flags.go:64] FLAG: --pods-per-core="0"
Jul 15 10:46:21.439112 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432028    2563 flags.go:64] FLAG: --port="10250"
Jul 15 10:46:21.439112 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432031    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 15 10:46:21.439112 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432036    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0147b43441a12f976"
Jul 15 10:46:21.439112 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432039    2563 flags.go:64] FLAG: --qos-reserved=""
Jul 15 10:46:21.439112 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432042    2563 flags.go:64] FLAG: --read-only-port="10255"
Jul 15 10:46:21.439112 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432045    2563 flags.go:64] FLAG: --register-node="true"
Jul 15 10:46:21.439112 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432048    2563 flags.go:64] FLAG: --register-with-taints=""
Jul 15 10:46:21.439112 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432051    2563 flags.go:64] FLAG: --registry-burst="10"
Jul 15 10:46:21.439112 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432054    2563 flags.go:64] FLAG: --registry-qps="5"
Jul 15 10:46:21.439112 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432057    2563 flags.go:64] FLAG: --reserved-cpus=""
Jul 15 10:46:21.439112 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432060    2563 flags.go:64] FLAG: --reserved-memory=""
Jul 15 10:46:21.439112 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432064    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 15 10:46:21.439112 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432067    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 15 10:46:21.439112 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432070    2563 flags.go:64] FLAG: --rotate-certificates="false"
Jul 15 10:46:21.439112 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432072    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 15 10:46:21.439112 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432076    2563 flags.go:64] FLAG: --runonce="false"
Jul 15 10:46:21.439112 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432078    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 15 10:46:21.439755 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432081    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 15 10:46:21.439755 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432085    2563 flags.go:64] FLAG: --seccomp-default="false"
Jul 15 10:46:21.439755 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432089    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 15 10:46:21.439755 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432092    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 15 10:46:21.439755 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432095    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 15 10:46:21.439755 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432098    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 15 10:46:21.439755 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432102    2563 flags.go:64] FLAG: --storage-driver-password="root"
Jul 15 10:46:21.439755 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432105    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 15 10:46:21.439755 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432108    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 15 10:46:21.439755 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432111    2563 flags.go:64] FLAG: --storage-driver-user="root"
Jul 15 10:46:21.439755 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432114    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 15 10:46:21.439755 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432117    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 15 10:46:21.439755 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432120    2563 flags.go:64] FLAG: --system-cgroups=""
Jul 15 10:46:21.439755 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432123    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 15 10:46:21.439755 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432128    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 15 10:46:21.439755 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432131    2563 flags.go:64] FLAG: --tls-cert-file=""
Jul 15 10:46:21.439755 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432134    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 15 10:46:21.439755 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432151    2563 flags.go:64] FLAG: --tls-min-version=""
Jul 15 10:46:21.439755 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432154    2563 flags.go:64] FLAG: --tls-private-key-file=""
Jul 15 10:46:21.439755 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432157    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 15 10:46:21.439755 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432160    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 15 10:46:21.439755 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432163    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 15 10:46:21.439755 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432166    2563 flags.go:64] FLAG: --v="2"
Jul 15 10:46:21.439755 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432170    2563 flags.go:64] FLAG: --version="false"
Jul 15 10:46:21.439755 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432174    2563 flags.go:64] FLAG: --vmodule=""
Jul 15 10:46:21.440405 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432178    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 15 10:46:21.440405 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.432181    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 15 10:46:21.440405 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432300    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 15 10:46:21.440405 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432304    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 15 10:46:21.440405 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432307    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 15 10:46:21.440405 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432310    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 15 10:46:21.440405 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432314    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 15 10:46:21.440405 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432318    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 15 10:46:21.440405 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432321    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 15 10:46:21.440405 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432325    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 15 10:46:21.440405 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432328    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 15 10:46:21.440405 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432331    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 15 10:46:21.440405 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432334    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 15 10:46:21.440405 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432337    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 15 10:46:21.440405 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432340    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 15 10:46:21.440405 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432343    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 15 10:46:21.440405 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432346    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 15 10:46:21.440405 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432349    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 15 10:46:21.440405 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432351    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 15 10:46:21.440405 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432354    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 15 10:46:21.441176 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432357    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 15 10:46:21.441176 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432360    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 15 10:46:21.441176 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432362    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 15 10:46:21.441176 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432364    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 15 10:46:21.441176 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432369    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 15 10:46:21.441176 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432371    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 15 10:46:21.441176 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432374    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 15 10:46:21.441176 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432376    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 15 10:46:21.441176 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432379    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 15 10:46:21.441176 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432381    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 15 10:46:21.441176 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432384    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 15 10:46:21.441176 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432387    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 15 10:46:21.441176 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432393    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 15 10:46:21.441176 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432397    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 15 10:46:21.441176 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432400    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 15 10:46:21.441176 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432403    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 15 10:46:21.441176 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432406    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 15 10:46:21.441176 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432409    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 15 10:46:21.441176 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432412    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 15 10:46:21.441936 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432414    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 15 10:46:21.441936 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432417    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 15 10:46:21.441936 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432422    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 15 10:46:21.441936 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432425    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 15 10:46:21.441936 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432427    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 15 10:46:21.441936 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432430    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 15 10:46:21.441936 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432433    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 15 10:46:21.441936 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432436    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 15 10:46:21.441936 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432439    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 15 10:46:21.441936 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432442    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 15 10:46:21.441936 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432444    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 15 10:46:21.441936 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432447    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 15 10:46:21.441936 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432449    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 15 10:46:21.441936 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432452    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 15 10:46:21.441936 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432455    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 15 10:46:21.441936 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432457    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 15 10:46:21.441936 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432460    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 15 10:46:21.441936 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432464    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 15 10:46:21.441936 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432467    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 15 10:46:21.441936 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432469    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 15 10:46:21.441936 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432472    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 15 10:46:21.442683 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432474    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 15 10:46:21.442683 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432477    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 15 10:46:21.442683 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432480    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 15 10:46:21.442683 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432483    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 15 10:46:21.442683 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432486    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 15 10:46:21.442683 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432488    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 15 10:46:21.442683 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432491    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 15 10:46:21.442683 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432493    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 15 10:46:21.442683 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432496    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 15 10:46:21.442683 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432498    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 15 10:46:21.442683 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432501    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 15 10:46:21.442683 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432503    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 15 10:46:21.442683 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432506    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 15 10:46:21.442683 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432511    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 15 10:46:21.442683 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432513    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 15 10:46:21.442683 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432516    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 15 10:46:21.442683 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432519    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 15 10:46:21.442683 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432521    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 15 10:46:21.442683 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432524    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 15 10:46:21.442683 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432527    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 15 10:46:21.443203 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432529    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 15 10:46:21.443203 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432532    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 15 10:46:21.443203 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432534    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 15 10:46:21.443203 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432537    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 15 10:46:21.443203 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432539    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 15 10:46:21.443203 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432542    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 15 10:46:21.443203 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432544    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 15 10:46:21.443203 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432547    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 15 10:46:21.443203 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432552    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 15 10:46:21.443203 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432555    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 15 10:46:21.443203 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432558    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 15 10:46:21.443203 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432561    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 15 10:46:21.443203 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432564    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 15 10:46:21.443203 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432566    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 15 10:46:21.443203 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432569    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 15 10:46:21.443203 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432571    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 15 10:46:21.443203 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432574    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 15 10:46:21.443203 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432577    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 15 10:46:21.443203 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432579    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 15 10:46:21.443654 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432582    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 15 10:46:21.443654 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.432584    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 15 10:46:21.443654 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.433227    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 15 10:46:21.443654 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.442252    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 15 10:46:21.443654 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.442273    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 15 10:46:21.443654 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.442294    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 15 10:46:21.443654 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.442301    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 15 10:46:21.443654 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442363    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 15 10:46:21.443654 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442368    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 15 10:46:21.443654 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442371    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 15 10:46:21.443654 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442375    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 15 10:46:21.443654 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442378    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 15 10:46:21.443654 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442380    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 15 10:46:21.443654 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442383    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 15 10:46:21.444000 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442386    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 15 10:46:21.444000 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442388    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 15 10:46:21.444000 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442392    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 15 10:46:21.444000 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442394    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 15 10:46:21.444000 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442397    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 15 10:46:21.444000 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442399    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 15 10:46:21.444000 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442402    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 15 10:46:21.444000 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442405    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 15 10:46:21.444000 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442408    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 15 10:46:21.444000 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442411    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 15 10:46:21.444000 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442414    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 15 10:46:21.444000 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442416    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 15 10:46:21.444000 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442419    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 15 10:46:21.444000 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442422    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 15 10:46:21.444000 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442425    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 15 10:46:21.444000 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442428    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 15 10:46:21.444000 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442431    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 15 10:46:21.444000 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442433    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 15 10:46:21.444000 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442436    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 15 10:46:21.444000 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442440    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 15 10:46:21.444494 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442443    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 15 10:46:21.444494 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442453    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 15 10:46:21.444494 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442456    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 15 10:46:21.444494 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442459    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 15 10:46:21.444494 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442462    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 15 10:46:21.444494 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442465    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 15 10:46:21.444494 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442468    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 15 10:46:21.444494 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442470    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 15 10:46:21.444494 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442473    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 15 10:46:21.444494 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442476    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 15 10:46:21.444494 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442479    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 15 10:46:21.444494 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442481    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 15 10:46:21.444494 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442484    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 15 10:46:21.444494 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442486    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 15 10:46:21.444494 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442489    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 15 10:46:21.444494 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442491    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 15 10:46:21.444494 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442494    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 15 10:46:21.444494 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442496    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 15 10:46:21.444494 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442499    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 15 10:46:21.444494 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442502    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 15 10:46:21.444977 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442505    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 15 10:46:21.444977 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442508    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 15 10:46:21.444977 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442510    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 15 10:46:21.444977 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442513    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 15 10:46:21.444977 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442516    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 15 10:46:21.444977 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442518    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 15 10:46:21.444977 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442521    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 15 10:46:21.444977 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442523    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 15 10:46:21.444977 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442526    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 15 10:46:21.444977 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442528    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 15 10:46:21.444977 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442531    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 15 10:46:21.444977 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442533    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 15 10:46:21.444977 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442536    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 15 10:46:21.444977 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442538    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 15 10:46:21.444977 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442541    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 15 10:46:21.444977 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442543    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 15 10:46:21.444977 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442546    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 15 10:46:21.444977 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442549    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 15 10:46:21.444977 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442551    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 15 10:46:21.444977 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442554    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 15 10:46:21.445478 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442556    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 15 10:46:21.445478 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442560    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 15 10:46:21.445478 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442564    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 15 10:46:21.445478 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442568    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 15 10:46:21.445478 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442570    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 15 10:46:21.445478 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442573    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 15 10:46:21.445478 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442576    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 15 10:46:21.445478 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442578    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 15 10:46:21.445478 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442582    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 15 10:46:21.445478 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442586    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 15 10:46:21.445478 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442589    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 15 10:46:21.445478 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442592    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 15 10:46:21.445478 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442602    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 15 10:46:21.445478 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442606    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 15 10:46:21.445478 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442609    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 15 10:46:21.445478 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442612    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 15 10:46:21.445478 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442615    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 15 10:46:21.445478 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442618    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 15 10:46:21.445478 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442621    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 15 10:46:21.445935 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442624    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 15 10:46:21.445935 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442626    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 15 10:46:21.445935 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442629    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 15 10:46:21.445935 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442632    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 15 10:46:21.445935 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442634    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 15 10:46:21.445935 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442637    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 15 10:46:21.445935 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442640    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 15 10:46:21.445935 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442642    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 15 10:46:21.445935 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442645    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 15 10:46:21.445935 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442648    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 15 10:46:21.445935 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442650    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 15 10:46:21.445935 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442653    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 15 10:46:21.445935 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442655    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 15 10:46:21.445935 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.442661    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 15 10:46:21.445935 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442783    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 15 10:46:21.446327 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442788    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 15 10:46:21.446327 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442791    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 15 10:46:21.446327 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442794    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 15 10:46:21.446327 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442797    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 15 10:46:21.446327 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442800    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 15 10:46:21.446327 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442802    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 15 10:46:21.446327 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442805    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 15 10:46:21.446327 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442808    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 15 10:46:21.446327 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442811    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 15 10:46:21.446327 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442813    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 15 10:46:21.446327 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442817    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 15 10:46:21.446327 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442827    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 15 10:46:21.446327 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442830    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 15 10:46:21.446327 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442833    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 15 10:46:21.446327 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442836    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 15 10:46:21.446327 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442838    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 15 10:46:21.446327 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442841    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 15 10:46:21.446327 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442843    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 15 10:46:21.446327 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442846    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 15 10:46:21.446803 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442849    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 15 10:46:21.446803 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442853    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 15 10:46:21.446803 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442856    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 15 10:46:21.446803 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442859    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 15 10:46:21.446803 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442862    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 15 10:46:21.446803 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442865    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 15 10:46:21.446803 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442869    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 15 10:46:21.446803 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442872    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 15 10:46:21.446803 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442875    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 15 10:46:21.446803 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442877    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 15 10:46:21.446803 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442880    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 15 10:46:21.446803 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442882    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 15 10:46:21.446803 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442885    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 15 10:46:21.446803 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442887    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 15 10:46:21.446803 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442891    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 15 10:46:21.446803 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442893    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 15 10:46:21.446803 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442896    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 15 10:46:21.446803 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442899    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 15 10:46:21.446803 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442901    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 15 10:46:21.446803 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442904    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 15 10:46:21.447299 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442906    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 15 10:46:21.447299 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442909    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 15 10:46:21.447299 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442911    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 15 10:46:21.447299 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442913    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 15 10:46:21.447299 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442916    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 15 10:46:21.447299 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442924    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 15 10:46:21.447299 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442927    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 15 10:46:21.447299 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442929    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 15 10:46:21.447299 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442932    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 15 10:46:21.447299 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442934    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 15 10:46:21.447299 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442937    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 15 10:46:21.447299 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442939    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 15 10:46:21.447299 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442942    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 15 10:46:21.447299 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442944    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 15 10:46:21.447299 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442947    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 15 10:46:21.447299 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442950    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 15 10:46:21.447299 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442952    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 15 10:46:21.447299 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442955    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 15 10:46:21.447299 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442957    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 15 10:46:21.447299 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442960    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 15 10:46:21.447784 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442963    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 15 10:46:21.447784 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442965    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 15 10:46:21.447784 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442968    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 15 10:46:21.447784 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442970    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 15 10:46:21.447784 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442973    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 15 10:46:21.447784 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442975    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 15 10:46:21.447784 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442978    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 15 10:46:21.447784 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442981    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 15 10:46:21.447784 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442983    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 15 10:46:21.447784 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442986    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 15 10:46:21.447784 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442988    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 15 10:46:21.447784 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442991    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 15 10:46:21.447784 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442993    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 15 10:46:21.447784 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442995    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 15 10:46:21.447784 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.442998    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 15 10:46:21.447784 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.443000    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 15 10:46:21.447784 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.443003    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 15 10:46:21.447784 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.443006    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 15 10:46:21.447784 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.443015    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 15 10:46:21.447784 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.443017    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 15 10:46:21.448289 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.443020    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 15 10:46:21.448289 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.443023    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 15 10:46:21.448289 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.443027    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 15 10:46:21.448289 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.443031    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 15 10:46:21.448289 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.443034    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 15 10:46:21.448289 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.443036    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 15 10:46:21.448289 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.443040    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 15 10:46:21.448289 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.443042    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 15 10:46:21.448289 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.443045    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 15 10:46:21.448289 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.443048    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 15 10:46:21.448289 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.443050    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 15 10:46:21.448289 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.443053    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 15 10:46:21.448289 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.443055    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 15 10:46:21.448289 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.443058    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 15 10:46:21.448289 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.443060    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 15 10:46:21.448289 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.443063    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 15 10:46:21.448289 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.443065    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 15 10:46:21.448289 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.443068    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 15 10:46:21.448289 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:21.443071    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 15 10:46:21.448753 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.443076    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 15 10:46:21.448753 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.443244    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 15 10:46:21.448753 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.445416    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 15 10:46:21.448753 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.446685    2563 server.go:1019] "Starting client certificate rotation"
Jul 15 10:46:21.448753 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.446785    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 15 10:46:21.448753 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.446820    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 15 10:46:21.449076 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.449049    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 15 10:46:21.450496 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.450464    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 15 10:46:21.460058 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.460034    2563 log.go:25] "Validated CRI v1 runtime API"
Jul 15 10:46:21.462468 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.462441    2563 log.go:25] "Validated CRI v1 image API"
Jul 15 10:46:21.464451 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.464435    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 15 10:46:21.468250 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.468230    2563 fs.go:135] Filesystem UUIDs: map[603f75ab-6b72-4405-b718-4ee5b856fdb1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 91519319-4eac-4fd7-b635-cd31eeb2d58a:/dev/nvme0n1p3]
Jul 15 10:46:21.468323 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.468250    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 15 10:46:21.474566 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.474427    2563 manager.go:216] Machine: {Timestamp:2026-07-15 10:46:21.473242418 +0000 UTC m=+0.302757814 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100072 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e40f3528cabe5be5fd7f04b9e38ff SystemUUID:ec2e40f3-528c-abe5-be5f-d7f04b9e38ff BootID:d7c930d7-4956-4450-b9e8-79d20f123860 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4c:20:72:c7:59 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4c:20:72:c7:59 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9e:bd:90:a2:e7:fc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 15 10:46:21.474566 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.474562    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 15 10:46:21.474721 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.474704    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 15 10:46:21.476497 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.476476    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 15 10:46:21.477407 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.477366    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 15 10:46:21.477587 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.477417    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-34.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 15 10:46:21.478110 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.478100    2563 topology_manager.go:138] "Creating topology manager with none policy"
Jul 15 10:46:21.478157 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.478113    2563 container_manager_linux.go:309] "Creating device plugin manager"
Jul 15 10:46:21.478157 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.478121    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 15 10:46:21.478810 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.478797    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 15 10:46:21.479598 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.479587    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 15 10:46:21.482526 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.482509    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 15 10:46:21.482677 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.482667    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 15 10:46:21.482708 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.482697    2563 kubelet.go:486] "Attempting to sync node with API server"
Jul 15 10:46:21.482745 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.482715    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 15 10:46:21.482745 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.482737    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 15 10:46:21.482803 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.482760    2563 kubelet.go:398] "Adding apiserver pod source"
Jul 15 10:46:21.482803 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.482774    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 15 10:46:21.483698 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.483510    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 15 10:46:21.483698 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.483704    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 15 10:46:21.488594 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.488567    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 15 10:46:21.490003 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.489982    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 15 10:46:21.490117 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.490021    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 15 10:46:21.490695 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.490680    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 15 10:46:21.490781 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.490702    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 15 10:46:21.490781 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.490711    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 15 10:46:21.490781 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.490719    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 15 10:46:21.490781 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.490733    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 15 10:46:21.490781 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.490741    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 15 10:46:21.490781 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.490754    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 15 10:46:21.490781 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.490762    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 15 10:46:21.490781 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.490773    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 15 10:46:21.490781 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.490781    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 15 10:46:21.491039 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.490794    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 15 10:46:21.491039 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.490804    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 15 10:46:21.491982 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.491969    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 15 10:46:21.492032 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.491987    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 15 10:46:21.493892 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:21.493864    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 15 10:46:21.493892 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:21.493861    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-34.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 15 10:46:21.495516 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.495504    2563 server.go:1268] "Started kubelet"
Jul 15 10:46:21.495629 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.495600    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 15 10:46:21.495629 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.495584    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 15 10:46:21.495741 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.495668    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 15 10:46:21.496478 ip-10-0-131-34 systemd[1]: Started Kubernetes Kubelet.
Jul 15 10:46:21.496833 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.496803    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 15 10:46:21.498973 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.498948    2563 server.go:310] "Adding debug handlers to kubelet server"
Jul 15 10:46:21.501399 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:21.501366    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 15 10:46:21.502638 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.502619    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dxlf4"
Jul 15 10:46:21.502983 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.502962    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 15 10:46:21.502983 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.502964    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 15 10:46:21.503156 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:21.503063    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-34.ec2.internal\" not found"
Jul 15 10:46:21.503684 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.503662    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 15 10:46:21.503684 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.503686    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 15 10:46:21.503841 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.503814    2563 reconstruct.go:97] "Volume reconstruction finished"
Jul 15 10:46:21.503909 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.503842    2563 reconciler.go:29] "Reconciler: start to sync state"
Jul 15 10:46:21.504215 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.504133    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 15 10:46:21.504355 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.504336    2563 factory.go:153] Registering CRI-O factory
Jul 15 10:46:21.504468 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.504358    2563 factory.go:223] Registration of the crio container factory successfully
Jul 15 10:46:21.504468 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.504418    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 15 10:46:21.504468 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.504428    2563 factory.go:55] Registering systemd factory
Jul 15 10:46:21.504468 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.504435    2563 factory.go:223] Registration of the systemd container factory successfully
Jul 15 10:46:21.504468 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.504457    2563 factory.go:103] Registering Raw factory
Jul 15 10:46:21.504682 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.504474    2563 manager.go:1196] Started watching for new ooms in manager
Jul 15 10:46:21.504682 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.504663    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-34.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 15 10:46:21.504931 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.504909    2563 manager.go:349] Starting recovery of all containers
Jul 15 10:46:21.505718 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:21.504831    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-34.ec2.internal.18c270c22269f4bb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-34.ec2.internal,UID:ip-10-0-131-34.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-34.ec2.internal,},FirstTimestamp:2026-07-15 10:46:21.495473339 +0000 UTC m=+0.324988736,LastTimestamp:2026-07-15 10:46:21.495473339 +0000 UTC m=+0.324988736,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-34.ec2.internal,}"
Jul 15 10:46:21.511773 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.511742    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dxlf4"
Jul 15 10:46:21.511976 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:21.511943    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-34.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 15 10:46:21.512095 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:21.512073    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 15 10:46:21.514442 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.514414    2563 manager.go:354] Recovery completed
Jul 15 10:46:21.520787 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.520773    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 15 10:46:21.522789 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.522769    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-34.ec2.internal" event="NodeHasSufficientMemory"
Jul 15 10:46:21.522891 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.522807    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-34.ec2.internal" event="NodeHasNoDiskPressure"
Jul 15 10:46:21.522891 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.522821    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-34.ec2.internal" event="NodeHasSufficientPID"
Jul 15 10:46:21.523389 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.523374    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 15 10:46:21.523389 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.523385    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 15 10:46:21.523492 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.523386    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 15 10:46:21.523492 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.523418    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 15 10:46:21.524108 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.524095    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-34.ec2.internal" event="NodeHasSufficientMemory"
Jul 15 10:46:21.524168 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.524118    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-34.ec2.internal" event="NodeHasNoDiskPressure"
Jul 15 10:46:21.524168 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.524127    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-34.ec2.internal" event="NodeHasSufficientPID"
Jul 15 10:46:21.524503 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.524491    2563 policy_none.go:49] "None policy: Start"
Jul 15 10:46:21.524532 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.524507    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 15 10:46:21.524532 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.524518    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 15 10:46:21.524842 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:21.524777    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-34.ec2.internal.18c270c2240ac3f4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-34.ec2.internal,UID:ip-10-0-131-34.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-34.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-34.ec2.internal,},FirstTimestamp:2026-07-15 10:46:21.522789364 +0000 UTC m=+0.352304767,LastTimestamp:2026-07-15 10:46:21.522789364 +0000 UTC m=+0.352304767,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-34.ec2.internal,}"
Jul 15 10:46:21.525578 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.525562    2563 policy_none.go:47] "Start"
Jul 15 10:46:21.561931 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.561907    2563 manager.go:337] "Starting Device Plugin manager"
Jul 15 10:46:21.562070 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:21.562013    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 15 10:46:21.562070 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.562027    2563 server.go:87] "Starting device plugin registration server"
Jul 15 10:46:21.562352 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.562340    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 15 10:46:21.562463 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.562353    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 15 10:46:21.562463 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.562445    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 15 10:46:21.562575 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.562561    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 15 10:46:21.562620 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.562575    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 15 10:46:21.563166 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:21.563125    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 15 10:46:21.563254 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:21.563191    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-34.ec2.internal\" not found"
Jul 15 10:46:21.663395 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.663352    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 15 10:46:21.665463 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.665443    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-34.ec2.internal" event="NodeHasSufficientMemory"
Jul 15 10:46:21.665553 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.665480    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-34.ec2.internal" event="NodeHasNoDiskPressure"
Jul 15 10:46:21.665553 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.665491    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-34.ec2.internal" event="NodeHasSufficientPID"
Jul 15 10:46:21.665553 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.665519    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-34.ec2.internal"
Jul 15 10:46:21.670802 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.670757    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 15 10:46:21.672080 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.672064    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 15 10:46:21.672174 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.672103    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 15 10:46:21.672174 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.672127    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 15 10:46:21.672277 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:21.672174    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 15 10:46:21.673111 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.673095    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-34.ec2.internal"
Jul 15 10:46:21.673257 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:21.673122    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-34.ec2.internal\": node \"ip-10-0-131-34.ec2.internal\" not found"
Jul 15 10:46:21.674997 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.674973    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 15 10:46:21.692270 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:21.692241    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-34.ec2.internal\" not found"
Jul 15 10:46:21.774056 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.774012    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-34.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-34.ec2.internal"]
Jul 15 10:46:21.774226 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.774103    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 15 10:46:21.775662 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.775645    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-34.ec2.internal" event="NodeHasSufficientMemory"
Jul 15 10:46:21.775727 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.775679    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-34.ec2.internal" event="NodeHasNoDiskPressure"
Jul 15 10:46:21.775727 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.775689    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-34.ec2.internal" event="NodeHasSufficientPID"
Jul 15 10:46:21.777065 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.777051    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 15 10:46:21.777199 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.777181    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-34.ec2.internal"
Jul 15 10:46:21.777254 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.777222    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 15 10:46:21.778090 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.778064    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-34.ec2.internal" event="NodeHasSufficientMemory"
Jul 15 10:46:21.778090 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.778079    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-34.ec2.internal" event="NodeHasSufficientMemory"
Jul 15 10:46:21.778090 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.778097    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-34.ec2.internal" event="NodeHasNoDiskPressure"
Jul 15 10:46:21.778361 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.778106    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-34.ec2.internal" event="NodeHasNoDiskPressure"
Jul 15 10:46:21.778361 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.778108    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-34.ec2.internal" event="NodeHasSufficientPID"
Jul 15 10:46:21.778361 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.778122    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-34.ec2.internal" event="NodeHasSufficientPID"
Jul 15 10:46:21.779481 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.779463    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-34.ec2.internal"
Jul 15 10:46:21.779583 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.779499    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 15 10:46:21.780521 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.780505    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-34.ec2.internal" event="NodeHasSufficientMemory"
Jul 15 10:46:21.780603 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.780536    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-34.ec2.internal" event="NodeHasNoDiskPressure"
Jul 15 10:46:21.780603 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.780550    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-34.ec2.internal" event="NodeHasSufficientPID"
Jul 15 10:46:21.792847 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:21.792819    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-34.ec2.internal\" not found"
Jul 15 10:46:21.813488 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:21.813459    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-34.ec2.internal\" not found" node="ip-10-0-131-34.ec2.internal"
Jul 15 10:46:21.818616 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:21.818593    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-34.ec2.internal\" not found" node="ip-10-0-131-34.ec2.internal"
Jul 15 10:46:21.892966 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:21.892921    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-34.ec2.internal\" not found"
Jul 15 10:46:21.905692 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.905658    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3d172d526ed473c2594efb0d6fe48f48-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-34.ec2.internal\" (UID: \"3d172d526ed473c2594efb0d6fe48f48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-34.ec2.internal"
Jul 15 10:46:21.905809 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.905700    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d172d526ed473c2594efb0d6fe48f48-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-34.ec2.internal\" (UID: \"3d172d526ed473c2594efb0d6fe48f48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-34.ec2.internal"
Jul 15 10:46:21.905809 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:21.905727    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a18a70f1e59122311577155ea3452e36-config\") pod \"kube-apiserver-proxy-ip-10-0-131-34.ec2.internal\" (UID: \"a18a70f1e59122311577155ea3452e36\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-34.ec2.internal"
Jul 15 10:46:21.993723 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:21.993618    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-34.ec2.internal\" not found"
Jul 15 10:46:22.006613 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:22.006575    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a18a70f1e59122311577155ea3452e36-config\") pod \"kube-apiserver-proxy-ip-10-0-131-34.ec2.internal\" (UID: \"a18a70f1e59122311577155ea3452e36\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-34.ec2.internal"
Jul 15 10:46:22.006613 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:22.006617    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3d172d526ed473c2594efb0d6fe48f48-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-34.ec2.internal\" (UID: \"3d172d526ed473c2594efb0d6fe48f48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-34.ec2.internal"
Jul 15 10:46:22.006826 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:22.006635    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d172d526ed473c2594efb0d6fe48f48-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-34.ec2.internal\" (UID: \"3d172d526ed473c2594efb0d6fe48f48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-34.ec2.internal"
Jul 15 10:46:22.006826 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:22.006694    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a18a70f1e59122311577155ea3452e36-config\") pod \"kube-apiserver-proxy-ip-10-0-131-34.ec2.internal\" (UID: \"a18a70f1e59122311577155ea3452e36\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-34.ec2.internal"
Jul 15 10:46:22.006826 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:22.006746    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d172d526ed473c2594efb0d6fe48f48-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-34.ec2.internal\" (UID: \"3d172d526ed473c2594efb0d6fe48f48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-34.ec2.internal"
Jul 15 10:46:22.006826 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:22.006773    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3d172d526ed473c2594efb0d6fe48f48-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-34.ec2.internal\" (UID: \"3d172d526ed473c2594efb0d6fe48f48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-34.ec2.internal"
Jul 15 10:46:22.100981 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:22.100926    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-34.ec2.internal\" not found"
Jul 15 10:46:22.116556 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:22.116526    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-34.ec2.internal"
Jul 15 10:46:22.122088 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:22.122065    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-34.ec2.internal"
Jul 15 10:46:22.202664 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:22.202613    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-34.ec2.internal\" not found"
Jul 15 10:46:22.310313 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:22.310218    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-34.ec2.internal\" not found"
Jul 15 10:46:22.411371 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:22.411338    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-34.ec2.internal\" not found"
Jul 15 10:46:22.448345 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:22.448307    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 15 10:46:22.449033 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:22.448523    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 15 10:46:22.495032 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:22.494994    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 15 10:46:22.503407 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:22.503372    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 15 10:46:22.503531 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:22.503387    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-34.ec2.internal"
Jul 15 10:46:22.517356 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:22.517321    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 15 10:46:22.518724 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:22.518690    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-14 10:41:21 +0000 UTC" deadline="2028-02-18 23:17:31.911855182 +0000 UTC"
Jul 15 10:46:22.518803 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:22.518725    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14004h31m9.393133939s"
Jul 15 10:46:22.519392 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:22.519373    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 15 10:46:22.520993 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:22.520974    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-34.ec2.internal"
Jul 15 10:46:22.533594 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:22.533570    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 15 10:46:22.540018 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:22.539988    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gl9gw"
Jul 15 10:46:22.552380 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:22.552354    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gl9gw"
Jul 15 10:46:22.787100 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:22.787057    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d172d526ed473c2594efb0d6fe48f48.slice/crio-e81bf91b78f915aa42b2e65e45cbcc51d57223c1070dc23a3e949dc3add0c159 WatchSource:0}: Error finding container e81bf91b78f915aa42b2e65e45cbcc51d57223c1070dc23a3e949dc3add0c159: Status 404 returned error can't find the container with id e81bf91b78f915aa42b2e65e45cbcc51d57223c1070dc23a3e949dc3add0c159
Jul 15 10:46:22.788735 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:22.788076    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda18a70f1e59122311577155ea3452e36.slice/crio-1e961cc3426942747aefbba2ba1ded1d350e9f16e8c2dfa1a45e56abf9d395e4 WatchSource:0}: Error finding container 1e961cc3426942747aefbba2ba1ded1d350e9f16e8c2dfa1a45e56abf9d395e4: Status 404 returned error can't find the container with id 1e961cc3426942747aefbba2ba1ded1d350e9f16e8c2dfa1a45e56abf9d395e4
Jul 15 10:46:22.794217 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:22.794195    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 15 10:46:22.826305 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:22.826081    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 15 10:46:23.062663 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.062573    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 15 10:46:23.483640 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.483529    2563 apiserver.go:52] "Watching apiserver"
Jul 15 10:46:23.496322 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.496287    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 15 10:46:23.496982 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.496928    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-b4gsj","openshift-network-diagnostics/network-check-target-9mhqc","openshift-network-operator/iptables-alerter-4jcc7","openshift-ovn-kubernetes/ovnkube-node-rk49n","kube-system/konnectivity-agent-wdxpd","kube-system/kube-apiserver-proxy-ip-10-0-131-34.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxh9r","openshift-cluster-node-tuning-operator/tuned-4445h","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-34.ec2.internal","openshift-multus/multus-additional-cni-plugins-l8v8c","openshift-multus/multus-fcnjt","openshift-multus/network-metrics-daemon-84n76","kube-system/global-pull-secret-syncer-rx95w","openshift-dns/node-resolver-klpd8"]
Jul 15 10:46:23.498984 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.498960    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-b4gsj"
Jul 15 10:46:23.504259 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.503480    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 15 10:46:23.504259 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.503770    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 15 10:46:23.504259 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.503977    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 15 10:46:23.504259 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.504202    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-h42xm\""
Jul 15 10:46:23.506599 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.506242    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:46:23.506599 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:23.506322    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mhqc" podUID="b5f21679-bed4-4e21-9a3b-22ec261aec90"
Jul 15 10:46:23.508265 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.508238    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4jcc7"
Jul 15 10:46:23.508484 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.508468    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.510057 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.510033    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wdxpd"
Jul 15 10:46:23.511407 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.511386    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxh9r"
Jul 15 10:46:23.513094 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.513071    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rsxk9\""
Jul 15 10:46:23.513461 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.513442    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 15 10:46:23.514317 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.513656    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 15 10:46:23.514317 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.513888    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 15 10:46:23.514317 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.514073    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 15 10:46:23.515001 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.514971    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 15 10:46:23.515092 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.515062    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.518307 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.515151    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 15 10:46:23.518307 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.515428    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r5p2x\""
Jul 15 10:46:23.518307 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.515648    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 15 10:46:23.518307 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.515683    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 15 10:46:23.518307 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.515810    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 15 10:46:23.518307 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.515936    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 15 10:46:23.518307 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.516048    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-l8v8c"
Jul 15 10:46:23.518307 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.515683    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 15 10:46:23.518307 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.516210    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rqzxb\""
Jul 15 10:46:23.518307 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.516461    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 15 10:46:23.518307 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.516053    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 15 10:46:23.518307 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.516731    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 15 10:46:23.518307 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.516746    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.518307 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.516958    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2v2vl\""
Jul 15 10:46:23.518307 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.517218    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a7699c1-9758-4c35-8b4c-aed46512560e-host\") pod \"node-ca-b4gsj\" (UID: \"2a7699c1-9758-4c35-8b4c-aed46512560e\") " pod="openshift-image-registry/node-ca-b4gsj"
Jul 15 10:46:23.518307 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.517251    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2a7699c1-9758-4c35-8b4c-aed46512560e-serviceca\") pod \"node-ca-b4gsj\" (UID: \"2a7699c1-9758-4c35-8b4c-aed46512560e\") " pod="openshift-image-registry/node-ca-b4gsj"
Jul 15 10:46:23.518307 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.517281    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz6dv\" (UniqueName: \"kubernetes.io/projected/2a7699c1-9758-4c35-8b4c-aed46512560e-kube-api-access-cz6dv\") pod \"node-ca-b4gsj\" (UID: \"2a7699c1-9758-4c35-8b4c-aed46512560e\") " pod="openshift-image-registry/node-ca-b4gsj"
Jul 15 10:46:23.518307 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.517308    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rwgk\" (UniqueName: \"kubernetes.io/projected/b5f21679-bed4-4e21-9a3b-22ec261aec90-kube-api-access-4rwgk\") pod \"network-check-target-9mhqc\" (UID: \"b5f21679-bed4-4e21-9a3b-22ec261aec90\") " pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:46:23.518307 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.517568    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:46:23.518307 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:23.517636    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-84n76" podUID="c8d1da87-af98-4f6b-af6a-27a5fbabba60"
Jul 15 10:46:23.519405 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.518925    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 15 10:46:23.519405 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.519379    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:46:23.519508 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:23.519447    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx95w" podUID="37ad4cef-91aa-459c-80f3-f67d36e697aa"
Jul 15 10:46:23.520066 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.520044    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 15 10:46:23.520145 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.520052    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 15 10:46:23.520634 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.520615    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-klpd8"
Jul 15 10:46:23.520858 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.520838    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 15 10:46:23.521073 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.521053    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 15 10:46:23.521394 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.521371    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 15 10:46:23.521994 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.521687    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 15 10:46:23.522664 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.522431    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5m7ms\""
Jul 15 10:46:23.522664 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.522484    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pp6jw\""
Jul 15 10:46:23.522664 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.522543    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 15 10:46:23.522664 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.522490    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-82xcj\""
Jul 15 10:46:23.525620 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.524643    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 15 10:46:23.525620 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.524930    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-x2qjk\""
Jul 15 10:46:23.525620 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.525125    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 15 10:46:23.554658 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.554597    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 10:41:22 +0000 UTC" deadline="2027-12-16 14:07:37.57204276 +0000 UTC"
Jul 15 10:46:23.554658 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.554656    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12459h21m14.017390651s"
Jul 15 10:46:23.605876 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.605837    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 15 10:46:23.617876 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.617767    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ad4cef-91aa-459c-80f3-f67d36e697aa-original-pull-secret\") pod \"global-pull-secret-syncer-rx95w\" (UID: \"37ad4cef-91aa-459c-80f3-f67d36e697aa\") " pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:46:23.617876 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.617818    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c40e765d-670d-40c3-925d-9585c28cbe72-registration-dir\") pod \"aws-ebs-csi-driver-node-lxh9r\" (UID: \"c40e765d-670d-40c3-925d-9585c28cbe72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxh9r"
Jul 15 10:46:23.617876 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.617847    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2456e018-71e0-42bc-8153-64c597b91f17-etc-kubernetes\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.617876 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.617876    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f949cb2-6105-448d-b4df-82ab30888fef-host-slash\") pod \"iptables-alerter-4jcc7\" (UID: \"2f949cb2-6105-448d-b4df-82ab30888fef\") " pod="openshift-network-operator/iptables-alerter-4jcc7"
Jul 15 10:46:23.618269 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.617902    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c40e765d-670d-40c3-925d-9585c28cbe72-device-dir\") pod \"aws-ebs-csi-driver-node-lxh9r\" (UID: \"c40e765d-670d-40c3-925d-9585c28cbe72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxh9r"
Jul 15 10:46:23.618269 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.617929    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2456e018-71e0-42bc-8153-64c597b91f17-tmp\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.618269 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.617953    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-run-systemd\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.618269 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.617973    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-node-log\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.618269 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.617996    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksq62\" (UniqueName: \"kubernetes.io/projected/2456e018-71e0-42bc-8153-64c597b91f17-kube-api-access-ksq62\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.618269 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.618020    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/902fd91e-2979-4d2c-b428-ff5e62d8203b-os-release\") pod \"multus-additional-cni-plugins-l8v8c\" (UID: \"902fd91e-2979-4d2c-b428-ff5e62d8203b\") " pod="openshift-multus/multus-additional-cni-plugins-l8v8c"
Jul 15 10:46:23.618269 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.618071    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/902fd91e-2979-4d2c-b428-ff5e62d8203b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l8v8c\" (UID: \"902fd91e-2979-4d2c-b428-ff5e62d8203b\") " pod="openshift-multus/multus-additional-cni-plugins-l8v8c"
Jul 15 10:46:23.618269 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.618147    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-host-run-netns\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.618269 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.618206    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/37ad4cef-91aa-459c-80f3-f67d36e697aa-dbus\") pod \"global-pull-secret-syncer-rx95w\" (UID: \"37ad4cef-91aa-459c-80f3-f67d36e697aa\") " pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:46:23.618269 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.618242    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhfrm\" (UniqueName: \"kubernetes.io/projected/c40e765d-670d-40c3-925d-9585c28cbe72-kube-api-access-dhfrm\") pod \"aws-ebs-csi-driver-node-lxh9r\" (UID: \"c40e765d-670d-40c3-925d-9585c28cbe72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxh9r"
Jul 15 10:46:23.618269 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.618268    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2456e018-71e0-42bc-8153-64c597b91f17-etc-tuned\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.618810 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.618307    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/902fd91e-2979-4d2c-b428-ff5e62d8203b-cnibin\") pod \"multus-additional-cni-plugins-l8v8c\" (UID: \"902fd91e-2979-4d2c-b428-ff5e62d8203b\") " pod="openshift-multus/multus-additional-cni-plugins-l8v8c"
Jul 15 10:46:23.618810 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.618352    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2a7699c1-9758-4c35-8b4c-aed46512560e-serviceca\") pod \"node-ca-b4gsj\" (UID: \"2a7699c1-9758-4c35-8b4c-aed46512560e\") " pod="openshift-image-registry/node-ca-b4gsj"
Jul 15 10:46:23.618810 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.618383    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-host-run-k8s-cni-cncf-io\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.618810 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.618436    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/beb68899-ffb3-436f-8e17-43597aad4030-env-overrides\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.618810 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.618472    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7e4a1af3-5ae7-495e-bd96-25df977fef6f-hosts-file\") pod \"node-resolver-klpd8\" (UID: \"7e4a1af3-5ae7-495e-bd96-25df977fef6f\") " pod="openshift-dns/node-resolver-klpd8"
Jul 15 10:46:23.618810 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.618516    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eaebac66-19db-44df-85f4-887969b5984e-agent-certs\") pod \"konnectivity-agent-wdxpd\" (UID: \"eaebac66-19db-44df-85f4-887969b5984e\") " pod="kube-system/konnectivity-agent-wdxpd"
Jul 15 10:46:23.618810 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.618564    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c40e765d-670d-40c3-925d-9585c28cbe72-sys-fs\") pod \"aws-ebs-csi-driver-node-lxh9r\" (UID: \"c40e765d-670d-40c3-925d-9585c28cbe72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxh9r"
Jul 15 10:46:23.618810 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.618591    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2456e018-71e0-42bc-8153-64c597b91f17-etc-sysconfig\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.618810 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.618620    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cae13509-00de-48df-ae3d-943755c4b2e9-cni-binary-copy\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.618810 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.618650    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-system-cni-dir\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.618810 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.618676    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-multus-cni-dir\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.618810 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.618703    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-host-slash\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.618810 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.618765    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-host-run-ovn-kubernetes\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.618810 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.618802    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/beb68899-ffb3-436f-8e17-43597aad4030-ovnkube-config\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.619459 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.618836    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lz28\" (UniqueName: \"kubernetes.io/projected/c8d1da87-af98-4f6b-af6a-27a5fbabba60-kube-api-access-4lz28\") pod \"network-metrics-daemon-84n76\" (UID: \"c8d1da87-af98-4f6b-af6a-27a5fbabba60\") " pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:46:23.619459 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.618864    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2456e018-71e0-42bc-8153-64c597b91f17-etc-modprobe-d\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.619459 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.618888    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-etc-kubernetes\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.619459 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.618916    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-run-openvswitch\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.619459 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.618942    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-host-cni-netd\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.619459 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.618970    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2456e018-71e0-42bc-8153-64c597b91f17-etc-sysctl-d\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.619459 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.619005    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2456e018-71e0-42bc-8153-64c597b91f17-etc-systemd\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.619459 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.619010    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2a7699c1-9758-4c35-8b4c-aed46512560e-serviceca\") pod \"node-ca-b4gsj\" (UID: \"2a7699c1-9758-4c35-8b4c-aed46512560e\") " pod="openshift-image-registry/node-ca-b4gsj"
Jul 15 10:46:23.619459 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.619030    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2456e018-71e0-42bc-8153-64c597b91f17-sys\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.619459 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.619109    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/902fd91e-2979-4d2c-b428-ff5e62d8203b-system-cni-dir\") pod \"multus-additional-cni-plugins-l8v8c\" (UID: \"902fd91e-2979-4d2c-b428-ff5e62d8203b\") " pod="openshift-multus/multus-additional-cni-plugins-l8v8c"
Jul 15 10:46:23.619459 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.619149    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fzfx\" (UniqueName: \"kubernetes.io/projected/902fd91e-2979-4d2c-b428-ff5e62d8203b-kube-api-access-7fzfx\") pod \"multus-additional-cni-plugins-l8v8c\" (UID: \"902fd91e-2979-4d2c-b428-ff5e62d8203b\") " pod="openshift-multus/multus-additional-cni-plugins-l8v8c"
Jul 15 10:46:23.619459 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.619214    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2f949cb2-6105-448d-b4df-82ab30888fef-iptables-alerter-script\") pod \"iptables-alerter-4jcc7\" (UID: \"2f949cb2-6105-448d-b4df-82ab30888fef\") " pod="openshift-network-operator/iptables-alerter-4jcc7"
Jul 15 10:46:23.619459 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.619241    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eaebac66-19db-44df-85f4-887969b5984e-konnectivity-ca\") pod \"konnectivity-agent-wdxpd\" (UID: \"eaebac66-19db-44df-85f4-887969b5984e\") " pod="kube-system/konnectivity-agent-wdxpd"
Jul 15 10:46:23.619459 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.619276    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-os-release\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.619459 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.619305    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cae13509-00de-48df-ae3d-943755c4b2e9-multus-daemon-config\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.619459 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.619334    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-host-kubelet\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.620302 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.619358    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-systemd-units\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.620302 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.619381    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klkk5\" (UniqueName: \"kubernetes.io/projected/7e4a1af3-5ae7-495e-bd96-25df977fef6f-kube-api-access-klkk5\") pod \"node-resolver-klpd8\" (UID: \"7e4a1af3-5ae7-495e-bd96-25df977fef6f\") " pod="openshift-dns/node-resolver-klpd8"
Jul 15 10:46:23.620302 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.619406    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/902fd91e-2979-4d2c-b428-ff5e62d8203b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l8v8c\" (UID: \"902fd91e-2979-4d2c-b428-ff5e62d8203b\") " pod="openshift-multus/multus-additional-cni-plugins-l8v8c"
Jul 15 10:46:23.620302 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.619447    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a7699c1-9758-4c35-8b4c-aed46512560e-host\") pod \"node-ca-b4gsj\" (UID: \"2a7699c1-9758-4c35-8b4c-aed46512560e\") " pod="openshift-image-registry/node-ca-b4gsj"
Jul 15 10:46:23.620302 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.619472    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-multus-conf-dir\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.620302 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.619505    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2456e018-71e0-42bc-8153-64c597b91f17-run\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.620302 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.619531    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/902fd91e-2979-4d2c-b428-ff5e62d8203b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l8v8c\" (UID: \"902fd91e-2979-4d2c-b428-ff5e62d8203b\") " pod="openshift-multus/multus-additional-cni-plugins-l8v8c"
Jul 15 10:46:23.620302 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.619557    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-hostroot\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.620302 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.619583    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq57f\" (UniqueName: \"kubernetes.io/projected/beb68899-ffb3-436f-8e17-43597aad4030-kube-api-access-tq57f\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.620302 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.619625    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-etc-openvswitch\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.620302 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.619630    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a7699c1-9758-4c35-8b4c-aed46512560e-host\") pod \"node-ca-b4gsj\" (UID: \"2a7699c1-9758-4c35-8b4c-aed46512560e\") " pod="openshift-image-registry/node-ca-b4gsj"
Jul 15 10:46:23.620302 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.619686    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-log-socket\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.620302 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.619720    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8d1da87-af98-4f6b-af6a-27a5fbabba60-metrics-certs\") pod \"network-metrics-daemon-84n76\" (UID: \"c8d1da87-af98-4f6b-af6a-27a5fbabba60\") " pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:46:23.620302 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.619746    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2456e018-71e0-42bc-8153-64c597b91f17-host\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.620302 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.619782    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v88cb\" (UniqueName: \"kubernetes.io/projected/cae13509-00de-48df-ae3d-943755c4b2e9-kube-api-access-v88cb\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.620302 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.619814    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-run-ovn\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.620302 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.619840    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/37ad4cef-91aa-459c-80f3-f67d36e697aa-kubelet-config\") pod \"global-pull-secret-syncer-rx95w\" (UID: \"37ad4cef-91aa-459c-80f3-f67d36e697aa\") " pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:46:23.621069 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.619872    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c40e765d-670d-40c3-925d-9585c28cbe72-socket-dir\") pod \"aws-ebs-csi-driver-node-lxh9r\" (UID: \"c40e765d-670d-40c3-925d-9585c28cbe72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxh9r"
Jul 15 10:46:23.621069 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.619897    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2456e018-71e0-42bc-8153-64c597b91f17-lib-modules\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.621069 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.619926    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz6dv\" (UniqueName: \"kubernetes.io/projected/2a7699c1-9758-4c35-8b4c-aed46512560e-kube-api-access-cz6dv\") pod \"node-ca-b4gsj\" (UID: \"2a7699c1-9758-4c35-8b4c-aed46512560e\") " pod="openshift-image-registry/node-ca-b4gsj"
Jul 15 10:46:23.621069 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.619973    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rwgk\" (UniqueName: \"kubernetes.io/projected/b5f21679-bed4-4e21-9a3b-22ec261aec90-kube-api-access-4rwgk\") pod \"network-check-target-9mhqc\" (UID: \"b5f21679-bed4-4e21-9a3b-22ec261aec90\") " pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:46:23.621069 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.620041    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-host-var-lib-kubelet\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.621069 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.620071    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/beb68899-ffb3-436f-8e17-43597aad4030-ovnkube-script-lib\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.621069 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.620097    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e4a1af3-5ae7-495e-bd96-25df977fef6f-tmp-dir\") pod \"node-resolver-klpd8\" (UID: \"7e4a1af3-5ae7-495e-bd96-25df977fef6f\") " pod="openshift-dns/node-resolver-klpd8"
Jul 15 10:46:23.621069 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.620133    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/902fd91e-2979-4d2c-b428-ff5e62d8203b-cni-binary-copy\") pod \"multus-additional-cni-plugins-l8v8c\" (UID: \"902fd91e-2979-4d2c-b428-ff5e62d8203b\") " pod="openshift-multus/multus-additional-cni-plugins-l8v8c"
Jul 15 10:46:23.621069 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.620158    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-host-cni-bin\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.621069 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.620206    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.621069 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.620232    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c40e765d-670d-40c3-925d-9585c28cbe72-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lxh9r\" (UID: \"c40e765d-670d-40c3-925d-9585c28cbe72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxh9r"
Jul 15 10:46:23.621069 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.620257    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c40e765d-670d-40c3-925d-9585c28cbe72-etc-selinux\") pod \"aws-ebs-csi-driver-node-lxh9r\" (UID: \"c40e765d-670d-40c3-925d-9585c28cbe72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxh9r"
Jul 15 10:46:23.621069 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.620281    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2456e018-71e0-42bc-8153-64c597b91f17-etc-sysctl-conf\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.621069 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.620334    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2456e018-71e0-42bc-8153-64c597b91f17-var-lib-kubelet\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.621069 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.620370    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-multus-socket-dir-parent\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.621069 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.620399    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-cnibin\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.621705 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.620425    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-host-run-netns\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.621705 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.620451    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-host-var-lib-cni-bin\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.621705 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.620476    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-host-var-lib-cni-multus\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.621705 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.620521    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxkdb\" (UniqueName: \"kubernetes.io/projected/2f949cb2-6105-448d-b4df-82ab30888fef-kube-api-access-nxkdb\") pod \"iptables-alerter-4jcc7\" (UID: \"2f949cb2-6105-448d-b4df-82ab30888fef\") " pod="openshift-network-operator/iptables-alerter-4jcc7"
Jul 15 10:46:23.621705 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.620564    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-var-lib-openvswitch\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.621705 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.620622    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/beb68899-ffb3-436f-8e17-43597aad4030-ovn-node-metrics-cert\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.621705 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.620674    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-host-run-multus-certs\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.630360 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:23.630309    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 15 10:46:23.630360 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:23.630350    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 15 10:46:23.630360 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:23.630364    2563 projected.go:196] Error preparing data for projected volume kube-api-access-4rwgk for pod openshift-network-diagnostics/network-check-target-9mhqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:46:23.630661 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:23.630451    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5f21679-bed4-4e21-9a3b-22ec261aec90-kube-api-access-4rwgk podName:b5f21679-bed4-4e21-9a3b-22ec261aec90 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:24.130428688 +0000 UTC m=+2.959944076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4rwgk" (UniqueName: "kubernetes.io/projected/b5f21679-bed4-4e21-9a3b-22ec261aec90-kube-api-access-4rwgk") pod "network-check-target-9mhqc" (UID: "b5f21679-bed4-4e21-9a3b-22ec261aec90") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:46:23.631881 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.631849    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 15 10:46:23.639086 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.639020    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz6dv\" (UniqueName: \"kubernetes.io/projected/2a7699c1-9758-4c35-8b4c-aed46512560e-kube-api-access-cz6dv\") pod \"node-ca-b4gsj\" (UID: \"2a7699c1-9758-4c35-8b4c-aed46512560e\") " pod="openshift-image-registry/node-ca-b4gsj"
Jul 15 10:46:23.679232 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.679158    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-34.ec2.internal" event={"ID":"3d172d526ed473c2594efb0d6fe48f48","Type":"ContainerStarted","Data":"e81bf91b78f915aa42b2e65e45cbcc51d57223c1070dc23a3e949dc3add0c159"}
Jul 15 10:46:23.680872 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.680846    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-34.ec2.internal" event={"ID":"a18a70f1e59122311577155ea3452e36","Type":"ContainerStarted","Data":"1e961cc3426942747aefbba2ba1ded1d350e9f16e8c2dfa1a45e56abf9d395e4"}
Jul 15 10:46:23.721738 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.721696    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klkk5\" (UniqueName: \"kubernetes.io/projected/7e4a1af3-5ae7-495e-bd96-25df977fef6f-kube-api-access-klkk5\") pod \"node-resolver-klpd8\" (UID: \"7e4a1af3-5ae7-495e-bd96-25df977fef6f\") " pod="openshift-dns/node-resolver-klpd8"
Jul 15 10:46:23.721738 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.721743    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/902fd91e-2979-4d2c-b428-ff5e62d8203b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l8v8c\" (UID: \"902fd91e-2979-4d2c-b428-ff5e62d8203b\") " pod="openshift-multus/multus-additional-cni-plugins-l8v8c"
Jul 15 10:46:23.721988 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.721771    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-multus-conf-dir\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.721988 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.721795    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2456e018-71e0-42bc-8153-64c597b91f17-run\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.721988 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.721820    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/902fd91e-2979-4d2c-b428-ff5e62d8203b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l8v8c\" (UID: \"902fd91e-2979-4d2c-b428-ff5e62d8203b\") " pod="openshift-multus/multus-additional-cni-plugins-l8v8c"
Jul 15 10:46:23.721988 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.721844    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-hostroot\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.721988 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.721870    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq57f\" (UniqueName: \"kubernetes.io/projected/beb68899-ffb3-436f-8e17-43597aad4030-kube-api-access-tq57f\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.721988 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.721895    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-etc-openvswitch\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.721988 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.721918    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-log-socket\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.721988 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.721944    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8d1da87-af98-4f6b-af6a-27a5fbabba60-metrics-certs\") pod \"network-metrics-daemon-84n76\" (UID: \"c8d1da87-af98-4f6b-af6a-27a5fbabba60\") " pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:46:23.721988 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.721969    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2456e018-71e0-42bc-8153-64c597b91f17-host\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.722549 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.721993    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v88cb\" (UniqueName: \"kubernetes.io/projected/cae13509-00de-48df-ae3d-943755c4b2e9-kube-api-access-v88cb\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.722549 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.721993    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-multus-conf-dir\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.722549 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.722005    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/902fd91e-2979-4d2c-b428-ff5e62d8203b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l8v8c\" (UID: \"902fd91e-2979-4d2c-b428-ff5e62d8203b\") " pod="openshift-multus/multus-additional-cni-plugins-l8v8c"
Jul 15 10:46:23.722549 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.722017    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-run-ovn\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.722549 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.722067    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-log-socket\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.722549 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.722108    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-hostroot\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.722549 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.722098    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2456e018-71e0-42bc-8153-64c597b91f17-host\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.722549 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.722117    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2456e018-71e0-42bc-8153-64c597b91f17-run\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.722549 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:23.722131    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:46:23.722549 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.722151    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/37ad4cef-91aa-459c-80f3-f67d36e697aa-kubelet-config\") pod \"global-pull-secret-syncer-rx95w\" (UID: \"37ad4cef-91aa-459c-80f3-f67d36e697aa\") " pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:46:23.722549 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.722179    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-etc-openvswitch\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.722549 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:23.722209    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8d1da87-af98-4f6b-af6a-27a5fbabba60-metrics-certs podName:c8d1da87-af98-4f6b-af6a-27a5fbabba60 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:24.22218585 +0000 UTC m=+3.051701254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c8d1da87-af98-4f6b-af6a-27a5fbabba60-metrics-certs") pod "network-metrics-daemon-84n76" (UID: "c8d1da87-af98-4f6b-af6a-27a5fbabba60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:46:23.722549 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.722228    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/37ad4cef-91aa-459c-80f3-f67d36e697aa-kubelet-config\") pod \"global-pull-secret-syncer-rx95w\" (UID: \"37ad4cef-91aa-459c-80f3-f67d36e697aa\") " pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:46:23.722549 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.722222    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-run-ovn\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.722549 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.722248    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c40e765d-670d-40c3-925d-9585c28cbe72-socket-dir\") pod \"aws-ebs-csi-driver-node-lxh9r\" (UID: \"c40e765d-670d-40c3-925d-9585c28cbe72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxh9r"
Jul 15 10:46:23.722549 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.722277    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2456e018-71e0-42bc-8153-64c597b91f17-lib-modules\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.722549 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.722317    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-host-var-lib-kubelet\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.723343 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.722345    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/beb68899-ffb3-436f-8e17-43597aad4030-ovnkube-script-lib\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.723343 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.722388    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e4a1af3-5ae7-495e-bd96-25df977fef6f-tmp-dir\") pod \"node-resolver-klpd8\" (UID: \"7e4a1af3-5ae7-495e-bd96-25df977fef6f\") " pod="openshift-dns/node-resolver-klpd8"
Jul 15 10:46:23.723343 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.722427    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/902fd91e-2979-4d2c-b428-ff5e62d8203b-cni-binary-copy\") pod \"multus-additional-cni-plugins-l8v8c\" (UID: \"902fd91e-2979-4d2c-b428-ff5e62d8203b\") " pod="openshift-multus/multus-additional-cni-plugins-l8v8c"
Jul 15 10:46:23.723343 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.722453    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-host-cni-bin\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.723343 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.722466    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c40e765d-670d-40c3-925d-9585c28cbe72-socket-dir\") pod \"aws-ebs-csi-driver-node-lxh9r\" (UID: \"c40e765d-670d-40c3-925d-9585c28cbe72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxh9r"
Jul 15 10:46:23.723343 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.722479    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.723343 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.722514    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-host-var-lib-kubelet\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.723343 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.722524    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c40e765d-670d-40c3-925d-9585c28cbe72-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lxh9r\" (UID: \"c40e765d-670d-40c3-925d-9585c28cbe72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxh9r"
Jul 15 10:46:23.723343 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.722567    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2456e018-71e0-42bc-8153-64c597b91f17-lib-modules\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.723343 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.722592    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-host-cni-bin\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.723343 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.722625    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.723343 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.722772    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c40e765d-670d-40c3-925d-9585c28cbe72-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lxh9r\" (UID: \"c40e765d-670d-40c3-925d-9585c28cbe72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxh9r"
Jul 15 10:46:23.723343 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.722943    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/902fd91e-2979-4d2c-b428-ff5e62d8203b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l8v8c\" (UID: \"902fd91e-2979-4d2c-b428-ff5e62d8203b\") " pod="openshift-multus/multus-additional-cni-plugins-l8v8c"
Jul 15 10:46:23.723343 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.722972    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/beb68899-ffb3-436f-8e17-43597aad4030-ovnkube-script-lib\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.723343 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.723030    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e4a1af3-5ae7-495e-bd96-25df977fef6f-tmp-dir\") pod \"node-resolver-klpd8\" (UID: \"7e4a1af3-5ae7-495e-bd96-25df977fef6f\") " pod="openshift-dns/node-resolver-klpd8"
Jul 15 10:46:23.723343 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.723126    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c40e765d-670d-40c3-925d-9585c28cbe72-etc-selinux\") pod \"aws-ebs-csi-driver-node-lxh9r\" (UID: \"c40e765d-670d-40c3-925d-9585c28cbe72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxh9r"
Jul 15 10:46:23.723343 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.723159    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2456e018-71e0-42bc-8153-64c597b91f17-etc-sysctl-conf\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.724194 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.723186    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2456e018-71e0-42bc-8153-64c597b91f17-var-lib-kubelet\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.724194 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.723187    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c40e765d-670d-40c3-925d-9585c28cbe72-etc-selinux\") pod \"aws-ebs-csi-driver-node-lxh9r\" (UID: \"c40e765d-670d-40c3-925d-9585c28cbe72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxh9r"
Jul 15 10:46:23.724194 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.723212    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-multus-socket-dir-parent\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.724194 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.723276    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-cnibin\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.724194 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.723304    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-host-run-netns\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.724194 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.723326    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-multus-socket-dir-parent\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.724194 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.723334    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2456e018-71e0-42bc-8153-64c597b91f17-etc-sysctl-conf\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.724194 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.723277    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2456e018-71e0-42bc-8153-64c597b91f17-var-lib-kubelet\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.724194 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.723330    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-host-var-lib-cni-bin\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.724194 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.723372    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-host-run-netns\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.724194 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.723381    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-cnibin\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.724194 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.723392    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-host-var-lib-cni-multus\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.724194 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.723421    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-host-var-lib-cni-bin\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.724194 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.723425    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-host-var-lib-cni-multus\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.724194 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.723425    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxkdb\" (UniqueName: \"kubernetes.io/projected/2f949cb2-6105-448d-b4df-82ab30888fef-kube-api-access-nxkdb\") pod \"iptables-alerter-4jcc7\" (UID: \"2f949cb2-6105-448d-b4df-82ab30888fef\") " pod="openshift-network-operator/iptables-alerter-4jcc7"
Jul 15 10:46:23.724194 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.723468    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-var-lib-openvswitch\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.724194 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.723484    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/beb68899-ffb3-436f-8e17-43597aad4030-ovn-node-metrics-cert\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.724194 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.723517    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-host-run-multus-certs\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.725176 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.723541    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-var-lib-openvswitch\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.725176 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.723544    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ad4cef-91aa-459c-80f3-f67d36e697aa-original-pull-secret\") pod \"global-pull-secret-syncer-rx95w\" (UID: \"37ad4cef-91aa-459c-80f3-f67d36e697aa\") " pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:46:23.725176 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.723583    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c40e765d-670d-40c3-925d-9585c28cbe72-registration-dir\") pod \"aws-ebs-csi-driver-node-lxh9r\" (UID: \"c40e765d-670d-40c3-925d-9585c28cbe72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxh9r"
Jul 15 10:46:23.725176 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.723609    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2456e018-71e0-42bc-8153-64c597b91f17-etc-kubernetes\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.725176 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:23.723613    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 15 10:46:23.725176 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:23.723678    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37ad4cef-91aa-459c-80f3-f67d36e697aa-original-pull-secret podName:37ad4cef-91aa-459c-80f3-f67d36e697aa nodeName:}" failed. No retries permitted until 2026-07-15 10:46:24.223660969 +0000 UTC m=+3.053176372 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37ad4cef-91aa-459c-80f3-f67d36e697aa-original-pull-secret") pod "global-pull-secret-syncer-rx95w" (UID: "37ad4cef-91aa-459c-80f3-f67d36e697aa") : object "kube-system"/"original-pull-secret" not registered
Jul 15 10:46:23.725176 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.723704    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c40e765d-670d-40c3-925d-9585c28cbe72-registration-dir\") pod \"aws-ebs-csi-driver-node-lxh9r\" (UID: \"c40e765d-670d-40c3-925d-9585c28cbe72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxh9r"
Jul 15 10:46:23.725176 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.723728    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2456e018-71e0-42bc-8153-64c597b91f17-etc-kubernetes\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.725176 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.723769    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-host-run-multus-certs\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.725176 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.723772    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f949cb2-6105-448d-b4df-82ab30888fef-host-slash\") pod \"iptables-alerter-4jcc7\" (UID: \"2f949cb2-6105-448d-b4df-82ab30888fef\") " pod="openshift-network-operator/iptables-alerter-4jcc7"
Jul 15 10:46:23.725176 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.723844    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c40e765d-670d-40c3-925d-9585c28cbe72-device-dir\") pod \"aws-ebs-csi-driver-node-lxh9r\" (UID: \"c40e765d-670d-40c3-925d-9585c28cbe72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxh9r"
Jul 15 10:46:23.725176 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.723881    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f949cb2-6105-448d-b4df-82ab30888fef-host-slash\") pod \"iptables-alerter-4jcc7\" (UID: \"2f949cb2-6105-448d-b4df-82ab30888fef\") " pod="openshift-network-operator/iptables-alerter-4jcc7"
Jul 15 10:46:23.725176 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.723886    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2456e018-71e0-42bc-8153-64c597b91f17-tmp\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.725176 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.723917    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-run-systemd\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.725176 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.723939    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-node-log\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.725176 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.723962    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksq62\" (UniqueName: \"kubernetes.io/projected/2456e018-71e0-42bc-8153-64c597b91f17-kube-api-access-ksq62\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.725176 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.723985    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/902fd91e-2979-4d2c-b428-ff5e62d8203b-os-release\") pod \"multus-additional-cni-plugins-l8v8c\" (UID: \"902fd91e-2979-4d2c-b428-ff5e62d8203b\") " pod="openshift-multus/multus-additional-cni-plugins-l8v8c"
Jul 15 10:46:23.725984 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.724010    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/902fd91e-2979-4d2c-b428-ff5e62d8203b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l8v8c\" (UID: \"902fd91e-2979-4d2c-b428-ff5e62d8203b\") " pod="openshift-multus/multus-additional-cni-plugins-l8v8c"
Jul 15 10:46:23.725984 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.724040    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-host-run-netns\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.725984 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.724063    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/37ad4cef-91aa-459c-80f3-f67d36e697aa-dbus\") pod \"global-pull-secret-syncer-rx95w\" (UID: \"37ad4cef-91aa-459c-80f3-f67d36e697aa\") " pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:46:23.725984 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.724089    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhfrm\" (UniqueName: \"kubernetes.io/projected/c40e765d-670d-40c3-925d-9585c28cbe72-kube-api-access-dhfrm\") pod \"aws-ebs-csi-driver-node-lxh9r\" (UID: \"c40e765d-670d-40c3-925d-9585c28cbe72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxh9r"
Jul 15 10:46:23.725984 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.724115    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2456e018-71e0-42bc-8153-64c597b91f17-etc-tuned\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.725984 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.724128    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/902fd91e-2979-4d2c-b428-ff5e62d8203b-cni-binary-copy\") pod \"multus-additional-cni-plugins-l8v8c\" (UID: \"902fd91e-2979-4d2c-b428-ff5e62d8203b\") " pod="openshift-multus/multus-additional-cni-plugins-l8v8c"
Jul 15 10:46:23.725984 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.724137    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/902fd91e-2979-4d2c-b428-ff5e62d8203b-cnibin\") pod \"multus-additional-cni-plugins-l8v8c\" (UID: \"902fd91e-2979-4d2c-b428-ff5e62d8203b\") " pod="openshift-multus/multus-additional-cni-plugins-l8v8c"
Jul 15 10:46:23.725984 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.724180    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-host-run-k8s-cni-cncf-io\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.725984 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.725386    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/beb68899-ffb3-436f-8e17-43597aad4030-env-overrides\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.725984 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.725470    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7e4a1af3-5ae7-495e-bd96-25df977fef6f-hosts-file\") pod \"node-resolver-klpd8\" (UID: \"7e4a1af3-5ae7-495e-bd96-25df977fef6f\") " pod="openshift-dns/node-resolver-klpd8"
Jul 15 10:46:23.725984 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.725513    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eaebac66-19db-44df-85f4-887969b5984e-agent-certs\") pod \"konnectivity-agent-wdxpd\" (UID: \"eaebac66-19db-44df-85f4-887969b5984e\") " pod="kube-system/konnectivity-agent-wdxpd"
Jul 15 10:46:23.726525 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.726297    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-run-systemd\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.726525 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.726372    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-node-log\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.729633 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.726653    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/902fd91e-2979-4d2c-b428-ff5e62d8203b-os-release\") pod \"multus-additional-cni-plugins-l8v8c\" (UID: \"902fd91e-2979-4d2c-b428-ff5e62d8203b\") " pod="openshift-multus/multus-additional-cni-plugins-l8v8c"
Jul 15 10:46:23.729633 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.724236    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c40e765d-670d-40c3-925d-9585c28cbe72-device-dir\") pod \"aws-ebs-csi-driver-node-lxh9r\" (UID: \"c40e765d-670d-40c3-925d-9585c28cbe72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxh9r"
Jul 15 10:46:23.729633 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.726787    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c40e765d-670d-40c3-925d-9585c28cbe72-sys-fs\") pod \"aws-ebs-csi-driver-node-lxh9r\" (UID: \"c40e765d-670d-40c3-925d-9585c28cbe72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxh9r"
Jul 15 10:46:23.729633 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.726819    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2456e018-71e0-42bc-8153-64c597b91f17-etc-sysconfig\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.729633 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.726854    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cae13509-00de-48df-ae3d-943755c4b2e9-cni-binary-copy\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.729633 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.726886    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-system-cni-dir\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.729633 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.726920    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-multus-cni-dir\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.729633 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.726921    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/beb68899-ffb3-436f-8e17-43597aad4030-ovn-node-metrics-cert\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.729633 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.726952    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-host-slash\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.729633 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.726984    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-host-run-ovn-kubernetes\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.729633 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.727017    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/beb68899-ffb3-436f-8e17-43597aad4030-ovnkube-config\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.729633 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.727019    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/902fd91e-2979-4d2c-b428-ff5e62d8203b-cnibin\") pod \"multus-additional-cni-plugins-l8v8c\" (UID: \"902fd91e-2979-4d2c-b428-ff5e62d8203b\") " pod="openshift-multus/multus-additional-cni-plugins-l8v8c"
Jul 15 10:46:23.729633 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.727090    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lz28\" (UniqueName: \"kubernetes.io/projected/c8d1da87-af98-4f6b-af6a-27a5fbabba60-kube-api-access-4lz28\") pod \"network-metrics-daemon-84n76\" (UID: \"c8d1da87-af98-4f6b-af6a-27a5fbabba60\") " pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:46:23.729633 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.727115    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-host-run-netns\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.729633 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.727128    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2456e018-71e0-42bc-8153-64c597b91f17-etc-modprobe-d\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.729633 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.727185    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-etc-kubernetes\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.729633 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.727237    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-run-openvswitch\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.732659 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.727273    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-host-cni-netd\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.732659 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.727288    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2456e018-71e0-42bc-8153-64c597b91f17-etc-modprobe-d\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.732659 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.727304    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2456e018-71e0-42bc-8153-64c597b91f17-etc-sysctl-d\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.732659 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.727365    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-multus-cni-dir\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.732659 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.727405    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2456e018-71e0-42bc-8153-64c597b91f17-etc-systemd\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.732659 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.727437    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2456e018-71e0-42bc-8153-64c597b91f17-sys\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.732659 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.727442    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/37ad4cef-91aa-459c-80f3-f67d36e697aa-dbus\") pod \"global-pull-secret-syncer-rx95w\" (UID: \"37ad4cef-91aa-459c-80f3-f67d36e697aa\") " pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:46:23.732659 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.727463    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/902fd91e-2979-4d2c-b428-ff5e62d8203b-system-cni-dir\") pod \"multus-additional-cni-plugins-l8v8c\" (UID: \"902fd91e-2979-4d2c-b428-ff5e62d8203b\") " pod="openshift-multus/multus-additional-cni-plugins-l8v8c"
Jul 15 10:46:23.732659 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.727516    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fzfx\" (UniqueName: \"kubernetes.io/projected/902fd91e-2979-4d2c-b428-ff5e62d8203b-kube-api-access-7fzfx\") pod \"multus-additional-cni-plugins-l8v8c\" (UID: \"902fd91e-2979-4d2c-b428-ff5e62d8203b\") " pod="openshift-multus/multus-additional-cni-plugins-l8v8c"
Jul 15 10:46:23.732659 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.727551    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2f949cb2-6105-448d-b4df-82ab30888fef-iptables-alerter-script\") pod \"iptables-alerter-4jcc7\" (UID: \"2f949cb2-6105-448d-b4df-82ab30888fef\") " pod="openshift-network-operator/iptables-alerter-4jcc7"
Jul 15 10:46:23.732659 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.727589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eaebac66-19db-44df-85f4-887969b5984e-konnectivity-ca\") pod \"konnectivity-agent-wdxpd\" (UID: \"eaebac66-19db-44df-85f4-887969b5984e\") " pod="kube-system/konnectivity-agent-wdxpd"
Jul 15 10:46:23.732659 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.727624    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-os-release\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.732659 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.727657    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cae13509-00de-48df-ae3d-943755c4b2e9-multus-daemon-config\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.732659 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.727683    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cae13509-00de-48df-ae3d-943755c4b2e9-cni-binary-copy\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.732659 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.727689    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-host-kubelet\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.732659 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.727754    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-host-run-ovn-kubernetes\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.732659 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.727765    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-systemd-units\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.732659 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.727814    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-host-slash\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.733584 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.727858    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-host-kubelet\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.733584 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.727975    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2456e018-71e0-42bc-8153-64c597b91f17-etc-sysctl-d\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.733584 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.728093    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-system-cni-dir\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.733584 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.728180    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-host-cni-netd\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.733584 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.728694    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cae13509-00de-48df-ae3d-943755c4b2e9-multus-daemon-config\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.733584 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.728789    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-os-release\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.733584 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.728836    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-run-openvswitch\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.733584 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.728841    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eaebac66-19db-44df-85f4-887969b5984e-konnectivity-ca\") pod \"konnectivity-agent-wdxpd\" (UID: \"eaebac66-19db-44df-85f4-887969b5984e\") " pod="kube-system/konnectivity-agent-wdxpd"
Jul 15 10:46:23.733584 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.728909    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c40e765d-670d-40c3-925d-9585c28cbe72-sys-fs\") pod \"aws-ebs-csi-driver-node-lxh9r\" (UID: \"c40e765d-670d-40c3-925d-9585c28cbe72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxh9r"
Jul 15 10:46:23.733584 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.728960    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-etc-kubernetes\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.733584 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.728988    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2456e018-71e0-42bc-8153-64c597b91f17-etc-sysconfig\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.733584 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.729010    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/902fd91e-2979-4d2c-b428-ff5e62d8203b-system-cni-dir\") pod \"multus-additional-cni-plugins-l8v8c\" (UID: \"902fd91e-2979-4d2c-b428-ff5e62d8203b\") " pod="openshift-multus/multus-additional-cni-plugins-l8v8c"
Jul 15 10:46:23.733584 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.729068    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2f949cb2-6105-448d-b4df-82ab30888fef-iptables-alerter-script\") pod \"iptables-alerter-4jcc7\" (UID: \"2f949cb2-6105-448d-b4df-82ab30888fef\") " pod="openshift-network-operator/iptables-alerter-4jcc7"
Jul 15 10:46:23.733584 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.729326    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cae13509-00de-48df-ae3d-943755c4b2e9-host-run-k8s-cni-cncf-io\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.733584 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.729405    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2456e018-71e0-42bc-8153-64c597b91f17-sys\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.733584 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.729456    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/beb68899-ffb3-436f-8e17-43597aad4030-systemd-units\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.733584 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.729566    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/beb68899-ffb3-436f-8e17-43597aad4030-ovnkube-config\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.733584 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.729574    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/902fd91e-2979-4d2c-b428-ff5e62d8203b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l8v8c\" (UID: \"902fd91e-2979-4d2c-b428-ff5e62d8203b\") " pod="openshift-multus/multus-additional-cni-plugins-l8v8c"
Jul 15 10:46:23.734493 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.729624    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2456e018-71e0-42bc-8153-64c597b91f17-etc-systemd\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.734493 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.729812    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2456e018-71e0-42bc-8153-64c597b91f17-tmp\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.734493 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.729834    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7e4a1af3-5ae7-495e-bd96-25df977fef6f-hosts-file\") pod \"node-resolver-klpd8\" (UID: \"7e4a1af3-5ae7-495e-bd96-25df977fef6f\") " pod="openshift-dns/node-resolver-klpd8"
Jul 15 10:46:23.734493 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.729953    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/beb68899-ffb3-436f-8e17-43597aad4030-env-overrides\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.734493 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.732067    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2456e018-71e0-42bc-8153-64c597b91f17-etc-tuned\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.734493 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.732890    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eaebac66-19db-44df-85f4-887969b5984e-agent-certs\") pod \"konnectivity-agent-wdxpd\" (UID: \"eaebac66-19db-44df-85f4-887969b5984e\") " pod="kube-system/konnectivity-agent-wdxpd"
Jul 15 10:46:23.742349 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.742312    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lz28\" (UniqueName: \"kubernetes.io/projected/c8d1da87-af98-4f6b-af6a-27a5fbabba60-kube-api-access-4lz28\") pod \"network-metrics-daemon-84n76\" (UID: \"c8d1da87-af98-4f6b-af6a-27a5fbabba60\") " pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:46:23.742349 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.742332    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fzfx\" (UniqueName: \"kubernetes.io/projected/902fd91e-2979-4d2c-b428-ff5e62d8203b-kube-api-access-7fzfx\") pod \"multus-additional-cni-plugins-l8v8c\" (UID: \"902fd91e-2979-4d2c-b428-ff5e62d8203b\") " pod="openshift-multus/multus-additional-cni-plugins-l8v8c"
Jul 15 10:46:23.742811 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.742773    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klkk5\" (UniqueName: \"kubernetes.io/projected/7e4a1af3-5ae7-495e-bd96-25df977fef6f-kube-api-access-klkk5\") pod \"node-resolver-klpd8\" (UID: \"7e4a1af3-5ae7-495e-bd96-25df977fef6f\") " pod="openshift-dns/node-resolver-klpd8"
Jul 15 10:46:23.743312 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.743280    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhfrm\" (UniqueName: \"kubernetes.io/projected/c40e765d-670d-40c3-925d-9585c28cbe72-kube-api-access-dhfrm\") pod \"aws-ebs-csi-driver-node-lxh9r\" (UID: \"c40e765d-670d-40c3-925d-9585c28cbe72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxh9r"
Jul 15 10:46:23.743555 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.743524    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v88cb\" (UniqueName: \"kubernetes.io/projected/cae13509-00de-48df-ae3d-943755c4b2e9-kube-api-access-v88cb\") pod \"multus-fcnjt\" (UID: \"cae13509-00de-48df-ae3d-943755c4b2e9\") " pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.744866 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.744824    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxkdb\" (UniqueName: \"kubernetes.io/projected/2f949cb2-6105-448d-b4df-82ab30888fef-kube-api-access-nxkdb\") pod \"iptables-alerter-4jcc7\" (UID: \"2f949cb2-6105-448d-b4df-82ab30888fef\") " pod="openshift-network-operator/iptables-alerter-4jcc7"
Jul 15 10:46:23.745461 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.745422    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksq62\" (UniqueName: \"kubernetes.io/projected/2456e018-71e0-42bc-8153-64c597b91f17-kube-api-access-ksq62\") pod \"tuned-4445h\" (UID: \"2456e018-71e0-42bc-8153-64c597b91f17\") " pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.745572 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.745501    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq57f\" (UniqueName: \"kubernetes.io/projected/beb68899-ffb3-436f-8e17-43597aad4030-kube-api-access-tq57f\") pod \"ovnkube-node-rk49n\" (UID: \"beb68899-ffb3-436f-8e17-43597aad4030\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.819602 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.819562    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-b4gsj"
Jul 15 10:46:23.831260 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.831202    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:23.848076 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.848040    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4jcc7"
Jul 15 10:46:23.855426 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.855385    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wdxpd"
Jul 15 10:46:23.862925 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.862890    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxh9r"
Jul 15 10:46:23.871288 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.871260    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4445h"
Jul 15 10:46:23.881303 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.881267    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-l8v8c"
Jul 15 10:46:23.890694 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.890662    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fcnjt"
Jul 15 10:46:23.898112 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.898079    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-klpd8"
Jul 15 10:46:23.996311 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:23.996214    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 15 10:46:24.131331 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:24.131287    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rwgk\" (UniqueName: \"kubernetes.io/projected/b5f21679-bed4-4e21-9a3b-22ec261aec90-kube-api-access-4rwgk\") pod \"network-check-target-9mhqc\" (UID: \"b5f21679-bed4-4e21-9a3b-22ec261aec90\") " pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:46:24.131533 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:24.131488    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 15 10:46:24.131533 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:24.131517    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 15 10:46:24.131533 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:24.131530    2563 projected.go:196] Error preparing data for projected volume kube-api-access-4rwgk for pod openshift-network-diagnostics/network-check-target-9mhqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:46:24.131687 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:24.131598    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5f21679-bed4-4e21-9a3b-22ec261aec90-kube-api-access-4rwgk podName:b5f21679-bed4-4e21-9a3b-22ec261aec90 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:25.131576026 +0000 UTC m=+3.961091428 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4rwgk" (UniqueName: "kubernetes.io/projected/b5f21679-bed4-4e21-9a3b-22ec261aec90-kube-api-access-4rwgk") pod "network-check-target-9mhqc" (UID: "b5f21679-bed4-4e21-9a3b-22ec261aec90") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:46:24.232344 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:24.232298    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8d1da87-af98-4f6b-af6a-27a5fbabba60-metrics-certs\") pod \"network-metrics-daemon-84n76\" (UID: \"c8d1da87-af98-4f6b-af6a-27a5fbabba60\") " pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:46:24.232564 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:24.232367    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ad4cef-91aa-459c-80f3-f67d36e697aa-original-pull-secret\") pod \"global-pull-secret-syncer-rx95w\" (UID: \"37ad4cef-91aa-459c-80f3-f67d36e697aa\") " pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:46:24.232564 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:24.232435    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:46:24.232564 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:24.232488    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 15 10:46:24.232564 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:24.232530    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8d1da87-af98-4f6b-af6a-27a5fbabba60-metrics-certs podName:c8d1da87-af98-4f6b-af6a-27a5fbabba60 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:25.232505179 +0000 UTC m=+4.062020568 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c8d1da87-af98-4f6b-af6a-27a5fbabba60-metrics-certs") pod "network-metrics-daemon-84n76" (UID: "c8d1da87-af98-4f6b-af6a-27a5fbabba60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:46:24.232564 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:24.232550    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37ad4cef-91aa-459c-80f3-f67d36e697aa-original-pull-secret podName:37ad4cef-91aa-459c-80f3-f67d36e697aa nodeName:}" failed. No retries permitted until 2026-07-15 10:46:25.232540976 +0000 UTC m=+4.062056363 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37ad4cef-91aa-459c-80f3-f67d36e697aa-original-pull-secret") pod "global-pull-secret-syncer-rx95w" (UID: "37ad4cef-91aa-459c-80f3-f67d36e697aa") : object "kube-system"/"original-pull-secret" not registered
Jul 15 10:46:24.555360 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:24.555307    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 10:41:22 +0000 UTC" deadline="2028-03-02 22:51:21.765592812 +0000 UTC"
Jul 15 10:46:24.555360 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:24.555353    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14316h4m57.210242584s"
Jul 15 10:46:24.774375 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:24.774343    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e4a1af3_5ae7_495e_bd96_25df977fef6f.slice/crio-7c07bcb644e1b104523b7676013cea31055a0c450022e2b4346e43482e9fbfdf WatchSource:0}: Error finding container 7c07bcb644e1b104523b7676013cea31055a0c450022e2b4346e43482e9fbfdf: Status 404 returned error can't find the container with id 7c07bcb644e1b104523b7676013cea31055a0c450022e2b4346e43482e9fbfdf
Jul 15 10:46:24.789985 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:24.789959    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc40e765d_670d_40c3_925d_9585c28cbe72.slice/crio-a6069947e6ec2c63a9566803c42189452bd3a23afa4f7002ab5f1d3a1a6f5c64 WatchSource:0}: Error finding container a6069947e6ec2c63a9566803c42189452bd3a23afa4f7002ab5f1d3a1a6f5c64: Status 404 returned error can't find the container with id a6069947e6ec2c63a9566803c42189452bd3a23afa4f7002ab5f1d3a1a6f5c64
Jul 15 10:46:24.794973 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:24.794853    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcae13509_00de_48df_ae3d_943755c4b2e9.slice/crio-6619864cbf5573d2b86ff999a5579d147d3594941a3fab5f70d8e1d39e36b200 WatchSource:0}: Error finding container 6619864cbf5573d2b86ff999a5579d147d3594941a3fab5f70d8e1d39e36b200: Status 404 returned error can't find the container with id 6619864cbf5573d2b86ff999a5579d147d3594941a3fab5f70d8e1d39e36b200
Jul 15 10:46:24.796993 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:24.796966    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2456e018_71e0_42bc_8153_64c597b91f17.slice/crio-7f3573e4f3793809016ede9fc06dd2124ca5ee76c2251841e19f822d8102adf8 WatchSource:0}: Error finding container 7f3573e4f3793809016ede9fc06dd2124ca5ee76c2251841e19f822d8102adf8: Status 404 returned error can't find the container with id 7f3573e4f3793809016ede9fc06dd2124ca5ee76c2251841e19f822d8102adf8
Jul 15 10:46:24.798384 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:24.798357    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod902fd91e_2979_4d2c_b428_ff5e62d8203b.slice/crio-89d4093ade851edc2c64112fd16a24d2b2149e748e880c661e58c7fcf338767f WatchSource:0}: Error finding container 89d4093ade851edc2c64112fd16a24d2b2149e748e880c661e58c7fcf338767f: Status 404 returned error can't find the container with id 89d4093ade851edc2c64112fd16a24d2b2149e748e880c661e58c7fcf338767f
Jul 15 10:46:24.799211 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:24.799130    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a7699c1_9758_4c35_8b4c_aed46512560e.slice/crio-d0c22a75ab4777c365f419e3b0e3a40201524342b91918af2fcb42370b58e770 WatchSource:0}: Error finding container d0c22a75ab4777c365f419e3b0e3a40201524342b91918af2fcb42370b58e770: Status 404 returned error can't find the container with id d0c22a75ab4777c365f419e3b0e3a40201524342b91918af2fcb42370b58e770
Jul 15 10:46:24.801512 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:24.801378    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeb68899_ffb3_436f_8e17_43597aad4030.slice/crio-21de722cfad70e8f9b9d03fba659aea6b871f08a47e2aff609791f66b11f6131 WatchSource:0}: Error finding container 21de722cfad70e8f9b9d03fba659aea6b871f08a47e2aff609791f66b11f6131: Status 404 returned error can't find the container with id 21de722cfad70e8f9b9d03fba659aea6b871f08a47e2aff609791f66b11f6131
Jul 15 10:46:24.801988 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:24.801949    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f949cb2_6105_448d_b4df_82ab30888fef.slice/crio-51e4a6ac2f0dcea81e3fd6d793c2625ac8a99dea9c05404ac3d92be2f2d17d36 WatchSource:0}: Error finding container 51e4a6ac2f0dcea81e3fd6d793c2625ac8a99dea9c05404ac3d92be2f2d17d36: Status 404 returned error can't find the container with id 51e4a6ac2f0dcea81e3fd6d793c2625ac8a99dea9c05404ac3d92be2f2d17d36
Jul 15 10:46:24.803686 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:46:24.803199    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaebac66_19db_44df_85f4_887969b5984e.slice/crio-5b2511cb710f820f13cd12cd64550fa1f463c5b172190bd3c1db4931fec11250 WatchSource:0}: Error finding container 5b2511cb710f820f13cd12cd64550fa1f463c5b172190bd3c1db4931fec11250: Status 404 returned error can't find the container with id 5b2511cb710f820f13cd12cd64550fa1f463c5b172190bd3c1db4931fec11250
Jul 15 10:46:25.141177 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:25.141071    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rwgk\" (UniqueName: \"kubernetes.io/projected/b5f21679-bed4-4e21-9a3b-22ec261aec90-kube-api-access-4rwgk\") pod \"network-check-target-9mhqc\" (UID: \"b5f21679-bed4-4e21-9a3b-22ec261aec90\") " pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:46:25.141326 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:25.141224    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 15 10:46:25.141326 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:25.141243    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 15 10:46:25.141326 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:25.141252    2563 projected.go:196] Error preparing data for projected volume kube-api-access-4rwgk for pod openshift-network-diagnostics/network-check-target-9mhqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:46:25.141326 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:25.141306    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5f21679-bed4-4e21-9a3b-22ec261aec90-kube-api-access-4rwgk podName:b5f21679-bed4-4e21-9a3b-22ec261aec90 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:27.141292123 +0000 UTC m=+5.970807523 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4rwgk" (UniqueName: "kubernetes.io/projected/b5f21679-bed4-4e21-9a3b-22ec261aec90-kube-api-access-4rwgk") pod "network-check-target-9mhqc" (UID: "b5f21679-bed4-4e21-9a3b-22ec261aec90") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:46:25.241609 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:25.241566    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ad4cef-91aa-459c-80f3-f67d36e697aa-original-pull-secret\") pod \"global-pull-secret-syncer-rx95w\" (UID: \"37ad4cef-91aa-459c-80f3-f67d36e697aa\") " pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:46:25.241801 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:25.241663    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8d1da87-af98-4f6b-af6a-27a5fbabba60-metrics-certs\") pod \"network-metrics-daemon-84n76\" (UID: \"c8d1da87-af98-4f6b-af6a-27a5fbabba60\") " pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:46:25.241867 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:25.241822    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:46:25.241927 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:25.241889    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8d1da87-af98-4f6b-af6a-27a5fbabba60-metrics-certs podName:c8d1da87-af98-4f6b-af6a-27a5fbabba60 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:27.241869416 +0000 UTC m=+6.071384816 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c8d1da87-af98-4f6b-af6a-27a5fbabba60-metrics-certs") pod "network-metrics-daemon-84n76" (UID: "c8d1da87-af98-4f6b-af6a-27a5fbabba60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:46:25.242553 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:25.242420    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 15 10:46:25.242553 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:25.242481    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37ad4cef-91aa-459c-80f3-f67d36e697aa-original-pull-secret podName:37ad4cef-91aa-459c-80f3-f67d36e697aa nodeName:}" failed. No retries permitted until 2026-07-15 10:46:27.242464097 +0000 UTC m=+6.071979505 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37ad4cef-91aa-459c-80f3-f67d36e697aa-original-pull-secret") pod "global-pull-secret-syncer-rx95w" (UID: "37ad4cef-91aa-459c-80f3-f67d36e697aa") : object "kube-system"/"original-pull-secret" not registered
Jul 15 10:46:25.675712 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:25.675640    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:46:25.676316 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:25.675808    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-84n76" podUID="c8d1da87-af98-4f6b-af6a-27a5fbabba60"
Jul 15 10:46:25.676316 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:25.675907    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:46:25.676316 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:25.675987    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx95w" podUID="37ad4cef-91aa-459c-80f3-f67d36e697aa"
Jul 15 10:46:25.676316 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:25.676046    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:46:25.676316 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:25.676119    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mhqc" podUID="b5f21679-bed4-4e21-9a3b-22ec261aec90"
Jul 15 10:46:25.693110 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:25.693070    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4jcc7" event={"ID":"2f949cb2-6105-448d-b4df-82ab30888fef","Type":"ContainerStarted","Data":"51e4a6ac2f0dcea81e3fd6d793c2625ac8a99dea9c05404ac3d92be2f2d17d36"}
Jul 15 10:46:25.703524 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:25.703481    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rk49n" event={"ID":"beb68899-ffb3-436f-8e17-43597aad4030","Type":"ContainerStarted","Data":"21de722cfad70e8f9b9d03fba659aea6b871f08a47e2aff609791f66b11f6131"}
Jul 15 10:46:25.710826 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:25.710624    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-b4gsj" event={"ID":"2a7699c1-9758-4c35-8b4c-aed46512560e","Type":"ContainerStarted","Data":"d0c22a75ab4777c365f419e3b0e3a40201524342b91918af2fcb42370b58e770"}
Jul 15 10:46:25.716331 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:25.716171    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4445h" event={"ID":"2456e018-71e0-42bc-8153-64c597b91f17","Type":"ContainerStarted","Data":"7f3573e4f3793809016ede9fc06dd2124ca5ee76c2251841e19f822d8102adf8"}
Jul 15 10:46:25.721577 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:25.721474    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fcnjt" event={"ID":"cae13509-00de-48df-ae3d-943755c4b2e9","Type":"ContainerStarted","Data":"6619864cbf5573d2b86ff999a5579d147d3594941a3fab5f70d8e1d39e36b200"}
Jul 15 10:46:25.737839 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:25.737718    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxh9r" event={"ID":"c40e765d-670d-40c3-925d-9585c28cbe72","Type":"ContainerStarted","Data":"a6069947e6ec2c63a9566803c42189452bd3a23afa4f7002ab5f1d3a1a6f5c64"}
Jul 15 10:46:25.743254 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:25.742884    2563 generic.go:356] "Generic (PLEG): container finished" podID="3d172d526ed473c2594efb0d6fe48f48" containerID="d15a6161a10820a296c8ddf06921867ef04c54bc9315a0fd4b4e4232ed6a8028" exitCode=0
Jul 15 10:46:25.743254 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:25.743182    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-34.ec2.internal" event={"ID":"3d172d526ed473c2594efb0d6fe48f48","Type":"ContainerDied","Data":"d15a6161a10820a296c8ddf06921867ef04c54bc9315a0fd4b4e4232ed6a8028"}
Jul 15 10:46:25.755412 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:25.755291    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-34.ec2.internal" event={"ID":"a18a70f1e59122311577155ea3452e36","Type":"ContainerStarted","Data":"883bcbe330a30ec15e88ce91f0e719bb8105560a5b4615f3f92b36b982f56fa7"}
Jul 15 10:46:25.762627 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:25.762479    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wdxpd" event={"ID":"eaebac66-19db-44df-85f4-887969b5984e","Type":"ContainerStarted","Data":"5b2511cb710f820f13cd12cd64550fa1f463c5b172190bd3c1db4931fec11250"}
Jul 15 10:46:25.775655 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:25.775607    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l8v8c" event={"ID":"902fd91e-2979-4d2c-b428-ff5e62d8203b","Type":"ContainerStarted","Data":"89d4093ade851edc2c64112fd16a24d2b2149e748e880c661e58c7fcf338767f"}
Jul 15 10:46:25.781754 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:25.781715    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-klpd8" event={"ID":"7e4a1af3-5ae7-495e-bd96-25df977fef6f","Type":"ContainerStarted","Data":"7c07bcb644e1b104523b7676013cea31055a0c450022e2b4346e43482e9fbfdf"}
Jul 15 10:46:25.788911 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:25.787531    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-34.ec2.internal" podStartSLOduration=3.787506012 podStartE2EDuration="3.787506012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:46:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:25.787463728 +0000 UTC m=+4.616979138" watchObservedRunningTime="2026-07-15 10:46:25.787506012 +0000 UTC m=+4.617021418"
Jul 15 10:46:26.813600 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:26.813556    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-34.ec2.internal" event={"ID":"3d172d526ed473c2594efb0d6fe48f48","Type":"ContainerStarted","Data":"36a59eeea50efdf8bc51408ead1b9cab238c6cba00f66a62e53b338b9e589611"}
Jul 15 10:46:27.163254 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:27.162391    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rwgk\" (UniqueName: \"kubernetes.io/projected/b5f21679-bed4-4e21-9a3b-22ec261aec90-kube-api-access-4rwgk\") pod \"network-check-target-9mhqc\" (UID: \"b5f21679-bed4-4e21-9a3b-22ec261aec90\") " pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:46:27.163254 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:27.162646    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 15 10:46:27.163254 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:27.162673    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 15 10:46:27.163254 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:27.162687    2563 projected.go:196] Error preparing data for projected volume kube-api-access-4rwgk for pod openshift-network-diagnostics/network-check-target-9mhqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:46:27.163254 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:27.162760    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5f21679-bed4-4e21-9a3b-22ec261aec90-kube-api-access-4rwgk podName:b5f21679-bed4-4e21-9a3b-22ec261aec90 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:31.162738099 +0000 UTC m=+9.992253486 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4rwgk" (UniqueName: "kubernetes.io/projected/b5f21679-bed4-4e21-9a3b-22ec261aec90-kube-api-access-4rwgk") pod "network-check-target-9mhqc" (UID: "b5f21679-bed4-4e21-9a3b-22ec261aec90") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:46:27.263626 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:27.263585    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8d1da87-af98-4f6b-af6a-27a5fbabba60-metrics-certs\") pod \"network-metrics-daemon-84n76\" (UID: \"c8d1da87-af98-4f6b-af6a-27a5fbabba60\") " pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:46:27.263834 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:27.263665    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ad4cef-91aa-459c-80f3-f67d36e697aa-original-pull-secret\") pod \"global-pull-secret-syncer-rx95w\" (UID: \"37ad4cef-91aa-459c-80f3-f67d36e697aa\") " pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:46:27.263834 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:27.263807    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 15 10:46:27.263952 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:27.263868    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37ad4cef-91aa-459c-80f3-f67d36e697aa-original-pull-secret podName:37ad4cef-91aa-459c-80f3-f67d36e697aa nodeName:}" failed. No retries permitted until 2026-07-15 10:46:31.263849861 +0000 UTC m=+10.093365250 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37ad4cef-91aa-459c-80f3-f67d36e697aa-original-pull-secret") pod "global-pull-secret-syncer-rx95w" (UID: "37ad4cef-91aa-459c-80f3-f67d36e697aa") : object "kube-system"/"original-pull-secret" not registered
Jul 15 10:46:27.264014 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:27.263954    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:46:27.264014 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:27.263988    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8d1da87-af98-4f6b-af6a-27a5fbabba60-metrics-certs podName:c8d1da87-af98-4f6b-af6a-27a5fbabba60 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:31.263977645 +0000 UTC m=+10.093493041 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c8d1da87-af98-4f6b-af6a-27a5fbabba60-metrics-certs") pod "network-metrics-daemon-84n76" (UID: "c8d1da87-af98-4f6b-af6a-27a5fbabba60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:46:27.673455 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:27.673395    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:46:27.673671 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:27.673560    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-84n76" podUID="c8d1da87-af98-4f6b-af6a-27a5fbabba60"
Jul 15 10:46:27.674038 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:27.673867    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:46:27.674038 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:27.673983    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx95w" podUID="37ad4cef-91aa-459c-80f3-f67d36e697aa"
Jul 15 10:46:27.674227 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:27.674043    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:46:27.674227 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:27.674157    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mhqc" podUID="b5f21679-bed4-4e21-9a3b-22ec261aec90"
Jul 15 10:46:29.672689 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:29.672652    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:46:29.673192 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:29.672786    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:46:29.673192 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:29.672796    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mhqc" podUID="b5f21679-bed4-4e21-9a3b-22ec261aec90"
Jul 15 10:46:29.673192 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:29.672931    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-84n76" podUID="c8d1da87-af98-4f6b-af6a-27a5fbabba60"
Jul 15 10:46:29.673192 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:29.673013    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:46:29.673192 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:29.673089    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx95w" podUID="37ad4cef-91aa-459c-80f3-f67d36e697aa"
Jul 15 10:46:31.199693 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:31.199653    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rwgk\" (UniqueName: \"kubernetes.io/projected/b5f21679-bed4-4e21-9a3b-22ec261aec90-kube-api-access-4rwgk\") pod \"network-check-target-9mhqc\" (UID: \"b5f21679-bed4-4e21-9a3b-22ec261aec90\") " pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:46:31.200092 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:31.199819    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 15 10:46:31.200092 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:31.199836    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 15 10:46:31.200092 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:31.199845    2563 projected.go:196] Error preparing data for projected volume kube-api-access-4rwgk for pod openshift-network-diagnostics/network-check-target-9mhqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:46:31.200092 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:31.199894    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5f21679-bed4-4e21-9a3b-22ec261aec90-kube-api-access-4rwgk podName:b5f21679-bed4-4e21-9a3b-22ec261aec90 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:39.199879516 +0000 UTC m=+18.029394913 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4rwgk" (UniqueName: "kubernetes.io/projected/b5f21679-bed4-4e21-9a3b-22ec261aec90-kube-api-access-4rwgk") pod "network-check-target-9mhqc" (UID: "b5f21679-bed4-4e21-9a3b-22ec261aec90") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:46:31.300927 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:31.300859    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8d1da87-af98-4f6b-af6a-27a5fbabba60-metrics-certs\") pod \"network-metrics-daemon-84n76\" (UID: \"c8d1da87-af98-4f6b-af6a-27a5fbabba60\") " pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:46:31.301176 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:31.300943    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ad4cef-91aa-459c-80f3-f67d36e697aa-original-pull-secret\") pod \"global-pull-secret-syncer-rx95w\" (UID: \"37ad4cef-91aa-459c-80f3-f67d36e697aa\") " pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:46:31.301176 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:31.301056    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 15 10:46:31.301176 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:31.301059    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:46:31.301176 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:31.301136    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37ad4cef-91aa-459c-80f3-f67d36e697aa-original-pull-secret podName:37ad4cef-91aa-459c-80f3-f67d36e697aa nodeName:}" failed. No retries permitted until 2026-07-15 10:46:39.301111807 +0000 UTC m=+18.130627207 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37ad4cef-91aa-459c-80f3-f67d36e697aa-original-pull-secret") pod "global-pull-secret-syncer-rx95w" (UID: "37ad4cef-91aa-459c-80f3-f67d36e697aa") : object "kube-system"/"original-pull-secret" not registered
Jul 15 10:46:31.301176 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:31.301178    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8d1da87-af98-4f6b-af6a-27a5fbabba60-metrics-certs podName:c8d1da87-af98-4f6b-af6a-27a5fbabba60 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:39.301164614 +0000 UTC m=+18.130679996 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c8d1da87-af98-4f6b-af6a-27a5fbabba60-metrics-certs") pod "network-metrics-daemon-84n76" (UID: "c8d1da87-af98-4f6b-af6a-27a5fbabba60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:46:31.674227 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:31.673353    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:46:31.674227 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:31.673495    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-84n76" podUID="c8d1da87-af98-4f6b-af6a-27a5fbabba60"
Jul 15 10:46:31.674227 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:31.673893    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:46:31.674227 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:31.674000    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx95w" podUID="37ad4cef-91aa-459c-80f3-f67d36e697aa"
Jul 15 10:46:31.674227 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:31.674048    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:46:31.674227 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:31.674112    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mhqc" podUID="b5f21679-bed4-4e21-9a3b-22ec261aec90"
Jul 15 10:46:33.673648 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:33.673164    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:46:33.673648 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:33.673201    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:46:33.673648 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:33.673164    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:46:33.673648 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:33.673325    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-84n76" podUID="c8d1da87-af98-4f6b-af6a-27a5fbabba60"
Jul 15 10:46:33.673648 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:33.673437    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mhqc" podUID="b5f21679-bed4-4e21-9a3b-22ec261aec90"
Jul 15 10:46:33.673648 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:33.673523    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx95w" podUID="37ad4cef-91aa-459c-80f3-f67d36e697aa"
Jul 15 10:46:35.676667 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:35.676628    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:46:35.676667 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:35.676653    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:46:35.677107 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:35.676644    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:46:35.677107 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:35.676788    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx95w" podUID="37ad4cef-91aa-459c-80f3-f67d36e697aa"
Jul 15 10:46:35.677107 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:35.676822    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mhqc" podUID="b5f21679-bed4-4e21-9a3b-22ec261aec90"
Jul 15 10:46:35.677107 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:35.676888    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-84n76" podUID="c8d1da87-af98-4f6b-af6a-27a5fbabba60"
Jul 15 10:46:37.673297 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:37.673255    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:46:37.673297 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:37.673292    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:46:37.673847 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:37.673255    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:46:37.673847 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:37.673411    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx95w" podUID="37ad4cef-91aa-459c-80f3-f67d36e697aa"
Jul 15 10:46:37.673847 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:37.673538    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mhqc" podUID="b5f21679-bed4-4e21-9a3b-22ec261aec90"
Jul 15 10:46:37.673847 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:37.673761    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-84n76" podUID="c8d1da87-af98-4f6b-af6a-27a5fbabba60"
Jul 15 10:46:39.268650 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:39.268606    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rwgk\" (UniqueName: \"kubernetes.io/projected/b5f21679-bed4-4e21-9a3b-22ec261aec90-kube-api-access-4rwgk\") pod \"network-check-target-9mhqc\" (UID: \"b5f21679-bed4-4e21-9a3b-22ec261aec90\") " pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:46:39.269214 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:39.268805    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 15 10:46:39.269214 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:39.268830    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 15 10:46:39.269214 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:39.268843    2563 projected.go:196] Error preparing data for projected volume kube-api-access-4rwgk for pod openshift-network-diagnostics/network-check-target-9mhqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:46:39.269214 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:39.268912    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5f21679-bed4-4e21-9a3b-22ec261aec90-kube-api-access-4rwgk podName:b5f21679-bed4-4e21-9a3b-22ec261aec90 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:55.268892041 +0000 UTC m=+34.098407424 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4rwgk" (UniqueName: "kubernetes.io/projected/b5f21679-bed4-4e21-9a3b-22ec261aec90-kube-api-access-4rwgk") pod "network-check-target-9mhqc" (UID: "b5f21679-bed4-4e21-9a3b-22ec261aec90") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:46:39.369110 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:39.369070    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ad4cef-91aa-459c-80f3-f67d36e697aa-original-pull-secret\") pod \"global-pull-secret-syncer-rx95w\" (UID: \"37ad4cef-91aa-459c-80f3-f67d36e697aa\") " pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:46:39.369324 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:39.369157    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8d1da87-af98-4f6b-af6a-27a5fbabba60-metrics-certs\") pod \"network-metrics-daemon-84n76\" (UID: \"c8d1da87-af98-4f6b-af6a-27a5fbabba60\") " pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:46:39.369324 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:39.369273    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 15 10:46:39.369444 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:39.369358    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37ad4cef-91aa-459c-80f3-f67d36e697aa-original-pull-secret podName:37ad4cef-91aa-459c-80f3-f67d36e697aa nodeName:}" failed. No retries permitted until 2026-07-15 10:46:55.369335249 +0000 UTC m=+34.198850656 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37ad4cef-91aa-459c-80f3-f67d36e697aa-original-pull-secret") pod "global-pull-secret-syncer-rx95w" (UID: "37ad4cef-91aa-459c-80f3-f67d36e697aa") : object "kube-system"/"original-pull-secret" not registered
Jul 15 10:46:39.369444 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:39.369276    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:46:39.369547 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:39.369462    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8d1da87-af98-4f6b-af6a-27a5fbabba60-metrics-certs podName:c8d1da87-af98-4f6b-af6a-27a5fbabba60 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:55.36944022 +0000 UTC m=+34.198955617 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c8d1da87-af98-4f6b-af6a-27a5fbabba60-metrics-certs") pod "network-metrics-daemon-84n76" (UID: "c8d1da87-af98-4f6b-af6a-27a5fbabba60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:46:39.673154 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:39.673054    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:46:39.673331 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:39.673054    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:46:39.673331 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:39.673185    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx95w" podUID="37ad4cef-91aa-459c-80f3-f67d36e697aa"
Jul 15 10:46:39.673331 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:39.673069    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:46:39.673331 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:39.673253    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-84n76" podUID="c8d1da87-af98-4f6b-af6a-27a5fbabba60"
Jul 15 10:46:39.673485 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:39.673339    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mhqc" podUID="b5f21679-bed4-4e21-9a3b-22ec261aec90"
Jul 15 10:46:41.673978 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:41.673941    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:46:41.673978 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:41.673965    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:46:41.674501 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:41.674093    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:46:41.674501 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:41.674093    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-84n76" podUID="c8d1da87-af98-4f6b-af6a-27a5fbabba60"
Jul 15 10:46:41.674501 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:41.674200    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mhqc" podUID="b5f21679-bed4-4e21-9a3b-22ec261aec90"
Jul 15 10:46:41.674501 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:41.674284    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx95w" podUID="37ad4cef-91aa-459c-80f3-f67d36e697aa"
Jul 15 10:46:43.673501 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:43.673461    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:46:43.674056 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:43.673469    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:46:43.674056 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:43.673600    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mhqc" podUID="b5f21679-bed4-4e21-9a3b-22ec261aec90"
Jul 15 10:46:43.674056 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:43.673469    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:46:43.674056 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:43.673708    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx95w" podUID="37ad4cef-91aa-459c-80f3-f67d36e697aa"
Jul 15 10:46:43.674056 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:43.673869    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-84n76" podUID="c8d1da87-af98-4f6b-af6a-27a5fbabba60"
Jul 15 10:46:44.853521 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:44.853189    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fcnjt" event={"ID":"cae13509-00de-48df-ae3d-943755c4b2e9","Type":"ContainerStarted","Data":"77fc9b1338898ff1681844be22f8b1c679ec0c018183ef03e89ec3465b54fc2a"}
Jul 15 10:46:44.855096 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:44.855067    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxh9r" event={"ID":"c40e765d-670d-40c3-925d-9585c28cbe72","Type":"ContainerStarted","Data":"e422843484325b3efc4c1ad85f6b6f105634b81a891489d113e2e6df61dc041d"}
Jul 15 10:46:44.856645 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:44.856615    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wdxpd" event={"ID":"eaebac66-19db-44df-85f4-887969b5984e","Type":"ContainerStarted","Data":"c97a7ed4716266ff7825f7ffad3093b365990354a3177da59eaf5a95eb14f239"}
Jul 15 10:46:44.858556 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:44.858519    2563 generic.go:356] "Generic (PLEG): container finished" podID="902fd91e-2979-4d2c-b428-ff5e62d8203b" containerID="edeb45c889e139a842b30ebb772a738ad9051a45d1c616d5ab8447640f92640b" exitCode=0
Jul 15 10:46:44.858681 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:44.858605    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l8v8c" event={"ID":"902fd91e-2979-4d2c-b428-ff5e62d8203b","Type":"ContainerDied","Data":"edeb45c889e139a842b30ebb772a738ad9051a45d1c616d5ab8447640f92640b"}
Jul 15 10:46:44.860510 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:44.860479    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-klpd8" event={"ID":"7e4a1af3-5ae7-495e-bd96-25df977fef6f","Type":"ContainerStarted","Data":"4705c4cc6f72f69e86203cbf3b0526215fb6840740dfcb5a22a5cd33f877cd7b"}
Jul 15 10:46:44.867264 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:44.867237    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk49n_beb68899-ffb3-436f-8e17-43597aad4030/ovn-acl-logging/0.log"
Jul 15 10:46:44.867653 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:44.867620    2563 generic.go:356] "Generic (PLEG): container finished" podID="beb68899-ffb3-436f-8e17-43597aad4030" containerID="16b795bc42da196edf305988ddd8b30517bfe1bd4c5141ba281cc94ab7386161" exitCode=1
Jul 15 10:46:44.867758 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:44.867696    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rk49n" event={"ID":"beb68899-ffb3-436f-8e17-43597aad4030","Type":"ContainerStarted","Data":"82991807dcb857d00b7bd696389e07d8a96dc81dba6682d220236e711252fdb2"}
Jul 15 10:46:44.867825 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:44.867792    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rk49n" event={"ID":"beb68899-ffb3-436f-8e17-43597aad4030","Type":"ContainerStarted","Data":"0c38a194256ff29b71baf289a228a9cf2e099fb6529313fcb48b6d1c53595e53"}
Jul 15 10:46:44.867875 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:44.867843    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rk49n" event={"ID":"beb68899-ffb3-436f-8e17-43597aad4030","Type":"ContainerStarted","Data":"256107711c40646be7f8afdebd5d04e1c60e9c2e60d67ba99392416ee14c338b"}
Jul 15 10:46:44.867912 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:44.867891    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rk49n" event={"ID":"beb68899-ffb3-436f-8e17-43597aad4030","Type":"ContainerStarted","Data":"032c3452f0e9053acae16c0c82c84e72d4a7a58081452096c158e240a8882874"}
Jul 15 10:46:44.867952 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:44.867940    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rk49n" event={"ID":"beb68899-ffb3-436f-8e17-43597aad4030","Type":"ContainerDied","Data":"16b795bc42da196edf305988ddd8b30517bfe1bd4c5141ba281cc94ab7386161"}
Jul 15 10:46:44.868040 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:44.868025    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rk49n" event={"ID":"beb68899-ffb3-436f-8e17-43597aad4030","Type":"ContainerStarted","Data":"29ae9412f98d74775cb05278c42db97aa766e7e1787ba74d8866a11e6606d51f"}
Jul 15 10:46:44.869448 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:44.869422    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-b4gsj" event={"ID":"2a7699c1-9758-4c35-8b4c-aed46512560e","Type":"ContainerStarted","Data":"45ced42e5fb0d6c6cae8776222ee0f9e9934f2b36f079cdda28bd0ce01ac1a84"}
Jul 15 10:46:44.871389 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:44.871362    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4445h" event={"ID":"2456e018-71e0-42bc-8153-64c597b91f17","Type":"ContainerStarted","Data":"82943820740b899b57a5845a18d64b177fc433c956323c7b845a6c4625371f23"}
Jul 15 10:46:44.875341 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:44.875284    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-34.ec2.internal" podStartSLOduration=22.875266321 podStartE2EDuration="22.875266321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:46:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:26.832308921 +0000 UTC m=+5.661824331" watchObservedRunningTime="2026-07-15 10:46:44.875266321 +0000 UTC m=+23.704781726"
Jul 15 10:46:44.875522 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:44.875444    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fcnjt" podStartSLOduration=3.572880326 podStartE2EDuration="22.875439485s" totalImagesPullingTime="19.302559159s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:44.874519929 +0000 UTC m=+23.704035334" watchObservedRunningTime="2026-07-15 10:46:44.875439485 +0000 UTC m=+23.704954889"
Jul 15 10:46:44.896922 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:44.896854    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wdxpd" podStartSLOduration=3.658038925 podStartE2EDuration="22.896830433s" totalImagesPullingTime="19.238791508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:44.89639283 +0000 UTC m=+23.725908235" watchObservedRunningTime="2026-07-15 10:46:44.896830433 +0000 UTC m=+23.726345838"
Jul 15 10:46:44.935465 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:44.935415    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-b4gsj" podStartSLOduration=3.6919254500000003 podStartE2EDuration="22.935397992s" totalImagesPullingTime="19.243472542s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:44.935272258 +0000 UTC m=+23.764787664" watchObservedRunningTime="2026-07-15 10:46:44.935397992 +0000 UTC m=+23.764913396"
Jul 15 10:46:44.951985 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:44.951957    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-b4gsj_2a7699c1-9758-4c35-8b4c-aed46512560e/node-ca/0.log"
Jul 15 10:46:44.954872 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:44.954819    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4445h" podStartSLOduration=3.709577513 podStartE2EDuration="22.954803311s" totalImagesPullingTime="19.245225798s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:44.954370731 +0000 UTC m=+23.783886135" watchObservedRunningTime="2026-07-15 10:46:44.954803311 +0000 UTC m=+23.784318717"
Jul 15 10:46:44.971595 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:44.971536    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-klpd8" podStartSLOduration=3.714111318 podStartE2EDuration="22.971517799s" totalImagesPullingTime="19.257406481s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:44.971314316 +0000 UTC m=+23.800829722" watchObservedRunningTime="2026-07-15 10:46:44.971517799 +0000 UTC m=+23.801033203"
Jul 15 10:46:45.490807 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:45.490758    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wdxpd_eaebac66-19db-44df-85f4-887969b5984e/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:45.593515 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:45.593440    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wdxpd_eaebac66-19db-44df-85f4-887969b5984e/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:45.646106 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:45.646051    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 15 10:46:45.673255 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:45.673221    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:46:45.673255 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:45.673236    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:46:45.673498 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:45.673222    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:46:45.673498 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:45.673336    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx95w" podUID="37ad4cef-91aa-459c-80f3-f67d36e697aa"
Jul 15 10:46:45.673498 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:45.673417    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mhqc" podUID="b5f21679-bed4-4e21-9a3b-22ec261aec90"
Jul 15 10:46:45.673611 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:45.673522    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-84n76" podUID="c8d1da87-af98-4f6b-af6a-27a5fbabba60"
Jul 15 10:46:45.875724 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:45.875626    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4jcc7" event={"ID":"2f949cb2-6105-448d-b4df-82ab30888fef","Type":"ContainerStarted","Data":"de7d6754f3c608541bfa2bb00024fb3844edb71db3665ea3499658323733889c"}
Jul 15 10:46:45.877724 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:45.877684    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxh9r" event={"ID":"c40e765d-670d-40c3-925d-9585c28cbe72","Type":"ContainerStarted","Data":"2413540237d0f2a486aad722e3e359c15f6a759a751ea3135f76e1f9baa13ed7"}
Jul 15 10:46:45.902088 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:45.901978    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4jcc7" podStartSLOduration=4.657542124 podStartE2EDuration="23.901952431s" totalImagesPullingTime="19.244410307s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:45.901573121 +0000 UTC m=+24.731088529" watchObservedRunningTime="2026-07-15 10:46:45.901952431 +0000 UTC m=+24.731467837"
Jul 15 10:46:46.022993 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:46.022960    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wdxpd_eaebac66-19db-44df-85f4-887969b5984e/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:46.574629 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:46.574514    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-15T10:46:45.646076998Z","UUID":"9018163f-567a-4302-b5e8-4c795d1097d1","Handler":null,"Name":"","Endpoint":""}
Jul 15 10:46:46.576838 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:46.576689    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 15 10:46:46.576838 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:46.576741    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 15 10:46:46.885175 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:46.885125    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk49n_beb68899-ffb3-436f-8e17-43597aad4030/ovn-acl-logging/0.log"
Jul 15 10:46:46.885860 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:46.885829    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rk49n" event={"ID":"beb68899-ffb3-436f-8e17-43597aad4030","Type":"ContainerStarted","Data":"c9687ad2e7b4dd6ac5a2879dcb31b209ea7bdc7873fabeb98a4c23e067e68583"}
Jul 15 10:46:47.673199 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:47.673164    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:46:47.673199 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:47.673190    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:46:47.673431 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:47.673164    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:46:47.673431 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:47.673294    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mhqc" podUID="b5f21679-bed4-4e21-9a3b-22ec261aec90"
Jul 15 10:46:47.673431 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:47.673360    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx95w" podUID="37ad4cef-91aa-459c-80f3-f67d36e697aa"
Jul 15 10:46:47.673431 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:47.673421    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-84n76" podUID="c8d1da87-af98-4f6b-af6a-27a5fbabba60"
Jul 15 10:46:48.008042 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:48.007994    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wdxpd"
Jul 15 10:46:48.008712 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:48.008684    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wdxpd"
Jul 15 10:46:48.891634 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:48.891600    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wdxpd"
Jul 15 10:46:48.892185 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:48.892160    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wdxpd"
Jul 15 10:46:49.673811 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:49.673554    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:46:49.674752 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:49.673552    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:46:49.674752 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:49.674059    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-84n76" podUID="c8d1da87-af98-4f6b-af6a-27a5fbabba60"
Jul 15 10:46:49.674752 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:49.673923    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx95w" podUID="37ad4cef-91aa-459c-80f3-f67d36e697aa"
Jul 15 10:46:49.674752 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:49.673679    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:46:49.674752 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:49.674182    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mhqc" podUID="b5f21679-bed4-4e21-9a3b-22ec261aec90"
Jul 15 10:46:49.895911 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:49.895877    2563 generic.go:356] "Generic (PLEG): container finished" podID="902fd91e-2979-4d2c-b428-ff5e62d8203b" containerID="52ff4bad98b2e397bba407d5e15f5c1423040c34b2e6d2121e689cda28a8c31b" exitCode=0
Jul 15 10:46:49.896059 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:49.895957    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l8v8c" event={"ID":"902fd91e-2979-4d2c-b428-ff5e62d8203b","Type":"ContainerDied","Data":"52ff4bad98b2e397bba407d5e15f5c1423040c34b2e6d2121e689cda28a8c31b"}
Jul 15 10:46:49.901708 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:49.901678    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk49n_beb68899-ffb3-436f-8e17-43597aad4030/ovn-acl-logging/0.log"
Jul 15 10:46:49.902348 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:49.902321    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rk49n" event={"ID":"beb68899-ffb3-436f-8e17-43597aad4030","Type":"ContainerStarted","Data":"c723b842b3c5a16bfea5252dfa4816611130ccc1351b1f806c93a14d64f5ea6e"}
Jul 15 10:46:49.902612 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:49.902592    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:49.902804 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:49.902782    2563 scope.go:117] "RemoveContainer" containerID="16b795bc42da196edf305988ddd8b30517bfe1bd4c5141ba281cc94ab7386161"
Jul 15 10:46:49.905204 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:49.905130    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxh9r" event={"ID":"c40e765d-670d-40c3-925d-9585c28cbe72","Type":"ContainerStarted","Data":"c5246d01fda7153c9a6da2fd1e727ae979000d49826a61cc69734ddadaa54357"}
Jul 15 10:46:49.921517 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:49.921490    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:49.966379 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:49.966322    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxh9r" podStartSLOduration=3.516942932 podStartE2EDuration="27.966304998s" totalImagesPullingTime="24.449362066s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:46:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:49.965706412 +0000 UTC m=+28.795221831" watchObservedRunningTime="2026-07-15 10:46:49.966304998 +0000 UTC m=+28.795820403"
Jul 15 10:46:50.915254 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:50.914940    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk49n_beb68899-ffb3-436f-8e17-43597aad4030/ovn-acl-logging/0.log"
Jul 15 10:46:50.915750 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:50.915657    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rk49n" event={"ID":"beb68899-ffb3-436f-8e17-43597aad4030","Type":"ContainerStarted","Data":"4fadd9215883a88f8e9f9e0da40c318678ca0c24dac7fec85546408f376bb8e3"}
Jul 15 10:46:50.915816 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:50.915781    2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 15 10:46:50.916222 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:50.916198    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:50.944428 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:50.937227    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:50.946585 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:50.946533    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rk49n" podStartSLOduration=9.608751803 podStartE2EDuration="28.946514554s" totalImagesPullingTime="19.337762751s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:50.944917568 +0000 UTC m=+29.774432974" watchObservedRunningTime="2026-07-15 10:46:50.946514554 +0000 UTC m=+29.776029960"
Jul 15 10:46:51.673624 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:51.673593    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:46:51.673805 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:51.673706    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:46:51.673805 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:51.673717    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-84n76" podUID="c8d1da87-af98-4f6b-af6a-27a5fbabba60"
Jul 15 10:46:51.673921 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:51.673848    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mhqc" podUID="b5f21679-bed4-4e21-9a3b-22ec261aec90"
Jul 15 10:46:51.673971 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:51.673933    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:46:51.674054 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:51.674029    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx95w" podUID="37ad4cef-91aa-459c-80f3-f67d36e697aa"
Jul 15 10:46:51.782897 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:51.782861    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rx95w"]
Jul 15 10:46:51.785480 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:51.785447    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-84n76"]
Jul 15 10:46:51.801335 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:51.801298    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9mhqc"]
Jul 15 10:46:51.920896 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:51.920810    2563 generic.go:356] "Generic (PLEG): container finished" podID="902fd91e-2979-4d2c-b428-ff5e62d8203b" containerID="dddab9186d9632173c3b026fc10d3354f365d963c97e887823721d5d65dfb0c4" exitCode=0
Jul 15 10:46:51.921734 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:51.920920    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l8v8c" event={"ID":"902fd91e-2979-4d2c-b428-ff5e62d8203b","Type":"ContainerDied","Data":"dddab9186d9632173c3b026fc10d3354f365d963c97e887823721d5d65dfb0c4"}
Jul 15 10:46:51.921734 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:51.921032    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:46:51.921734 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:51.921183    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx95w" podUID="37ad4cef-91aa-459c-80f3-f67d36e697aa"
Jul 15 10:46:51.921734 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:51.921293    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:46:51.921734 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:51.921300    2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 15 10:46:51.921734 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:51.921405    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:46:51.921734 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:51.921407    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mhqc" podUID="b5f21679-bed4-4e21-9a3b-22ec261aec90"
Jul 15 10:46:51.922023 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:51.921890    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-84n76" podUID="c8d1da87-af98-4f6b-af6a-27a5fbabba60"
Jul 15 10:46:52.923687 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:52.923653    2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 15 10:46:53.524996 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:53.524809    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:46:53.673253 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:53.673222    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:46:53.673421 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:53.673225    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:46:53.673421 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:53.673330    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mhqc" podUID="b5f21679-bed4-4e21-9a3b-22ec261aec90"
Jul 15 10:46:53.673421 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:53.673234    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:46:53.673519 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:53.673422    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-84n76" podUID="c8d1da87-af98-4f6b-af6a-27a5fbabba60"
Jul 15 10:46:53.673519 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:53.673469    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx95w" podUID="37ad4cef-91aa-459c-80f3-f67d36e697aa"
Jul 15 10:46:53.928498 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:53.928460    2563 generic.go:356] "Generic (PLEG): container finished" podID="902fd91e-2979-4d2c-b428-ff5e62d8203b" containerID="8e66350b70231d599848bbf357076c5c61ae013e2562d9b209c5dc3b49a6e96e" exitCode=0
Jul 15 10:46:53.928908 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:53.928523    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l8v8c" event={"ID":"902fd91e-2979-4d2c-b428-ff5e62d8203b","Type":"ContainerDied","Data":"8e66350b70231d599848bbf357076c5c61ae013e2562d9b209c5dc3b49a6e96e"}
Jul 15 10:46:55.293536 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:55.293490    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rwgk\" (UniqueName: \"kubernetes.io/projected/b5f21679-bed4-4e21-9a3b-22ec261aec90-kube-api-access-4rwgk\") pod \"network-check-target-9mhqc\" (UID: \"b5f21679-bed4-4e21-9a3b-22ec261aec90\") " pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:46:55.293999 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:55.293696    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 15 10:46:55.293999 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:55.293723    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 15 10:46:55.293999 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:55.293737    2563 projected.go:196] Error preparing data for projected volume kube-api-access-4rwgk for pod openshift-network-diagnostics/network-check-target-9mhqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:46:55.293999 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:55.293807    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5f21679-bed4-4e21-9a3b-22ec261aec90-kube-api-access-4rwgk podName:b5f21679-bed4-4e21-9a3b-22ec261aec90 nodeName:}" failed. No retries permitted until 2026-07-15 10:47:27.293785879 +0000 UTC m=+66.123301276 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4rwgk" (UniqueName: "kubernetes.io/projected/b5f21679-bed4-4e21-9a3b-22ec261aec90-kube-api-access-4rwgk") pod "network-check-target-9mhqc" (UID: "b5f21679-bed4-4e21-9a3b-22ec261aec90") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:46:55.393883 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:55.393845    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ad4cef-91aa-459c-80f3-f67d36e697aa-original-pull-secret\") pod \"global-pull-secret-syncer-rx95w\" (UID: \"37ad4cef-91aa-459c-80f3-f67d36e697aa\") " pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:46:55.394068 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:55.393911    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8d1da87-af98-4f6b-af6a-27a5fbabba60-metrics-certs\") pod \"network-metrics-daemon-84n76\" (UID: \"c8d1da87-af98-4f6b-af6a-27a5fbabba60\") " pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:46:55.394068 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:55.394011    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 15 10:46:55.394193 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:55.394084    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37ad4cef-91aa-459c-80f3-f67d36e697aa-original-pull-secret podName:37ad4cef-91aa-459c-80f3-f67d36e697aa nodeName:}" failed. No retries permitted until 2026-07-15 10:47:27.394065295 +0000 UTC m=+66.223580695 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37ad4cef-91aa-459c-80f3-f67d36e697aa-original-pull-secret") pod "global-pull-secret-syncer-rx95w" (UID: "37ad4cef-91aa-459c-80f3-f67d36e697aa") : object "kube-system"/"original-pull-secret" not registered
Jul 15 10:46:55.394193 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:55.394011    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:46:55.394298 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:55.394212    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8d1da87-af98-4f6b-af6a-27a5fbabba60-metrics-certs podName:c8d1da87-af98-4f6b-af6a-27a5fbabba60 nodeName:}" failed. No retries permitted until 2026-07-15 10:47:27.394186918 +0000 UTC m=+66.223702315 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c8d1da87-af98-4f6b-af6a-27a5fbabba60-metrics-certs") pod "network-metrics-daemon-84n76" (UID: "c8d1da87-af98-4f6b-af6a-27a5fbabba60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:46:55.673519 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:55.673435    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:46:55.673706 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:55.673446    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:46:55.673706 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:55.673577    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mhqc" podUID="b5f21679-bed4-4e21-9a3b-22ec261aec90"
Jul 15 10:46:55.673706 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:55.673446    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:46:55.673706 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:55.673654    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-84n76" podUID="c8d1da87-af98-4f6b-af6a-27a5fbabba60"
Jul 15 10:46:55.673892 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:55.673733    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx95w" podUID="37ad4cef-91aa-459c-80f3-f67d36e697aa"
Jul 15 10:46:57.678347 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:57.678316    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:46:57.678799 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:57.678318    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:46:57.678799 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:57.678433    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx95w" podUID="37ad4cef-91aa-459c-80f3-f67d36e697aa"
Jul 15 10:46:57.678799 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:57.678536    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mhqc" podUID="b5f21679-bed4-4e21-9a3b-22ec261aec90"
Jul 15 10:46:57.678799 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:57.678318    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:46:57.678799 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:57.678650    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-84n76" podUID="c8d1da87-af98-4f6b-af6a-27a5fbabba60"
Jul 15 10:46:58.944733 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:58.944695    2563 generic.go:356] "Generic (PLEG): container finished" podID="902fd91e-2979-4d2c-b428-ff5e62d8203b" containerID="e8476d2036831870ea73f2d216018985d0b076fe432fbf00b1e1304072b74ab6" exitCode=0
Jul 15 10:46:58.945155 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:58.944740    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l8v8c" event={"ID":"902fd91e-2979-4d2c-b428-ff5e62d8203b","Type":"ContainerDied","Data":"e8476d2036831870ea73f2d216018985d0b076fe432fbf00b1e1304072b74ab6"}
Jul 15 10:46:59.673346 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:59.673307    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:46:59.673508 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:59.673324    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:46:59.673508 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:59.673419    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-84n76" podUID="c8d1da87-af98-4f6b-af6a-27a5fbabba60"
Jul 15 10:46:59.673508 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:59.673324    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:46:59.673642 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:59.673525    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx95w" podUID="37ad4cef-91aa-459c-80f3-f67d36e697aa"
Jul 15 10:46:59.673642 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:46:59.673561    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mhqc" podUID="b5f21679-bed4-4e21-9a3b-22ec261aec90"
Jul 15 10:46:59.950414 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:59.950376    2563 generic.go:356] "Generic (PLEG): container finished" podID="902fd91e-2979-4d2c-b428-ff5e62d8203b" containerID="9da7409f7720cb3f35c6a5033b99dcb3c845a6b7d24802e31ebb1bf9a162aa28" exitCode=0
Jul 15 10:46:59.950947 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:46:59.950442    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l8v8c" event={"ID":"902fd91e-2979-4d2c-b428-ff5e62d8203b","Type":"ContainerDied","Data":"9da7409f7720cb3f35c6a5033b99dcb3c845a6b7d24802e31ebb1bf9a162aa28"}
Jul 15 10:47:00.956079 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:00.956045    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l8v8c" event={"ID":"902fd91e-2979-4d2c-b428-ff5e62d8203b","Type":"ContainerStarted","Data":"80430cf5b390dee32184369bc5f3385714a3bd5689dc4f4b0b80e7ef047ed882"}
Jul 15 10:47:00.981202 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:00.981125    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-l8v8c" podStartSLOduration=8.165305403 podStartE2EDuration="38.981110348s" totalImagesPullingTime="30.815804945s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:46:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:00.980863898 +0000 UTC m=+39.810379304" watchObservedRunningTime="2026-07-15 10:47:00.981110348 +0000 UTC m=+39.810625753"
Jul 15 10:47:01.673401 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:01.673124    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:47:01.673548 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:01.673189    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:47:01.673548 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:47:01.673422    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mhqc" podUID="b5f21679-bed4-4e21-9a3b-22ec261aec90"
Jul 15 10:47:01.673548 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:01.673218    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:47:01.673548 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:47:01.673512    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx95w" podUID="37ad4cef-91aa-459c-80f3-f67d36e697aa"
Jul 15 10:47:01.673770 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:47:01.673638    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-84n76" podUID="c8d1da87-af98-4f6b-af6a-27a5fbabba60"
Jul 15 10:47:03.672637 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:03.672600    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:47:03.673086 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:03.672600    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:47:03.673086 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:47:03.672715    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-84n76" podUID="c8d1da87-af98-4f6b-af6a-27a5fbabba60"
Jul 15 10:47:03.673086 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:47:03.672807    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx95w" podUID="37ad4cef-91aa-459c-80f3-f67d36e697aa"
Jul 15 10:47:03.673086 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:03.672610    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:47:03.673086 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:47:03.672910    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mhqc" podUID="b5f21679-bed4-4e21-9a3b-22ec261aec90"
Jul 15 10:47:05.672691 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:05.672662    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:47:05.673164 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:05.672691    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:47:05.673164 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:47:05.672778    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mhqc" podUID="b5f21679-bed4-4e21-9a3b-22ec261aec90"
Jul 15 10:47:05.673164 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:47:05.672823    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-84n76" podUID="c8d1da87-af98-4f6b-af6a-27a5fbabba60"
Jul 15 10:47:05.673164 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:05.672852    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:47:05.673164 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:47:05.672954    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx95w" podUID="37ad4cef-91aa-459c-80f3-f67d36e697aa"
Jul 15 10:47:07.672723 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:07.672693    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:47:07.673208 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:07.672692    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:47:07.673208 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:47:07.672882    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mhqc" podUID="b5f21679-bed4-4e21-9a3b-22ec261aec90"
Jul 15 10:47:07.673208 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:47:07.672790    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-84n76" podUID="c8d1da87-af98-4f6b-af6a-27a5fbabba60"
Jul 15 10:47:07.673208 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:07.672939    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:47:07.673208 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:47:07.672990    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx95w" podUID="37ad4cef-91aa-459c-80f3-f67d36e697aa"
Jul 15 10:47:09.673349 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:09.673322    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:47:09.673349 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:09.673334    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:47:09.673686 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:09.673341    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:47:09.673686 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:47:09.673412    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mhqc" podUID="b5f21679-bed4-4e21-9a3b-22ec261aec90"
Jul 15 10:47:09.673686 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:47:09.673493    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-84n76" podUID="c8d1da87-af98-4f6b-af6a-27a5fbabba60"
Jul 15 10:47:09.673686 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:47:09.673570    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx95w" podUID="37ad4cef-91aa-459c-80f3-f67d36e697aa"
Jul 15 10:47:10.018584 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.018559    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-34.ec2.internal" event="NodeReady"
Jul 15 10:47:10.018741 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.018670    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 15 10:47:10.069871 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.069841    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7ltzc"]
Jul 15 10:47:10.105407 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.105369    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-f8rzs"]
Jul 15 10:47:10.105584 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.105512    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7ltzc"
Jul 15 10:47:10.109110 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.109092    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 15 10:47:10.109253 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.109208    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 15 10:47:10.109500 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.109482    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 15 10:47:10.109620 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.109604    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ngctx\""
Jul 15 10:47:10.120967 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.120910    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7ltzc"]
Jul 15 10:47:10.121057 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.120979    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f8rzs"
Jul 15 10:47:10.121057 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.120993    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-r52qn"]
Jul 15 10:47:10.123860 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.123842    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 15 10:47:10.124018 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.124004    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ghxdw\""
Jul 15 10:47:10.124191 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.124179    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 15 10:47:10.124270 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.124256    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 15 10:47:10.124397 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.124382    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 15 10:47:10.139006 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.138986    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f8rzs"]
Jul 15 10:47:10.139097 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.139028    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r52qn"]
Jul 15 10:47:10.139097 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.139092    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r52qn"
Jul 15 10:47:10.141968 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.141952    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 15 10:47:10.142071 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.141954    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sw5h4\""
Jul 15 10:47:10.142071 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.142042    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 15 10:47:10.201252 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.201232    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c72e99f-bc5f-4851-a890-2605a51de0e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8rzs\" (UID: \"8c72e99f-bc5f-4851-a890-2605a51de0e0\") " pod="openshift-insights/insights-runtime-extractor-f8rzs"
Jul 15 10:47:10.201346 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.201258    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c72e99f-bc5f-4851-a890-2605a51de0e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8rzs\" (UID: \"8c72e99f-bc5f-4851-a890-2605a51de0e0\") " pod="openshift-insights/insights-runtime-extractor-f8rzs"
Jul 15 10:47:10.201346 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.201278    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b38e6e47-2551-4b35-a480-cda496f5af1f-config-volume\") pod \"dns-default-r52qn\" (UID: \"b38e6e47-2551-4b35-a480-cda496f5af1f\") " pod="openshift-dns/dns-default-r52qn"
Jul 15 10:47:10.201346 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.201293    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8jxc\" (UniqueName: \"kubernetes.io/projected/b38e6e47-2551-4b35-a480-cda496f5af1f-kube-api-access-x8jxc\") pod \"dns-default-r52qn\" (UID: \"b38e6e47-2551-4b35-a480-cda496f5af1f\") " pod="openshift-dns/dns-default-r52qn"
Jul 15 10:47:10.201346 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.201345    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8c72e99f-bc5f-4851-a890-2605a51de0e0-crio-socket\") pod \"insights-runtime-extractor-f8rzs\" (UID: \"8c72e99f-bc5f-4851-a890-2605a51de0e0\") " pod="openshift-insights/insights-runtime-extractor-f8rzs"
Jul 15 10:47:10.201490 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.201371    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8c72e99f-bc5f-4851-a890-2605a51de0e0-data-volume\") pod \"insights-runtime-extractor-f8rzs\" (UID: \"8c72e99f-bc5f-4851-a890-2605a51de0e0\") " pod="openshift-insights/insights-runtime-extractor-f8rzs"
Jul 15 10:47:10.201490 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.201405    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rv8k\" (UniqueName: \"kubernetes.io/projected/92ddad92-166b-43ce-a067-0007c9f8b30d-kube-api-access-8rv8k\") pod \"ingress-canary-7ltzc\" (UID: \"92ddad92-166b-43ce-a067-0007c9f8b30d\") " pod="openshift-ingress-canary/ingress-canary-7ltzc"
Jul 15 10:47:10.201490 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.201421    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b38e6e47-2551-4b35-a480-cda496f5af1f-tmp-dir\") pod \"dns-default-r52qn\" (UID: \"b38e6e47-2551-4b35-a480-cda496f5af1f\") " pod="openshift-dns/dns-default-r52qn"
Jul 15 10:47:10.201490 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.201442    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92ddad92-166b-43ce-a067-0007c9f8b30d-cert\") pod \"ingress-canary-7ltzc\" (UID: \"92ddad92-166b-43ce-a067-0007c9f8b30d\") " pod="openshift-ingress-canary/ingress-canary-7ltzc"
Jul 15 10:47:10.201618 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.201525    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt4tr\" (UniqueName: \"kubernetes.io/projected/8c72e99f-bc5f-4851-a890-2605a51de0e0-kube-api-access-mt4tr\") pod \"insights-runtime-extractor-f8rzs\" (UID: \"8c72e99f-bc5f-4851-a890-2605a51de0e0\") " pod="openshift-insights/insights-runtime-extractor-f8rzs"
Jul 15 10:47:10.201618 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.201553    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b38e6e47-2551-4b35-a480-cda496f5af1f-metrics-tls\") pod \"dns-default-r52qn\" (UID: \"b38e6e47-2551-4b35-a480-cda496f5af1f\") " pod="openshift-dns/dns-default-r52qn"
Jul 15 10:47:10.302601 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.302532    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c72e99f-bc5f-4851-a890-2605a51de0e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8rzs\" (UID: \"8c72e99f-bc5f-4851-a890-2605a51de0e0\") " pod="openshift-insights/insights-runtime-extractor-f8rzs"
Jul 15 10:47:10.302601 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.302569    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c72e99f-bc5f-4851-a890-2605a51de0e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8rzs\" (UID: \"8c72e99f-bc5f-4851-a890-2605a51de0e0\") " pod="openshift-insights/insights-runtime-extractor-f8rzs"
Jul 15 10:47:10.302601 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.302591    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b38e6e47-2551-4b35-a480-cda496f5af1f-config-volume\") pod \"dns-default-r52qn\" (UID: \"b38e6e47-2551-4b35-a480-cda496f5af1f\") " pod="openshift-dns/dns-default-r52qn"
Jul 15 10:47:10.302816 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.302609    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8jxc\" (UniqueName: \"kubernetes.io/projected/b38e6e47-2551-4b35-a480-cda496f5af1f-kube-api-access-x8jxc\") pod \"dns-default-r52qn\" (UID: \"b38e6e47-2551-4b35-a480-cda496f5af1f\") " pod="openshift-dns/dns-default-r52qn"
Jul 15 10:47:10.302816 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.302645    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8c72e99f-bc5f-4851-a890-2605a51de0e0-crio-socket\") pod \"insights-runtime-extractor-f8rzs\" (UID: \"8c72e99f-bc5f-4851-a890-2605a51de0e0\") " pod="openshift-insights/insights-runtime-extractor-f8rzs"
Jul 15 10:47:10.302816 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.302684    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8c72e99f-bc5f-4851-a890-2605a51de0e0-data-volume\") pod \"insights-runtime-extractor-f8rzs\" (UID: \"8c72e99f-bc5f-4851-a890-2605a51de0e0\") " pod="openshift-insights/insights-runtime-extractor-f8rzs"
Jul 15 10:47:10.302816 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.302725    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rv8k\" (UniqueName: \"kubernetes.io/projected/92ddad92-166b-43ce-a067-0007c9f8b30d-kube-api-access-8rv8k\") pod \"ingress-canary-7ltzc\" (UID: \"92ddad92-166b-43ce-a067-0007c9f8b30d\") " pod="openshift-ingress-canary/ingress-canary-7ltzc"
Jul 15 10:47:10.302816 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.302751    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b38e6e47-2551-4b35-a480-cda496f5af1f-tmp-dir\") pod \"dns-default-r52qn\" (UID: \"b38e6e47-2551-4b35-a480-cda496f5af1f\") " pod="openshift-dns/dns-default-r52qn"
Jul 15 10:47:10.302816 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.302781    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92ddad92-166b-43ce-a067-0007c9f8b30d-cert\") pod \"ingress-canary-7ltzc\" (UID: \"92ddad92-166b-43ce-a067-0007c9f8b30d\") " pod="openshift-ingress-canary/ingress-canary-7ltzc"
Jul 15 10:47:10.303109 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.302825    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt4tr\" (UniqueName: \"kubernetes.io/projected/8c72e99f-bc5f-4851-a890-2605a51de0e0-kube-api-access-mt4tr\") pod \"insights-runtime-extractor-f8rzs\" (UID: \"8c72e99f-bc5f-4851-a890-2605a51de0e0\") " pod="openshift-insights/insights-runtime-extractor-f8rzs"
Jul 15 10:47:10.303109 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.302847    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b38e6e47-2551-4b35-a480-cda496f5af1f-metrics-tls\") pod \"dns-default-r52qn\" (UID: \"b38e6e47-2551-4b35-a480-cda496f5af1f\") " pod="openshift-dns/dns-default-r52qn"
Jul 15 10:47:10.303109 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.302908    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8c72e99f-bc5f-4851-a890-2605a51de0e0-crio-socket\") pod \"insights-runtime-extractor-f8rzs\" (UID: \"8c72e99f-bc5f-4851-a890-2605a51de0e0\") " pod="openshift-insights/insights-runtime-extractor-f8rzs"
Jul 15 10:47:10.303109 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.303068    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8c72e99f-bc5f-4851-a890-2605a51de0e0-data-volume\") pod \"insights-runtime-extractor-f8rzs\" (UID: \"8c72e99f-bc5f-4851-a890-2605a51de0e0\") " pod="openshift-insights/insights-runtime-extractor-f8rzs"
Jul 15 10:47:10.303364 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.303248    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b38e6e47-2551-4b35-a480-cda496f5af1f-config-volume\") pod \"dns-default-r52qn\" (UID: \"b38e6e47-2551-4b35-a480-cda496f5af1f\") " pod="openshift-dns/dns-default-r52qn"
Jul 15 10:47:10.303364 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.303342    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b38e6e47-2551-4b35-a480-cda496f5af1f-tmp-dir\") pod \"dns-default-r52qn\" (UID: \"b38e6e47-2551-4b35-a480-cda496f5af1f\") " pod="openshift-dns/dns-default-r52qn"
Jul 15 10:47:10.303472 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.303421    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c72e99f-bc5f-4851-a890-2605a51de0e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8rzs\" (UID: \"8c72e99f-bc5f-4851-a890-2605a51de0e0\") " pod="openshift-insights/insights-runtime-extractor-f8rzs"
Jul 15 10:47:10.306700 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.306678    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c72e99f-bc5f-4851-a890-2605a51de0e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8rzs\" (UID: \"8c72e99f-bc5f-4851-a890-2605a51de0e0\") " pod="openshift-insights/insights-runtime-extractor-f8rzs"
Jul 15 10:47:10.306785 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.306681    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b38e6e47-2551-4b35-a480-cda496f5af1f-metrics-tls\") pod \"dns-default-r52qn\" (UID: \"b38e6e47-2551-4b35-a480-cda496f5af1f\") " pod="openshift-dns/dns-default-r52qn"
Jul 15 10:47:10.306950 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.306929    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92ddad92-166b-43ce-a067-0007c9f8b30d-cert\") pod \"ingress-canary-7ltzc\" (UID: \"92ddad92-166b-43ce-a067-0007c9f8b30d\") " pod="openshift-ingress-canary/ingress-canary-7ltzc"
Jul 15 10:47:10.312290 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.312269    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8jxc\" (UniqueName: \"kubernetes.io/projected/b38e6e47-2551-4b35-a480-cda496f5af1f-kube-api-access-x8jxc\") pod \"dns-default-r52qn\" (UID: \"b38e6e47-2551-4b35-a480-cda496f5af1f\") " pod="openshift-dns/dns-default-r52qn"
Jul 15 10:47:10.312574 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.312540    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt4tr\" (UniqueName: \"kubernetes.io/projected/8c72e99f-bc5f-4851-a890-2605a51de0e0-kube-api-access-mt4tr\") pod \"insights-runtime-extractor-f8rzs\" (UID: \"8c72e99f-bc5f-4851-a890-2605a51de0e0\") " pod="openshift-insights/insights-runtime-extractor-f8rzs"
Jul 15 10:47:10.312668 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.312650    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rv8k\" (UniqueName: \"kubernetes.io/projected/92ddad92-166b-43ce-a067-0007c9f8b30d-kube-api-access-8rv8k\") pod \"ingress-canary-7ltzc\" (UID: \"92ddad92-166b-43ce-a067-0007c9f8b30d\") " pod="openshift-ingress-canary/ingress-canary-7ltzc"
Jul 15 10:47:10.414724 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.414699    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7ltzc"
Jul 15 10:47:10.429869 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.429850    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f8rzs"
Jul 15 10:47:10.447386 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.447371    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r52qn"
Jul 15 10:47:10.640124 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.640086    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7ltzc"]
Jul 15 10:47:10.644606 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:47:10.644572    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92ddad92_166b_43ce_a067_0007c9f8b30d.slice/crio-80fa0e0661aba96d3f809062d4b5b8ec7de618025c8991efc31a2832b010332b WatchSource:0}: Error finding container 80fa0e0661aba96d3f809062d4b5b8ec7de618025c8991efc31a2832b010332b: Status 404 returned error can't find the container with id 80fa0e0661aba96d3f809062d4b5b8ec7de618025c8991efc31a2832b010332b
Jul 15 10:47:10.656600 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.656570    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f8rzs"]
Jul 15 10:47:10.660563 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:47:10.660535    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c72e99f_bc5f_4851_a890_2605a51de0e0.slice/crio-8df27808a44fd8cebf3da6013c333e698671e194ffd227c9c19f7edb5698a99e WatchSource:0}: Error finding container 8df27808a44fd8cebf3da6013c333e698671e194ffd227c9c19f7edb5698a99e: Status 404 returned error can't find the container with id 8df27808a44fd8cebf3da6013c333e698671e194ffd227c9c19f7edb5698a99e
Jul 15 10:47:10.674765 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.674743    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r52qn"]
Jul 15 10:47:10.677370 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:47:10.677349    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb38e6e47_2551_4b35_a480_cda496f5af1f.slice/crio-e253c24cef86aba80e2a3ef1db30b78fc35a8948c7822a404b6f653cc5866377 WatchSource:0}: Error finding container e253c24cef86aba80e2a3ef1db30b78fc35a8948c7822a404b6f653cc5866377: Status 404 returned error can't find the container with id e253c24cef86aba80e2a3ef1db30b78fc35a8948c7822a404b6f653cc5866377
Jul 15 10:47:10.883259 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.883191    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5ff9595488-rmsmj"]
Jul 15 10:47:10.907580 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.907559    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ff9595488-rmsmj"]
Jul 15 10:47:10.907709 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.907644    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff9595488-rmsmj"
Jul 15 10:47:10.911030 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.911005    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 15 10:47:10.911204 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.911048    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bkt8n\""
Jul 15 10:47:10.911204 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.911112    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 15 10:47:10.912123 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.912100    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 15 10:47:10.912244 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.912129    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 15 10:47:10.912244 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.912167    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 15 10:47:10.912244 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.912214    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 15 10:47:10.912385 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.912259    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 15 10:47:10.983106 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.983083    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r52qn" event={"ID":"b38e6e47-2551-4b35-a480-cda496f5af1f","Type":"ContainerStarted","Data":"e253c24cef86aba80e2a3ef1db30b78fc35a8948c7822a404b6f653cc5866377"}
Jul 15 10:47:10.984176 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.984154    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f8rzs" event={"ID":"8c72e99f-bc5f-4851-a890-2605a51de0e0","Type":"ContainerStarted","Data":"8df27808a44fd8cebf3da6013c333e698671e194ffd227c9c19f7edb5698a99e"}
Jul 15 10:47:10.985187 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:10.985165    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7ltzc" event={"ID":"92ddad92-166b-43ce-a067-0007c9f8b30d","Type":"ContainerStarted","Data":"80fa0e0661aba96d3f809062d4b5b8ec7de618025c8991efc31a2832b010332b"}
Jul 15 10:47:11.006651 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:11.006628    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/482314fa-4e2c-45e3-b506-053bd6d63b6c-console-serving-cert\") pod \"console-5ff9595488-rmsmj\" (UID: \"482314fa-4e2c-45e3-b506-053bd6d63b6c\") " pod="openshift-console/console-5ff9595488-rmsmj"
Jul 15 10:47:11.006761 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:11.006654    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/482314fa-4e2c-45e3-b506-053bd6d63b6c-console-oauth-config\") pod \"console-5ff9595488-rmsmj\" (UID: \"482314fa-4e2c-45e3-b506-053bd6d63b6c\") " pod="openshift-console/console-5ff9595488-rmsmj"
Jul 15 10:47:11.006761 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:11.006695    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mbsk\" (UniqueName: \"kubernetes.io/projected/482314fa-4e2c-45e3-b506-053bd6d63b6c-kube-api-access-7mbsk\") pod \"console-5ff9595488-rmsmj\" (UID: \"482314fa-4e2c-45e3-b506-053bd6d63b6c\") " pod="openshift-console/console-5ff9595488-rmsmj"
Jul 15 10:47:11.006761 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:11.006743    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/482314fa-4e2c-45e3-b506-053bd6d63b6c-console-config\") pod \"console-5ff9595488-rmsmj\" (UID: \"482314fa-4e2c-45e3-b506-053bd6d63b6c\") " pod="openshift-console/console-5ff9595488-rmsmj"
Jul 15 10:47:11.006903 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:11.006791    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/482314fa-4e2c-45e3-b506-053bd6d63b6c-service-ca\") pod \"console-5ff9595488-rmsmj\" (UID: \"482314fa-4e2c-45e3-b506-053bd6d63b6c\") " pod="openshift-console/console-5ff9595488-rmsmj"
Jul 15 10:47:11.006903 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:11.006809    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/482314fa-4e2c-45e3-b506-053bd6d63b6c-oauth-serving-cert\") pod \"console-5ff9595488-rmsmj\" (UID: \"482314fa-4e2c-45e3-b506-053bd6d63b6c\") " pod="openshift-console/console-5ff9595488-rmsmj"
Jul 15 10:47:11.108007 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:11.107952    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/482314fa-4e2c-45e3-b506-053bd6d63b6c-service-ca\") pod \"console-5ff9595488-rmsmj\" (UID: \"482314fa-4e2c-45e3-b506-053bd6d63b6c\") " pod="openshift-console/console-5ff9595488-rmsmj"
Jul 15 10:47:11.108007 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:11.108004    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/482314fa-4e2c-45e3-b506-053bd6d63b6c-oauth-serving-cert\") pod \"console-5ff9595488-rmsmj\" (UID: \"482314fa-4e2c-45e3-b506-053bd6d63b6c\") " pod="openshift-console/console-5ff9595488-rmsmj"
Jul 15 10:47:11.108261 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:11.108061    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/482314fa-4e2c-45e3-b506-053bd6d63b6c-console-serving-cert\") pod \"console-5ff9595488-rmsmj\" (UID: \"482314fa-4e2c-45e3-b506-053bd6d63b6c\") " pod="openshift-console/console-5ff9595488-rmsmj"
Jul 15 10:47:11.108261 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:11.108084    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/482314fa-4e2c-45e3-b506-053bd6d63b6c-console-oauth-config\") pod \"console-5ff9595488-rmsmj\" (UID: \"482314fa-4e2c-45e3-b506-053bd6d63b6c\") " pod="openshift-console/console-5ff9595488-rmsmj"
Jul 15 10:47:11.108261 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:11.108152    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mbsk\" (UniqueName: \"kubernetes.io/projected/482314fa-4e2c-45e3-b506-053bd6d63b6c-kube-api-access-7mbsk\") pod \"console-5ff9595488-rmsmj\" (UID: \"482314fa-4e2c-45e3-b506-053bd6d63b6c\") " pod="openshift-console/console-5ff9595488-rmsmj"
Jul 15 10:47:11.108261 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:11.108180    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/482314fa-4e2c-45e3-b506-053bd6d63b6c-console-config\") pod \"console-5ff9595488-rmsmj\" (UID: \"482314fa-4e2c-45e3-b506-053bd6d63b6c\") " pod="openshift-console/console-5ff9595488-rmsmj"
Jul 15 10:47:11.108786 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:11.108755    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/482314fa-4e2c-45e3-b506-053bd6d63b6c-service-ca\") pod \"console-5ff9595488-rmsmj\" (UID: \"482314fa-4e2c-45e3-b506-053bd6d63b6c\") " pod="openshift-console/console-5ff9595488-rmsmj"
Jul 15 10:47:11.108915 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:11.108882    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/482314fa-4e2c-45e3-b506-053bd6d63b6c-console-config\") pod \"console-5ff9595488-rmsmj\" (UID: \"482314fa-4e2c-45e3-b506-053bd6d63b6c\") " pod="openshift-console/console-5ff9595488-rmsmj"
Jul 15 10:47:11.109156 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:11.109115    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/482314fa-4e2c-45e3-b506-053bd6d63b6c-oauth-serving-cert\") pod \"console-5ff9595488-rmsmj\" (UID: \"482314fa-4e2c-45e3-b506-053bd6d63b6c\") " pod="openshift-console/console-5ff9595488-rmsmj"
Jul 15 10:47:11.121068 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:11.121047    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mbsk\" (UniqueName: \"kubernetes.io/projected/482314fa-4e2c-45e3-b506-053bd6d63b6c-kube-api-access-7mbsk\") pod \"console-5ff9595488-rmsmj\" (UID: \"482314fa-4e2c-45e3-b506-053bd6d63b6c\") " pod="openshift-console/console-5ff9595488-rmsmj"
Jul 15 10:47:11.127383 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:11.127357    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/482314fa-4e2c-45e3-b506-053bd6d63b6c-console-oauth-config\") pod \"console-5ff9595488-rmsmj\" (UID: \"482314fa-4e2c-45e3-b506-053bd6d63b6c\") " pod="openshift-console/console-5ff9595488-rmsmj"
Jul 15 10:47:11.127468 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:11.127431    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/482314fa-4e2c-45e3-b506-053bd6d63b6c-console-serving-cert\") pod \"console-5ff9595488-rmsmj\" (UID: \"482314fa-4e2c-45e3-b506-053bd6d63b6c\") " pod="openshift-console/console-5ff9595488-rmsmj"
Jul 15 10:47:11.216545 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:11.216513    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff9595488-rmsmj"
Jul 15 10:47:11.413068 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:11.412978    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ff9595488-rmsmj"]
Jul 15 10:47:11.423161 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:47:11.423103    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod482314fa_4e2c_45e3_b506_053bd6d63b6c.slice/crio-4d755bef17816e1d356400d283d9f3618a797532338ac25541cebf49cfd6c4b3 WatchSource:0}: Error finding container 4d755bef17816e1d356400d283d9f3618a797532338ac25541cebf49cfd6c4b3: Status 404 returned error can't find the container with id 4d755bef17816e1d356400d283d9f3618a797532338ac25541cebf49cfd6c4b3
Jul 15 10:47:11.678129 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:11.677932    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:47:11.678129 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:11.678116    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:47:11.678621 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:11.678125    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:47:11.681156 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:11.681119    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 15 10:47:11.685538 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:11.683965    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qpwxt\""
Jul 15 10:47:11.685538 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:11.685365    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 15 10:47:11.685538 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:11.685407    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 15 10:47:11.685763 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:11.685548    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 15 10:47:11.685763 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:11.685738    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tlzgw\""
Jul 15 10:47:11.991975 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:11.991863    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff9595488-rmsmj" event={"ID":"482314fa-4e2c-45e3-b506-053bd6d63b6c","Type":"ContainerStarted","Data":"4d755bef17816e1d356400d283d9f3618a797532338ac25541cebf49cfd6c4b3"}
Jul 15 10:47:19.014734 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:19.014696    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f8rzs" event={"ID":"8c72e99f-bc5f-4851-a890-2605a51de0e0","Type":"ContainerStarted","Data":"5c12899fc23d453d7e3b4d6e24e4359da1b128a0b687d2c3035bacd4a43cabf1"}
Jul 15 10:47:19.016204 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:19.016169    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7ltzc" event={"ID":"92ddad92-166b-43ce-a067-0007c9f8b30d","Type":"ContainerStarted","Data":"3b0ea712d386ae2772d89fd86538004e88fc13e27e141d1f4215b7982ec16c1b"}
Jul 15 10:47:19.017854 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:19.017819    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r52qn" event={"ID":"b38e6e47-2551-4b35-a480-cda496f5af1f","Type":"ContainerStarted","Data":"c5476e3f446e6c8432a3edc63888b88cc8cedd6e92ae2ca6da6bc13664c12280"}
Jul 15 10:47:19.017993 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:19.017885    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r52qn" event={"ID":"b38e6e47-2551-4b35-a480-cda496f5af1f","Type":"ContainerStarted","Data":"ea6721cb3cae7b1b527861430822b3bd30202e00b759b35f9484674f50b390f7"}
Jul 15 10:47:19.017993 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:19.017947    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-r52qn"
Jul 15 10:47:19.019372 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:19.019347    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff9595488-rmsmj" event={"ID":"482314fa-4e2c-45e3-b506-053bd6d63b6c","Type":"ContainerStarted","Data":"4c3fade0383aed5c22f5a4ce0a19d05d868837c5a53a462dedab93d9efcc8378"}
Jul 15 10:47:19.045945 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:19.045886    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7ltzc" podStartSLOduration=1.645404309 podStartE2EDuration="9.045867879s" totalImagesPullingTime="7.40046357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:47:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:19.045511642 +0000 UTC m=+57.875027046" watchObservedRunningTime="2026-07-15 10:47:19.045867879 +0000 UTC m=+57.875383284"
Jul 15 10:47:19.088757 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:19.088699    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5ff9595488-rmsmj" podStartSLOduration=2.467570418 podStartE2EDuration="9.08868234s" totalImagesPullingTime="6.621111922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:47:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:19.088038337 +0000 UTC m=+57.917553743" watchObservedRunningTime="2026-07-15 10:47:19.08868234 +0000 UTC m=+57.918197723"
Jul 15 10:47:19.151925 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:19.151865    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-r52qn" podStartSLOduration=1.7838944730000001 podStartE2EDuration="9.151845481s" totalImagesPullingTime="7.367951008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:47:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:19.146975978 +0000 UTC m=+57.976491384" watchObservedRunningTime="2026-07-15 10:47:19.151845481 +0000 UTC m=+57.981360884"
Jul 15 10:47:19.516889 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:19.516853    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8rwts"]
Jul 15 10:47:19.518862 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:19.518842    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8rwts"
Jul 15 10:47:19.523452 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:19.523423    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 15 10:47:19.523668 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:19.523651    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-vgsbx\""
Jul 15 10:47:19.533386 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:19.533342    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8rwts"]
Jul 15 10:47:19.579206 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:19.579160    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cf2605ca-f1b1-4931-8b11-c3badbcdb1cd-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-8rwts\" (UID: \"cf2605ca-f1b1-4931-8b11-c3badbcdb1cd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8rwts"
Jul 15 10:47:19.680396 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:19.680363    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cf2605ca-f1b1-4931-8b11-c3badbcdb1cd-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-8rwts\" (UID: \"cf2605ca-f1b1-4931-8b11-c3badbcdb1cd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8rwts"
Jul 15 10:47:19.680608 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:47:19.680523    2563 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 15 10:47:19.680608 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:47:19.680591    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf2605ca-f1b1-4931-8b11-c3badbcdb1cd-tls-certificates podName:cf2605ca-f1b1-4931-8b11-c3badbcdb1cd nodeName:}" failed. No retries permitted until 2026-07-15 10:47:20.180575364 +0000 UTC m=+59.010090751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/cf2605ca-f1b1-4931-8b11-c3badbcdb1cd-tls-certificates") pod "prometheus-operator-admission-webhook-77d98bc786-8rwts" (UID: "cf2605ca-f1b1-4931-8b11-c3badbcdb1cd") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 15 10:47:20.185264 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:20.185211    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cf2605ca-f1b1-4931-8b11-c3badbcdb1cd-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-8rwts\" (UID: \"cf2605ca-f1b1-4931-8b11-c3badbcdb1cd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8rwts"
Jul 15 10:47:20.187877 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:20.187845    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cf2605ca-f1b1-4931-8b11-c3badbcdb1cd-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-8rwts\" (UID: \"cf2605ca-f1b1-4931-8b11-c3badbcdb1cd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8rwts"
Jul 15 10:47:20.427684 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:20.427633    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8rwts"
Jul 15 10:47:20.569720 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:20.569679    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8rwts"]
Jul 15 10:47:20.573127 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:47:20.573089    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf2605ca_f1b1_4931_8b11_c3badbcdb1cd.slice/crio-dc3dd90c4fb21dc30176e8d4197d93095620ea030af8e5e4255b24212e9a2edd WatchSource:0}: Error finding container dc3dd90c4fb21dc30176e8d4197d93095620ea030af8e5e4255b24212e9a2edd: Status 404 returned error can't find the container with id dc3dd90c4fb21dc30176e8d4197d93095620ea030af8e5e4255b24212e9a2edd
Jul 15 10:47:21.031471 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:21.031433    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8rwts" event={"ID":"cf2605ca-f1b1-4931-8b11-c3badbcdb1cd","Type":"ContainerStarted","Data":"dc3dd90c4fb21dc30176e8d4197d93095620ea030af8e5e4255b24212e9a2edd"}
Jul 15 10:47:21.084776 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:21.084507    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-88d4d74bd-v4zn7"]
Jul 15 10:47:21.102176 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:21.102119    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-88d4d74bd-v4zn7"]
Jul 15 10:47:21.102374 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:21.102263    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88d4d74bd-v4zn7"
Jul 15 10:47:21.113215 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:21.113187    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 15 10:47:21.191261 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:21.191217    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a522d3d-13c9-4a9c-949d-f30304744f77-console-serving-cert\") pod \"console-88d4d74bd-v4zn7\" (UID: \"3a522d3d-13c9-4a9c-949d-f30304744f77\") " pod="openshift-console/console-88d4d74bd-v4zn7"
Jul 15 10:47:21.191261 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:21.191274    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a522d3d-13c9-4a9c-949d-f30304744f77-console-config\") pod \"console-88d4d74bd-v4zn7\" (UID: \"3a522d3d-13c9-4a9c-949d-f30304744f77\") " pod="openshift-console/console-88d4d74bd-v4zn7"
Jul 15 10:47:21.191801 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:21.191303    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xmwv\" (UniqueName: \"kubernetes.io/projected/3a522d3d-13c9-4a9c-949d-f30304744f77-kube-api-access-6xmwv\") pod \"console-88d4d74bd-v4zn7\" (UID: \"3a522d3d-13c9-4a9c-949d-f30304744f77\") " pod="openshift-console/console-88d4d74bd-v4zn7"
Jul 15 10:47:21.191801 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:21.191337    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a522d3d-13c9-4a9c-949d-f30304744f77-console-oauth-config\") pod \"console-88d4d74bd-v4zn7\" (UID: \"3a522d3d-13c9-4a9c-949d-f30304744f77\") " pod="openshift-console/console-88d4d74bd-v4zn7"
Jul 15 10:47:21.191801 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:21.191366    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a522d3d-13c9-4a9c-949d-f30304744f77-service-ca\") pod \"console-88d4d74bd-v4zn7\" (UID: \"3a522d3d-13c9-4a9c-949d-f30304744f77\") " pod="openshift-console/console-88d4d74bd-v4zn7"
Jul 15 10:47:21.191801 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:21.191418    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a522d3d-13c9-4a9c-949d-f30304744f77-oauth-serving-cert\") pod \"console-88d4d74bd-v4zn7\" (UID: \"3a522d3d-13c9-4a9c-949d-f30304744f77\") " pod="openshift-console/console-88d4d74bd-v4zn7"
Jul 15 10:47:21.191801 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:21.191462    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a522d3d-13c9-4a9c-949d-f30304744f77-trusted-ca-bundle\") pod \"console-88d4d74bd-v4zn7\" (UID: \"3a522d3d-13c9-4a9c-949d-f30304744f77\") " pod="openshift-console/console-88d4d74bd-v4zn7"
Jul 15 10:47:21.217289 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:21.217246    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5ff9595488-rmsmj"
Jul 15 10:47:21.217474 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:21.217413    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5ff9595488-rmsmj"
Jul 15 10:47:21.219088 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:21.219045    2563 patch_prober.go:28] interesting pod/console-5ff9595488-rmsmj container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.9:8443/health\": dial tcp 10.133.0.9:8443: connect: connection refused" start-of-body=
Jul 15 10:47:21.219289 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:21.219108    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-5ff9595488-rmsmj" podUID="482314fa-4e2c-45e3-b506-053bd6d63b6c" containerName="console" probeResult="failure" output="Get \"https://10.133.0.9:8443/health\": dial tcp 10.133.0.9:8443: connect: connection refused"
Jul 15 10:47:21.292761 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:21.292651    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a522d3d-13c9-4a9c-949d-f30304744f77-service-ca\") pod \"console-88d4d74bd-v4zn7\" (UID: \"3a522d3d-13c9-4a9c-949d-f30304744f77\") " pod="openshift-console/console-88d4d74bd-v4zn7"
Jul 15 10:47:21.292988 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:21.292784    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a522d3d-13c9-4a9c-949d-f30304744f77-oauth-serving-cert\") pod \"console-88d4d74bd-v4zn7\" (UID: \"3a522d3d-13c9-4a9c-949d-f30304744f77\") " pod="openshift-console/console-88d4d74bd-v4zn7"
Jul 15 10:47:21.292988 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:21.292832    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a522d3d-13c9-4a9c-949d-f30304744f77-trusted-ca-bundle\") pod \"console-88d4d74bd-v4zn7\" (UID: \"3a522d3d-13c9-4a9c-949d-f30304744f77\") " pod="openshift-console/console-88d4d74bd-v4zn7"
Jul 15 10:47:21.292988 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:21.292939    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a522d3d-13c9-4a9c-949d-f30304744f77-console-serving-cert\") pod \"console-88d4d74bd-v4zn7\" (UID: \"3a522d3d-13c9-4a9c-949d-f30304744f77\") " pod="openshift-console/console-88d4d74bd-v4zn7"
Jul 15 10:47:21.292988 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:21.292961    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a522d3d-13c9-4a9c-949d-f30304744f77-console-config\") pod \"console-88d4d74bd-v4zn7\" (UID: \"3a522d3d-13c9-4a9c-949d-f30304744f77\") " pod="openshift-console/console-88d4d74bd-v4zn7"
Jul 15 10:47:21.292988 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:21.292984    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xmwv\" (UniqueName: \"kubernetes.io/projected/3a522d3d-13c9-4a9c-949d-f30304744f77-kube-api-access-6xmwv\") pod \"console-88d4d74bd-v4zn7\" (UID: \"3a522d3d-13c9-4a9c-949d-f30304744f77\") " pod="openshift-console/console-88d4d74bd-v4zn7"
Jul 15 10:47:21.293325 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:21.293021    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a522d3d-13c9-4a9c-949d-f30304744f77-console-oauth-config\") pod \"console-88d4d74bd-v4zn7\" (UID: \"3a522d3d-13c9-4a9c-949d-f30304744f77\") " pod="openshift-console/console-88d4d74bd-v4zn7"
Jul 15 10:47:21.294259 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:21.293913    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a522d3d-13c9-4a9c-949d-f30304744f77-service-ca\") pod \"console-88d4d74bd-v4zn7\" (UID: \"3a522d3d-13c9-4a9c-949d-f30304744f77\") " pod="openshift-console/console-88d4d74bd-v4zn7"
Jul 15 10:47:21.294259 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:21.294013    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a522d3d-13c9-4a9c-949d-f30304744f77-oauth-serving-cert\") pod \"console-88d4d74bd-v4zn7\" (UID: \"3a522d3d-13c9-4a9c-949d-f30304744f77\") " pod="openshift-console/console-88d4d74bd-v4zn7"
Jul 15 10:47:21.294464 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:21.294435    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a522d3d-13c9-4a9c-949d-f30304744f77-console-config\") pod \"console-88d4d74bd-v4zn7\" (UID: \"3a522d3d-13c9-4a9c-949d-f30304744f77\") " pod="openshift-console/console-88d4d74bd-v4zn7"
Jul 15 10:47:21.294743 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:21.294692    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a522d3d-13c9-4a9c-949d-f30304744f77-trusted-ca-bundle\") pod \"console-88d4d74bd-v4zn7\" (UID: \"3a522d3d-13c9-4a9c-949d-f30304744f77\") " pod="openshift-console/console-88d4d74bd-v4zn7"
Jul 15 10:47:21.296047 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:21.296000    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a522d3d-13c9-4a9c-949d-f30304744f77-console-oauth-config\") pod \"console-88d4d74bd-v4zn7\" (UID: \"3a522d3d-13c9-4a9c-949d-f30304744f77\") " pod="openshift-console/console-88d4d74bd-v4zn7"
Jul 15 10:47:21.296356 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:21.296331    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a522d3d-13c9-4a9c-949d-f30304744f77-console-serving-cert\") pod \"console-88d4d74bd-v4zn7\" (UID: \"3a522d3d-13c9-4a9c-949d-f30304744f77\") " pod="openshift-console/console-88d4d74bd-v4zn7"
Jul 15 10:47:21.305548 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:21.305476    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xmwv\" (UniqueName: \"kubernetes.io/projected/3a522d3d-13c9-4a9c-949d-f30304744f77-kube-api-access-6xmwv\") pod \"console-88d4d74bd-v4zn7\" (UID: \"3a522d3d-13c9-4a9c-949d-f30304744f77\") " pod="openshift-console/console-88d4d74bd-v4zn7"
Jul 15 10:47:21.415664 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:21.415621    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88d4d74bd-v4zn7"
Jul 15 10:47:23.910624 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:23.910585    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-88d4d74bd-v4zn7"]
Jul 15 10:47:23.914841 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:47:23.914812    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a522d3d_13c9_4a9c_949d_f30304744f77.slice/crio-654986954ed09dda528b29a0e018174f700e7c8405dfd683ded597e5ab6f0a2f WatchSource:0}: Error finding container 654986954ed09dda528b29a0e018174f700e7c8405dfd683ded597e5ab6f0a2f: Status 404 returned error can't find the container with id 654986954ed09dda528b29a0e018174f700e7c8405dfd683ded597e5ab6f0a2f
Jul 15 10:47:23.954516 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:23.954479    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rk49n"
Jul 15 10:47:24.044434 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:24.044334    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f8rzs" event={"ID":"8c72e99f-bc5f-4851-a890-2605a51de0e0","Type":"ContainerStarted","Data":"2efdd66ce5278b709082b170eb5e63f0ebd79aac118402695aa8f04f3ded391c"}
Jul 15 10:47:24.045942 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:24.045904    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8rwts" event={"ID":"cf2605ca-f1b1-4931-8b11-c3badbcdb1cd","Type":"ContainerStarted","Data":"5ab66be182ac02de3b39867c1584e19936627c9c9fa22d9ce9ca25d7ae65c169"}
Jul 15 10:47:24.046130 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:24.046104    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8rwts"
Jul 15 10:47:24.047803 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:24.047773    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88d4d74bd-v4zn7" event={"ID":"3a522d3d-13c9-4a9c-949d-f30304744f77","Type":"ContainerStarted","Data":"b9bbf0793522807d7b81612f1805a46913fd3756eab86c224eb238777f42fee0"}
Jul 15 10:47:24.047943 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:24.047821    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88d4d74bd-v4zn7" event={"ID":"3a522d3d-13c9-4a9c-949d-f30304744f77","Type":"ContainerStarted","Data":"654986954ed09dda528b29a0e018174f700e7c8405dfd683ded597e5ab6f0a2f"}
Jul 15 10:47:24.051634 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:24.051604    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8rwts"
Jul 15 10:47:24.066378 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:24.066291    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8rwts" podStartSLOduration=1.875378939 podStartE2EDuration="5.066270152s" totalImagesPullingTime="3.190891213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:47:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:24.065347136 +0000 UTC m=+62.894862545" watchObservedRunningTime="2026-07-15 10:47:24.066270152 +0000 UTC m=+62.895785556"
Jul 15 10:47:24.086758 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:24.086692    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-88d4d74bd-v4zn7" podStartSLOduration=3.086672162 podStartE2EDuration="3.086672162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:24.08658692 +0000 UTC m=+62.916102326" watchObservedRunningTime="2026-07-15 10:47:24.086672162 +0000 UTC m=+62.916187570"
Jul 15 10:47:24.559109 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:24.559047    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-kd448"]
Jul 15 10:47:24.561381 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:24.561345    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-kd448"
Jul 15 10:47:24.564054 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:24.564029    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 15 10:47:24.564200 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:24.564032    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 15 10:47:24.564756 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:24.564738    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 15 10:47:24.565331 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:24.565295    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 15 10:47:24.565845 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:24.565819    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-ljr7d\""
Jul 15 10:47:24.565941 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:24.565867    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 15 10:47:24.571651 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:24.571615    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-kd448"]
Jul 15 10:47:24.621052 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:24.621016    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07d78928-8e15-4a72-93e3-af348e9593b2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-kd448\" (UID: \"07d78928-8e15-4a72-93e3-af348e9593b2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kd448"
Jul 15 10:47:24.621286 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:24.621082    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/07d78928-8e15-4a72-93e3-af348e9593b2-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-kd448\" (UID: \"07d78928-8e15-4a72-93e3-af348e9593b2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kd448"
Jul 15 10:47:24.621286 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:24.621103    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07d78928-8e15-4a72-93e3-af348e9593b2-metrics-client-ca\") pod \"prometheus-operator-64785b6675-kd448\" (UID: \"07d78928-8e15-4a72-93e3-af348e9593b2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kd448"
Jul 15 10:47:24.621286 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:24.621267    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnh4v\" (UniqueName: \"kubernetes.io/projected/07d78928-8e15-4a72-93e3-af348e9593b2-kube-api-access-wnh4v\") pod \"prometheus-operator-64785b6675-kd448\" (UID: \"07d78928-8e15-4a72-93e3-af348e9593b2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kd448"
Jul 15 10:47:24.721833 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:24.721794    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07d78928-8e15-4a72-93e3-af348e9593b2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-kd448\" (UID: \"07d78928-8e15-4a72-93e3-af348e9593b2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kd448"
Jul 15 10:47:24.722038 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:24.721951    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/07d78928-8e15-4a72-93e3-af348e9593b2-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-kd448\" (UID: \"07d78928-8e15-4a72-93e3-af348e9593b2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kd448"
Jul 15 10:47:24.722038 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:24.721981    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07d78928-8e15-4a72-93e3-af348e9593b2-metrics-client-ca\") pod \"prometheus-operator-64785b6675-kd448\" (UID: \"07d78928-8e15-4a72-93e3-af348e9593b2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kd448"
Jul 15 10:47:24.722175 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:24.722048    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnh4v\" (UniqueName: \"kubernetes.io/projected/07d78928-8e15-4a72-93e3-af348e9593b2-kube-api-access-wnh4v\") pod \"prometheus-operator-64785b6675-kd448\" (UID: \"07d78928-8e15-4a72-93e3-af348e9593b2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kd448"
Jul 15 10:47:24.722766 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:24.722737    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07d78928-8e15-4a72-93e3-af348e9593b2-metrics-client-ca\") pod \"prometheus-operator-64785b6675-kd448\" (UID: \"07d78928-8e15-4a72-93e3-af348e9593b2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kd448"
Jul 15 10:47:24.724983 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:24.724955    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/07d78928-8e15-4a72-93e3-af348e9593b2-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-kd448\" (UID: \"07d78928-8e15-4a72-93e3-af348e9593b2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kd448"
Jul 15 10:47:24.724983 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:24.724967    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07d78928-8e15-4a72-93e3-af348e9593b2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-kd448\" (UID: \"07d78928-8e15-4a72-93e3-af348e9593b2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kd448"
Jul 15 10:47:24.732406 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:24.732373    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnh4v\" (UniqueName: \"kubernetes.io/projected/07d78928-8e15-4a72-93e3-af348e9593b2-kube-api-access-wnh4v\") pod \"prometheus-operator-64785b6675-kd448\" (UID: \"07d78928-8e15-4a72-93e3-af348e9593b2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kd448"
Jul 15 10:47:24.877623 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:24.877526    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-kd448"
Jul 15 10:47:25.036392 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:25.036350    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-kd448"]
Jul 15 10:47:25.039917 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:47:25.039877    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d78928_8e15_4a72_93e3_af348e9593b2.slice/crio-4811c2f44bd85f26b8505efddfca87e90f68aeee75b7be56341cf779fa5e7aeb WatchSource:0}: Error finding container 4811c2f44bd85f26b8505efddfca87e90f68aeee75b7be56341cf779fa5e7aeb: Status 404 returned error can't find the container with id 4811c2f44bd85f26b8505efddfca87e90f68aeee75b7be56341cf779fa5e7aeb
Jul 15 10:47:25.053190 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:25.053131    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-kd448" event={"ID":"07d78928-8e15-4a72-93e3-af348e9593b2","Type":"ContainerStarted","Data":"4811c2f44bd85f26b8505efddfca87e90f68aeee75b7be56341cf779fa5e7aeb"}
Jul 15 10:47:27.348717 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:27.348668    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rwgk\" (UniqueName: \"kubernetes.io/projected/b5f21679-bed4-4e21-9a3b-22ec261aec90-kube-api-access-4rwgk\") pod \"network-check-target-9mhqc\" (UID: \"b5f21679-bed4-4e21-9a3b-22ec261aec90\") " pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:47:27.352546 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:27.352511    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 15 10:47:27.362649 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:27.362616    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 15 10:47:27.373356 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:27.373318    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rwgk\" (UniqueName: \"kubernetes.io/projected/b5f21679-bed4-4e21-9a3b-22ec261aec90-kube-api-access-4rwgk\") pod \"network-check-target-9mhqc\" (UID: \"b5f21679-bed4-4e21-9a3b-22ec261aec90\") " pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:47:27.449391 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:27.449346    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8d1da87-af98-4f6b-af6a-27a5fbabba60-metrics-certs\") pod \"network-metrics-daemon-84n76\" (UID: \"c8d1da87-af98-4f6b-af6a-27a5fbabba60\") " pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:47:27.449391 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:27.449399    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ad4cef-91aa-459c-80f3-f67d36e697aa-original-pull-secret\") pod \"global-pull-secret-syncer-rx95w\" (UID: \"37ad4cef-91aa-459c-80f3-f67d36e697aa\") " pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:47:27.452612 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:27.452589    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 15 10:47:27.453660 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:27.453641    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 15 10:47:27.462559 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:27.462519    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ad4cef-91aa-459c-80f3-f67d36e697aa-original-pull-secret\") pod \"global-pull-secret-syncer-rx95w\" (UID: \"37ad4cef-91aa-459c-80f3-f67d36e697aa\") " pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:47:27.462559 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:27.462540    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8d1da87-af98-4f6b-af6a-27a5fbabba60-metrics-certs\") pod \"network-metrics-daemon-84n76\" (UID: \"c8d1da87-af98-4f6b-af6a-27a5fbabba60\") " pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:47:27.599547 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:27.599459    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qpwxt\""
Jul 15 10:47:27.605116 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:27.605075    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx95w"
Jul 15 10:47:27.607246 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:27.607204    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-84n76"
Jul 15 10:47:27.614483 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:27.614456    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tlzgw\""
Jul 15 10:47:27.622091 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:27.622056    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:47:28.388693 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:28.388308    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rx95w"]
Jul 15 10:47:28.603865 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:28.603830    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-84n76"]
Jul 15 10:47:28.606797 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:47:28.606764    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8d1da87_af98_4f6b_af6a_27a5fbabba60.slice/crio-cd637b269b714a047ab273ecff6cbd7d7ac4a9c36029c22f5546d644ec4b7dbd WatchSource:0}: Error finding container cd637b269b714a047ab273ecff6cbd7d7ac4a9c36029c22f5546d644ec4b7dbd: Status 404 returned error can't find the container with id cd637b269b714a047ab273ecff6cbd7d7ac4a9c36029c22f5546d644ec4b7dbd
Jul 15 10:47:28.607334 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:28.607313    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9mhqc"]
Jul 15 10:47:28.610358 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:47:28.610337    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f21679_bed4_4e21_9a3b_22ec261aec90.slice/crio-a4496580af492462ce5b70fa8f820f2d91e957bbc23124ee6edc6ee83f19d5d6 WatchSource:0}: Error finding container a4496580af492462ce5b70fa8f820f2d91e957bbc23124ee6edc6ee83f19d5d6: Status 404 returned error can't find the container with id a4496580af492462ce5b70fa8f820f2d91e957bbc23124ee6edc6ee83f19d5d6
Jul 15 10:47:29.027444 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:29.027408    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-r52qn"
Jul 15 10:47:29.072852 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:29.072809    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f8rzs" event={"ID":"8c72e99f-bc5f-4851-a890-2605a51de0e0","Type":"ContainerStarted","Data":"a5404654b72b166c167766f3f9d046b0eb202f6a4200967bab40142a3dd61ea0"}
Jul 15 10:47:29.077864 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:29.077796    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9mhqc" event={"ID":"b5f21679-bed4-4e21-9a3b-22ec261aec90","Type":"ContainerStarted","Data":"a4496580af492462ce5b70fa8f820f2d91e957bbc23124ee6edc6ee83f19d5d6"}
Jul 15 10:47:29.082878 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:29.082515    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-kd448" event={"ID":"07d78928-8e15-4a72-93e3-af348e9593b2","Type":"ContainerStarted","Data":"31b45030085b99c8244303460fc934f134325cfa21c90bdf07afb8faf0c1099e"}
Jul 15 10:47:29.082878 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:29.082581    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-kd448" event={"ID":"07d78928-8e15-4a72-93e3-af348e9593b2","Type":"ContainerStarted","Data":"5231f874bb704694a088e4b2eb8c33e9c8924c82a35c6b084c69a8d757f431d2"}
Jul 15 10:47:29.084835 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:29.084803    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-84n76" event={"ID":"c8d1da87-af98-4f6b-af6a-27a5fbabba60","Type":"ContainerStarted","Data":"cd637b269b714a047ab273ecff6cbd7d7ac4a9c36029c22f5546d644ec4b7dbd"}
Jul 15 10:47:29.086341 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:29.086315    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rx95w" event={"ID":"37ad4cef-91aa-459c-80f3-f67d36e697aa","Type":"ContainerStarted","Data":"f27a75067a59fad04ae1b9caadb24074d7cd1e1a44df452c87bc93ad6e9f7999"}
Jul 15 10:47:29.094559 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:29.094421    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-f8rzs" podStartSLOduration=1.7005875339999998 podStartE2EDuration="19.094399428s" totalImagesPullingTime="17.393811894s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:47:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:29.093205314 +0000 UTC m=+67.922720713" watchObservedRunningTime="2026-07-15 10:47:29.094399428 +0000 UTC m=+67.923914835"
Jul 15 10:47:29.111687 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:29.111459    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-kd448" podStartSLOduration=1.927982941 podStartE2EDuration="5.111438658s" totalImagesPullingTime="3.183455717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:47:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:29.111315157 +0000 UTC m=+67.940830563" watchObservedRunningTime="2026-07-15 10:47:29.111438658 +0000 UTC m=+67.940954065"
Jul 15 10:47:30.093524 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:30.093477    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-84n76" event={"ID":"c8d1da87-af98-4f6b-af6a-27a5fbabba60","Type":"ContainerStarted","Data":"a2b0de15751423cff64e96361b2b4fba219b914354129fc22275edf407028118"}
Jul 15 10:47:30.093995 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:30.093563    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-84n76" event={"ID":"c8d1da87-af98-4f6b-af6a-27a5fbabba60","Type":"ContainerStarted","Data":"410c97223aa87e81a03f59c97d37d69c017cc949b3e704045c9001bd2d3875cc"}
Jul 15 10:47:30.115817 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:30.115753    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-84n76" podStartSLOduration=67.042218339 podStartE2EDuration="1m8.115733084s" totalImagesPullingTime="1.073514745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:30.112667985 +0000 UTC m=+68.942183392" watchObservedRunningTime="2026-07-15 10:47:30.115733084 +0000 UTC m=+68.945248490"
Jul 15 10:47:30.952470 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:30.952414    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-zcbpw"]
Jul 15 10:47:30.956294 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:30.956259    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-zcbpw"
Jul 15 10:47:30.960503 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:30.960240    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 15 10:47:30.960503 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:30.960246    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 15 10:47:30.960503 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:30.960246    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 15 10:47:30.960503 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:30.960355    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-qrlxv\""
Jul 15 10:47:30.972012 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:30.971984    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-zcbpw"]
Jul 15 10:47:30.972184 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:30.972046    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-j2hs4"]
Jul 15 10:47:30.978581 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:30.978549    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-j2hs4"
Jul 15 10:47:30.981244 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:30.981216    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 15 10:47:30.981483 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:30.981442    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 15 10:47:30.981554 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:30.981273    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-m8kxz\""
Jul 15 10:47:30.981619 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:30.981485    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 15 10:47:31.078562 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.078520    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb7xg\" (UniqueName: \"kubernetes.io/projected/d5bb1c2f-6490-4e67-a722-0c360b3cc058-kube-api-access-nb7xg\") pod \"node-exporter-j2hs4\" (UID: \"d5bb1c2f-6490-4e67-a722-0c360b3cc058\") " pod="openshift-monitoring/node-exporter-j2hs4"
Jul 15 10:47:31.078744 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.078575    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/96740e9c-b52a-4571-92eb-60c936c00d7c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-zcbpw\" (UID: \"96740e9c-b52a-4571-92eb-60c936c00d7c\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zcbpw"
Jul 15 10:47:31.078744 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.078645    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d5bb1c2f-6490-4e67-a722-0c360b3cc058-root\") pod \"node-exporter-j2hs4\" (UID: \"d5bb1c2f-6490-4e67-a722-0c360b3cc058\") " pod="openshift-monitoring/node-exporter-j2hs4"
Jul 15 10:47:31.078744 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.078670    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5bb1c2f-6490-4e67-a722-0c360b3cc058-metrics-client-ca\") pod \"node-exporter-j2hs4\" (UID: \"d5bb1c2f-6490-4e67-a722-0c360b3cc058\") " pod="openshift-monitoring/node-exporter-j2hs4"
Jul 15 10:47:31.078744 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.078692    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/96740e9c-b52a-4571-92eb-60c936c00d7c-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-zcbpw\" (UID: \"96740e9c-b52a-4571-92eb-60c936c00d7c\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zcbpw"
Jul 15 10:47:31.078744 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.078718    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5bb1c2f-6490-4e67-a722-0c360b3cc058-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j2hs4\" (UID: \"d5bb1c2f-6490-4e67-a722-0c360b3cc058\") " pod="openshift-monitoring/node-exporter-j2hs4"
Jul 15 10:47:31.078960 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.078752    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d5bb1c2f-6490-4e67-a722-0c360b3cc058-node-exporter-accelerators-collector-config\") pod \"node-exporter-j2hs4\" (UID: \"d5bb1c2f-6490-4e67-a722-0c360b3cc058\") " pod="openshift-monitoring/node-exporter-j2hs4"
Jul 15 10:47:31.078960 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.078777    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d5bb1c2f-6490-4e67-a722-0c360b3cc058-node-exporter-tls\") pod \"node-exporter-j2hs4\" (UID: \"d5bb1c2f-6490-4e67-a722-0c360b3cc058\") " pod="openshift-monitoring/node-exporter-j2hs4"
Jul 15 10:47:31.078960 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.078796    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5bb1c2f-6490-4e67-a722-0c360b3cc058-sys\") pod \"node-exporter-j2hs4\" (UID: \"d5bb1c2f-6490-4e67-a722-0c360b3cc058\") " pod="openshift-monitoring/node-exporter-j2hs4"
Jul 15 10:47:31.078960 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.078850    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96740e9c-b52a-4571-92eb-60c936c00d7c-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-zcbpw\" (UID: \"96740e9c-b52a-4571-92eb-60c936c00d7c\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zcbpw"
Jul 15 10:47:31.078960 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.078870    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/96740e9c-b52a-4571-92eb-60c936c00d7c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-zcbpw\" (UID: \"96740e9c-b52a-4571-92eb-60c936c00d7c\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zcbpw"
Jul 15 10:47:31.078960 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.078891    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ps52\" (UniqueName: \"kubernetes.io/projected/96740e9c-b52a-4571-92eb-60c936c00d7c-kube-api-access-6ps52\") pod \"kube-state-metrics-785778ff74-zcbpw\" (UID: \"96740e9c-b52a-4571-92eb-60c936c00d7c\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zcbpw"
Jul 15 10:47:31.078960 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.078949    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/96740e9c-b52a-4571-92eb-60c936c00d7c-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-zcbpw\" (UID: \"96740e9c-b52a-4571-92eb-60c936c00d7c\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zcbpw"
Jul 15 10:47:31.079247 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.078967    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d5bb1c2f-6490-4e67-a722-0c360b3cc058-node-exporter-textfile\") pod \"node-exporter-j2hs4\" (UID: \"d5bb1c2f-6490-4e67-a722-0c360b3cc058\") " pod="openshift-monitoring/node-exporter-j2hs4"
Jul 15 10:47:31.079247 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.078986    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d5bb1c2f-6490-4e67-a722-0c360b3cc058-node-exporter-wtmp\") pod \"node-exporter-j2hs4\" (UID: \"d5bb1c2f-6490-4e67-a722-0c360b3cc058\") " pod="openshift-monitoring/node-exporter-j2hs4"
Jul 15 10:47:31.180318 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.180279    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96740e9c-b52a-4571-92eb-60c936c00d7c-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-zcbpw\" (UID: \"96740e9c-b52a-4571-92eb-60c936c00d7c\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zcbpw"
Jul 15 10:47:31.180318 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.180324    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/96740e9c-b52a-4571-92eb-60c936c00d7c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-zcbpw\" (UID: \"96740e9c-b52a-4571-92eb-60c936c00d7c\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zcbpw"
Jul 15 10:47:31.180860 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.180347    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ps52\" (UniqueName: \"kubernetes.io/projected/96740e9c-b52a-4571-92eb-60c936c00d7c-kube-api-access-6ps52\") pod \"kube-state-metrics-785778ff74-zcbpw\" (UID: \"96740e9c-b52a-4571-92eb-60c936c00d7c\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zcbpw"
Jul 15 10:47:31.180860 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.180467    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/96740e9c-b52a-4571-92eb-60c936c00d7c-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-zcbpw\" (UID: \"96740e9c-b52a-4571-92eb-60c936c00d7c\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zcbpw"
Jul 15 10:47:31.180860 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.180505    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d5bb1c2f-6490-4e67-a722-0c360b3cc058-node-exporter-textfile\") pod \"node-exporter-j2hs4\" (UID: \"d5bb1c2f-6490-4e67-a722-0c360b3cc058\") " pod="openshift-monitoring/node-exporter-j2hs4"
Jul 15 10:47:31.180860 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.180530    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d5bb1c2f-6490-4e67-a722-0c360b3cc058-node-exporter-wtmp\") pod \"node-exporter-j2hs4\" (UID: \"d5bb1c2f-6490-4e67-a722-0c360b3cc058\") " pod="openshift-monitoring/node-exporter-j2hs4"
Jul 15 10:47:31.180860 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.180571    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb7xg\" (UniqueName: \"kubernetes.io/projected/d5bb1c2f-6490-4e67-a722-0c360b3cc058-kube-api-access-nb7xg\") pod \"node-exporter-j2hs4\" (UID: \"d5bb1c2f-6490-4e67-a722-0c360b3cc058\") " pod="openshift-monitoring/node-exporter-j2hs4"
Jul 15 10:47:31.180860 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.180597    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/96740e9c-b52a-4571-92eb-60c936c00d7c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-zcbpw\" (UID: \"96740e9c-b52a-4571-92eb-60c936c00d7c\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zcbpw"
Jul 15 10:47:31.180860 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.180632    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d5bb1c2f-6490-4e67-a722-0c360b3cc058-root\") pod \"node-exporter-j2hs4\" (UID: \"d5bb1c2f-6490-4e67-a722-0c360b3cc058\") " pod="openshift-monitoring/node-exporter-j2hs4"
Jul 15 10:47:31.180860 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.180656    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5bb1c2f-6490-4e67-a722-0c360b3cc058-metrics-client-ca\") pod \"node-exporter-j2hs4\" (UID: \"d5bb1c2f-6490-4e67-a722-0c360b3cc058\") " pod="openshift-monitoring/node-exporter-j2hs4"
Jul 15 10:47:31.180860 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.180680    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/96740e9c-b52a-4571-92eb-60c936c00d7c-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-zcbpw\" (UID: \"96740e9c-b52a-4571-92eb-60c936c00d7c\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zcbpw"
Jul 15 10:47:31.180860 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.180713    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5bb1c2f-6490-4e67-a722-0c360b3cc058-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j2hs4\" (UID: \"d5bb1c2f-6490-4e67-a722-0c360b3cc058\") " pod="openshift-monitoring/node-exporter-j2hs4"
Jul 15 10:47:31.180860 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.180760    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d5bb1c2f-6490-4e67-a722-0c360b3cc058-node-exporter-accelerators-collector-config\") pod \"node-exporter-j2hs4\" (UID: \"d5bb1c2f-6490-4e67-a722-0c360b3cc058\") " pod="openshift-monitoring/node-exporter-j2hs4"
Jul 15 10:47:31.180860 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.180793    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d5bb1c2f-6490-4e67-a722-0c360b3cc058-node-exporter-tls\") pod \"node-exporter-j2hs4\" (UID: \"d5bb1c2f-6490-4e67-a722-0c360b3cc058\") " pod="openshift-monitoring/node-exporter-j2hs4"
Jul 15 10:47:31.180860 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.180825    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5bb1c2f-6490-4e67-a722-0c360b3cc058-sys\") pod \"node-exporter-j2hs4\" (UID: \"d5bb1c2f-6490-4e67-a722-0c360b3cc058\") " pod="openshift-monitoring/node-exporter-j2hs4"
Jul 15 10:47:31.181508 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.180927    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5bb1c2f-6490-4e67-a722-0c360b3cc058-sys\") pod \"node-exporter-j2hs4\" (UID: \"d5bb1c2f-6490-4e67-a722-0c360b3cc058\") " pod="openshift-monitoring/node-exporter-j2hs4"
Jul 15 10:47:31.181508 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.181202    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/96740e9c-b52a-4571-92eb-60c936c00d7c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-zcbpw\" (UID: \"96740e9c-b52a-4571-92eb-60c936c00d7c\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zcbpw"
Jul 15 10:47:31.181508 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.181305    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/96740e9c-b52a-4571-92eb-60c936c00d7c-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-zcbpw\" (UID: \"96740e9c-b52a-4571-92eb-60c936c00d7c\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zcbpw"
Jul 15 10:47:31.181508 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.181202    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96740e9c-b52a-4571-92eb-60c936c00d7c-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-zcbpw\" (UID: \"96740e9c-b52a-4571-92eb-60c936c00d7c\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zcbpw"
Jul 15 10:47:31.181508 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.181464    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d5bb1c2f-6490-4e67-a722-0c360b3cc058-node-exporter-textfile\") pod \"node-exporter-j2hs4\" (UID: \"d5bb1c2f-6490-4e67-a722-0c360b3cc058\") " pod="openshift-monitoring/node-exporter-j2hs4"
Jul 15 10:47:31.181508 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:47:31.181505    2563 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 15 10:47:31.181816 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.181520    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d5bb1c2f-6490-4e67-a722-0c360b3cc058-root\") pod \"node-exporter-j2hs4\" (UID: \"d5bb1c2f-6490-4e67-a722-0c360b3cc058\") " pod="openshift-monitoring/node-exporter-j2hs4"
Jul 15 10:47:31.181816 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.181521    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d5bb1c2f-6490-4e67-a722-0c360b3cc058-node-exporter-wtmp\") pod \"node-exporter-j2hs4\" (UID: \"d5bb1c2f-6490-4e67-a722-0c360b3cc058\") " pod="openshift-monitoring/node-exporter-j2hs4"
Jul 15 10:47:31.181816 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:47:31.181576    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96740e9c-b52a-4571-92eb-60c936c00d7c-kube-state-metrics-tls podName:96740e9c-b52a-4571-92eb-60c936c00d7c nodeName:}" failed. No retries permitted until 2026-07-15 10:47:31.681557882 +0000 UTC m=+70.511073276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/96740e9c-b52a-4571-92eb-60c936c00d7c-kube-state-metrics-tls") pod "kube-state-metrics-785778ff74-zcbpw" (UID: "96740e9c-b52a-4571-92eb-60c936c00d7c") : secret "kube-state-metrics-tls" not found
Jul 15 10:47:31.181984 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.181881    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5bb1c2f-6490-4e67-a722-0c360b3cc058-metrics-client-ca\") pod \"node-exporter-j2hs4\" (UID: \"d5bb1c2f-6490-4e67-a722-0c360b3cc058\") " pod="openshift-monitoring/node-exporter-j2hs4"
Jul 15 10:47:31.182191 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.182124    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d5bb1c2f-6490-4e67-a722-0c360b3cc058-node-exporter-accelerators-collector-config\") pod \"node-exporter-j2hs4\" (UID: \"d5bb1c2f-6490-4e67-a722-0c360b3cc058\") " pod="openshift-monitoring/node-exporter-j2hs4"
Jul 15 10:47:31.184019 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.183988    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5bb1c2f-6490-4e67-a722-0c360b3cc058-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j2hs4\" (UID: \"d5bb1c2f-6490-4e67-a722-0c360b3cc058\") " pod="openshift-monitoring/node-exporter-j2hs4"
Jul 15 10:47:31.184160 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.184094    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/96740e9c-b52a-4571-92eb-60c936c00d7c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-zcbpw\" (UID: \"96740e9c-b52a-4571-92eb-60c936c00d7c\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zcbpw"
Jul 15 10:47:31.184519 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.184496    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d5bb1c2f-6490-4e67-a722-0c360b3cc058-node-exporter-tls\") pod \"node-exporter-j2hs4\" (UID: \"d5bb1c2f-6490-4e67-a722-0c360b3cc058\") " pod="openshift-monitoring/node-exporter-j2hs4"
Jul 15 10:47:31.195079 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.195038    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb7xg\" (UniqueName: \"kubernetes.io/projected/d5bb1c2f-6490-4e67-a722-0c360b3cc058-kube-api-access-nb7xg\") pod \"node-exporter-j2hs4\" (UID: \"d5bb1c2f-6490-4e67-a722-0c360b3cc058\") " pod="openshift-monitoring/node-exporter-j2hs4"
Jul 15 10:47:31.197974 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.197947    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ps52\" (UniqueName: \"kubernetes.io/projected/96740e9c-b52a-4571-92eb-60c936c00d7c-kube-api-access-6ps52\") pod \"kube-state-metrics-785778ff74-zcbpw\" (UID: \"96740e9c-b52a-4571-92eb-60c936c00d7c\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zcbpw"
Jul 15 10:47:31.222453 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.222364    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5ff9595488-rmsmj"
Jul 15 10:47:31.226902 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.226860    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5ff9595488-rmsmj"
Jul 15 10:47:31.290797 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.290752    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-j2hs4"
Jul 15 10:47:31.416049 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.416012    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-88d4d74bd-v4zn7"
Jul 15 10:47:31.416270 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.416082    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-88d4d74bd-v4zn7"
Jul 15 10:47:31.417656 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.417622    2563 patch_prober.go:28] interesting pod/console-88d4d74bd-v4zn7 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.11:8443/health\": dial tcp 10.133.0.11:8443: connect: connection refused" start-of-body=
Jul 15 10:47:31.417806 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.417678    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-88d4d74bd-v4zn7" podUID="3a522d3d-13c9-4a9c-949d-f30304744f77" containerName="console" probeResult="failure" output="Get \"https://10.133.0.11:8443/health\": dial tcp 10.133.0.11:8443: connect: connection refused"
Jul 15 10:47:31.684391 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.684182    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/96740e9c-b52a-4571-92eb-60c936c00d7c-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-zcbpw\" (UID: \"96740e9c-b52a-4571-92eb-60c936c00d7c\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zcbpw"
Jul 15 10:47:31.687424 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.687387    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/96740e9c-b52a-4571-92eb-60c936c00d7c-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-zcbpw\" (UID: \"96740e9c-b52a-4571-92eb-60c936c00d7c\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zcbpw"
Jul 15 10:47:31.867954 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:31.867901    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-zcbpw"
Jul 15 10:47:33.291280 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:47:33.291230    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5bb1c2f_6490_4e67_a722_0c360b3cc058.slice/crio-298d1e9ead582a06971b111bcc4f527161465d927621ee00331fe576493556d0 WatchSource:0}: Error finding container 298d1e9ead582a06971b111bcc4f527161465d927621ee00331fe576493556d0: Status 404 returned error can't find the container with id 298d1e9ead582a06971b111bcc4f527161465d927621ee00331fe576493556d0
Jul 15 10:47:33.445712 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:33.445675    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-zcbpw"]
Jul 15 10:47:33.450116 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:47:33.449981    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96740e9c_b52a_4571_92eb_60c936c00d7c.slice/crio-2b10c3ef25108105c7d6bbfee37bdd88e021f5c0c27fe67666068b1cfe101153 WatchSource:0}: Error finding container 2b10c3ef25108105c7d6bbfee37bdd88e021f5c0c27fe67666068b1cfe101153: Status 404 returned error can't find the container with id 2b10c3ef25108105c7d6bbfee37bdd88e021f5c0c27fe67666068b1cfe101153
Jul 15 10:47:34.114195 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:34.114159    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9mhqc" event={"ID":"b5f21679-bed4-4e21-9a3b-22ec261aec90","Type":"ContainerStarted","Data":"f93b05531c57212c682ce7eef28d241cc80b1785fe84d14510b86efce7642043"}
Jul 15 10:47:34.114402 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:34.114245    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:47:34.116065 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:34.116021    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j2hs4" event={"ID":"d5bb1c2f-6490-4e67-a722-0c360b3cc058","Type":"ContainerStarted","Data":"298d1e9ead582a06971b111bcc4f527161465d927621ee00331fe576493556d0"}
Jul 15 10:47:34.117871 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:34.117833    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-zcbpw" event={"ID":"96740e9c-b52a-4571-92eb-60c936c00d7c","Type":"ContainerStarted","Data":"2b10c3ef25108105c7d6bbfee37bdd88e021f5c0c27fe67666068b1cfe101153"}
Jul 15 10:47:34.120106 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:34.120075    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rx95w" event={"ID":"37ad4cef-91aa-459c-80f3-f67d36e697aa","Type":"ContainerStarted","Data":"7a863a1d3af91f8bf925f62f0792d6877ff9e0f34f6670b1c146ef6f47f25c50"}
Jul 15 10:47:34.132345 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:34.132294    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9mhqc" podStartSLOduration=67.45310167 podStartE2EDuration="1m12.132275113s" totalImagesPullingTime="4.679173443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:34.131401825 +0000 UTC m=+72.960917232" watchObservedRunningTime="2026-07-15 10:47:34.132275113 +0000 UTC m=+72.961790519"
Jul 15 10:47:34.149654 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:34.149590    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rx95w" podStartSLOduration=67.239505473 podStartE2EDuration="1m12.149564868s" totalImagesPullingTime="4.910059395s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:34.148339828 +0000 UTC m=+72.977855235" watchObservedRunningTime="2026-07-15 10:47:34.149564868 +0000 UTC m=+72.979080272"
Jul 15 10:47:35.126583 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:35.126548    2563 generic.go:356] "Generic (PLEG): container finished" podID="d5bb1c2f-6490-4e67-a722-0c360b3cc058" containerID="a1fa05cb7e240410f17e941a662f7d82872d6e2336f543de4fb6e612a5c423e6" exitCode=0
Jul 15 10:47:35.127080 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:35.126632    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j2hs4" event={"ID":"d5bb1c2f-6490-4e67-a722-0c360b3cc058","Type":"ContainerDied","Data":"a1fa05cb7e240410f17e941a662f7d82872d6e2336f543de4fb6e612a5c423e6"}
Jul 15 10:47:35.129329 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:35.129291    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-zcbpw" event={"ID":"96740e9c-b52a-4571-92eb-60c936c00d7c","Type":"ContainerStarted","Data":"aea85d93df3ce2fe1cac8bc55ee656c4352969330f29806faf53b75ec636f985"}
Jul 15 10:47:35.129494 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:35.129372    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-zcbpw" event={"ID":"96740e9c-b52a-4571-92eb-60c936c00d7c","Type":"ContainerStarted","Data":"cea030df8a44e6f5a83a533443ad4113e946a0f30f13d4203c1e66c4b78e5d46"}
Jul 15 10:47:35.129494 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:35.129407    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-zcbpw" event={"ID":"96740e9c-b52a-4571-92eb-60c936c00d7c","Type":"ContainerStarted","Data":"51255f58f96ec916a1cd3c964a81e7490bf41b768a3d72a8daf07748226f01df"}
Jul 15 10:47:35.175310 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:35.174911    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-zcbpw" podStartSLOduration=3.9940645200000002 podStartE2EDuration="5.1748829s" totalImagesPullingTime="1.18081838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:47:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:35.173358845 +0000 UTC m=+74.002874252" watchObservedRunningTime="2026-07-15 10:47:35.1748829 +0000 UTC m=+74.004398309"
Jul 15 10:47:35.762055 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:35.761997    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-l6qwd"]
Jul 15 10:47:35.768245 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:35.768201    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-l6qwd"
Jul 15 10:47:35.771224 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:35.771200    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-n86gx\""
Jul 15 10:47:35.771343 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:35.771222    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 15 10:47:35.774700 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:35.774670    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-l6qwd"]
Jul 15 10:47:35.919817 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:35.919773    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4b1598b-43de-43d0-98e9-0ec886523487-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-l6qwd\" (UID: \"d4b1598b-43de-43d0-98e9-0ec886523487\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-l6qwd"
Jul 15 10:47:35.919817 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:35.919826    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lslx\" (UniqueName: \"kubernetes.io/projected/d4b1598b-43de-43d0-98e9-0ec886523487-kube-api-access-2lslx\") pod \"monitoring-plugin-5f68df7f58-l6qwd\" (UID: \"d4b1598b-43de-43d0-98e9-0ec886523487\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-l6qwd"
Jul 15 10:47:35.954820 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:35.954781    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-88d4d74bd-v4zn7"]
Jul 15 10:47:36.004795 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:36.004752    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f8488489b-q92p9"]
Jul 15 10:47:36.007033 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:36.007010    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f8488489b-q92p9"
Jul 15 10:47:36.020666 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:36.020625    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4b1598b-43de-43d0-98e9-0ec886523487-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-l6qwd\" (UID: \"d4b1598b-43de-43d0-98e9-0ec886523487\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-l6qwd"
Jul 15 10:47:36.020846 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:36.020677    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lslx\" (UniqueName: \"kubernetes.io/projected/d4b1598b-43de-43d0-98e9-0ec886523487-kube-api-access-2lslx\") pod \"monitoring-plugin-5f68df7f58-l6qwd\" (UID: \"d4b1598b-43de-43d0-98e9-0ec886523487\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-l6qwd"
Jul 15 10:47:36.020846 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:47:36.020797    2563 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 15 10:47:36.021007 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:47:36.020883    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4b1598b-43de-43d0-98e9-0ec886523487-monitoring-plugin-cert podName:d4b1598b-43de-43d0-98e9-0ec886523487 nodeName:}" failed. No retries permitted until 2026-07-15 10:47:36.520859955 +0000 UTC m=+75.350375350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/d4b1598b-43de-43d0-98e9-0ec886523487-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-l6qwd" (UID: "d4b1598b-43de-43d0-98e9-0ec886523487") : secret "monitoring-plugin-cert" not found
Jul 15 10:47:36.025812 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:36.025778    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f8488489b-q92p9"]
Jul 15 10:47:36.051034 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:36.050996    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lslx\" (UniqueName: \"kubernetes.io/projected/d4b1598b-43de-43d0-98e9-0ec886523487-kube-api-access-2lslx\") pod \"monitoring-plugin-5f68df7f58-l6qwd\" (UID: \"d4b1598b-43de-43d0-98e9-0ec886523487\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-l6qwd"
Jul 15 10:47:36.121999 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:36.121961    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-service-ca\") pod \"console-f8488489b-q92p9\" (UID: \"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1\") " pod="openshift-console/console-f8488489b-q92p9"
Jul 15 10:47:36.121999 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:36.122004    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-console-oauth-config\") pod \"console-f8488489b-q92p9\" (UID: \"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1\") " pod="openshift-console/console-f8488489b-q92p9"
Jul 15 10:47:36.122240 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:36.122080    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9fsh\" (UniqueName: \"kubernetes.io/projected/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-kube-api-access-b9fsh\") pod \"console-f8488489b-q92p9\" (UID: \"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1\") " pod="openshift-console/console-f8488489b-q92p9"
Jul 15 10:47:36.122240 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:36.122099    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-oauth-serving-cert\") pod \"console-f8488489b-q92p9\" (UID: \"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1\") " pod="openshift-console/console-f8488489b-q92p9"
Jul 15 10:47:36.122240 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:36.122152    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-trusted-ca-bundle\") pod \"console-f8488489b-q92p9\" (UID: \"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1\") " pod="openshift-console/console-f8488489b-q92p9"
Jul 15 10:47:36.122240 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:36.122179    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-console-serving-cert\") pod \"console-f8488489b-q92p9\" (UID: \"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1\") " pod="openshift-console/console-f8488489b-q92p9"
Jul 15 10:47:36.122240 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:36.122202    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-console-config\") pod \"console-f8488489b-q92p9\" (UID: \"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1\") " pod="openshift-console/console-f8488489b-q92p9"
Jul 15 10:47:36.136762 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:36.136725    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j2hs4" event={"ID":"d5bb1c2f-6490-4e67-a722-0c360b3cc058","Type":"ContainerStarted","Data":"4aae63d8240cd3ce33a7fa6488580f4c126779fb92d1d75a448440c6ff190f1d"}
Jul 15 10:47:36.137228 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:36.136789    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j2hs4" event={"ID":"d5bb1c2f-6490-4e67-a722-0c360b3cc058","Type":"ContainerStarted","Data":"fd832c770ce9ea2b8ea028d10bb7eeb2c26b6850d4c3bc27ab2118fbf088d8a9"}
Jul 15 10:47:36.170081 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:36.169971    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-j2hs4" podStartSLOduration=5.224968097 podStartE2EDuration="6.16995292s" totalImagesPullingTime="861.896583ms" totalInitContainerRuntime="83.08824ms" isStatefulPod=true podCreationTimestamp="2026-07-15 10:47:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:36.168824817 +0000 UTC m=+74.998340221" watchObservedRunningTime="2026-07-15 10:47:36.16995292 +0000 UTC m=+74.999468324"
Jul 15 10:47:36.222962 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:36.222927    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-trusted-ca-bundle\") pod \"console-f8488489b-q92p9\" (UID: \"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1\") " pod="openshift-console/console-f8488489b-q92p9"
Jul 15 10:47:36.222962 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:36.222965    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-console-serving-cert\") pod \"console-f8488489b-q92p9\" (UID: \"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1\") " pod="openshift-console/console-f8488489b-q92p9"
Jul 15 10:47:36.223243 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:36.222986    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-console-config\") pod \"console-f8488489b-q92p9\" (UID: \"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1\") " pod="openshift-console/console-f8488489b-q92p9"
Jul 15 10:47:36.223243 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:36.223129    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-service-ca\") pod \"console-f8488489b-q92p9\" (UID: \"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1\") " pod="openshift-console/console-f8488489b-q92p9"
Jul 15 10:47:36.223243 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:36.223195    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-console-oauth-config\") pod \"console-f8488489b-q92p9\" (UID: \"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1\") " pod="openshift-console/console-f8488489b-q92p9"
Jul 15 10:47:36.223390 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:36.223341    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9fsh\" (UniqueName: \"kubernetes.io/projected/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-kube-api-access-b9fsh\") pod \"console-f8488489b-q92p9\" (UID: \"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1\") " pod="openshift-console/console-f8488489b-q92p9"
Jul 15 10:47:36.223443 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:36.223414    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-oauth-serving-cert\") pod \"console-f8488489b-q92p9\" (UID: \"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1\") " pod="openshift-console/console-f8488489b-q92p9"
Jul 15 10:47:36.223922 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:36.223883    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-console-config\") pod \"console-f8488489b-q92p9\" (UID: \"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1\") " pod="openshift-console/console-f8488489b-q92p9"
Jul 15 10:47:36.224107 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:36.224078    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-oauth-serving-cert\") pod \"console-f8488489b-q92p9\" (UID: \"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1\") " pod="openshift-console/console-f8488489b-q92p9"
Jul 15 10:47:36.224257 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:36.224233    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-service-ca\") pod \"console-f8488489b-q92p9\" (UID: \"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1\") " pod="openshift-console/console-f8488489b-q92p9"
Jul 15 10:47:36.224516 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:36.224494    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-trusted-ca-bundle\") pod \"console-f8488489b-q92p9\" (UID: \"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1\") " pod="openshift-console/console-f8488489b-q92p9"
Jul 15 10:47:36.225782 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:36.225762    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-console-serving-cert\") pod \"console-f8488489b-q92p9\" (UID: \"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1\") " pod="openshift-console/console-f8488489b-q92p9"
Jul 15 10:47:36.225862 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:36.225812    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-console-oauth-config\") pod \"console-f8488489b-q92p9\" (UID: \"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1\") " pod="openshift-console/console-f8488489b-q92p9"
Jul 15 10:47:36.256778 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:36.256742    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9fsh\" (UniqueName: \"kubernetes.io/projected/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-kube-api-access-b9fsh\") pod \"console-f8488489b-q92p9\" (UID: \"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1\") " pod="openshift-console/console-f8488489b-q92p9"
Jul 15 10:47:36.317834 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:36.317725    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f8488489b-q92p9"
Jul 15 10:47:36.479648 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:36.479614    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f8488489b-q92p9"]
Jul 15 10:47:36.482781 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:47:36.482748    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5bb1b91_6f58_4406_b7cd_2ed2cb4289f1.slice/crio-b5ef821ca2d5e792d6e8bcfcff272bcb647781592090d601e6ba671d08ab5ea0 WatchSource:0}: Error finding container b5ef821ca2d5e792d6e8bcfcff272bcb647781592090d601e6ba671d08ab5ea0: Status 404 returned error can't find the container with id b5ef821ca2d5e792d6e8bcfcff272bcb647781592090d601e6ba671d08ab5ea0
Jul 15 10:47:36.525227 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:36.525180    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4b1598b-43de-43d0-98e9-0ec886523487-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-l6qwd\" (UID: \"d4b1598b-43de-43d0-98e9-0ec886523487\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-l6qwd"
Jul 15 10:47:36.527713 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:36.527682    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4b1598b-43de-43d0-98e9-0ec886523487-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-l6qwd\" (UID: \"d4b1598b-43de-43d0-98e9-0ec886523487\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-l6qwd"
Jul 15 10:47:36.679227 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:36.679096    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-l6qwd"
Jul 15 10:47:36.829109 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:36.829075    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-l6qwd"]
Jul 15 10:47:36.832332 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:47:36.832296    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b1598b_43de_43d0_98e9_0ec886523487.slice/crio-ba20be01dc1ed2059aa483290cbb6288aa30b95cde08d58fb215f2fa793ad2e7 WatchSource:0}: Error finding container ba20be01dc1ed2059aa483290cbb6288aa30b95cde08d58fb215f2fa793ad2e7: Status 404 returned error can't find the container with id ba20be01dc1ed2059aa483290cbb6288aa30b95cde08d58fb215f2fa793ad2e7
Jul 15 10:47:37.142618 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.142581    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f8488489b-q92p9" event={"ID":"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1","Type":"ContainerStarted","Data":"61fbeebccf765180d57dbe2570667638b3dbf6f014e1ec17e87acf61da3d2899"}
Jul 15 10:47:37.143092 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.142638    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f8488489b-q92p9" event={"ID":"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1","Type":"ContainerStarted","Data":"b5ef821ca2d5e792d6e8bcfcff272bcb647781592090d601e6ba671d08ab5ea0"}
Jul 15 10:47:37.143897 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.143865    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-l6qwd" event={"ID":"d4b1598b-43de-43d0-98e9-0ec886523487","Type":"ContainerStarted","Data":"ba20be01dc1ed2059aa483290cbb6288aa30b95cde08d58fb215f2fa793ad2e7"}
Jul 15 10:47:37.169673 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.169607    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f8488489b-q92p9" podStartSLOduration=2.169583831 podStartE2EDuration="2.169583831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:47:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:37.168919013 +0000 UTC m=+75.998434429" watchObservedRunningTime="2026-07-15 10:47:37.169583831 +0000 UTC m=+75.999099240"
Jul 15 10:47:37.375257 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.375216    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 15 10:47:37.379497 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.379461    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.386417 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.386383    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-83lebucbq1nsu\""
Jul 15 10:47:37.386634 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.386613    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-q8jkb\""
Jul 15 10:47:37.386708 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.386633    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 15 10:47:37.386993 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.386969    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 15 10:47:37.387108 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.386992    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 15 10:47:37.387108 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.386999    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 15 10:47:37.387513 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.387439    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 15 10:47:37.387513 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.387474    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 15 10:47:37.387513 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.387483    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 15 10:47:37.387723 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.387491    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 15 10:47:37.388040 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.387780    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 15 10:47:37.388040 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.387983    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 15 10:47:37.388244 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.388203    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 15 10:47:37.388340 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.388305    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 15 10:47:37.393400 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.393315    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 15 10:47:37.398575 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.398540    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 15 10:47:37.402783 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.402750    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 15 10:47:37.408220 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.408119    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 15 10:47:37.534032 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.533990    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d4f1d838-0655-4218-8247-4233d2e0ac04-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.534032 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.534035    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d4f1d838-0655-4218-8247-4233d2e0ac04-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.534311 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.534065    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d4f1d838-0655-4218-8247-4233d2e0ac04-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.534311 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.534092    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d4f1d838-0655-4218-8247-4233d2e0ac04-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.534311 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.534195    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4f1d838-0655-4218-8247-4233d2e0ac04-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.534311 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.534241    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d4f1d838-0655-4218-8247-4233d2e0ac04-web-config\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.534311 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.534260    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svvbv\" (UniqueName: \"kubernetes.io/projected/d4f1d838-0655-4218-8247-4233d2e0ac04-kube-api-access-svvbv\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.534311 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.534282    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d4f1d838-0655-4218-8247-4233d2e0ac04-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.534311 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.534306    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d4f1d838-0655-4218-8247-4233d2e0ac04-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.534513 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.534324    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d4f1d838-0655-4218-8247-4233d2e0ac04-config-out\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.534513 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.534364    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d4f1d838-0655-4218-8247-4233d2e0ac04-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.534513 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.534430    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4f1d838-0655-4218-8247-4233d2e0ac04-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.534513 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.534470    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d4f1d838-0655-4218-8247-4233d2e0ac04-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.534513 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.534487    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d4f1d838-0655-4218-8247-4233d2e0ac04-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.534513 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.534506    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4f1d838-0655-4218-8247-4233d2e0ac04-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.534689 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.534525    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d4f1d838-0655-4218-8247-4233d2e0ac04-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.534689 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.534546    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d4f1d838-0655-4218-8247-4233d2e0ac04-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.534689 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.534565    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d4f1d838-0655-4218-8247-4233d2e0ac04-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.534689 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.534606    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d4f1d838-0655-4218-8247-4233d2e0ac04-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.534689 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.534641    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d4f1d838-0655-4218-8247-4233d2e0ac04-config\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.635723 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.635674    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d4f1d838-0655-4218-8247-4233d2e0ac04-web-config\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.635723 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.635717    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svvbv\" (UniqueName: \"kubernetes.io/projected/d4f1d838-0655-4218-8247-4233d2e0ac04-kube-api-access-svvbv\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.635723 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.635735    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d4f1d838-0655-4218-8247-4233d2e0ac04-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.636051 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.635896    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d4f1d838-0655-4218-8247-4233d2e0ac04-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.636051 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.635944    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d4f1d838-0655-4218-8247-4233d2e0ac04-config-out\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.636051 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.635979    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d4f1d838-0655-4218-8247-4233d2e0ac04-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.636051 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.636011    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4f1d838-0655-4218-8247-4233d2e0ac04-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.636277 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.636062    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d4f1d838-0655-4218-8247-4233d2e0ac04-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.636277 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.636087    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d4f1d838-0655-4218-8247-4233d2e0ac04-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.636277 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.636166    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4f1d838-0655-4218-8247-4233d2e0ac04-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.636277 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.636208    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d4f1d838-0655-4218-8247-4233d2e0ac04-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.636277 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.636249    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d4f1d838-0655-4218-8247-4233d2e0ac04-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.636510 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.636280    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d4f1d838-0655-4218-8247-4233d2e0ac04-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.636510 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.636305    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d4f1d838-0655-4218-8247-4233d2e0ac04-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.636510 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.636331    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d4f1d838-0655-4218-8247-4233d2e0ac04-config\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.636510 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.636357    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d4f1d838-0655-4218-8247-4233d2e0ac04-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.636510 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.636384    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d4f1d838-0655-4218-8247-4233d2e0ac04-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.636510 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.636422    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d4f1d838-0655-4218-8247-4233d2e0ac04-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.636510 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.636470    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d4f1d838-0655-4218-8247-4233d2e0ac04-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.636829 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.636518    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4f1d838-0655-4218-8247-4233d2e0ac04-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.637133 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.637000    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4f1d838-0655-4218-8247-4233d2e0ac04-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.638019 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.637674    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d4f1d838-0655-4218-8247-4233d2e0ac04-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.638019 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.637855    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4f1d838-0655-4218-8247-4233d2e0ac04-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.639187 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.639134    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d4f1d838-0655-4218-8247-4233d2e0ac04-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.639304 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.639245    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d4f1d838-0655-4218-8247-4233d2e0ac04-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.639817 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.639790    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d4f1d838-0655-4218-8247-4233d2e0ac04-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.639978 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.639795    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d4f1d838-0655-4218-8247-4233d2e0ac04-web-config\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.639978 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.639819    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4f1d838-0655-4218-8247-4233d2e0ac04-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.640442 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.640330    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d4f1d838-0655-4218-8247-4233d2e0ac04-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.643153 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.641402    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d4f1d838-0655-4218-8247-4233d2e0ac04-config-out\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.643153 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.641488    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d4f1d838-0655-4218-8247-4233d2e0ac04-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.643153 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.642003    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d4f1d838-0655-4218-8247-4233d2e0ac04-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.643153 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.642040    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d4f1d838-0655-4218-8247-4233d2e0ac04-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.643153 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.642523    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d4f1d838-0655-4218-8247-4233d2e0ac04-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.643153 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.642893    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d4f1d838-0655-4218-8247-4233d2e0ac04-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.644213 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.644096    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d4f1d838-0655-4218-8247-4233d2e0ac04-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.644213 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.644162    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d4f1d838-0655-4218-8247-4233d2e0ac04-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.645325 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.645301    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d4f1d838-0655-4218-8247-4233d2e0ac04-config\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.645613 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.645590    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d4f1d838-0655-4218-8247-4233d2e0ac04-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.646941 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.646918    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svvbv\" (UniqueName: \"kubernetes.io/projected/d4f1d838-0655-4218-8247-4233d2e0ac04-kube-api-access-svvbv\") pod \"prometheus-k8s-0\" (UID: \"d4f1d838-0655-4218-8247-4233d2e0ac04\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.691594 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.691554    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:37.927783 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:37.927701    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 15 10:47:37.930357 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:47:37.930318    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f1d838_0655_4218_8247_4233d2e0ac04.slice/crio-56a996c787bde3d8a7c3389e6e101d12ed146e80efba3d2f6f04984b7412428b WatchSource:0}: Error finding container 56a996c787bde3d8a7c3389e6e101d12ed146e80efba3d2f6f04984b7412428b: Status 404 returned error can't find the container with id 56a996c787bde3d8a7c3389e6e101d12ed146e80efba3d2f6f04984b7412428b
Jul 15 10:47:38.149934 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:38.149900    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d4f1d838-0655-4218-8247-4233d2e0ac04","Type":"ContainerStarted","Data":"56a996c787bde3d8a7c3389e6e101d12ed146e80efba3d2f6f04984b7412428b"}
Jul 15 10:47:39.157119 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:39.157068    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-l6qwd" event={"ID":"d4b1598b-43de-43d0-98e9-0ec886523487","Type":"ContainerStarted","Data":"d9636ebf2260cf3bd69baf92e4831fedf39977324ddf9baabace61793a72daed"}
Jul 15 10:47:39.157633 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:39.157200    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-l6qwd"
Jul 15 10:47:39.164503 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:39.164465    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-l6qwd"
Jul 15 10:47:39.191204 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:39.191117    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-l6qwd" podStartSLOduration=2.337714627 podStartE2EDuration="4.191093307s" totalImagesPullingTime="1.85337868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:47:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:39.190335064 +0000 UTC m=+78.019850469" watchObservedRunningTime="2026-07-15 10:47:39.191093307 +0000 UTC m=+78.020608691"
Jul 15 10:47:40.162848 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:40.162810    2563 generic.go:356] "Generic (PLEG): container finished" podID="d4f1d838-0655-4218-8247-4233d2e0ac04" containerID="b28bbff745531dd66280a0906127044e551762ec21d5c789aad485814043b94c" exitCode=0
Jul 15 10:47:40.163480 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:40.162894    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d4f1d838-0655-4218-8247-4233d2e0ac04","Type":"ContainerDied","Data":"b28bbff745531dd66280a0906127044e551762ec21d5c789aad485814043b94c"}
Jul 15 10:47:41.157617 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:41.157574    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f8488489b-q92p9"]
Jul 15 10:47:41.243441 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:41.243402    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8677894d4d-zwdn4"]
Jul 15 10:47:41.247312 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:41.247273    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8677894d4d-zwdn4"
Jul 15 10:47:41.263501 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:41.263462    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8677894d4d-zwdn4"]
Jul 15 10:47:41.369667 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:41.369625    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6294ad35-0d5d-4ba7-9d44-15d91effa356-console-config\") pod \"console-8677894d4d-zwdn4\" (UID: \"6294ad35-0d5d-4ba7-9d44-15d91effa356\") " pod="openshift-console/console-8677894d4d-zwdn4"
Jul 15 10:47:41.369859 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:41.369674    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6294ad35-0d5d-4ba7-9d44-15d91effa356-console-serving-cert\") pod \"console-8677894d4d-zwdn4\" (UID: \"6294ad35-0d5d-4ba7-9d44-15d91effa356\") " pod="openshift-console/console-8677894d4d-zwdn4"
Jul 15 10:47:41.369859 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:41.369700    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thvfs\" (UniqueName: \"kubernetes.io/projected/6294ad35-0d5d-4ba7-9d44-15d91effa356-kube-api-access-thvfs\") pod \"console-8677894d4d-zwdn4\" (UID: \"6294ad35-0d5d-4ba7-9d44-15d91effa356\") " pod="openshift-console/console-8677894d4d-zwdn4"
Jul 15 10:47:41.369859 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:41.369839    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6294ad35-0d5d-4ba7-9d44-15d91effa356-service-ca\") pod \"console-8677894d4d-zwdn4\" (UID: \"6294ad35-0d5d-4ba7-9d44-15d91effa356\") " pod="openshift-console/console-8677894d4d-zwdn4"
Jul 15 10:47:41.369995 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:41.369877    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6294ad35-0d5d-4ba7-9d44-15d91effa356-oauth-serving-cert\") pod \"console-8677894d4d-zwdn4\" (UID: \"6294ad35-0d5d-4ba7-9d44-15d91effa356\") " pod="openshift-console/console-8677894d4d-zwdn4"
Jul 15 10:47:41.370038 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:41.370017    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6294ad35-0d5d-4ba7-9d44-15d91effa356-trusted-ca-bundle\") pod \"console-8677894d4d-zwdn4\" (UID: \"6294ad35-0d5d-4ba7-9d44-15d91effa356\") " pod="openshift-console/console-8677894d4d-zwdn4"
Jul 15 10:47:41.370091 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:41.370058    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6294ad35-0d5d-4ba7-9d44-15d91effa356-console-oauth-config\") pod \"console-8677894d4d-zwdn4\" (UID: \"6294ad35-0d5d-4ba7-9d44-15d91effa356\") " pod="openshift-console/console-8677894d4d-zwdn4"
Jul 15 10:47:41.471430 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:41.471378    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6294ad35-0d5d-4ba7-9d44-15d91effa356-trusted-ca-bundle\") pod \"console-8677894d4d-zwdn4\" (UID: \"6294ad35-0d5d-4ba7-9d44-15d91effa356\") " pod="openshift-console/console-8677894d4d-zwdn4"
Jul 15 10:47:41.471430 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:41.471439    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6294ad35-0d5d-4ba7-9d44-15d91effa356-console-oauth-config\") pod \"console-8677894d4d-zwdn4\" (UID: \"6294ad35-0d5d-4ba7-9d44-15d91effa356\") " pod="openshift-console/console-8677894d4d-zwdn4"
Jul 15 10:47:41.473006 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:41.471515    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6294ad35-0d5d-4ba7-9d44-15d91effa356-console-config\") pod \"console-8677894d4d-zwdn4\" (UID: \"6294ad35-0d5d-4ba7-9d44-15d91effa356\") " pod="openshift-console/console-8677894d4d-zwdn4"
Jul 15 10:47:41.473006 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:41.471539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6294ad35-0d5d-4ba7-9d44-15d91effa356-console-serving-cert\") pod \"console-8677894d4d-zwdn4\" (UID: \"6294ad35-0d5d-4ba7-9d44-15d91effa356\") " pod="openshift-console/console-8677894d4d-zwdn4"
Jul 15 10:47:41.473006 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:41.471561    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thvfs\" (UniqueName: \"kubernetes.io/projected/6294ad35-0d5d-4ba7-9d44-15d91effa356-kube-api-access-thvfs\") pod \"console-8677894d4d-zwdn4\" (UID: \"6294ad35-0d5d-4ba7-9d44-15d91effa356\") " pod="openshift-console/console-8677894d4d-zwdn4"
Jul 15 10:47:41.473006 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:41.471620    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6294ad35-0d5d-4ba7-9d44-15d91effa356-service-ca\") pod \"console-8677894d4d-zwdn4\" (UID: \"6294ad35-0d5d-4ba7-9d44-15d91effa356\") " pod="openshift-console/console-8677894d4d-zwdn4"
Jul 15 10:47:41.473006 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:41.471642    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6294ad35-0d5d-4ba7-9d44-15d91effa356-oauth-serving-cert\") pod \"console-8677894d4d-zwdn4\" (UID: \"6294ad35-0d5d-4ba7-9d44-15d91effa356\") " pod="openshift-console/console-8677894d4d-zwdn4"
Jul 15 10:47:41.473006 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:41.472647    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6294ad35-0d5d-4ba7-9d44-15d91effa356-oauth-serving-cert\") pod \"console-8677894d4d-zwdn4\" (UID: \"6294ad35-0d5d-4ba7-9d44-15d91effa356\") " pod="openshift-console/console-8677894d4d-zwdn4"
Jul 15 10:47:41.473394 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:41.473342    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6294ad35-0d5d-4ba7-9d44-15d91effa356-console-config\") pod \"console-8677894d4d-zwdn4\" (UID: \"6294ad35-0d5d-4ba7-9d44-15d91effa356\") " pod="openshift-console/console-8677894d4d-zwdn4"
Jul 15 10:47:41.473517 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:41.473434    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6294ad35-0d5d-4ba7-9d44-15d91effa356-trusted-ca-bundle\") pod \"console-8677894d4d-zwdn4\" (UID: \"6294ad35-0d5d-4ba7-9d44-15d91effa356\") " pod="openshift-console/console-8677894d4d-zwdn4"
Jul 15 10:47:41.474014 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:41.473896    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6294ad35-0d5d-4ba7-9d44-15d91effa356-service-ca\") pod \"console-8677894d4d-zwdn4\" (UID: \"6294ad35-0d5d-4ba7-9d44-15d91effa356\") " pod="openshift-console/console-8677894d4d-zwdn4"
Jul 15 10:47:41.476428 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:41.476364    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6294ad35-0d5d-4ba7-9d44-15d91effa356-console-oauth-config\") pod \"console-8677894d4d-zwdn4\" (UID: \"6294ad35-0d5d-4ba7-9d44-15d91effa356\") " pod="openshift-console/console-8677894d4d-zwdn4"
Jul 15 10:47:41.476575 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:41.476432    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6294ad35-0d5d-4ba7-9d44-15d91effa356-console-serving-cert\") pod \"console-8677894d4d-zwdn4\" (UID: \"6294ad35-0d5d-4ba7-9d44-15d91effa356\") " pod="openshift-console/console-8677894d4d-zwdn4"
Jul 15 10:47:41.488578 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:41.488535    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thvfs\" (UniqueName: \"kubernetes.io/projected/6294ad35-0d5d-4ba7-9d44-15d91effa356-kube-api-access-thvfs\") pod \"console-8677894d4d-zwdn4\" (UID: \"6294ad35-0d5d-4ba7-9d44-15d91effa356\") " pod="openshift-console/console-8677894d4d-zwdn4"
Jul 15 10:47:41.559867 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:41.559810    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8677894d4d-zwdn4"
Jul 15 10:47:41.743780 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:41.743749    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8677894d4d-zwdn4"]
Jul 15 10:47:41.746468 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:47:41.746425    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6294ad35_0d5d_4ba7_9d44_15d91effa356.slice/crio-4d38c44ae44f4b9bc197df797f4c9d412c33b8d8a3d6e7f26388ef82e79533b7 WatchSource:0}: Error finding container 4d38c44ae44f4b9bc197df797f4c9d412c33b8d8a3d6e7f26388ef82e79533b7: Status 404 returned error can't find the container with id 4d38c44ae44f4b9bc197df797f4c9d412c33b8d8a3d6e7f26388ef82e79533b7
Jul 15 10:47:42.180456 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:42.180362    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8677894d4d-zwdn4" event={"ID":"6294ad35-0d5d-4ba7-9d44-15d91effa356","Type":"ContainerStarted","Data":"0ff21212972d3fd980190a45822f3aa3f09ebee058cc0ed85f7c6091c539dd54"}
Jul 15 10:47:42.180456 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:42.180428    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8677894d4d-zwdn4" event={"ID":"6294ad35-0d5d-4ba7-9d44-15d91effa356","Type":"ContainerStarted","Data":"4d38c44ae44f4b9bc197df797f4c9d412c33b8d8a3d6e7f26388ef82e79533b7"}
Jul 15 10:47:42.222103 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:42.222043    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8677894d4d-zwdn4" podStartSLOduration=1.222019673 podStartE2EDuration="1.222019673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:42.220908124 +0000 UTC m=+81.050423530" watchObservedRunningTime="2026-07-15 10:47:42.222019673 +0000 UTC m=+81.051535080"
Jul 15 10:47:44.192837 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:44.192795    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d4f1d838-0655-4218-8247-4233d2e0ac04","Type":"ContainerStarted","Data":"54f8627cf75ba116391a864d29a824305ddcdd5f3328cb2e5bced1650d8e050b"}
Jul 15 10:47:44.193240 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:44.192871    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d4f1d838-0655-4218-8247-4233d2e0ac04","Type":"ContainerStarted","Data":"1c1731dc53358c58ddc1bd80b4c66a7a40f4c71a6fe19d63ee4f4147c8727962"}
Jul 15 10:47:46.207684 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:46.207645    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d4f1d838-0655-4218-8247-4233d2e0ac04","Type":"ContainerStarted","Data":"0319e2caeb3f2f54020763872ac7900cd290c3c1bed7a323e868c8592a1b8832"}
Jul 15 10:47:46.208063 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:46.207727    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d4f1d838-0655-4218-8247-4233d2e0ac04","Type":"ContainerStarted","Data":"4b54276e75d2a73ee6f2fb07339fe2b3c8ad93110f60631001883b37c76cc9c9"}
Jul 15 10:47:46.208063 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:46.207755    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d4f1d838-0655-4218-8247-4233d2e0ac04","Type":"ContainerStarted","Data":"890dc9b1b461fce071bbe4bc66b3ea86a71be45b2c65e667d144f38f7b3d0fea"}
Jul 15 10:47:46.208063 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:46.207781    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d4f1d838-0655-4218-8247-4233d2e0ac04","Type":"ContainerStarted","Data":"226e0e35ea12ffb196d15732fda55e17b60cf8eb33afdfd0c293a84ab2f7e874"}
Jul 15 10:47:46.250580 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:46.250522    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.72816188 podStartE2EDuration="9.250499403s" totalImagesPullingTime="6.500601082s" totalInitContainerRuntime="21.736441ms" isStatefulPod=false podCreationTimestamp="2026-07-15 10:47:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:46.247519121 +0000 UTC m=+85.077034563" watchObservedRunningTime="2026-07-15 10:47:46.250499403 +0000 UTC m=+85.080014810"
Jul 15 10:47:46.318129 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:46.318092    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f8488489b-q92p9"
Jul 15 10:47:47.692464 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:47.692417    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:47:51.560498 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:51.560456    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8677894d4d-zwdn4"
Jul 15 10:47:51.561090 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:51.560539    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8677894d4d-zwdn4"
Jul 15 10:47:51.565614 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:51.565588    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8677894d4d-zwdn4"
Jul 15 10:47:52.240381 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:52.240353    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8677894d4d-zwdn4"
Jul 15 10:47:52.310541 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:47:52.310507    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5ff9595488-rmsmj"]
Jul 15 10:48:00.975806 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:00.975730    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-88d4d74bd-v4zn7" podUID="3a522d3d-13c9-4a9c-949d-f30304744f77" containerName="console" containerID="cri-o://b9bbf0793522807d7b81612f1805a46913fd3756eab86c224eb238777f42fee0" gracePeriod=15
Jul 15 10:48:01.218905 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:01.218879    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88d4d74bd-v4zn7"
Jul 15 10:48:01.243028 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:01.242939    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xmwv\" (UniqueName: \"kubernetes.io/projected/3a522d3d-13c9-4a9c-949d-f30304744f77-kube-api-access-6xmwv\") pod \"3a522d3d-13c9-4a9c-949d-f30304744f77\" (UID: \"3a522d3d-13c9-4a9c-949d-f30304744f77\") "
Jul 15 10:48:01.243028 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:01.242994    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a522d3d-13c9-4a9c-949d-f30304744f77-trusted-ca-bundle\") pod \"3a522d3d-13c9-4a9c-949d-f30304744f77\" (UID: \"3a522d3d-13c9-4a9c-949d-f30304744f77\") "
Jul 15 10:48:01.243503 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:01.243306    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a522d3d-13c9-4a9c-949d-f30304744f77-console-oauth-config\") pod \"3a522d3d-13c9-4a9c-949d-f30304744f77\" (UID: \"3a522d3d-13c9-4a9c-949d-f30304744f77\") "
Jul 15 10:48:01.243503 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:01.243372    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a522d3d-13c9-4a9c-949d-f30304744f77-console-config\") pod \"3a522d3d-13c9-4a9c-949d-f30304744f77\" (UID: \"3a522d3d-13c9-4a9c-949d-f30304744f77\") "
Jul 15 10:48:01.243503 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:01.243413    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a522d3d-13c9-4a9c-949d-f30304744f77-console-serving-cert\") pod \"3a522d3d-13c9-4a9c-949d-f30304744f77\" (UID: \"3a522d3d-13c9-4a9c-949d-f30304744f77\") "
Jul 15 10:48:01.243503 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:01.243440    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a522d3d-13c9-4a9c-949d-f30304744f77-service-ca\") pod \"3a522d3d-13c9-4a9c-949d-f30304744f77\" (UID: \"3a522d3d-13c9-4a9c-949d-f30304744f77\") "
Jul 15 10:48:01.243503 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:01.243440    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a522d3d-13c9-4a9c-949d-f30304744f77-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3a522d3d-13c9-4a9c-949d-f30304744f77" (UID: "3a522d3d-13c9-4a9c-949d-f30304744f77"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:48:01.243503 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:01.243494    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a522d3d-13c9-4a9c-949d-f30304744f77-oauth-serving-cert\") pod \"3a522d3d-13c9-4a9c-949d-f30304744f77\" (UID: \"3a522d3d-13c9-4a9c-949d-f30304744f77\") "
Jul 15 10:48:01.243831 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:01.243768    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a522d3d-13c9-4a9c-949d-f30304744f77-trusted-ca-bundle\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:48:01.243831 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:01.243728    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a522d3d-13c9-4a9c-949d-f30304744f77-console-config" (OuterVolumeSpecName: "console-config") pod "3a522d3d-13c9-4a9c-949d-f30304744f77" (UID: "3a522d3d-13c9-4a9c-949d-f30304744f77"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:48:01.244063 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:01.243917    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a522d3d-13c9-4a9c-949d-f30304744f77-service-ca" (OuterVolumeSpecName: "service-ca") pod "3a522d3d-13c9-4a9c-949d-f30304744f77" (UID: "3a522d3d-13c9-4a9c-949d-f30304744f77"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:48:01.244509 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:01.244075    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a522d3d-13c9-4a9c-949d-f30304744f77-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3a522d3d-13c9-4a9c-949d-f30304744f77" (UID: "3a522d3d-13c9-4a9c-949d-f30304744f77"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:48:01.245749 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:01.245725    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a522d3d-13c9-4a9c-949d-f30304744f77-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3a522d3d-13c9-4a9c-949d-f30304744f77" (UID: "3a522d3d-13c9-4a9c-949d-f30304744f77"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 10:48:01.245867 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:01.245812    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a522d3d-13c9-4a9c-949d-f30304744f77-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3a522d3d-13c9-4a9c-949d-f30304744f77" (UID: "3a522d3d-13c9-4a9c-949d-f30304744f77"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 10:48:01.245930 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:01.245912    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a522d3d-13c9-4a9c-949d-f30304744f77-kube-api-access-6xmwv" (OuterVolumeSpecName: "kube-api-access-6xmwv") pod "3a522d3d-13c9-4a9c-949d-f30304744f77" (UID: "3a522d3d-13c9-4a9c-949d-f30304744f77"). InnerVolumeSpecName "kube-api-access-6xmwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 10:48:01.271439 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:01.271404    2563 generic.go:356] "Generic (PLEG): container finished" podID="3a522d3d-13c9-4a9c-949d-f30304744f77" containerID="b9bbf0793522807d7b81612f1805a46913fd3756eab86c224eb238777f42fee0" exitCode=0
Jul 15 10:48:01.271605 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:01.271450    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88d4d74bd-v4zn7"
Jul 15 10:48:01.271605 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:01.271464    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88d4d74bd-v4zn7" event={"ID":"3a522d3d-13c9-4a9c-949d-f30304744f77","Type":"ContainerDied","Data":"b9bbf0793522807d7b81612f1805a46913fd3756eab86c224eb238777f42fee0"}
Jul 15 10:48:01.271605 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:01.271505    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88d4d74bd-v4zn7" event={"ID":"3a522d3d-13c9-4a9c-949d-f30304744f77","Type":"ContainerDied","Data":"654986954ed09dda528b29a0e018174f700e7c8405dfd683ded597e5ab6f0a2f"}
Jul 15 10:48:01.271605 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:01.271524    2563 scope.go:117] "RemoveContainer" containerID="b9bbf0793522807d7b81612f1805a46913fd3756eab86c224eb238777f42fee0"
Jul 15 10:48:01.280090 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:01.280072    2563 scope.go:117] "RemoveContainer" containerID="b9bbf0793522807d7b81612f1805a46913fd3756eab86c224eb238777f42fee0"
Jul 15 10:48:01.280401 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:48:01.280382    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9bbf0793522807d7b81612f1805a46913fd3756eab86c224eb238777f42fee0\": container with ID starting with b9bbf0793522807d7b81612f1805a46913fd3756eab86c224eb238777f42fee0 not found: ID does not exist" containerID="b9bbf0793522807d7b81612f1805a46913fd3756eab86c224eb238777f42fee0"
Jul 15 10:48:01.280451 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:01.280410    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9bbf0793522807d7b81612f1805a46913fd3756eab86c224eb238777f42fee0"} err="failed to get container status \"b9bbf0793522807d7b81612f1805a46913fd3756eab86c224eb238777f42fee0\": rpc error: code = NotFound desc = could not find container \"b9bbf0793522807d7b81612f1805a46913fd3756eab86c224eb238777f42fee0\": container with ID starting with b9bbf0793522807d7b81612f1805a46913fd3756eab86c224eb238777f42fee0 not found: ID does not exist"
Jul 15 10:48:01.294882 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:01.294852    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-88d4d74bd-v4zn7"]
Jul 15 10:48:01.298776 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:01.298752    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-88d4d74bd-v4zn7"]
Jul 15 10:48:01.344947 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:01.344916    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xmwv\" (UniqueName: \"kubernetes.io/projected/3a522d3d-13c9-4a9c-949d-f30304744f77-kube-api-access-6xmwv\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:48:01.344947 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:01.344946    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a522d3d-13c9-4a9c-949d-f30304744f77-console-oauth-config\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:48:01.345109 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:01.344955    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a522d3d-13c9-4a9c-949d-f30304744f77-console-config\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:48:01.345109 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:01.344964    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a522d3d-13c9-4a9c-949d-f30304744f77-console-serving-cert\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:48:01.345109 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:01.344973    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a522d3d-13c9-4a9c-949d-f30304744f77-service-ca\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:48:01.345109 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:01.344981    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a522d3d-13c9-4a9c-949d-f30304744f77-oauth-serving-cert\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:48:01.677735 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:01.677637    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a522d3d-13c9-4a9c-949d-f30304744f77" path="/var/lib/kubelet/pods/3a522d3d-13c9-4a9c-949d-f30304744f77/volumes"
Jul 15 10:48:05.132239 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:05.132211    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9mhqc"
Jul 15 10:48:06.186728 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:06.183518    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-f8488489b-q92p9" podUID="a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1" containerName="console" containerID="cri-o://61fbeebccf765180d57dbe2570667638b3dbf6f014e1ec17e87acf61da3d2899" gracePeriod=15
Jul 15 10:48:06.475256 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:06.475220    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f8488489b-q92p9"
Jul 15 10:48:06.586438 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:06.586397    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-trusted-ca-bundle\") pod \"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1\" (UID: \"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1\") "
Jul 15 10:48:06.586638 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:06.586472    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9fsh\" (UniqueName: \"kubernetes.io/projected/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-kube-api-access-b9fsh\") pod \"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1\" (UID: \"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1\") "
Jul 15 10:48:06.586638 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:06.586500    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-console-oauth-config\") pod \"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1\" (UID: \"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1\") "
Jul 15 10:48:06.586638 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:06.586520    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-console-serving-cert\") pod \"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1\" (UID: \"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1\") "
Jul 15 10:48:06.586638 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:06.586545    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-service-ca\") pod \"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1\" (UID: \"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1\") "
Jul 15 10:48:06.586638 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:06.586581    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-oauth-serving-cert\") pod \"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1\" (UID: \"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1\") "
Jul 15 10:48:06.586638 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:06.586608    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-console-config\") pod \"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1\" (UID: \"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1\") "
Jul 15 10:48:06.587023 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:06.586901    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1" (UID: "a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:48:06.587211 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:06.587169    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1" (UID: "a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:48:06.587297 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:06.587220    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-service-ca" (OuterVolumeSpecName: "service-ca") pod "a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1" (UID: "a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:48:06.587297 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:06.587238    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-console-config" (OuterVolumeSpecName: "console-config") pod "a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1" (UID: "a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:48:06.589072 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:06.589039    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1" (UID: "a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 10:48:06.589353 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:06.589333    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1" (UID: "a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 10:48:06.589485 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:06.589463    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-kube-api-access-b9fsh" (OuterVolumeSpecName: "kube-api-access-b9fsh") pod "a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1" (UID: "a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1"). InnerVolumeSpecName "kube-api-access-b9fsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 10:48:06.688265 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:06.688165    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-console-config\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:48:06.688265 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:06.688200    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-trusted-ca-bundle\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:48:06.688265 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:06.688212    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9fsh\" (UniqueName: \"kubernetes.io/projected/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-kube-api-access-b9fsh\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:48:06.688265 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:06.688222    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-console-oauth-config\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:48:06.688265 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:06.688233    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-console-serving-cert\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:48:06.688265 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:06.688241    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-service-ca\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:48:06.688265 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:06.688250    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1-oauth-serving-cert\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:48:07.297688 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:07.297651    2563 generic.go:356] "Generic (PLEG): container finished" podID="a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1" containerID="61fbeebccf765180d57dbe2570667638b3dbf6f014e1ec17e87acf61da3d2899" exitCode=0
Jul 15 10:48:07.298249 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:07.297701    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f8488489b-q92p9" event={"ID":"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1","Type":"ContainerDied","Data":"61fbeebccf765180d57dbe2570667638b3dbf6f014e1ec17e87acf61da3d2899"}
Jul 15 10:48:07.298249 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:07.297748    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f8488489b-q92p9" event={"ID":"a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1","Type":"ContainerDied","Data":"b5ef821ca2d5e792d6e8bcfcff272bcb647781592090d601e6ba671d08ab5ea0"}
Jul 15 10:48:07.298249 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:07.297771    2563 scope.go:117] "RemoveContainer" containerID="61fbeebccf765180d57dbe2570667638b3dbf6f014e1ec17e87acf61da3d2899"
Jul 15 10:48:07.298249 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:07.297707    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f8488489b-q92p9"
Jul 15 10:48:07.307109 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:07.307087    2563 scope.go:117] "RemoveContainer" containerID="61fbeebccf765180d57dbe2570667638b3dbf6f014e1ec17e87acf61da3d2899"
Jul 15 10:48:07.307486 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:48:07.307464    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61fbeebccf765180d57dbe2570667638b3dbf6f014e1ec17e87acf61da3d2899\": container with ID starting with 61fbeebccf765180d57dbe2570667638b3dbf6f014e1ec17e87acf61da3d2899 not found: ID does not exist" containerID="61fbeebccf765180d57dbe2570667638b3dbf6f014e1ec17e87acf61da3d2899"
Jul 15 10:48:07.307558 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:07.307497    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61fbeebccf765180d57dbe2570667638b3dbf6f014e1ec17e87acf61da3d2899"} err="failed to get container status \"61fbeebccf765180d57dbe2570667638b3dbf6f014e1ec17e87acf61da3d2899\": rpc error: code = NotFound desc = could not find container \"61fbeebccf765180d57dbe2570667638b3dbf6f014e1ec17e87acf61da3d2899\": container with ID starting with 61fbeebccf765180d57dbe2570667638b3dbf6f014e1ec17e87acf61da3d2899 not found: ID does not exist"
Jul 15 10:48:07.321936 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:07.321895    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f8488489b-q92p9"]
Jul 15 10:48:07.324369 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:07.324332    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f8488489b-q92p9"]
Jul 15 10:48:07.678053 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:07.677954    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1" path="/var/lib/kubelet/pods/a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1/volumes"
Jul 15 10:48:17.333164 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:17.333099    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5ff9595488-rmsmj" podUID="482314fa-4e2c-45e3-b506-053bd6d63b6c" containerName="console" containerID="cri-o://4c3fade0383aed5c22f5a4ce0a19d05d868837c5a53a462dedab93d9efcc8378" gracePeriod=15
Jul 15 10:48:17.633586 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:17.633540    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff9595488-rmsmj"
Jul 15 10:48:17.693317 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:17.693274    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/482314fa-4e2c-45e3-b506-053bd6d63b6c-service-ca\") pod \"482314fa-4e2c-45e3-b506-053bd6d63b6c\" (UID: \"482314fa-4e2c-45e3-b506-053bd6d63b6c\") "
Jul 15 10:48:17.693317 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:17.693320    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mbsk\" (UniqueName: \"kubernetes.io/projected/482314fa-4e2c-45e3-b506-053bd6d63b6c-kube-api-access-7mbsk\") pod \"482314fa-4e2c-45e3-b506-053bd6d63b6c\" (UID: \"482314fa-4e2c-45e3-b506-053bd6d63b6c\") "
Jul 15 10:48:17.693626 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:17.693343    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/482314fa-4e2c-45e3-b506-053bd6d63b6c-console-oauth-config\") pod \"482314fa-4e2c-45e3-b506-053bd6d63b6c\" (UID: \"482314fa-4e2c-45e3-b506-053bd6d63b6c\") "
Jul 15 10:48:17.693626 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:17.693380    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/482314fa-4e2c-45e3-b506-053bd6d63b6c-console-serving-cert\") pod \"482314fa-4e2c-45e3-b506-053bd6d63b6c\" (UID: \"482314fa-4e2c-45e3-b506-053bd6d63b6c\") "
Jul 15 10:48:17.693626 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:17.693428    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/482314fa-4e2c-45e3-b506-053bd6d63b6c-console-config\") pod \"482314fa-4e2c-45e3-b506-053bd6d63b6c\" (UID: \"482314fa-4e2c-45e3-b506-053bd6d63b6c\") "
Jul 15 10:48:17.693626 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:17.693471    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/482314fa-4e2c-45e3-b506-053bd6d63b6c-oauth-serving-cert\") pod \"482314fa-4e2c-45e3-b506-053bd6d63b6c\" (UID: \"482314fa-4e2c-45e3-b506-053bd6d63b6c\") "
Jul 15 10:48:17.693901 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:17.693870    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/482314fa-4e2c-45e3-b506-053bd6d63b6c-service-ca" (OuterVolumeSpecName: "service-ca") pod "482314fa-4e2c-45e3-b506-053bd6d63b6c" (UID: "482314fa-4e2c-45e3-b506-053bd6d63b6c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:48:17.693957 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:17.693912    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/482314fa-4e2c-45e3-b506-053bd6d63b6c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "482314fa-4e2c-45e3-b506-053bd6d63b6c" (UID: "482314fa-4e2c-45e3-b506-053bd6d63b6c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:48:17.693957 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:17.693932    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/482314fa-4e2c-45e3-b506-053bd6d63b6c-console-config" (OuterVolumeSpecName: "console-config") pod "482314fa-4e2c-45e3-b506-053bd6d63b6c" (UID: "482314fa-4e2c-45e3-b506-053bd6d63b6c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:48:17.695899 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:17.695855    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/482314fa-4e2c-45e3-b506-053bd6d63b6c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "482314fa-4e2c-45e3-b506-053bd6d63b6c" (UID: "482314fa-4e2c-45e3-b506-053bd6d63b6c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 10:48:17.695899 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:17.695864    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/482314fa-4e2c-45e3-b506-053bd6d63b6c-kube-api-access-7mbsk" (OuterVolumeSpecName: "kube-api-access-7mbsk") pod "482314fa-4e2c-45e3-b506-053bd6d63b6c" (UID: "482314fa-4e2c-45e3-b506-053bd6d63b6c"). InnerVolumeSpecName "kube-api-access-7mbsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 10:48:17.696229 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:17.695916    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/482314fa-4e2c-45e3-b506-053bd6d63b6c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "482314fa-4e2c-45e3-b506-053bd6d63b6c" (UID: "482314fa-4e2c-45e3-b506-053bd6d63b6c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 10:48:17.794670 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:17.794614    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/482314fa-4e2c-45e3-b506-053bd6d63b6c-console-config\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:48:17.794670 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:17.794673    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/482314fa-4e2c-45e3-b506-053bd6d63b6c-oauth-serving-cert\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:48:17.794920 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:17.794694    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/482314fa-4e2c-45e3-b506-053bd6d63b6c-service-ca\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:48:17.794920 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:17.794713    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mbsk\" (UniqueName: \"kubernetes.io/projected/482314fa-4e2c-45e3-b506-053bd6d63b6c-kube-api-access-7mbsk\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:48:17.794920 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:17.794733    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/482314fa-4e2c-45e3-b506-053bd6d63b6c-console-oauth-config\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:48:17.794920 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:17.794752    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/482314fa-4e2c-45e3-b506-053bd6d63b6c-console-serving-cert\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:48:18.352271 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:18.352232    2563 generic.go:356] "Generic (PLEG): container finished" podID="482314fa-4e2c-45e3-b506-053bd6d63b6c" containerID="4c3fade0383aed5c22f5a4ce0a19d05d868837c5a53a462dedab93d9efcc8378" exitCode=0
Jul 15 10:48:18.352271 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:18.352283    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff9595488-rmsmj"
Jul 15 10:48:18.352762 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:18.352283    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff9595488-rmsmj" event={"ID":"482314fa-4e2c-45e3-b506-053bd6d63b6c","Type":"ContainerDied","Data":"4c3fade0383aed5c22f5a4ce0a19d05d868837c5a53a462dedab93d9efcc8378"}
Jul 15 10:48:18.352762 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:18.352416    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff9595488-rmsmj" event={"ID":"482314fa-4e2c-45e3-b506-053bd6d63b6c","Type":"ContainerDied","Data":"4d755bef17816e1d356400d283d9f3618a797532338ac25541cebf49cfd6c4b3"}
Jul 15 10:48:18.352762 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:18.352457    2563 scope.go:117] "RemoveContainer" containerID="4c3fade0383aed5c22f5a4ce0a19d05d868837c5a53a462dedab93d9efcc8378"
Jul 15 10:48:18.371816 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:18.371793    2563 scope.go:117] "RemoveContainer" containerID="4c3fade0383aed5c22f5a4ce0a19d05d868837c5a53a462dedab93d9efcc8378"
Jul 15 10:48:18.372179 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:48:18.372153    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c3fade0383aed5c22f5a4ce0a19d05d868837c5a53a462dedab93d9efcc8378\": container with ID starting with 4c3fade0383aed5c22f5a4ce0a19d05d868837c5a53a462dedab93d9efcc8378 not found: ID does not exist" containerID="4c3fade0383aed5c22f5a4ce0a19d05d868837c5a53a462dedab93d9efcc8378"
Jul 15 10:48:18.372238 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:18.372191    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c3fade0383aed5c22f5a4ce0a19d05d868837c5a53a462dedab93d9efcc8378"} err="failed to get container status \"4c3fade0383aed5c22f5a4ce0a19d05d868837c5a53a462dedab93d9efcc8378\": rpc error: code = NotFound desc = could not find container \"4c3fade0383aed5c22f5a4ce0a19d05d868837c5a53a462dedab93d9efcc8378\": container with ID starting with 4c3fade0383aed5c22f5a4ce0a19d05d868837c5a53a462dedab93d9efcc8378 not found: ID does not exist"
Jul 15 10:48:18.380432 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:18.380393    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5ff9595488-rmsmj"]
Jul 15 10:48:18.383416 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:18.383376    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5ff9595488-rmsmj"]
Jul 15 10:48:19.677861 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:19.677823    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="482314fa-4e2c-45e3-b506-053bd6d63b6c" path="/var/lib/kubelet/pods/482314fa-4e2c-45e3-b506-053bd6d63b6c/volumes"
Jul 15 10:48:37.692629 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:37.692593    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:48:37.712529 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:37.712496    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:48:38.442449 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:48:38.442413    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 10:49:01.410872 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:01.410832    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-699c49876d-94ksh"]
Jul 15 10:49:01.411352 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:01.411213    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1" containerName="console"
Jul 15 10:49:01.411352 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:01.411230    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1" containerName="console"
Jul 15 10:49:01.411352 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:01.411260    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="482314fa-4e2c-45e3-b506-053bd6d63b6c" containerName="console"
Jul 15 10:49:01.411352 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:01.411270    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="482314fa-4e2c-45e3-b506-053bd6d63b6c" containerName="console"
Jul 15 10:49:01.411352 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:01.411279    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a522d3d-13c9-4a9c-949d-f30304744f77" containerName="console"
Jul 15 10:49:01.411352 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:01.411284    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a522d3d-13c9-4a9c-949d-f30304744f77" containerName="console"
Jul 15 10:49:01.411352 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:01.411330    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5bb1b91-6f58-4406-b7cd-2ed2cb4289f1" containerName="console"
Jul 15 10:49:01.411352 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:01.411343    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="482314fa-4e2c-45e3-b506-053bd6d63b6c" containerName="console"
Jul 15 10:49:01.411352 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:01.411353    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a522d3d-13c9-4a9c-949d-f30304744f77" containerName="console"
Jul 15 10:49:01.413576 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:01.413544    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-699c49876d-94ksh"
Jul 15 10:49:01.441116 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:01.441079    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-699c49876d-94ksh"]
Jul 15 10:49:01.446711 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:01.446666    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-console-config\") pod \"console-699c49876d-94ksh\" (UID: \"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89\") " pod="openshift-console/console-699c49876d-94ksh"
Jul 15 10:49:01.446711 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:01.446717    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-console-oauth-config\") pod \"console-699c49876d-94ksh\" (UID: \"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89\") " pod="openshift-console/console-699c49876d-94ksh"
Jul 15 10:49:01.446948 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:01.446830    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-service-ca\") pod \"console-699c49876d-94ksh\" (UID: \"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89\") " pod="openshift-console/console-699c49876d-94ksh"
Jul 15 10:49:01.446948 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:01.446885    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-console-serving-cert\") pod \"console-699c49876d-94ksh\" (UID: \"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89\") " pod="openshift-console/console-699c49876d-94ksh"
Jul 15 10:49:01.446948 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:01.446913    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-trusted-ca-bundle\") pod \"console-699c49876d-94ksh\" (UID: \"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89\") " pod="openshift-console/console-699c49876d-94ksh"
Jul 15 10:49:01.446948 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:01.446944    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn78w\" (UniqueName: \"kubernetes.io/projected/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-kube-api-access-vn78w\") pod \"console-699c49876d-94ksh\" (UID: \"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89\") " pod="openshift-console/console-699c49876d-94ksh"
Jul 15 10:49:01.447109 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:01.446972    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-oauth-serving-cert\") pod \"console-699c49876d-94ksh\" (UID: \"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89\") " pod="openshift-console/console-699c49876d-94ksh"
Jul 15 10:49:01.548353 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:01.548304    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn78w\" (UniqueName: \"kubernetes.io/projected/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-kube-api-access-vn78w\") pod \"console-699c49876d-94ksh\" (UID: \"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89\") " pod="openshift-console/console-699c49876d-94ksh"
Jul 15 10:49:01.548353 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:01.548347    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-oauth-serving-cert\") pod \"console-699c49876d-94ksh\" (UID: \"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89\") " pod="openshift-console/console-699c49876d-94ksh"
Jul 15 10:49:01.548589 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:01.548460    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-console-config\") pod \"console-699c49876d-94ksh\" (UID: \"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89\") " pod="openshift-console/console-699c49876d-94ksh"
Jul 15 10:49:01.548589 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:01.548497    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-console-oauth-config\") pod \"console-699c49876d-94ksh\" (UID: \"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89\") " pod="openshift-console/console-699c49876d-94ksh"
Jul 15 10:49:01.548589 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:01.548553    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-service-ca\") pod \"console-699c49876d-94ksh\" (UID: \"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89\") " pod="openshift-console/console-699c49876d-94ksh"
Jul 15 10:49:01.548722 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:01.548602    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-console-serving-cert\") pod \"console-699c49876d-94ksh\" (UID: \"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89\") " pod="openshift-console/console-699c49876d-94ksh"
Jul 15 10:49:01.548722 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:01.548645    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-trusted-ca-bundle\") pod \"console-699c49876d-94ksh\" (UID: \"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89\") " pod="openshift-console/console-699c49876d-94ksh"
Jul 15 10:49:01.549206 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:01.549172    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-oauth-serving-cert\") pod \"console-699c49876d-94ksh\" (UID: \"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89\") " pod="openshift-console/console-699c49876d-94ksh"
Jul 15 10:49:01.549349 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:01.549232    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-console-config\") pod \"console-699c49876d-94ksh\" (UID: \"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89\") " pod="openshift-console/console-699c49876d-94ksh"
Jul 15 10:49:01.549416 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:01.549355    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-service-ca\") pod \"console-699c49876d-94ksh\" (UID: \"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89\") " pod="openshift-console/console-699c49876d-94ksh"
Jul 15 10:49:01.549473 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:01.549456    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-trusted-ca-bundle\") pod \"console-699c49876d-94ksh\" (UID: \"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89\") " pod="openshift-console/console-699c49876d-94ksh"
Jul 15 10:49:01.551266 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:01.551235    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-console-serving-cert\") pod \"console-699c49876d-94ksh\" (UID: \"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89\") " pod="openshift-console/console-699c49876d-94ksh"
Jul 15 10:49:01.551374 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:01.551242    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-console-oauth-config\") pod \"console-699c49876d-94ksh\" (UID: \"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89\") " pod="openshift-console/console-699c49876d-94ksh"
Jul 15 10:49:01.559658 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:01.559624    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn78w\" (UniqueName: \"kubernetes.io/projected/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-kube-api-access-vn78w\") pod \"console-699c49876d-94ksh\" (UID: \"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89\") " pod="openshift-console/console-699c49876d-94ksh"
Jul 15 10:49:01.723902 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:01.723854    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-699c49876d-94ksh"
Jul 15 10:49:01.871352 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:01.871312    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-699c49876d-94ksh"]
Jul 15 10:49:01.874610 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:49:01.874580    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a93162d_3cf8_4cd3_bc95_e0fab2b75c89.slice/crio-a68367cd31185b55a4eb1f9987bf3a7971755fea17ae6eb6afb46e4355057392 WatchSource:0}: Error finding container a68367cd31185b55a4eb1f9987bf3a7971755fea17ae6eb6afb46e4355057392: Status 404 returned error can't find the container with id a68367cd31185b55a4eb1f9987bf3a7971755fea17ae6eb6afb46e4355057392
Jul 15 10:49:02.519266 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:02.519228    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-699c49876d-94ksh" event={"ID":"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89","Type":"ContainerStarted","Data":"eee94e9b5d9b0efeaa1164ca525e03430bd4281b2a6dfa3bc05b70a33378ad3c"}
Jul 15 10:49:02.519806 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:02.519295    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-699c49876d-94ksh" event={"ID":"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89","Type":"ContainerStarted","Data":"a68367cd31185b55a4eb1f9987bf3a7971755fea17ae6eb6afb46e4355057392"}
Jul 15 10:49:02.560332 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:02.560279    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-699c49876d-94ksh" podStartSLOduration=1.560259719 podStartE2EDuration="1.560259719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:49:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:49:02.559839915 +0000 UTC m=+161.389355319" watchObservedRunningTime="2026-07-15 10:49:02.560259719 +0000 UTC m=+161.389775123"
Jul 15 10:49:11.724548 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:11.724498    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-699c49876d-94ksh"
Jul 15 10:49:11.725004 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:11.724581    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-699c49876d-94ksh"
Jul 15 10:49:11.729757 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:11.729728    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-699c49876d-94ksh"
Jul 15 10:49:12.562558 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:12.562522    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-699c49876d-94ksh"
Jul 15 10:49:12.625208 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:12.625174    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8677894d4d-zwdn4"]
Jul 15 10:49:37.647845 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:37.647754    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8677894d4d-zwdn4" podUID="6294ad35-0d5d-4ba7-9d44-15d91effa356" containerName="console" containerID="cri-o://0ff21212972d3fd980190a45822f3aa3f09ebee058cc0ed85f7c6091c539dd54" gracePeriod=15
Jul 15 10:49:37.886475 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:37.886449    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8677894d4d-zwdn4"
Jul 15 10:49:37.963030 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:37.962997    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thvfs\" (UniqueName: \"kubernetes.io/projected/6294ad35-0d5d-4ba7-9d44-15d91effa356-kube-api-access-thvfs\") pod \"6294ad35-0d5d-4ba7-9d44-15d91effa356\" (UID: \"6294ad35-0d5d-4ba7-9d44-15d91effa356\") "
Jul 15 10:49:37.963229 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:37.963054    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6294ad35-0d5d-4ba7-9d44-15d91effa356-console-config\") pod \"6294ad35-0d5d-4ba7-9d44-15d91effa356\" (UID: \"6294ad35-0d5d-4ba7-9d44-15d91effa356\") "
Jul 15 10:49:37.963229 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:37.963128    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6294ad35-0d5d-4ba7-9d44-15d91effa356-trusted-ca-bundle\") pod \"6294ad35-0d5d-4ba7-9d44-15d91effa356\" (UID: \"6294ad35-0d5d-4ba7-9d44-15d91effa356\") "
Jul 15 10:49:37.963229 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:37.963193    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6294ad35-0d5d-4ba7-9d44-15d91effa356-console-serving-cert\") pod \"6294ad35-0d5d-4ba7-9d44-15d91effa356\" (UID: \"6294ad35-0d5d-4ba7-9d44-15d91effa356\") "
Jul 15 10:49:37.963229 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:37.963225    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6294ad35-0d5d-4ba7-9d44-15d91effa356-console-oauth-config\") pod \"6294ad35-0d5d-4ba7-9d44-15d91effa356\" (UID: \"6294ad35-0d5d-4ba7-9d44-15d91effa356\") "
Jul 15 10:49:37.963429 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:37.963256    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6294ad35-0d5d-4ba7-9d44-15d91effa356-oauth-serving-cert\") pod \"6294ad35-0d5d-4ba7-9d44-15d91effa356\" (UID: \"6294ad35-0d5d-4ba7-9d44-15d91effa356\") "
Jul 15 10:49:37.963429 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:37.963284    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6294ad35-0d5d-4ba7-9d44-15d91effa356-service-ca\") pod \"6294ad35-0d5d-4ba7-9d44-15d91effa356\" (UID: \"6294ad35-0d5d-4ba7-9d44-15d91effa356\") "
Jul 15 10:49:37.963559 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:37.963527    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6294ad35-0d5d-4ba7-9d44-15d91effa356-console-config" (OuterVolumeSpecName: "console-config") pod "6294ad35-0d5d-4ba7-9d44-15d91effa356" (UID: "6294ad35-0d5d-4ba7-9d44-15d91effa356"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:49:37.963622 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:37.963598    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6294ad35-0d5d-4ba7-9d44-15d91effa356-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6294ad35-0d5d-4ba7-9d44-15d91effa356" (UID: "6294ad35-0d5d-4ba7-9d44-15d91effa356"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:49:37.963815 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:37.963791    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6294ad35-0d5d-4ba7-9d44-15d91effa356-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6294ad35-0d5d-4ba7-9d44-15d91effa356" (UID: "6294ad35-0d5d-4ba7-9d44-15d91effa356"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:49:37.963815 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:37.963804    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6294ad35-0d5d-4ba7-9d44-15d91effa356-service-ca" (OuterVolumeSpecName: "service-ca") pod "6294ad35-0d5d-4ba7-9d44-15d91effa356" (UID: "6294ad35-0d5d-4ba7-9d44-15d91effa356"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:49:37.965388 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:37.965354    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6294ad35-0d5d-4ba7-9d44-15d91effa356-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6294ad35-0d5d-4ba7-9d44-15d91effa356" (UID: "6294ad35-0d5d-4ba7-9d44-15d91effa356"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 10:49:37.965760 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:37.965736    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6294ad35-0d5d-4ba7-9d44-15d91effa356-kube-api-access-thvfs" (OuterVolumeSpecName: "kube-api-access-thvfs") pod "6294ad35-0d5d-4ba7-9d44-15d91effa356" (UID: "6294ad35-0d5d-4ba7-9d44-15d91effa356"). InnerVolumeSpecName "kube-api-access-thvfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 10:49:37.965832 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:37.965737    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6294ad35-0d5d-4ba7-9d44-15d91effa356-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6294ad35-0d5d-4ba7-9d44-15d91effa356" (UID: "6294ad35-0d5d-4ba7-9d44-15d91effa356"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 10:49:38.063980 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:38.063946    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thvfs\" (UniqueName: \"kubernetes.io/projected/6294ad35-0d5d-4ba7-9d44-15d91effa356-kube-api-access-thvfs\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:49:38.063980 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:38.063977    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6294ad35-0d5d-4ba7-9d44-15d91effa356-console-config\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:49:38.063980 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:38.063988    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6294ad35-0d5d-4ba7-9d44-15d91effa356-trusted-ca-bundle\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:49:38.064229 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:38.063997    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6294ad35-0d5d-4ba7-9d44-15d91effa356-console-serving-cert\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:49:38.064229 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:38.064006    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6294ad35-0d5d-4ba7-9d44-15d91effa356-console-oauth-config\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:49:38.064229 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:38.064015    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6294ad35-0d5d-4ba7-9d44-15d91effa356-oauth-serving-cert\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:49:38.064229 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:38.064023    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6294ad35-0d5d-4ba7-9d44-15d91effa356-service-ca\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:49:38.656125 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:38.656092    2563 generic.go:356] "Generic (PLEG): container finished" podID="6294ad35-0d5d-4ba7-9d44-15d91effa356" containerID="0ff21212972d3fd980190a45822f3aa3f09ebee058cc0ed85f7c6091c539dd54" exitCode=0
Jul 15 10:49:38.656539 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:38.656130    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8677894d4d-zwdn4" event={"ID":"6294ad35-0d5d-4ba7-9d44-15d91effa356","Type":"ContainerDied","Data":"0ff21212972d3fd980190a45822f3aa3f09ebee058cc0ed85f7c6091c539dd54"}
Jul 15 10:49:38.656539 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:38.656164    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8677894d4d-zwdn4"
Jul 15 10:49:38.656539 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:38.656202    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8677894d4d-zwdn4" event={"ID":"6294ad35-0d5d-4ba7-9d44-15d91effa356","Type":"ContainerDied","Data":"4d38c44ae44f4b9bc197df797f4c9d412c33b8d8a3d6e7f26388ef82e79533b7"}
Jul 15 10:49:38.656539 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:38.656225    2563 scope.go:117] "RemoveContainer" containerID="0ff21212972d3fd980190a45822f3aa3f09ebee058cc0ed85f7c6091c539dd54"
Jul 15 10:49:38.664603 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:38.664582    2563 scope.go:117] "RemoveContainer" containerID="0ff21212972d3fd980190a45822f3aa3f09ebee058cc0ed85f7c6091c539dd54"
Jul 15 10:49:38.664928 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:49:38.664906    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ff21212972d3fd980190a45822f3aa3f09ebee058cc0ed85f7c6091c539dd54\": container with ID starting with 0ff21212972d3fd980190a45822f3aa3f09ebee058cc0ed85f7c6091c539dd54 not found: ID does not exist" containerID="0ff21212972d3fd980190a45822f3aa3f09ebee058cc0ed85f7c6091c539dd54"
Jul 15 10:49:38.664974 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:38.664938    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff21212972d3fd980190a45822f3aa3f09ebee058cc0ed85f7c6091c539dd54"} err="failed to get container status \"0ff21212972d3fd980190a45822f3aa3f09ebee058cc0ed85f7c6091c539dd54\": rpc error: code = NotFound desc = could not find container \"0ff21212972d3fd980190a45822f3aa3f09ebee058cc0ed85f7c6091c539dd54\": container with ID starting with 0ff21212972d3fd980190a45822f3aa3f09ebee058cc0ed85f7c6091c539dd54 not found: ID does not exist"
Jul 15 10:49:38.676969 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:38.676938    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8677894d4d-zwdn4"]
Jul 15 10:49:38.681569 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:38.681540    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8677894d4d-zwdn4"]
Jul 15 10:49:39.677502 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:49:39.677457    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6294ad35-0d5d-4ba7-9d44-15d91effa356" path="/var/lib/kubelet/pods/6294ad35-0d5d-4ba7-9d44-15d91effa356/volumes"
Jul 15 10:51:21.517235 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:51:21.517198    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk49n_beb68899-ffb3-436f-8e17-43597aad4030/ovn-acl-logging/0.log"
Jul 15 10:51:21.519735 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:51:21.519717    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk49n_beb68899-ffb3-436f-8e17-43597aad4030/ovn-acl-logging/0.log"
Jul 15 10:51:21.524260 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:51:21.524237    2563 kubelet.go:1648] "Image garbage collection succeeded"
Jul 15 10:53:00.096374 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:00.096338    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8b6dfdb7d-z5qxq"]
Jul 15 10:53:00.096957 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:00.096704    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6294ad35-0d5d-4ba7-9d44-15d91effa356" containerName="console"
Jul 15 10:53:00.096957 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:00.096721    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6294ad35-0d5d-4ba7-9d44-15d91effa356" containerName="console"
Jul 15 10:53:00.096957 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:00.096816    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6294ad35-0d5d-4ba7-9d44-15d91effa356" containerName="console"
Jul 15 10:53:00.098845 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:00.098825    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8b6dfdb7d-z5qxq"
Jul 15 10:53:00.110059 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:00.110035    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8b6dfdb7d-z5qxq"]
Jul 15 10:53:00.130232 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:00.130202    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e27b46b3-6e7b-422b-ba47-9c4df4d9a99d-service-ca\") pod \"console-8b6dfdb7d-z5qxq\" (UID: \"e27b46b3-6e7b-422b-ba47-9c4df4d9a99d\") " pod="openshift-console/console-8b6dfdb7d-z5qxq"
Jul 15 10:53:00.130338 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:00.130242    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e27b46b3-6e7b-422b-ba47-9c4df4d9a99d-console-oauth-config\") pod \"console-8b6dfdb7d-z5qxq\" (UID: \"e27b46b3-6e7b-422b-ba47-9c4df4d9a99d\") " pod="openshift-console/console-8b6dfdb7d-z5qxq"
Jul 15 10:53:00.130338 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:00.130257    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e27b46b3-6e7b-422b-ba47-9c4df4d9a99d-oauth-serving-cert\") pod \"console-8b6dfdb7d-z5qxq\" (UID: \"e27b46b3-6e7b-422b-ba47-9c4df4d9a99d\") " pod="openshift-console/console-8b6dfdb7d-z5qxq"
Jul 15 10:53:00.130338 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:00.130323    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e27b46b3-6e7b-422b-ba47-9c4df4d9a99d-trusted-ca-bundle\") pod \"console-8b6dfdb7d-z5qxq\" (UID: \"e27b46b3-6e7b-422b-ba47-9c4df4d9a99d\") " pod="openshift-console/console-8b6dfdb7d-z5qxq"
Jul 15 10:53:00.130460 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:00.130379    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-476nl\" (UniqueName: \"kubernetes.io/projected/e27b46b3-6e7b-422b-ba47-9c4df4d9a99d-kube-api-access-476nl\") pod \"console-8b6dfdb7d-z5qxq\" (UID: \"e27b46b3-6e7b-422b-ba47-9c4df4d9a99d\") " pod="openshift-console/console-8b6dfdb7d-z5qxq"
Jul 15 10:53:00.130460 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:00.130402    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e27b46b3-6e7b-422b-ba47-9c4df4d9a99d-console-serving-cert\") pod \"console-8b6dfdb7d-z5qxq\" (UID: \"e27b46b3-6e7b-422b-ba47-9c4df4d9a99d\") " pod="openshift-console/console-8b6dfdb7d-z5qxq"
Jul 15 10:53:00.130460 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:00.130431    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e27b46b3-6e7b-422b-ba47-9c4df4d9a99d-console-config\") pod \"console-8b6dfdb7d-z5qxq\" (UID: \"e27b46b3-6e7b-422b-ba47-9c4df4d9a99d\") " pod="openshift-console/console-8b6dfdb7d-z5qxq"
Jul 15 10:53:00.231248 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:00.231219    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e27b46b3-6e7b-422b-ba47-9c4df4d9a99d-console-oauth-config\") pod \"console-8b6dfdb7d-z5qxq\" (UID: \"e27b46b3-6e7b-422b-ba47-9c4df4d9a99d\") " pod="openshift-console/console-8b6dfdb7d-z5qxq"
Jul 15 10:53:00.231248 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:00.231250    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e27b46b3-6e7b-422b-ba47-9c4df4d9a99d-oauth-serving-cert\") pod \"console-8b6dfdb7d-z5qxq\" (UID: \"e27b46b3-6e7b-422b-ba47-9c4df4d9a99d\") " pod="openshift-console/console-8b6dfdb7d-z5qxq"
Jul 15 10:53:00.231486 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:00.231280    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e27b46b3-6e7b-422b-ba47-9c4df4d9a99d-trusted-ca-bundle\") pod \"console-8b6dfdb7d-z5qxq\" (UID: \"e27b46b3-6e7b-422b-ba47-9c4df4d9a99d\") " pod="openshift-console/console-8b6dfdb7d-z5qxq"
Jul 15 10:53:00.231486 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:00.231315    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-476nl\" (UniqueName: \"kubernetes.io/projected/e27b46b3-6e7b-422b-ba47-9c4df4d9a99d-kube-api-access-476nl\") pod \"console-8b6dfdb7d-z5qxq\" (UID: \"e27b46b3-6e7b-422b-ba47-9c4df4d9a99d\") " pod="openshift-console/console-8b6dfdb7d-z5qxq"
Jul 15 10:53:00.231486 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:00.231340    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e27b46b3-6e7b-422b-ba47-9c4df4d9a99d-console-serving-cert\") pod \"console-8b6dfdb7d-z5qxq\" (UID: \"e27b46b3-6e7b-422b-ba47-9c4df4d9a99d\") " pod="openshift-console/console-8b6dfdb7d-z5qxq"
Jul 15 10:53:00.231486 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:00.231373    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e27b46b3-6e7b-422b-ba47-9c4df4d9a99d-console-config\") pod \"console-8b6dfdb7d-z5qxq\" (UID: \"e27b46b3-6e7b-422b-ba47-9c4df4d9a99d\") " pod="openshift-console/console-8b6dfdb7d-z5qxq"
Jul 15 10:53:00.231486 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:00.231417    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e27b46b3-6e7b-422b-ba47-9c4df4d9a99d-service-ca\") pod \"console-8b6dfdb7d-z5qxq\" (UID: \"e27b46b3-6e7b-422b-ba47-9c4df4d9a99d\") " pod="openshift-console/console-8b6dfdb7d-z5qxq"
Jul 15 10:53:00.232065 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:00.232037    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e27b46b3-6e7b-422b-ba47-9c4df4d9a99d-service-ca\") pod \"console-8b6dfdb7d-z5qxq\" (UID: \"e27b46b3-6e7b-422b-ba47-9c4df4d9a99d\") " pod="openshift-console/console-8b6dfdb7d-z5qxq"
Jul 15 10:53:00.232217 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:00.232037    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e27b46b3-6e7b-422b-ba47-9c4df4d9a99d-oauth-serving-cert\") pod \"console-8b6dfdb7d-z5qxq\" (UID: \"e27b46b3-6e7b-422b-ba47-9c4df4d9a99d\") " pod="openshift-console/console-8b6dfdb7d-z5qxq"
Jul 15 10:53:00.232217 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:00.232197    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e27b46b3-6e7b-422b-ba47-9c4df4d9a99d-trusted-ca-bundle\") pod \"console-8b6dfdb7d-z5qxq\" (UID: \"e27b46b3-6e7b-422b-ba47-9c4df4d9a99d\") " pod="openshift-console/console-8b6dfdb7d-z5qxq"
Jul 15 10:53:00.232217 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:00.232199    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e27b46b3-6e7b-422b-ba47-9c4df4d9a99d-console-config\") pod \"console-8b6dfdb7d-z5qxq\" (UID: \"e27b46b3-6e7b-422b-ba47-9c4df4d9a99d\") " pod="openshift-console/console-8b6dfdb7d-z5qxq"
Jul 15 10:53:00.233692 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:00.233672    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e27b46b3-6e7b-422b-ba47-9c4df4d9a99d-console-serving-cert\") pod \"console-8b6dfdb7d-z5qxq\" (UID: \"e27b46b3-6e7b-422b-ba47-9c4df4d9a99d\") " pod="openshift-console/console-8b6dfdb7d-z5qxq"
Jul 15 10:53:00.233815 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:00.233797    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e27b46b3-6e7b-422b-ba47-9c4df4d9a99d-console-oauth-config\") pod \"console-8b6dfdb7d-z5qxq\" (UID: \"e27b46b3-6e7b-422b-ba47-9c4df4d9a99d\") " pod="openshift-console/console-8b6dfdb7d-z5qxq"
Jul 15 10:53:00.242868 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:00.242839    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-476nl\" (UniqueName: \"kubernetes.io/projected/e27b46b3-6e7b-422b-ba47-9c4df4d9a99d-kube-api-access-476nl\") pod \"console-8b6dfdb7d-z5qxq\" (UID: \"e27b46b3-6e7b-422b-ba47-9c4df4d9a99d\") " pod="openshift-console/console-8b6dfdb7d-z5qxq"
Jul 15 10:53:00.407823 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:00.407740    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8b6dfdb7d-z5qxq"
Jul 15 10:53:00.536945 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:00.536871    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8b6dfdb7d-z5qxq"]
Jul 15 10:53:00.539511 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:53:00.539482    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode27b46b3_6e7b_422b_ba47_9c4df4d9a99d.slice/crio-a2b7113ffd8ed2baea7ac39f438d34c1b7548177cd6e5d36b959a64572e9ca62 WatchSource:0}: Error finding container a2b7113ffd8ed2baea7ac39f438d34c1b7548177cd6e5d36b959a64572e9ca62: Status 404 returned error can't find the container with id a2b7113ffd8ed2baea7ac39f438d34c1b7548177cd6e5d36b959a64572e9ca62
Jul 15 10:53:01.350594 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:01.350563    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8b6dfdb7d-z5qxq" event={"ID":"e27b46b3-6e7b-422b-ba47-9c4df4d9a99d","Type":"ContainerStarted","Data":"bdb0d350513f9a8d0c8e7bf428ed72f75ce9cde4ef0093882b71378acb835472"}
Jul 15 10:53:01.350930 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:01.350611    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8b6dfdb7d-z5qxq" event={"ID":"e27b46b3-6e7b-422b-ba47-9c4df4d9a99d","Type":"ContainerStarted","Data":"a2b7113ffd8ed2baea7ac39f438d34c1b7548177cd6e5d36b959a64572e9ca62"}
Jul 15 10:53:01.372816 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:01.372773    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8b6dfdb7d-z5qxq" podStartSLOduration=1.3727574169999999 podStartE2EDuration="1.372757417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:53:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:53:01.37125282 +0000 UTC m=+400.200768224" watchObservedRunningTime="2026-07-15 10:53:01.372757417 +0000 UTC m=+400.202272878"
Jul 15 10:53:10.408732 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:10.408697    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8b6dfdb7d-z5qxq"
Jul 15 10:53:10.409206 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:10.408770    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8b6dfdb7d-z5qxq"
Jul 15 10:53:10.413499 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:10.413477    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8b6dfdb7d-z5qxq"
Jul 15 10:53:11.389449 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:11.389422    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8b6dfdb7d-z5qxq"
Jul 15 10:53:11.440919 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:11.440886    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-699c49876d-94ksh"]
Jul 15 10:53:11.537555 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:11.537518    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-7zmgw"]
Jul 15 10:53:11.540208 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:11.540192    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-7zmgw"
Jul 15 10:53:11.543589 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:11.543569    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 15 10:53:11.543700 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:11.543573    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 15 10:53:11.543886 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:11.543872    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-2rm8s\""
Jul 15 10:53:11.544096 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:11.544082    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 15 10:53:11.551774 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:11.551753    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-7zmgw"]
Jul 15 10:53:11.617427 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:11.617390    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krzc6\" (UniqueName: \"kubernetes.io/projected/b1ca34b8-2a49-454b-9dcf-e684a84d9198-kube-api-access-krzc6\") pod \"s3-init-7zmgw\" (UID: \"b1ca34b8-2a49-454b-9dcf-e684a84d9198\") " pod="kserve/s3-init-7zmgw"
Jul 15 10:53:11.617603 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:11.617450    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1ca34b8-2a49-454b-9dcf-e684a84d9198-model-cache\") pod \"s3-init-7zmgw\" (UID: \"b1ca34b8-2a49-454b-9dcf-e684a84d9198\") " pod="kserve/s3-init-7zmgw"
Jul 15 10:53:11.718003 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:11.717975    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krzc6\" (UniqueName: \"kubernetes.io/projected/b1ca34b8-2a49-454b-9dcf-e684a84d9198-kube-api-access-krzc6\") pod \"s3-init-7zmgw\" (UID: \"b1ca34b8-2a49-454b-9dcf-e684a84d9198\") " pod="kserve/s3-init-7zmgw"
Jul 15 10:53:11.718178 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:11.718039    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1ca34b8-2a49-454b-9dcf-e684a84d9198-model-cache\") pod \"s3-init-7zmgw\" (UID: \"b1ca34b8-2a49-454b-9dcf-e684a84d9198\") " pod="kserve/s3-init-7zmgw"
Jul 15 10:53:11.718364 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:11.718349    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1ca34b8-2a49-454b-9dcf-e684a84d9198-model-cache\") pod \"s3-init-7zmgw\" (UID: \"b1ca34b8-2a49-454b-9dcf-e684a84d9198\") " pod="kserve/s3-init-7zmgw"
Jul 15 10:53:11.727311 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:11.727285    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krzc6\" (UniqueName: \"kubernetes.io/projected/b1ca34b8-2a49-454b-9dcf-e684a84d9198-kube-api-access-krzc6\") pod \"s3-init-7zmgw\" (UID: \"b1ca34b8-2a49-454b-9dcf-e684a84d9198\") " pod="kserve/s3-init-7zmgw"
Jul 15 10:53:11.863276 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:11.863223    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-7zmgw"
Jul 15 10:53:11.986883 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:11.986808    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-7zmgw"]
Jul 15 10:53:11.990347 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:53:11.990320    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1ca34b8_2a49_454b_9dcf_e684a84d9198.slice/crio-c87d93836a233209230213b052628e8d35def8dbb3fc591e2a5ff92d4ac72ccc WatchSource:0}: Error finding container c87d93836a233209230213b052628e8d35def8dbb3fc591e2a5ff92d4ac72ccc: Status 404 returned error can't find the container with id c87d93836a233209230213b052628e8d35def8dbb3fc591e2a5ff92d4ac72ccc
Jul 15 10:53:11.992522 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:11.992505    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 15 10:53:12.390682 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:12.390600    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-7zmgw" event={"ID":"b1ca34b8-2a49-454b-9dcf-e684a84d9198","Type":"ContainerStarted","Data":"c87d93836a233209230213b052628e8d35def8dbb3fc591e2a5ff92d4ac72ccc"}
Jul 15 10:53:16.412154 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:16.412034    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-7zmgw" event={"ID":"b1ca34b8-2a49-454b-9dcf-e684a84d9198","Type":"ContainerStarted","Data":"762458b8b1730497759681264f6e6e1a824069d80cf5a9468f60d74788533e13"}
Jul 15 10:53:17.417162 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:17.417108    2563 generic.go:356] "Generic (PLEG): container finished" podID="b1ca34b8-2a49-454b-9dcf-e684a84d9198" containerID="762458b8b1730497759681264f6e6e1a824069d80cf5a9468f60d74788533e13" exitCode=0
Jul 15 10:53:17.417511 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:17.417185    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-7zmgw" event={"ID":"b1ca34b8-2a49-454b-9dcf-e684a84d9198","Type":"ContainerDied","Data":"762458b8b1730497759681264f6e6e1a824069d80cf5a9468f60d74788533e13"}
Jul 15 10:53:21.848489 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:21.848466    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 15 10:53:22.440058 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:22.440026    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-7zmgw" event={"ID":"b1ca34b8-2a49-454b-9dcf-e684a84d9198","Type":"ContainerStarted","Data":"49e779d36954845e5f6448202691a6ea6521ae3efa527e30f7e88659fa670895"}
Jul 15 10:53:22.459135 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:22.459081    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-7zmgw" podStartSLOduration=2.41237158 podStartE2EDuration="11.459065826s" totalImagesPullingTime="8.044486203s" totalInitContainerRuntime="1.002208043s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:53:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:53:22.457750291 +0000 UTC m=+421.287265696" watchObservedRunningTime="2026-07-15 10:53:22.459065826 +0000 UTC m=+421.288581230"
Jul 15 10:53:36.464079 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:36.464020    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-699c49876d-94ksh" podUID="4a93162d-3cf8-4cd3-bc95-e0fab2b75c89" containerName="console" containerID="cri-o://eee94e9b5d9b0efeaa1164ca525e03430bd4281b2a6dfa3bc05b70a33378ad3c" gracePeriod=15
Jul 15 10:53:37.198033 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:37.198009    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-699c49876d-94ksh"
Jul 15 10:53:37.350838 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:37.350760    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-console-serving-cert\") pod \"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89\" (UID: \"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89\") "
Jul 15 10:53:37.350838 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:37.350796    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-console-oauth-config\") pod \"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89\" (UID: \"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89\") "
Jul 15 10:53:37.351050 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:37.350863    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-console-config\") pod \"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89\" (UID: \"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89\") "
Jul 15 10:53:37.351050 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:37.350886    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-trusted-ca-bundle\") pod \"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89\" (UID: \"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89\") "
Jul 15 10:53:37.351050 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:37.350938    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vn78w\" (UniqueName: \"kubernetes.io/projected/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-kube-api-access-vn78w\") pod \"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89\" (UID: \"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89\") "
Jul 15 10:53:37.351050 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:37.350959    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-oauth-serving-cert\") pod \"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89\" (UID: \"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89\") "
Jul 15 10:53:37.351050 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:37.350992    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-service-ca\") pod \"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89\" (UID: \"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89\") "
Jul 15 10:53:37.351442 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:37.351395    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-console-config" (OuterVolumeSpecName: "console-config") pod "4a93162d-3cf8-4cd3-bc95-e0fab2b75c89" (UID: "4a93162d-3cf8-4cd3-bc95-e0fab2b75c89"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:53:37.351539 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:37.351444    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4a93162d-3cf8-4cd3-bc95-e0fab2b75c89" (UID: "4a93162d-3cf8-4cd3-bc95-e0fab2b75c89"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:53:37.351539 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:37.351458    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4a93162d-3cf8-4cd3-bc95-e0fab2b75c89" (UID: "4a93162d-3cf8-4cd3-bc95-e0fab2b75c89"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:53:37.351539 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:37.351508    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-service-ca" (OuterVolumeSpecName: "service-ca") pod "4a93162d-3cf8-4cd3-bc95-e0fab2b75c89" (UID: "4a93162d-3cf8-4cd3-bc95-e0fab2b75c89"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:53:37.353052 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:37.353031    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4a93162d-3cf8-4cd3-bc95-e0fab2b75c89" (UID: "4a93162d-3cf8-4cd3-bc95-e0fab2b75c89"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 10:53:37.353128 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:37.353056    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4a93162d-3cf8-4cd3-bc95-e0fab2b75c89" (UID: "4a93162d-3cf8-4cd3-bc95-e0fab2b75c89"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 10:53:37.353128 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:37.353107    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-kube-api-access-vn78w" (OuterVolumeSpecName: "kube-api-access-vn78w") pod "4a93162d-3cf8-4cd3-bc95-e0fab2b75c89" (UID: "4a93162d-3cf8-4cd3-bc95-e0fab2b75c89"). InnerVolumeSpecName "kube-api-access-vn78w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 10:53:37.451806 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:37.451761    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn78w\" (UniqueName: \"kubernetes.io/projected/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-kube-api-access-vn78w\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:53:37.451806 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:37.451803    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-oauth-serving-cert\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:53:37.451806 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:37.451813    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-service-ca\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:53:37.451806 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:37.451822    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-console-serving-cert\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:53:37.452055 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:37.451832    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-console-oauth-config\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:53:37.452055 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:37.451840    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-console-config\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:53:37.452055 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:37.451848    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89-trusted-ca-bundle\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:53:37.493728 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:37.493699    2563 generic.go:356] "Generic (PLEG): container finished" podID="4a93162d-3cf8-4cd3-bc95-e0fab2b75c89" containerID="eee94e9b5d9b0efeaa1164ca525e03430bd4281b2a6dfa3bc05b70a33378ad3c" exitCode=0
Jul 15 10:53:37.494113 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:37.493746    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-699c49876d-94ksh"
Jul 15 10:53:37.494113 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:37.493782    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-699c49876d-94ksh" event={"ID":"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89","Type":"ContainerDied","Data":"eee94e9b5d9b0efeaa1164ca525e03430bd4281b2a6dfa3bc05b70a33378ad3c"}
Jul 15 10:53:37.494113 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:37.493850    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-699c49876d-94ksh" event={"ID":"4a93162d-3cf8-4cd3-bc95-e0fab2b75c89","Type":"ContainerDied","Data":"a68367cd31185b55a4eb1f9987bf3a7971755fea17ae6eb6afb46e4355057392"}
Jul 15 10:53:37.494113 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:37.493870    2563 scope.go:117] "RemoveContainer" containerID="eee94e9b5d9b0efeaa1164ca525e03430bd4281b2a6dfa3bc05b70a33378ad3c"
Jul 15 10:53:37.502229 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:37.502210    2563 scope.go:117] "RemoveContainer" containerID="eee94e9b5d9b0efeaa1164ca525e03430bd4281b2a6dfa3bc05b70a33378ad3c"
Jul 15 10:53:37.502473 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:53:37.502455    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eee94e9b5d9b0efeaa1164ca525e03430bd4281b2a6dfa3bc05b70a33378ad3c\": container with ID starting with eee94e9b5d9b0efeaa1164ca525e03430bd4281b2a6dfa3bc05b70a33378ad3c not found: ID does not exist" containerID="eee94e9b5d9b0efeaa1164ca525e03430bd4281b2a6dfa3bc05b70a33378ad3c"
Jul 15 10:53:37.502535 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:37.502487    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee94e9b5d9b0efeaa1164ca525e03430bd4281b2a6dfa3bc05b70a33378ad3c"} err="failed to get container status \"eee94e9b5d9b0efeaa1164ca525e03430bd4281b2a6dfa3bc05b70a33378ad3c\": rpc error: code = NotFound desc = could not find container \"eee94e9b5d9b0efeaa1164ca525e03430bd4281b2a6dfa3bc05b70a33378ad3c\": container with ID starting with eee94e9b5d9b0efeaa1164ca525e03430bd4281b2a6dfa3bc05b70a33378ad3c not found: ID does not exist"
Jul 15 10:53:37.514307 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:37.514283    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-699c49876d-94ksh"]
Jul 15 10:53:37.518351 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:37.518329    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-699c49876d-94ksh"]
Jul 15 10:53:37.678358 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:53:37.678267    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a93162d-3cf8-4cd3-bc95-e0fab2b75c89" path="/var/lib/kubelet/pods/4a93162d-3cf8-4cd3-bc95-e0fab2b75c89/volumes"
Jul 15 10:54:27.682511 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:27.682475    2563 generic.go:356] "Generic (PLEG): container finished" podID="b1ca34b8-2a49-454b-9dcf-e684a84d9198" containerID="49e779d36954845e5f6448202691a6ea6521ae3efa527e30f7e88659fa670895" exitCode=0
Jul 15 10:54:27.682896 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:27.682540    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-7zmgw" event={"ID":"b1ca34b8-2a49-454b-9dcf-e684a84d9198","Type":"ContainerDied","Data":"49e779d36954845e5f6448202691a6ea6521ae3efa527e30f7e88659fa670895"}
Jul 15 10:54:28.810487 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:28.810465    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-7zmgw"
Jul 15 10:54:28.949622 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:28.949593    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krzc6\" (UniqueName: \"kubernetes.io/projected/b1ca34b8-2a49-454b-9dcf-e684a84d9198-kube-api-access-krzc6\") pod \"b1ca34b8-2a49-454b-9dcf-e684a84d9198\" (UID: \"b1ca34b8-2a49-454b-9dcf-e684a84d9198\") "
Jul 15 10:54:28.949769 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:28.949647    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1ca34b8-2a49-454b-9dcf-e684a84d9198-model-cache\") pod \"b1ca34b8-2a49-454b-9dcf-e684a84d9198\" (UID: \"b1ca34b8-2a49-454b-9dcf-e684a84d9198\") "
Jul 15 10:54:28.951793 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:28.951767    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1ca34b8-2a49-454b-9dcf-e684a84d9198-kube-api-access-krzc6" (OuterVolumeSpecName: "kube-api-access-krzc6") pod "b1ca34b8-2a49-454b-9dcf-e684a84d9198" (UID: "b1ca34b8-2a49-454b-9dcf-e684a84d9198"). InnerVolumeSpecName "kube-api-access-krzc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 10:54:28.967478 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:28.967448    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1ca34b8-2a49-454b-9dcf-e684a84d9198-model-cache" (OuterVolumeSpecName: "model-cache") pod "b1ca34b8-2a49-454b-9dcf-e684a84d9198" (UID: "b1ca34b8-2a49-454b-9dcf-e684a84d9198"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:54:29.051102 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:29.051068    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1ca34b8-2a49-454b-9dcf-e684a84d9198-model-cache\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:54:29.051102 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:29.051096    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krzc6\" (UniqueName: \"kubernetes.io/projected/b1ca34b8-2a49-454b-9dcf-e684a84d9198-kube-api-access-krzc6\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:54:29.691214 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:29.691181    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-7zmgw" event={"ID":"b1ca34b8-2a49-454b-9dcf-e684a84d9198","Type":"ContainerDied","Data":"c87d93836a233209230213b052628e8d35def8dbb3fc591e2a5ff92d4ac72ccc"}
Jul 15 10:54:29.691338 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:29.691227    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c87d93836a233209230213b052628e8d35def8dbb3fc591e2a5ff92d4ac72ccc"
Jul 15 10:54:29.691338 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:29.691197    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-7zmgw"
Jul 15 10:54:42.794216 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:42.794184    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr"]
Jul 15 10:54:42.794575 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:42.794439    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1ca34b8-2a49-454b-9dcf-e684a84d9198" containerName="download-hf-model"
Jul 15 10:54:42.794575 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:42.794449    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1ca34b8-2a49-454b-9dcf-e684a84d9198" containerName="download-hf-model"
Jul 15 10:54:42.794575 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:42.794459    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a93162d-3cf8-4cd3-bc95-e0fab2b75c89" containerName="console"
Jul 15 10:54:42.794575 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:42.794467    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a93162d-3cf8-4cd3-bc95-e0fab2b75c89" containerName="console"
Jul 15 10:54:42.794575 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:42.794478    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1ca34b8-2a49-454b-9dcf-e684a84d9198" containerName="s3-init"
Jul 15 10:54:42.794575 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:42.794484    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1ca34b8-2a49-454b-9dcf-e684a84d9198" containerName="s3-init"
Jul 15 10:54:42.794575 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:42.794531    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1ca34b8-2a49-454b-9dcf-e684a84d9198" containerName="s3-init"
Jul 15 10:54:42.794575 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:42.794540    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a93162d-3cf8-4cd3-bc95-e0fab2b75c89" containerName="console"
Jul 15 10:54:42.797017 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:42.797003    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr"
Jul 15 10:54:42.801162 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:42.801129    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 15 10:54:42.801347 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:42.801331    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nfb6q\""
Jul 15 10:54:42.802435 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:42.802418    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-6db9e-kube-rbac-proxy-sar-config\""
Jul 15 10:54:42.802658 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:42.802644    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-6db9e-predictor-serving-cert\""
Jul 15 10:54:42.803632 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:42.803535    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 15 10:54:42.815840 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:42.815812    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr"]
Jul 15 10:54:42.960929 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:42.960888    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9436c84-deca-4753-a2be-5ec45980638f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6db9e-predictor-58b664446b-fmthr\" (UID: \"e9436c84-deca-4753-a2be-5ec45980638f\") " pod="kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr"
Jul 15 10:54:42.961104 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:42.960965    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9436c84-deca-4753-a2be-5ec45980638f-proxy-tls\") pod \"success-200-isvc-6db9e-predictor-58b664446b-fmthr\" (UID: \"e9436c84-deca-4753-a2be-5ec45980638f\") " pod="kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr"
Jul 15 10:54:42.961104 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:42.961008    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4tr8\" (UniqueName: \"kubernetes.io/projected/e9436c84-deca-4753-a2be-5ec45980638f-kube-api-access-g4tr8\") pod \"success-200-isvc-6db9e-predictor-58b664446b-fmthr\" (UID: \"e9436c84-deca-4753-a2be-5ec45980638f\") " pod="kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr"
Jul 15 10:54:43.062059 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.061976    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9436c84-deca-4753-a2be-5ec45980638f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6db9e-predictor-58b664446b-fmthr\" (UID: \"e9436c84-deca-4753-a2be-5ec45980638f\") " pod="kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr"
Jul 15 10:54:43.062059 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.062038    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9436c84-deca-4753-a2be-5ec45980638f-proxy-tls\") pod \"success-200-isvc-6db9e-predictor-58b664446b-fmthr\" (UID: \"e9436c84-deca-4753-a2be-5ec45980638f\") " pod="kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr"
Jul 15 10:54:43.062303 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.062063    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4tr8\" (UniqueName: \"kubernetes.io/projected/e9436c84-deca-4753-a2be-5ec45980638f-kube-api-access-g4tr8\") pod \"success-200-isvc-6db9e-predictor-58b664446b-fmthr\" (UID: \"e9436c84-deca-4753-a2be-5ec45980638f\") " pod="kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr"
Jul 15 10:54:43.062683 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.062659    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9436c84-deca-4753-a2be-5ec45980638f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6db9e-predictor-58b664446b-fmthr\" (UID: \"e9436c84-deca-4753-a2be-5ec45980638f\") " pod="kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr"
Jul 15 10:54:43.064499 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.064478    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9436c84-deca-4753-a2be-5ec45980638f-proxy-tls\") pod \"success-200-isvc-6db9e-predictor-58b664446b-fmthr\" (UID: \"e9436c84-deca-4753-a2be-5ec45980638f\") " pod="kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr"
Jul 15 10:54:43.079961 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.079934    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4tr8\" (UniqueName: \"kubernetes.io/projected/e9436c84-deca-4753-a2be-5ec45980638f-kube-api-access-g4tr8\") pod \"success-200-isvc-6db9e-predictor-58b664446b-fmthr\" (UID: \"e9436c84-deca-4753-a2be-5ec45980638f\") " pod="kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr"
Jul 15 10:54:43.106924 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.106886    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr"
Jul 15 10:54:43.182908 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.182880    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb"]
Jul 15 10:54:43.185281 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.185256    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb"
Jul 15 10:54:43.189565 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.189545    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 15 10:54:43.189729 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.189711    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\""
Jul 15 10:54:43.190353 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.190330    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 15 10:54:43.190462 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.190358    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\""
Jul 15 10:54:43.212470 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.212445    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb"]
Jul 15 10:54:43.266259 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.266226    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr"]
Jul 15 10:54:43.269961 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:54:43.269927    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9436c84_deca_4753_a2be_5ec45980638f.slice/crio-6f6ed26a2ed2fabf46832d39bbb93db9a5fb320b95a5256cce0906bc002a61dc WatchSource:0}: Error finding container 6f6ed26a2ed2fabf46832d39bbb93db9a5fb320b95a5256cce0906bc002a61dc: Status 404 returned error can't find the container with id 6f6ed26a2ed2fabf46832d39bbb93db9a5fb320b95a5256cce0906bc002a61dc
Jul 15 10:54:43.361625 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.361529    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g"]
Jul 15 10:54:43.367036 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.366153    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g"
Jul 15 10:54:43.367036 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.366385    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d952e58-6e36-4634-8acd-fdc7dd140fc3-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb\" (UID: \"0d952e58-6e36-4634-8acd-fdc7dd140fc3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb"
Jul 15 10:54:43.367036 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.366453    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d952e58-6e36-4634-8acd-fdc7dd140fc3-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb\" (UID: \"0d952e58-6e36-4634-8acd-fdc7dd140fc3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb"
Jul 15 10:54:43.367036 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.366486    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkcml\" (UniqueName: \"kubernetes.io/projected/0d952e58-6e36-4634-8acd-fdc7dd140fc3-kube-api-access-qkcml\") pod \"isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb\" (UID: \"0d952e58-6e36-4634-8acd-fdc7dd140fc3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb"
Jul 15 10:54:43.367036 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.366531    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d952e58-6e36-4634-8acd-fdc7dd140fc3-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb\" (UID: \"0d952e58-6e36-4634-8acd-fdc7dd140fc3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb"
Jul 15 10:54:43.367036 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.366573    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d952e58-6e36-4634-8acd-fdc7dd140fc3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb\" (UID: \"0d952e58-6e36-4634-8acd-fdc7dd140fc3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb"
Jul 15 10:54:43.373098 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.373063    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-6db9e-kube-rbac-proxy-sar-config\""
Jul 15 10:54:43.373374 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.373346    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-6db9e-predictor-serving-cert\""
Jul 15 10:54:43.383325 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.383295    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g"]
Jul 15 10:54:43.467123 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.467095    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d952e58-6e36-4634-8acd-fdc7dd140fc3-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb\" (UID: \"0d952e58-6e36-4634-8acd-fdc7dd140fc3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb"
Jul 15 10:54:43.467296 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.467135    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f43e5ac-68b6-4b9d-af22-40156d0c31da-proxy-tls\") pod \"error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g\" (UID: \"8f43e5ac-68b6-4b9d-af22-40156d0c31da\") " pod="kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g"
Jul 15 10:54:43.467296 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.467186    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d952e58-6e36-4634-8acd-fdc7dd140fc3-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb\" (UID: \"0d952e58-6e36-4634-8acd-fdc7dd140fc3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb"
Jul 15 10:54:43.467429 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.467299    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkcml\" (UniqueName: \"kubernetes.io/projected/0d952e58-6e36-4634-8acd-fdc7dd140fc3-kube-api-access-qkcml\") pod \"isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb\" (UID: \"0d952e58-6e36-4634-8acd-fdc7dd140fc3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb"
Jul 15 10:54:43.467429 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.467382    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d952e58-6e36-4634-8acd-fdc7dd140fc3-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb\" (UID: \"0d952e58-6e36-4634-8acd-fdc7dd140fc3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb"
Jul 15 10:54:43.467527 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.467432    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d952e58-6e36-4634-8acd-fdc7dd140fc3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb\" (UID: \"0d952e58-6e36-4634-8acd-fdc7dd140fc3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb"
Jul 15 10:54:43.467527 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.467465    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f43e5ac-68b6-4b9d-af22-40156d0c31da-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g\" (UID: \"8f43e5ac-68b6-4b9d-af22-40156d0c31da\") " pod="kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g"
Jul 15 10:54:43.467527 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.467498    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d952e58-6e36-4634-8acd-fdc7dd140fc3-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb\" (UID: \"0d952e58-6e36-4634-8acd-fdc7dd140fc3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb"
Jul 15 10:54:43.467527 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.467515    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq9tz\" (UniqueName: \"kubernetes.io/projected/8f43e5ac-68b6-4b9d-af22-40156d0c31da-kube-api-access-nq9tz\") pod \"error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g\" (UID: \"8f43e5ac-68b6-4b9d-af22-40156d0c31da\") " pod="kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g"
Jul 15 10:54:43.467845 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.467827    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d952e58-6e36-4634-8acd-fdc7dd140fc3-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb\" (UID: \"0d952e58-6e36-4634-8acd-fdc7dd140fc3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb"
Jul 15 10:54:43.468085 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.468067    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d952e58-6e36-4634-8acd-fdc7dd140fc3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb\" (UID: \"0d952e58-6e36-4634-8acd-fdc7dd140fc3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb"
Jul 15 10:54:43.469685 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.469666    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d952e58-6e36-4634-8acd-fdc7dd140fc3-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb\" (UID: \"0d952e58-6e36-4634-8acd-fdc7dd140fc3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb"
Jul 15 10:54:43.481076 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.481052    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkcml\" (UniqueName: \"kubernetes.io/projected/0d952e58-6e36-4634-8acd-fdc7dd140fc3-kube-api-access-qkcml\") pod \"isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb\" (UID: \"0d952e58-6e36-4634-8acd-fdc7dd140fc3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb"
Jul 15 10:54:43.498191 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.498165    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb"
Jul 15 10:54:43.563134 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.563099    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn"]
Jul 15 10:54:43.566661 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.566642    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn"
Jul 15 10:54:43.567947 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.567922    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f43e5ac-68b6-4b9d-af22-40156d0c31da-proxy-tls\") pod \"error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g\" (UID: \"8f43e5ac-68b6-4b9d-af22-40156d0c31da\") " pod="kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g"
Jul 15 10:54:43.568088 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:54:43.568068    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-serving-cert: secret "error-404-isvc-6db9e-predictor-serving-cert" not found
Jul 15 10:54:43.568183 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:54:43.568127    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f43e5ac-68b6-4b9d-af22-40156d0c31da-proxy-tls podName:8f43e5ac-68b6-4b9d-af22-40156d0c31da nodeName:}" failed. No retries permitted until 2026-07-15 10:54:44.068110344 +0000 UTC m=+502.897625727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8f43e5ac-68b6-4b9d-af22-40156d0c31da-proxy-tls") pod "error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g" (UID: "8f43e5ac-68b6-4b9d-af22-40156d0c31da") : secret "error-404-isvc-6db9e-predictor-serving-cert" not found
Jul 15 10:54:43.568257 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.568075    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f43e5ac-68b6-4b9d-af22-40156d0c31da-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g\" (UID: \"8f43e5ac-68b6-4b9d-af22-40156d0c31da\") " pod="kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g"
Jul 15 10:54:43.568316 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.568292    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq9tz\" (UniqueName: \"kubernetes.io/projected/8f43e5ac-68b6-4b9d-af22-40156d0c31da-kube-api-access-nq9tz\") pod \"error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g\" (UID: \"8f43e5ac-68b6-4b9d-af22-40156d0c31da\") " pod="kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g"
Jul 15 10:54:43.568705 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.568684    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f43e5ac-68b6-4b9d-af22-40156d0c31da-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g\" (UID: \"8f43e5ac-68b6-4b9d-af22-40156d0c31da\") " pod="kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g"
Jul 15 10:54:43.570046 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.570029    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\""
Jul 15 10:54:43.570134 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.570048    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\""
Jul 15 10:54:43.577016 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.576989    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq9tz\" (UniqueName: \"kubernetes.io/projected/8f43e5ac-68b6-4b9d-af22-40156d0c31da-kube-api-access-nq9tz\") pod \"error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g\" (UID: \"8f43e5ac-68b6-4b9d-af22-40156d0c31da\") " pod="kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g"
Jul 15 10:54:43.586306 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.586278    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn"]
Jul 15 10:54:43.638357 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.638283    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb"]
Jul 15 10:54:43.640988 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:54:43.640964    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d952e58_6e36_4634_8acd_fdc7dd140fc3.slice/crio-fbf0918df8d2769093cb2ffcbb13440ee01266301e89ce95c79653241c8eefa1 WatchSource:0}: Error finding container fbf0918df8d2769093cb2ffcbb13440ee01266301e89ce95c79653241c8eefa1: Status 404 returned error can't find the container with id fbf0918df8d2769093cb2ffcbb13440ee01266301e89ce95c79653241c8eefa1
Jul 15 10:54:43.669680 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.669655    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnbth\" (UniqueName: \"kubernetes.io/projected/478795bd-bfa8-449e-8ba0-e9d93d900364-kube-api-access-gnbth\") pod \"isvc-xgboost-graph-predictor-555fc69d56-j57gn\" (UID: \"478795bd-bfa8-449e-8ba0-e9d93d900364\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn"
Jul 15 10:54:43.669778 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.669693    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/478795bd-bfa8-449e-8ba0-e9d93d900364-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-j57gn\" (UID: \"478795bd-bfa8-449e-8ba0-e9d93d900364\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn"
Jul 15 10:54:43.669778 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.669717    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/478795bd-bfa8-449e-8ba0-e9d93d900364-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-j57gn\" (UID: \"478795bd-bfa8-449e-8ba0-e9d93d900364\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn"
Jul 15 10:54:43.669846 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.669814    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/478795bd-bfa8-449e-8ba0-e9d93d900364-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-j57gn\" (UID: \"478795bd-bfa8-449e-8ba0-e9d93d900364\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn"
Jul 15 10:54:43.669885 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.669842    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/478795bd-bfa8-449e-8ba0-e9d93d900364-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-j57gn\" (UID: \"478795bd-bfa8-449e-8ba0-e9d93d900364\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn"
Jul 15 10:54:43.707719 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.707697    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh"]
Jul 15 10:54:43.710260 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.710245    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh"
Jul 15 10:54:43.713503 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.713475    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\""
Jul 15 10:54:43.713503 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.713494    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\""
Jul 15 10:54:43.740851 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.740794    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh"]
Jul 15 10:54:43.750498 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.750429    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr" event={"ID":"e9436c84-deca-4753-a2be-5ec45980638f","Type":"ContainerStarted","Data":"6f6ed26a2ed2fabf46832d39bbb93db9a5fb320b95a5256cce0906bc002a61dc"}
Jul 15 10:54:43.752427 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.752350    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb" event={"ID":"0d952e58-6e36-4634-8acd-fdc7dd140fc3","Type":"ContainerStarted","Data":"214ad0f6f8b57a505c276f285068921693817cb4e3b4eecad43a8800443fce1e"}
Jul 15 10:54:43.752796 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.752771    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb" event={"ID":"0d952e58-6e36-4634-8acd-fdc7dd140fc3","Type":"ContainerStarted","Data":"fbf0918df8d2769093cb2ffcbb13440ee01266301e89ce95c79653241c8eefa1"}
Jul 15 10:54:43.770965 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.770936    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/478795bd-bfa8-449e-8ba0-e9d93d900364-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-j57gn\" (UID: \"478795bd-bfa8-449e-8ba0-e9d93d900364\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn"
Jul 15 10:54:43.771136 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.770993    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/478795bd-bfa8-449e-8ba0-e9d93d900364-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-j57gn\" (UID: \"478795bd-bfa8-449e-8ba0-e9d93d900364\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn"
Jul 15 10:54:43.771136 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.771053    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/478795bd-bfa8-449e-8ba0-e9d93d900364-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-j57gn\" (UID: \"478795bd-bfa8-449e-8ba0-e9d93d900364\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn"
Jul 15 10:54:43.771136 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.771077    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/478795bd-bfa8-449e-8ba0-e9d93d900364-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-j57gn\" (UID: \"478795bd-bfa8-449e-8ba0-e9d93d900364\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn"
Jul 15 10:54:43.771330 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.771165    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnbth\" (UniqueName: \"kubernetes.io/projected/478795bd-bfa8-449e-8ba0-e9d93d900364-kube-api-access-gnbth\") pod \"isvc-xgboost-graph-predictor-555fc69d56-j57gn\" (UID: \"478795bd-bfa8-449e-8ba0-e9d93d900364\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn"
Jul 15 10:54:43.771568 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.771527    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/478795bd-bfa8-449e-8ba0-e9d93d900364-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-j57gn\" (UID: \"478795bd-bfa8-449e-8ba0-e9d93d900364\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn"
Jul 15 10:54:43.771826 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.771804    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/478795bd-bfa8-449e-8ba0-e9d93d900364-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-j57gn\" (UID: \"478795bd-bfa8-449e-8ba0-e9d93d900364\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn"
Jul 15 10:54:43.771901 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.771887    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/478795bd-bfa8-449e-8ba0-e9d93d900364-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-j57gn\" (UID: \"478795bd-bfa8-449e-8ba0-e9d93d900364\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn"
Jul 15 10:54:43.773807 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.773782    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/478795bd-bfa8-449e-8ba0-e9d93d900364-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-j57gn\" (UID: \"478795bd-bfa8-449e-8ba0-e9d93d900364\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn"
Jul 15 10:54:43.787715 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.787660    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnbth\" (UniqueName: \"kubernetes.io/projected/478795bd-bfa8-449e-8ba0-e9d93d900364-kube-api-access-gnbth\") pod \"isvc-xgboost-graph-predictor-555fc69d56-j57gn\" (UID: \"478795bd-bfa8-449e-8ba0-e9d93d900364\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn"
Jul 15 10:54:43.872393 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.872351    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sng9b\" (UniqueName: \"kubernetes.io/projected/da453fef-0f6a-43b1-a04a-e5fabd061460-kube-api-access-sng9b\") pod \"isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh\" (UID: \"da453fef-0f6a-43b1-a04a-e5fabd061460\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh"
Jul 15 10:54:43.872914 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.872412    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da453fef-0f6a-43b1-a04a-e5fabd061460-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh\" (UID: \"da453fef-0f6a-43b1-a04a-e5fabd061460\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh"
Jul 15 10:54:43.872914 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.872526    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da453fef-0f6a-43b1-a04a-e5fabd061460-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh\" (UID: \"da453fef-0f6a-43b1-a04a-e5fabd061460\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh"
Jul 15 10:54:43.872914 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.872568    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da453fef-0f6a-43b1-a04a-e5fabd061460-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh\" (UID: \"da453fef-0f6a-43b1-a04a-e5fabd061460\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh"
Jul 15 10:54:43.872914 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.872652    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da453fef-0f6a-43b1-a04a-e5fabd061460-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh\" (UID: \"da453fef-0f6a-43b1-a04a-e5fabd061460\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh"
Jul 15 10:54:43.876828 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.876760    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn"
Jul 15 10:54:43.979189 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.976599    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da453fef-0f6a-43b1-a04a-e5fabd061460-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh\" (UID: \"da453fef-0f6a-43b1-a04a-e5fabd061460\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh"
Jul 15 10:54:43.979189 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.976649    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sng9b\" (UniqueName: \"kubernetes.io/projected/da453fef-0f6a-43b1-a04a-e5fabd061460-kube-api-access-sng9b\") pod \"isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh\" (UID: \"da453fef-0f6a-43b1-a04a-e5fabd061460\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh"
Jul 15 10:54:43.979189 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.976683    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da453fef-0f6a-43b1-a04a-e5fabd061460-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh\" (UID: \"da453fef-0f6a-43b1-a04a-e5fabd061460\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh"
Jul 15 10:54:43.979189 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.976768    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da453fef-0f6a-43b1-a04a-e5fabd061460-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh\" (UID: \"da453fef-0f6a-43b1-a04a-e5fabd061460\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh"
Jul 15 10:54:43.979189 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.976809    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da453fef-0f6a-43b1-a04a-e5fabd061460-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh\" (UID: \"da453fef-0f6a-43b1-a04a-e5fabd061460\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh"
Jul 15 10:54:43.979189 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.977254    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da453fef-0f6a-43b1-a04a-e5fabd061460-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh\" (UID: \"da453fef-0f6a-43b1-a04a-e5fabd061460\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh"
Jul 15 10:54:43.979189 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:54:43.977371    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-serving-cert: secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Jul 15 10:54:43.979189 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:54:43.977443    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da453fef-0f6a-43b1-a04a-e5fabd061460-proxy-tls podName:da453fef-0f6a-43b1-a04a-e5fabd061460 nodeName:}" failed. No retries permitted until 2026-07-15 10:54:44.477422432 +0000 UTC m=+503.306937817 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/da453fef-0f6a-43b1-a04a-e5fabd061460-proxy-tls") pod "isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh" (UID: "da453fef-0f6a-43b1-a04a-e5fabd061460") : secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Jul 15 10:54:43.979189 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.978495    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da453fef-0f6a-43b1-a04a-e5fabd061460-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh\" (UID: \"da453fef-0f6a-43b1-a04a-e5fabd061460\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh"
Jul 15 10:54:43.979189 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.979041    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da453fef-0f6a-43b1-a04a-e5fabd061460-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh\" (UID: \"da453fef-0f6a-43b1-a04a-e5fabd061460\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh"
Jul 15 10:54:43.994400 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:43.994369    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sng9b\" (UniqueName: \"kubernetes.io/projected/da453fef-0f6a-43b1-a04a-e5fabd061460-kube-api-access-sng9b\") pod \"isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh\" (UID: \"da453fef-0f6a-43b1-a04a-e5fabd061460\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh"
Jul 15 10:54:44.077692 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:44.077658    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f43e5ac-68b6-4b9d-af22-40156d0c31da-proxy-tls\") pod \"error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g\" (UID: \"8f43e5ac-68b6-4b9d-af22-40156d0c31da\") " pod="kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g"
Jul 15 10:54:44.083519 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:44.083460    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f43e5ac-68b6-4b9d-af22-40156d0c31da-proxy-tls\") pod \"error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g\" (UID: \"8f43e5ac-68b6-4b9d-af22-40156d0c31da\") " pod="kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g"
Jul 15 10:54:44.139753 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:44.138314    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn"]
Jul 15 10:54:44.152392 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:54:44.152350    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod478795bd_bfa8_449e_8ba0_e9d93d900364.slice/crio-0b208f2bc2233c2f2781bc894b38eedafd07d87ea18bdeae1716a766c6f64247 WatchSource:0}: Error finding container 0b208f2bc2233c2f2781bc894b38eedafd07d87ea18bdeae1716a766c6f64247: Status 404 returned error can't find the container with id 0b208f2bc2233c2f2781bc894b38eedafd07d87ea18bdeae1716a766c6f64247
Jul 15 10:54:44.277160 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:44.276864    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g"
Jul 15 10:54:44.481418 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:44.480787    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da453fef-0f6a-43b1-a04a-e5fabd061460-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh\" (UID: \"da453fef-0f6a-43b1-a04a-e5fabd061460\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh"
Jul 15 10:54:44.494428 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:44.494362    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da453fef-0f6a-43b1-a04a-e5fabd061460-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh\" (UID: \"da453fef-0f6a-43b1-a04a-e5fabd061460\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh"
Jul 15 10:54:44.537423 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:44.537396    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g"]
Jul 15 10:54:44.540456 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:54:44.540427    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f43e5ac_68b6_4b9d_af22_40156d0c31da.slice/crio-8035dcaac3799bdf892bd0a95e2cfe4e61fdcc0238288bc0ce5b6d0bc4c195ed WatchSource:0}: Error finding container 8035dcaac3799bdf892bd0a95e2cfe4e61fdcc0238288bc0ce5b6d0bc4c195ed: Status 404 returned error can't find the container with id 8035dcaac3799bdf892bd0a95e2cfe4e61fdcc0238288bc0ce5b6d0bc4c195ed
Jul 15 10:54:44.621327 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:44.621294    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh"
Jul 15 10:54:44.760421 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:44.760341    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g" event={"ID":"8f43e5ac-68b6-4b9d-af22-40156d0c31da","Type":"ContainerStarted","Data":"8035dcaac3799bdf892bd0a95e2cfe4e61fdcc0238288bc0ce5b6d0bc4c195ed"}
Jul 15 10:54:44.765255 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:44.764650    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn" event={"ID":"478795bd-bfa8-449e-8ba0-e9d93d900364","Type":"ContainerStarted","Data":"475459a60b80016148d07ab53b820a50988036888948a0d80e7d2b7ae4ad2742"}
Jul 15 10:54:44.765255 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:44.764712    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn" event={"ID":"478795bd-bfa8-449e-8ba0-e9d93d900364","Type":"ContainerStarted","Data":"0b208f2bc2233c2f2781bc894b38eedafd07d87ea18bdeae1716a766c6f64247"}
Jul 15 10:54:44.817431 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:44.817338    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh"]
Jul 15 10:54:45.781318 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:45.781189    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh" event={"ID":"da453fef-0f6a-43b1-a04a-e5fabd061460","Type":"ContainerStarted","Data":"dba88f6821ea153fbb60afe2d6d067fcf6c533a466b2fd392eb27354c4422547"}
Jul 15 10:54:45.781318 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:45.781272    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh" event={"ID":"da453fef-0f6a-43b1-a04a-e5fabd061460","Type":"ContainerStarted","Data":"716d7a916b796d4dd5b46a1cc9ab23f00ebdbaf0878f388fbd12890d4bbc8d83"}
Jul 15 10:54:50.816074 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:50.816035    2563 generic.go:356] "Generic (PLEG): container finished" podID="478795bd-bfa8-449e-8ba0-e9d93d900364" containerID="475459a60b80016148d07ab53b820a50988036888948a0d80e7d2b7ae4ad2742" exitCode=0
Jul 15 10:54:50.816539 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:50.816211    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn" event={"ID":"478795bd-bfa8-449e-8ba0-e9d93d900364","Type":"ContainerDied","Data":"475459a60b80016148d07ab53b820a50988036888948a0d80e7d2b7ae4ad2742"}
Jul 15 10:54:50.818692 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:50.818638    2563 generic.go:356] "Generic (PLEG): container finished" podID="0d952e58-6e36-4634-8acd-fdc7dd140fc3" containerID="214ad0f6f8b57a505c276f285068921693817cb4e3b4eecad43a8800443fce1e" exitCode=0
Jul 15 10:54:50.818799 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:50.818717    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb" event={"ID":"0d952e58-6e36-4634-8acd-fdc7dd140fc3","Type":"ContainerDied","Data":"214ad0f6f8b57a505c276f285068921693817cb4e3b4eecad43a8800443fce1e"}
Jul 15 10:54:50.821057 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:50.820972    2563 generic.go:356] "Generic (PLEG): container finished" podID="da453fef-0f6a-43b1-a04a-e5fabd061460" containerID="dba88f6821ea153fbb60afe2d6d067fcf6c533a466b2fd392eb27354c4422547" exitCode=0
Jul 15 10:54:50.821057 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:54:50.821001    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh" event={"ID":"da453fef-0f6a-43b1-a04a-e5fabd061460","Type":"ContainerDied","Data":"dba88f6821ea153fbb60afe2d6d067fcf6c533a466b2fd392eb27354c4422547"}
Jul 15 10:55:02.883704 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:02.883655    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb" event={"ID":"0d952e58-6e36-4634-8acd-fdc7dd140fc3","Type":"ContainerStarted","Data":"178050ef87da42773905f493e7b2eafb5fb250775ae3f192e5fea9efc0b0c941"}
Jul 15 10:55:02.885834 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:02.885788    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh" event={"ID":"da453fef-0f6a-43b1-a04a-e5fabd061460","Type":"ContainerStarted","Data":"58deac8f5693b704c7a4e9fb0e5fc14adc6eb34dfd95ed17ad19532e9d406caf"}
Jul 15 10:55:02.887511 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:02.887456    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr" event={"ID":"e9436c84-deca-4753-a2be-5ec45980638f","Type":"ContainerStarted","Data":"96b59e654a18fbc8c0d88641efdcf7f47c276503bb1a88da47d219de8bffac6c"}
Jul 15 10:55:02.889236 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:02.889210    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g" event={"ID":"8f43e5ac-68b6-4b9d-af22-40156d0c31da","Type":"ContainerStarted","Data":"378d487bbafa0a5b3723b5eeee09119d0c4db0b0f6c0ad8cb10acc0c1b7bc418"}
Jul 15 10:55:07.921631 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:07.921535    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb" event={"ID":"0d952e58-6e36-4634-8acd-fdc7dd140fc3","Type":"ContainerStarted","Data":"e87708530f6a5f64d691f250d2a1099cdac496591a689e20e635c5ec5fa3d990"}
Jul 15 10:55:07.922103 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:07.921646    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb"
Jul 15 10:55:07.924615 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:07.924584    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh" event={"ID":"da453fef-0f6a-43b1-a04a-e5fabd061460","Type":"ContainerStarted","Data":"2b24e8de1cb7c670e751f63bd3601a3ef2a65293665492f056af3cdf7c740175"}
Jul 15 10:55:07.924805 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:07.924790    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh"
Jul 15 10:55:07.924875 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:07.924838    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh"
Jul 15 10:55:07.926280 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:07.926254    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh" podUID="da453fef-0f6a-43b1-a04a-e5fabd061460" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 15 10:55:07.927486 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:07.927461    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr" event={"ID":"e9436c84-deca-4753-a2be-5ec45980638f","Type":"ContainerStarted","Data":"b664a49e9b4b941a40e250484962655846cfa4bb6319a42abde806e154ae8e6e"}
Jul 15 10:55:07.927600 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:07.927516    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr"
Jul 15 10:55:07.927600 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:07.927587    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr"
Jul 15 10:55:07.928757 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:07.928696    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr" podUID="e9436c84-deca-4753-a2be-5ec45980638f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 15 10:55:07.929465 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:07.929444    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g" event={"ID":"8f43e5ac-68b6-4b9d-af22-40156d0c31da","Type":"ContainerStarted","Data":"418a0e153095a3fd90278b7ec59240112017718c0a47fd60557ac3be239c1519"}
Jul 15 10:55:07.929692 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:07.929673    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g"
Jul 15 10:55:07.929789 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:07.929720    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g"
Jul 15 10:55:07.930485 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:07.930467    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g" podUID="8f43e5ac-68b6-4b9d-af22-40156d0c31da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 15 10:55:07.998573 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:07.998513    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g" podStartSLOduration=1.993892564 podStartE2EDuration="24.998492839s" totalImagesPullingTime="23.004600275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:54:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:55:07.996700412 +0000 UTC m=+526.826215816" watchObservedRunningTime="2026-07-15 10:55:07.998492839 +0000 UTC m=+526.828008245"
Jul 15 10:55:07.998744 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:07.998719    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb" podStartSLOduration=4.446246053 podStartE2EDuration="24.998710292s" totalImagesPullingTime="16.565770652s" totalInitContainerRuntime="3.986693587s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:54:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:55:07.961733355 +0000 UTC m=+526.791248761" watchObservedRunningTime="2026-07-15 10:55:07.998710292 +0000 UTC m=+526.828225697"
Jul 15 10:55:08.054811 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:08.054744    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh" podStartSLOduration=4.169642176 podStartE2EDuration="25.054723379s" totalImagesPullingTime="16.841857581s" totalInitContainerRuntime="4.043223622s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:54:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:55:08.0537857 +0000 UTC m=+526.883301107" watchObservedRunningTime="2026-07-15 10:55:08.054723379 +0000 UTC m=+526.884238784"
Jul 15 10:55:08.933769 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:08.933722    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr" podUID="e9436c84-deca-4753-a2be-5ec45980638f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 15 10:55:08.933769 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:08.933764    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh" podUID="da453fef-0f6a-43b1-a04a-e5fabd061460" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 15 10:55:08.934284 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:08.933837    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g" podUID="8f43e5ac-68b6-4b9d-af22-40156d0c31da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 15 10:55:08.934335 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:08.934293    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb"
Jul 15 10:55:08.934737 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:08.934716    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb" podUID="0d952e58-6e36-4634-8acd-fdc7dd140fc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 15 10:55:09.938787 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:09.938738    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb" podUID="0d952e58-6e36-4634-8acd-fdc7dd140fc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 15 10:55:13.937792 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:13.937710    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh"
Jul 15 10:55:13.938380 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:13.938352    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh" podUID="da453fef-0f6a-43b1-a04a-e5fabd061460" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 15 10:55:13.938665 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:13.938642    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr"
Jul 15 10:55:13.938766 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:13.938710    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g"
Jul 15 10:55:13.939032 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:13.939006    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g" podUID="8f43e5ac-68b6-4b9d-af22-40156d0c31da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 15 10:55:13.939107 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:13.939052    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr" podUID="e9436c84-deca-4753-a2be-5ec45980638f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 15 10:55:13.964253 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:13.964201    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr" podStartSLOduration=7.970642594 podStartE2EDuration="31.964185101s" totalImagesPullingTime="23.993542507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:54:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:55:08.084228134 +0000 UTC m=+526.913743537" watchObservedRunningTime="2026-07-15 10:55:13.964185101 +0000 UTC m=+532.793700505"
Jul 15 10:55:14.944344 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:14.944314    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb"
Jul 15 10:55:14.944915 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:14.944878    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb" podUID="0d952e58-6e36-4634-8acd-fdc7dd140fc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 15 10:55:22.999508 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:22.999472    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn" event={"ID":"478795bd-bfa8-449e-8ba0-e9d93d900364","Type":"ContainerStarted","Data":"3964554c7fa1aa6966209465bae0c6d608367c1b4b2bc2137970730ab7797d46"}
Jul 15 10:55:23.000010 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:22.999543    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn" event={"ID":"478795bd-bfa8-449e-8ba0-e9d93d900364","Type":"ContainerStarted","Data":"6bdccddd93b43bd16b99efa59394000efbf2369d74e4ee4e5369f5be5767f2f3"}
Jul 15 10:55:23.000010 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:22.999773    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn"
Jul 15 10:55:23.025525 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:23.025477    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn" podStartSLOduration=4.905747171 podStartE2EDuration="40.025461562s" totalImagesPullingTime="31.346261002s" totalInitContainerRuntime="3.773453389s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:54:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:55:23.023189562 +0000 UTC m=+541.852704970" watchObservedRunningTime="2026-07-15 10:55:23.025461562 +0000 UTC m=+541.854976965"
Jul 15 10:55:23.938996 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:23.938952    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh" podUID="da453fef-0f6a-43b1-a04a-e5fabd061460" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 15 10:55:23.939200 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:23.938959    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr" podUID="e9436c84-deca-4753-a2be-5ec45980638f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 15 10:55:23.939200 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:23.939082    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g" podUID="8f43e5ac-68b6-4b9d-af22-40156d0c31da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 15 10:55:24.003360 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:24.003323    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn"
Jul 15 10:55:24.004552 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:24.004526    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn" podUID="478795bd-bfa8-449e-8ba0-e9d93d900364" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 15 10:55:24.944995 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:24.944957    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb" podUID="0d952e58-6e36-4634-8acd-fdc7dd140fc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 15 10:55:25.007440 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:25.007396    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn" podUID="478795bd-bfa8-449e-8ba0-e9d93d900364" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 15 10:55:30.012520 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:30.012490    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn"
Jul 15 10:55:30.013185 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:30.013156    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn" podUID="478795bd-bfa8-449e-8ba0-e9d93d900364" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 15 10:55:33.938783 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:33.938743    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh" podUID="da453fef-0f6a-43b1-a04a-e5fabd061460" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 15 10:55:33.939325 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:33.938990    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr" podUID="e9436c84-deca-4753-a2be-5ec45980638f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 15 10:55:33.939325 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:33.939021    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g" podUID="8f43e5ac-68b6-4b9d-af22-40156d0c31da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 15 10:55:34.945545 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:34.945509    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb" podUID="0d952e58-6e36-4634-8acd-fdc7dd140fc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 15 10:55:40.013604 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:40.013563    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn" podUID="478795bd-bfa8-449e-8ba0-e9d93d900364" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 15 10:55:43.938724 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:43.938683    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh" podUID="da453fef-0f6a-43b1-a04a-e5fabd061460" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 15 10:55:43.939213 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:43.938966    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr" podUID="e9436c84-deca-4753-a2be-5ec45980638f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 15 10:55:43.939213 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:43.939091    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g" podUID="8f43e5ac-68b6-4b9d-af22-40156d0c31da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 15 10:55:44.945509 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:44.945469    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb" podUID="0d952e58-6e36-4634-8acd-fdc7dd140fc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 15 10:55:50.013822 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:50.013781    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn" podUID="478795bd-bfa8-449e-8ba0-e9d93d900364" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 15 10:55:53.938969 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:53.938937    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh" podUID="da453fef-0f6a-43b1-a04a-e5fabd061460" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 15 10:55:53.939416 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:53.939106    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr" podUID="e9436c84-deca-4753-a2be-5ec45980638f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 15 10:55:53.939416 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:53.939300    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g"
Jul 15 10:55:54.945235 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:55:54.945195    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb" podUID="0d952e58-6e36-4634-8acd-fdc7dd140fc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 15 10:56:00.013638 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:00.013588    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn" podUID="478795bd-bfa8-449e-8ba0-e9d93d900364" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 15 10:56:03.938285 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:03.938247    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh" podUID="da453fef-0f6a-43b1-a04a-e5fabd061460" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 15 10:56:03.939977 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:03.939954    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr"
Jul 15 10:56:04.944894 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:04.944858    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb" podUID="0d952e58-6e36-4634-8acd-fdc7dd140fc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 15 10:56:10.013639 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:10.013594    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn" podUID="478795bd-bfa8-449e-8ba0-e9d93d900364" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 15 10:56:13.938288 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:13.938252    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh" podUID="da453fef-0f6a-43b1-a04a-e5fabd061460" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 15 10:56:14.945647 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:14.945608    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb" podUID="0d952e58-6e36-4634-8acd-fdc7dd140fc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 15 10:56:20.013864 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:20.013821    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn" podUID="478795bd-bfa8-449e-8ba0-e9d93d900364" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 15 10:56:21.544218 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:21.544194    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk49n_beb68899-ffb3-436f-8e17-43597aad4030/ovn-acl-logging/0.log"
Jul 15 10:56:21.544602 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:21.544579    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk49n_beb68899-ffb3-436f-8e17-43597aad4030/ovn-acl-logging/0.log"
Jul 15 10:56:23.673788 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:23.673746    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh" podUID="da453fef-0f6a-43b1-a04a-e5fabd061460" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 15 10:56:24.945289 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:24.945251    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb" podUID="0d952e58-6e36-4634-8acd-fdc7dd140fc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 15 10:56:30.013280 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:30.013253    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn"
Jul 15 10:56:33.115746 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.115713    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr"]
Jul 15 10:56:33.116218 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.116019    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr" podUID="e9436c84-deca-4753-a2be-5ec45980638f" containerName="kserve-container" containerID="cri-o://96b59e654a18fbc8c0d88641efdcf7f47c276503bb1a88da47d219de8bffac6c" gracePeriod=30
Jul 15 10:56:33.116218 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.116079    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr" podUID="e9436c84-deca-4753-a2be-5ec45980638f" containerName="kube-rbac-proxy" containerID="cri-o://b664a49e9b4b941a40e250484962655846cfa4bb6319a42abde806e154ae8e6e" gracePeriod=30
Jul 15 10:56:33.199461 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.199427    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9"]
Jul 15 10:56:33.202080 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.202065    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9"
Jul 15 10:56:33.204807 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.204783    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ff4f8-kube-rbac-proxy-sar-config\""
Jul 15 10:56:33.204945 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.204832    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ff4f8-predictor-serving-cert\""
Jul 15 10:56:33.221635 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.221609    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9"]
Jul 15 10:56:33.244461 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.244419    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c456ffb4-785e-468a-9715-dd367745b6af-proxy-tls\") pod \"success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9\" (UID: \"c456ffb4-785e-468a-9715-dd367745b6af\") " pod="kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9"
Jul 15 10:56:33.244618 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.244532    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c456ffb4-785e-468a-9715-dd367745b6af-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9\" (UID: \"c456ffb4-785e-468a-9715-dd367745b6af\") " pod="kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9"
Jul 15 10:56:33.244618 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.244578    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b58zv\" (UniqueName: \"kubernetes.io/projected/c456ffb4-785e-468a-9715-dd367745b6af-kube-api-access-b58zv\") pod \"success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9\" (UID: \"c456ffb4-785e-468a-9715-dd367745b6af\") " pod="kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9"
Jul 15 10:56:33.276111 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.276081    2563 generic.go:356] "Generic (PLEG): container finished" podID="e9436c84-deca-4753-a2be-5ec45980638f" containerID="b664a49e9b4b941a40e250484962655846cfa4bb6319a42abde806e154ae8e6e" exitCode=2
Jul 15 10:56:33.276275 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.276161    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr" event={"ID":"e9436c84-deca-4753-a2be-5ec45980638f","Type":"ContainerDied","Data":"b664a49e9b4b941a40e250484962655846cfa4bb6319a42abde806e154ae8e6e"}
Jul 15 10:56:33.281097 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.281072    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g"]
Jul 15 10:56:33.281306 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.281284    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g" podUID="8f43e5ac-68b6-4b9d-af22-40156d0c31da" containerName="kserve-container" containerID="cri-o://378d487bbafa0a5b3723b5eeee09119d0c4db0b0f6c0ad8cb10acc0c1b7bc418" gracePeriod=30
Jul 15 10:56:33.281374 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.281337    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g" podUID="8f43e5ac-68b6-4b9d-af22-40156d0c31da" containerName="kube-rbac-proxy" containerID="cri-o://418a0e153095a3fd90278b7ec59240112017718c0a47fd60557ac3be239c1519" gracePeriod=30
Jul 15 10:56:33.299290 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.299267    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg"]
Jul 15 10:56:33.301613 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.301596    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg"
Jul 15 10:56:33.304462 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.304445    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ff4f8-kube-rbac-proxy-sar-config\""
Jul 15 10:56:33.304575 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.304486    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ff4f8-predictor-serving-cert\""
Jul 15 10:56:33.312670 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.312649    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg"]
Jul 15 10:56:33.345719 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.345687    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m9z7\" (UniqueName: \"kubernetes.io/projected/ae6f7d2a-dde4-4d1a-8931-9f1434e352f7-kube-api-access-7m9z7\") pod \"error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg\" (UID: \"ae6f7d2a-dde4-4d1a-8931-9f1434e352f7\") " pod="kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg"
Jul 15 10:56:33.345847 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.345737    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae6f7d2a-dde4-4d1a-8931-9f1434e352f7-proxy-tls\") pod \"error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg\" (UID: \"ae6f7d2a-dde4-4d1a-8931-9f1434e352f7\") " pod="kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg"
Jul 15 10:56:33.345847 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.345762    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c456ffb4-785e-468a-9715-dd367745b6af-proxy-tls\") pod \"success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9\" (UID: \"c456ffb4-785e-468a-9715-dd367745b6af\") " pod="kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9"
Jul 15 10:56:33.345847 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.345789    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae6f7d2a-dde4-4d1a-8931-9f1434e352f7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg\" (UID: \"ae6f7d2a-dde4-4d1a-8931-9f1434e352f7\") " pod="kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg"
Jul 15 10:56:33.345847 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.345826    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c456ffb4-785e-468a-9715-dd367745b6af-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9\" (UID: \"c456ffb4-785e-468a-9715-dd367745b6af\") " pod="kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9"
Jul 15 10:56:33.345979 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.345855    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b58zv\" (UniqueName: \"kubernetes.io/projected/c456ffb4-785e-468a-9715-dd367745b6af-kube-api-access-b58zv\") pod \"success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9\" (UID: \"c456ffb4-785e-468a-9715-dd367745b6af\") " pod="kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9"
Jul 15 10:56:33.346460 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.346437    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c456ffb4-785e-468a-9715-dd367745b6af-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9\" (UID: \"c456ffb4-785e-468a-9715-dd367745b6af\") " pod="kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9"
Jul 15 10:56:33.348043 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.348022    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c456ffb4-785e-468a-9715-dd367745b6af-proxy-tls\") pod \"success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9\" (UID: \"c456ffb4-785e-468a-9715-dd367745b6af\") " pod="kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9"
Jul 15 10:56:33.354343 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.354322    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b58zv\" (UniqueName: \"kubernetes.io/projected/c456ffb4-785e-468a-9715-dd367745b6af-kube-api-access-b58zv\") pod \"success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9\" (UID: \"c456ffb4-785e-468a-9715-dd367745b6af\") " pod="kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9"
Jul 15 10:56:33.446624 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.446594    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae6f7d2a-dde4-4d1a-8931-9f1434e352f7-proxy-tls\") pod \"error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg\" (UID: \"ae6f7d2a-dde4-4d1a-8931-9f1434e352f7\") " pod="kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg"
Jul 15 10:56:33.446787 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.446634    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae6f7d2a-dde4-4d1a-8931-9f1434e352f7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg\" (UID: \"ae6f7d2a-dde4-4d1a-8931-9f1434e352f7\") " pod="kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg"
Jul 15 10:56:33.446787 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.446693    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m9z7\" (UniqueName: \"kubernetes.io/projected/ae6f7d2a-dde4-4d1a-8931-9f1434e352f7-kube-api-access-7m9z7\") pod \"error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg\" (UID: \"ae6f7d2a-dde4-4d1a-8931-9f1434e352f7\") " pod="kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg"
Jul 15 10:56:33.447286 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.447263    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae6f7d2a-dde4-4d1a-8931-9f1434e352f7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg\" (UID: \"ae6f7d2a-dde4-4d1a-8931-9f1434e352f7\") " pod="kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg"
Jul 15 10:56:33.448673 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.448651    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae6f7d2a-dde4-4d1a-8931-9f1434e352f7-proxy-tls\") pod \"error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg\" (UID: \"ae6f7d2a-dde4-4d1a-8931-9f1434e352f7\") " pod="kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg"
Jul 15 10:56:33.455625 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.455602    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m9z7\" (UniqueName: \"kubernetes.io/projected/ae6f7d2a-dde4-4d1a-8931-9f1434e352f7-kube-api-access-7m9z7\") pod \"error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg\" (UID: \"ae6f7d2a-dde4-4d1a-8931-9f1434e352f7\") " pod="kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg"
Jul 15 10:56:33.511868 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.511846    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9"
Jul 15 10:56:33.612636 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.612594    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg"
Jul 15 10:56:33.639535 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.639505    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9"]
Jul 15 10:56:33.643127 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:56:33.643085    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc456ffb4_785e_468a_9715_dd367745b6af.slice/crio-9e01f44bf8af27bec12adc813fd307df7a2d573f3ee14e751784888f3b15a6a0 WatchSource:0}: Error finding container 9e01f44bf8af27bec12adc813fd307df7a2d573f3ee14e751784888f3b15a6a0: Status 404 returned error can't find the container with id 9e01f44bf8af27bec12adc813fd307df7a2d573f3ee14e751784888f3b15a6a0
Jul 15 10:56:33.678068 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.678042    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh"
Jul 15 10:56:33.756061 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.756030    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg"]
Jul 15 10:56:33.760652 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:56:33.760628    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae6f7d2a_dde4_4d1a_8931_9f1434e352f7.slice/crio-48a8c520cbebb0042089670be010741d232e5171bce75ededc8f9f885ee2950f WatchSource:0}: Error finding container 48a8c520cbebb0042089670be010741d232e5171bce75ededc8f9f885ee2950f: Status 404 returned error can't find the container with id 48a8c520cbebb0042089670be010741d232e5171bce75ededc8f9f885ee2950f
Jul 15 10:56:33.934584 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.934498    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr" podUID="e9436c84-deca-4753-a2be-5ec45980638f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Jul 15 10:56:33.934584 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.934514    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g" podUID="8f43e5ac-68b6-4b9d-af22-40156d0c31da" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 15 10:56:33.939622 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.939597    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g" podUID="8f43e5ac-68b6-4b9d-af22-40156d0c31da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 15 10:56:33.939710 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:33.939600    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr" podUID="e9436c84-deca-4753-a2be-5ec45980638f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 15 10:56:34.282320 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:34.282281    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg" event={"ID":"ae6f7d2a-dde4-4d1a-8931-9f1434e352f7","Type":"ContainerStarted","Data":"a57a0c822df0eeac9d6daae008bd498dd64c6934d106490a8e14ad3526bdf464"}
Jul 15 10:56:34.282733 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:34.282384    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg"
Jul 15 10:56:34.282733 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:34.282415    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg" event={"ID":"ae6f7d2a-dde4-4d1a-8931-9f1434e352f7","Type":"ContainerStarted","Data":"7687d00931774a2b9531d1bd71147f94790b247bf54e2c5ecfa87ec89f2e0d40"}
Jul 15 10:56:34.282733 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:34.282444    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg" event={"ID":"ae6f7d2a-dde4-4d1a-8931-9f1434e352f7","Type":"ContainerStarted","Data":"48a8c520cbebb0042089670be010741d232e5171bce75ededc8f9f885ee2950f"}
Jul 15 10:56:34.282733 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:34.282472    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg"
Jul 15 10:56:34.283938 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:34.283905    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg" podUID="ae6f7d2a-dde4-4d1a-8931-9f1434e352f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 15 10:56:34.284254 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:34.284229    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9" event={"ID":"c456ffb4-785e-468a-9715-dd367745b6af","Type":"ContainerStarted","Data":"0d0ca43c47e66a865b1d29a4a7ee5fe77dd69c9862fc7c60a961bf4a6f27a363"}
Jul 15 10:56:34.284340 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:34.284291    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9" event={"ID":"c456ffb4-785e-468a-9715-dd367745b6af","Type":"ContainerStarted","Data":"d4dd31a414a9293b8576f79468cbb2e3c9243af0672f8a49ac96f81091a54a1c"}
Jul 15 10:56:34.284340 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:34.284328    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9"
Jul 15 10:56:34.284421 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:34.284354    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9" event={"ID":"c456ffb4-785e-468a-9715-dd367745b6af","Type":"ContainerStarted","Data":"9e01f44bf8af27bec12adc813fd307df7a2d573f3ee14e751784888f3b15a6a0"}
Jul 15 10:56:34.285919 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:34.285898    2563 generic.go:356] "Generic (PLEG): container finished" podID="8f43e5ac-68b6-4b9d-af22-40156d0c31da" containerID="418a0e153095a3fd90278b7ec59240112017718c0a47fd60557ac3be239c1519" exitCode=2
Jul 15 10:56:34.285983 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:34.285931    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g" event={"ID":"8f43e5ac-68b6-4b9d-af22-40156d0c31da","Type":"ContainerDied","Data":"418a0e153095a3fd90278b7ec59240112017718c0a47fd60557ac3be239c1519"}
Jul 15 10:56:34.306526 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:34.306482    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg" podStartSLOduration=1.3064687689999999 podStartE2EDuration="1.306468769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:56:34.305080929 +0000 UTC m=+613.134596344" watchObservedRunningTime="2026-07-15 10:56:34.306468769 +0000 UTC m=+613.135984172"
Jul 15 10:56:34.328673 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:34.328631    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9" podStartSLOduration=1.328618515 podStartE2EDuration="1.328618515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:56:34.326362558 +0000 UTC m=+613.155877961" watchObservedRunningTime="2026-07-15 10:56:34.328618515 +0000 UTC m=+613.158133976"
Jul 15 10:56:34.945463 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:34.945435    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb"
Jul 15 10:56:35.291029 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:35.290933    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg" podUID="ae6f7d2a-dde4-4d1a-8931-9f1434e352f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 15 10:56:35.291029 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:35.290959    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9"
Jul 15 10:56:35.292221 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:35.292194    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9" podUID="c456ffb4-785e-468a-9715-dd367745b6af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 15 10:56:36.261020 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:36.260993    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr"
Jul 15 10:56:36.296180 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:36.296128    2563 generic.go:356] "Generic (PLEG): container finished" podID="e9436c84-deca-4753-a2be-5ec45980638f" containerID="96b59e654a18fbc8c0d88641efdcf7f47c276503bb1a88da47d219de8bffac6c" exitCode=0
Jul 15 10:56:36.296180 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:36.296173    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr" event={"ID":"e9436c84-deca-4753-a2be-5ec45980638f","Type":"ContainerDied","Data":"96b59e654a18fbc8c0d88641efdcf7f47c276503bb1a88da47d219de8bffac6c"}
Jul 15 10:56:36.296180 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:36.296185    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr"
Jul 15 10:56:36.296697 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:36.296233    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr" event={"ID":"e9436c84-deca-4753-a2be-5ec45980638f","Type":"ContainerDied","Data":"6f6ed26a2ed2fabf46832d39bbb93db9a5fb320b95a5256cce0906bc002a61dc"}
Jul 15 10:56:36.296697 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:36.296261    2563 scope.go:117] "RemoveContainer" containerID="b664a49e9b4b941a40e250484962655846cfa4bb6319a42abde806e154ae8e6e"
Jul 15 10:56:36.298583 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:36.298555    2563 generic.go:356] "Generic (PLEG): container finished" podID="8f43e5ac-68b6-4b9d-af22-40156d0c31da" containerID="378d487bbafa0a5b3723b5eeee09119d0c4db0b0f6c0ad8cb10acc0c1b7bc418" exitCode=0
Jul 15 10:56:36.298699 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:36.298615    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g" event={"ID":"8f43e5ac-68b6-4b9d-af22-40156d0c31da","Type":"ContainerDied","Data":"378d487bbafa0a5b3723b5eeee09119d0c4db0b0f6c0ad8cb10acc0c1b7bc418"}
Jul 15 10:56:36.298966 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:36.298934    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9" podUID="c456ffb4-785e-468a-9715-dd367745b6af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 15 10:56:36.305744 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:36.305720    2563 scope.go:117] "RemoveContainer" containerID="96b59e654a18fbc8c0d88641efdcf7f47c276503bb1a88da47d219de8bffac6c"
Jul 15 10:56:36.313287 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:36.313265    2563 scope.go:117] "RemoveContainer" containerID="b664a49e9b4b941a40e250484962655846cfa4bb6319a42abde806e154ae8e6e"
Jul 15 10:56:36.313570 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:56:36.313543    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b664a49e9b4b941a40e250484962655846cfa4bb6319a42abde806e154ae8e6e\": container with ID starting with b664a49e9b4b941a40e250484962655846cfa4bb6319a42abde806e154ae8e6e not found: ID does not exist" containerID="b664a49e9b4b941a40e250484962655846cfa4bb6319a42abde806e154ae8e6e"
Jul 15 10:56:36.313651 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:36.313586    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b664a49e9b4b941a40e250484962655846cfa4bb6319a42abde806e154ae8e6e"} err="failed to get container status \"b664a49e9b4b941a40e250484962655846cfa4bb6319a42abde806e154ae8e6e\": rpc error: code = NotFound desc = could not find container \"b664a49e9b4b941a40e250484962655846cfa4bb6319a42abde806e154ae8e6e\": container with ID starting with b664a49e9b4b941a40e250484962655846cfa4bb6319a42abde806e154ae8e6e not found: ID does not exist"
Jul 15 10:56:36.313651 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:36.313609    2563 scope.go:117] "RemoveContainer" containerID="96b59e654a18fbc8c0d88641efdcf7f47c276503bb1a88da47d219de8bffac6c"
Jul 15 10:56:36.313876 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:56:36.313855    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96b59e654a18fbc8c0d88641efdcf7f47c276503bb1a88da47d219de8bffac6c\": container with ID starting with 96b59e654a18fbc8c0d88641efdcf7f47c276503bb1a88da47d219de8bffac6c not found: ID does not exist" containerID="96b59e654a18fbc8c0d88641efdcf7f47c276503bb1a88da47d219de8bffac6c"
Jul 15 10:56:36.313965 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:36.313882    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96b59e654a18fbc8c0d88641efdcf7f47c276503bb1a88da47d219de8bffac6c"} err="failed to get container status \"96b59e654a18fbc8c0d88641efdcf7f47c276503bb1a88da47d219de8bffac6c\": rpc error: code = NotFound desc = could not find container \"96b59e654a18fbc8c0d88641efdcf7f47c276503bb1a88da47d219de8bffac6c\": container with ID starting with 96b59e654a18fbc8c0d88641efdcf7f47c276503bb1a88da47d219de8bffac6c not found: ID does not exist"
Jul 15 10:56:36.326254 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:36.326235    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g"
Jul 15 10:56:36.372477 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:36.372454    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nq9tz\" (UniqueName: \"kubernetes.io/projected/8f43e5ac-68b6-4b9d-af22-40156d0c31da-kube-api-access-nq9tz\") pod \"8f43e5ac-68b6-4b9d-af22-40156d0c31da\" (UID: \"8f43e5ac-68b6-4b9d-af22-40156d0c31da\") "
Jul 15 10:56:36.372598 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:36.372488    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f43e5ac-68b6-4b9d-af22-40156d0c31da-kube-rbac-proxy-sar-config\") pod \"8f43e5ac-68b6-4b9d-af22-40156d0c31da\" (UID: \"8f43e5ac-68b6-4b9d-af22-40156d0c31da\") "
Jul 15 10:56:36.372598 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:36.372521    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4tr8\" (UniqueName: \"kubernetes.io/projected/e9436c84-deca-4753-a2be-5ec45980638f-kube-api-access-g4tr8\") pod \"e9436c84-deca-4753-a2be-5ec45980638f\" (UID: \"e9436c84-deca-4753-a2be-5ec45980638f\") "
Jul 15 10:56:36.372598 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:36.372567    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9436c84-deca-4753-a2be-5ec45980638f-proxy-tls\") pod \"e9436c84-deca-4753-a2be-5ec45980638f\" (UID: \"e9436c84-deca-4753-a2be-5ec45980638f\") "
Jul 15 10:56:36.372598 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:36.372586    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9436c84-deca-4753-a2be-5ec45980638f-kube-rbac-proxy-sar-config\") pod \"e9436c84-deca-4753-a2be-5ec45980638f\" (UID: \"e9436c84-deca-4753-a2be-5ec45980638f\") "
Jul 15 10:56:36.372736 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:36.372634    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f43e5ac-68b6-4b9d-af22-40156d0c31da-proxy-tls\") pod \"8f43e5ac-68b6-4b9d-af22-40156d0c31da\" (UID: \"8f43e5ac-68b6-4b9d-af22-40156d0c31da\") "
Jul 15 10:56:36.373022 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:36.372994    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f43e5ac-68b6-4b9d-af22-40156d0c31da-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8f43e5ac-68b6-4b9d-af22-40156d0c31da" (UID: "8f43e5ac-68b6-4b9d-af22-40156d0c31da"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:56:36.373126 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:36.373042    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9436c84-deca-4753-a2be-5ec45980638f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e9436c84-deca-4753-a2be-5ec45980638f" (UID: "e9436c84-deca-4753-a2be-5ec45980638f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:56:36.374762 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:36.374738    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9436c84-deca-4753-a2be-5ec45980638f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e9436c84-deca-4753-a2be-5ec45980638f" (UID: "e9436c84-deca-4753-a2be-5ec45980638f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 10:56:36.374762 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:36.374741    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9436c84-deca-4753-a2be-5ec45980638f-kube-api-access-g4tr8" (OuterVolumeSpecName: "kube-api-access-g4tr8") pod "e9436c84-deca-4753-a2be-5ec45980638f" (UID: "e9436c84-deca-4753-a2be-5ec45980638f"). InnerVolumeSpecName "kube-api-access-g4tr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 10:56:36.375029 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:36.375005    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f43e5ac-68b6-4b9d-af22-40156d0c31da-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8f43e5ac-68b6-4b9d-af22-40156d0c31da" (UID: "8f43e5ac-68b6-4b9d-af22-40156d0c31da"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 10:56:36.375029 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:36.375008    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f43e5ac-68b6-4b9d-af22-40156d0c31da-kube-api-access-nq9tz" (OuterVolumeSpecName: "kube-api-access-nq9tz") pod "8f43e5ac-68b6-4b9d-af22-40156d0c31da" (UID: "8f43e5ac-68b6-4b9d-af22-40156d0c31da"). InnerVolumeSpecName "kube-api-access-nq9tz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 10:56:36.473861 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:36.473817    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9436c84-deca-4753-a2be-5ec45980638f-proxy-tls\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:56:36.473861 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:36.473857    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9436c84-deca-4753-a2be-5ec45980638f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:56:36.473861 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:36.473868    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f43e5ac-68b6-4b9d-af22-40156d0c31da-proxy-tls\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:56:36.473861 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:36.473877    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq9tz\" (UniqueName: \"kubernetes.io/projected/8f43e5ac-68b6-4b9d-af22-40156d0c31da-kube-api-access-nq9tz\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:56:36.474136 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:36.473886    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f43e5ac-68b6-4b9d-af22-40156d0c31da-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:56:36.474136 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:36.473895    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4tr8\" (UniqueName: \"kubernetes.io/projected/e9436c84-deca-4753-a2be-5ec45980638f-kube-api-access-g4tr8\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:56:36.627152 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:36.627119    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr"]
Jul 15 10:56:36.629754 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:36.629733    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr"]
Jul 15 10:56:37.304075 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:37.304047    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g"
Jul 15 10:56:37.304500 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:37.304080    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g" event={"ID":"8f43e5ac-68b6-4b9d-af22-40156d0c31da","Type":"ContainerDied","Data":"8035dcaac3799bdf892bd0a95e2cfe4e61fdcc0238288bc0ce5b6d0bc4c195ed"}
Jul 15 10:56:37.304500 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:37.304166    2563 scope.go:117] "RemoveContainer" containerID="418a0e153095a3fd90278b7ec59240112017718c0a47fd60557ac3be239c1519"
Jul 15 10:56:37.312539 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:37.312521    2563 scope.go:117] "RemoveContainer" containerID="378d487bbafa0a5b3723b5eeee09119d0c4db0b0f6c0ad8cb10acc0c1b7bc418"
Jul 15 10:56:37.338153 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:37.338116    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g"]
Jul 15 10:56:37.342861 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:37.342839    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g"]
Jul 15 10:56:37.677875 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:37.677800    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f43e5ac-68b6-4b9d-af22-40156d0c31da" path="/var/lib/kubelet/pods/8f43e5ac-68b6-4b9d-af22-40156d0c31da/volumes"
Jul 15 10:56:37.678236 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:37.678221    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9436c84-deca-4753-a2be-5ec45980638f" path="/var/lib/kubelet/pods/e9436c84-deca-4753-a2be-5ec45980638f/volumes"
Jul 15 10:56:40.295485 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:40.295456    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg"
Jul 15 10:56:40.295978 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:40.295947    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg" podUID="ae6f7d2a-dde4-4d1a-8931-9f1434e352f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 15 10:56:41.303541 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:41.303515    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9"
Jul 15 10:56:41.303968 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:41.303911    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9" podUID="c456ffb4-785e-468a-9715-dd367745b6af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 15 10:56:50.296704 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:50.296662    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg" podUID="ae6f7d2a-dde4-4d1a-8931-9f1434e352f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 15 10:56:51.304178 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:56:51.304122    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9" podUID="c456ffb4-785e-468a-9715-dd367745b6af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 15 10:57:00.296001 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:00.295955    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg" podUID="ae6f7d2a-dde4-4d1a-8931-9f1434e352f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 15 10:57:01.304176 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:01.304117    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9" podUID="c456ffb4-785e-468a-9715-dd367745b6af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 15 10:57:10.296052 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:10.296013    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg" podUID="ae6f7d2a-dde4-4d1a-8931-9f1434e352f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 15 10:57:11.304643 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:11.304605    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9" podUID="c456ffb4-785e-468a-9715-dd367745b6af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 15 10:57:13.004211 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.004172    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh"]
Jul 15 10:57:13.004584 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.004421    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh" podUID="da453fef-0f6a-43b1-a04a-e5fabd061460" containerName="kserve-container" containerID="cri-o://58deac8f5693b704c7a4e9fb0e5fc14adc6eb34dfd95ed17ad19532e9d406caf" gracePeriod=30
Jul 15 10:57:13.004584 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.004439    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh" podUID="da453fef-0f6a-43b1-a04a-e5fabd061460" containerName="kube-rbac-proxy" containerID="cri-o://2b24e8de1cb7c670e751f63bd3601a3ef2a65293665492f056af3cdf7c740175" gracePeriod=30
Jul 15 10:57:13.060543 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.060512    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb"]
Jul 15 10:57:13.060746 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.060724    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb" podUID="0d952e58-6e36-4634-8acd-fdc7dd140fc3" containerName="kserve-container" containerID="cri-o://178050ef87da42773905f493e7b2eafb5fb250775ae3f192e5fea9efc0b0c941" gracePeriod=30
Jul 15 10:57:13.060827 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.060763    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb" podUID="0d952e58-6e36-4634-8acd-fdc7dd140fc3" containerName="kube-rbac-proxy" containerID="cri-o://e87708530f6a5f64d691f250d2a1099cdac496591a689e20e635c5ec5fa3d990" gracePeriod=30
Jul 15 10:57:13.095151 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.095099    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp"]
Jul 15 10:57:13.095482 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.095468    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f43e5ac-68b6-4b9d-af22-40156d0c31da" containerName="kube-rbac-proxy"
Jul 15 10:57:13.095532 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.095485    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f43e5ac-68b6-4b9d-af22-40156d0c31da" containerName="kube-rbac-proxy"
Jul 15 10:57:13.095532 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.095505    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f43e5ac-68b6-4b9d-af22-40156d0c31da" containerName="kserve-container"
Jul 15 10:57:13.095532 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.095513    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f43e5ac-68b6-4b9d-af22-40156d0c31da" containerName="kserve-container"
Jul 15 10:57:13.095532 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.095525    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9436c84-deca-4753-a2be-5ec45980638f" containerName="kube-rbac-proxy"
Jul 15 10:57:13.095532 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.095531    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9436c84-deca-4753-a2be-5ec45980638f" containerName="kube-rbac-proxy"
Jul 15 10:57:13.095675 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.095542    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9436c84-deca-4753-a2be-5ec45980638f" containerName="kserve-container"
Jul 15 10:57:13.095675 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.095548    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9436c84-deca-4753-a2be-5ec45980638f" containerName="kserve-container"
Jul 15 10:57:13.095675 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.095620    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f43e5ac-68b6-4b9d-af22-40156d0c31da" containerName="kserve-container"
Jul 15 10:57:13.095675 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.095628    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f43e5ac-68b6-4b9d-af22-40156d0c31da" containerName="kube-rbac-proxy"
Jul 15 10:57:13.095675 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.095638    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9436c84-deca-4753-a2be-5ec45980638f" containerName="kserve-container"
Jul 15 10:57:13.095675 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.095644    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9436c84-deca-4753-a2be-5ec45980638f" containerName="kube-rbac-proxy"
Jul 15 10:57:13.097832 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.097814    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp"
Jul 15 10:57:13.100810 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.100789    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-074d2-predictor-serving-cert\""
Jul 15 10:57:13.100919 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.100900    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-074d2-kube-rbac-proxy-sar-config\""
Jul 15 10:57:13.110687 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.110664    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp"]
Jul 15 10:57:13.136572 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.136542    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn"]
Jul 15 10:57:13.136820 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.136795    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn" podUID="478795bd-bfa8-449e-8ba0-e9d93d900364" containerName="kube-rbac-proxy" containerID="cri-o://3964554c7fa1aa6966209465bae0c6d608367c1b4b2bc2137970730ab7797d46" gracePeriod=30
Jul 15 10:57:13.136873 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.136776    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn" podUID="478795bd-bfa8-449e-8ba0-e9d93d900364" containerName="kserve-container" containerID="cri-o://6bdccddd93b43bd16b99efa59394000efbf2369d74e4ee4e5369f5be5767f2f3" gracePeriod=30
Jul 15 10:57:13.217878 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.217849    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp"]
Jul 15 10:57:13.220671 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.220654    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp"
Jul 15 10:57:13.223333 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.223305    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-074d2-predictor-serving-cert\""
Jul 15 10:57:13.223333 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.223322    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-074d2-kube-rbac-proxy-sar-config\""
Jul 15 10:57:13.231627 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.231605    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp"]
Jul 15 10:57:13.284481 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.284427    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/582d624f-7f4f-459e-af00-7e5555fe81fd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp\" (UID: \"582d624f-7f4f-459e-af00-7e5555fe81fd\") " pod="kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp"
Jul 15 10:57:13.284481 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.284461    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/582d624f-7f4f-459e-af00-7e5555fe81fd-proxy-tls\") pod \"success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp\" (UID: \"582d624f-7f4f-459e-af00-7e5555fe81fd\") " pod="kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp"
Jul 15 10:57:13.284599 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.284482    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl7lk\" (UniqueName: \"kubernetes.io/projected/582d624f-7f4f-459e-af00-7e5555fe81fd-kube-api-access-dl7lk\") pod \"success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp\" (UID: \"582d624f-7f4f-459e-af00-7e5555fe81fd\") " pod="kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp"
Jul 15 10:57:13.385889 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.385861    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/582d624f-7f4f-459e-af00-7e5555fe81fd-proxy-tls\") pod \"success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp\" (UID: \"582d624f-7f4f-459e-af00-7e5555fe81fd\") " pod="kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp"
Jul 15 10:57:13.386002 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.385893    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl7lk\" (UniqueName: \"kubernetes.io/projected/582d624f-7f4f-459e-af00-7e5555fe81fd-kube-api-access-dl7lk\") pod \"success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp\" (UID: \"582d624f-7f4f-459e-af00-7e5555fe81fd\") " pod="kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp"
Jul 15 10:57:13.386002 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.385950    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfpsf\" (UniqueName: \"kubernetes.io/projected/bff3e308-3999-4ede-8379-1fdc81f06ae9-kube-api-access-lfpsf\") pod \"error-404-isvc-074d2-predictor-576646885c-d8cfp\" (UID: \"bff3e308-3999-4ede-8379-1fdc81f06ae9\") " pod="kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp"
Jul 15 10:57:13.386103 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.385997    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bff3e308-3999-4ede-8379-1fdc81f06ae9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-074d2-predictor-576646885c-d8cfp\" (UID: \"bff3e308-3999-4ede-8379-1fdc81f06ae9\") " pod="kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp"
Jul 15 10:57:13.386184 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.386096    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bff3e308-3999-4ede-8379-1fdc81f06ae9-proxy-tls\") pod \"error-404-isvc-074d2-predictor-576646885c-d8cfp\" (UID: \"bff3e308-3999-4ede-8379-1fdc81f06ae9\") " pod="kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp"
Jul 15 10:57:13.386184 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.386162    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/582d624f-7f4f-459e-af00-7e5555fe81fd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp\" (UID: \"582d624f-7f4f-459e-af00-7e5555fe81fd\") " pod="kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp"
Jul 15 10:57:13.386737 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.386719    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/582d624f-7f4f-459e-af00-7e5555fe81fd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp\" (UID: \"582d624f-7f4f-459e-af00-7e5555fe81fd\") " pod="kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp"
Jul 15 10:57:13.388117 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.388099    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/582d624f-7f4f-459e-af00-7e5555fe81fd-proxy-tls\") pod \"success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp\" (UID: \"582d624f-7f4f-459e-af00-7e5555fe81fd\") " pod="kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp"
Jul 15 10:57:13.393978 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.393956    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl7lk\" (UniqueName: \"kubernetes.io/projected/582d624f-7f4f-459e-af00-7e5555fe81fd-kube-api-access-dl7lk\") pod \"success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp\" (UID: \"582d624f-7f4f-459e-af00-7e5555fe81fd\") " pod="kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp"
Jul 15 10:57:13.407823 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.407804    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp"
Jul 15 10:57:13.457516 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.457410    2563 generic.go:356] "Generic (PLEG): container finished" podID="0d952e58-6e36-4634-8acd-fdc7dd140fc3" containerID="e87708530f6a5f64d691f250d2a1099cdac496591a689e20e635c5ec5fa3d990" exitCode=2
Jul 15 10:57:13.457516 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.457475    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb" event={"ID":"0d952e58-6e36-4634-8acd-fdc7dd140fc3","Type":"ContainerDied","Data":"e87708530f6a5f64d691f250d2a1099cdac496591a689e20e635c5ec5fa3d990"}
Jul 15 10:57:13.460514 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.460487    2563 generic.go:356] "Generic (PLEG): container finished" podID="da453fef-0f6a-43b1-a04a-e5fabd061460" containerID="2b24e8de1cb7c670e751f63bd3601a3ef2a65293665492f056af3cdf7c740175" exitCode=2
Jul 15 10:57:13.460611 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.460575    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh" event={"ID":"da453fef-0f6a-43b1-a04a-e5fabd061460","Type":"ContainerDied","Data":"2b24e8de1cb7c670e751f63bd3601a3ef2a65293665492f056af3cdf7c740175"}
Jul 15 10:57:13.463046 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.463021    2563 generic.go:356] "Generic (PLEG): container finished" podID="478795bd-bfa8-449e-8ba0-e9d93d900364" containerID="3964554c7fa1aa6966209465bae0c6d608367c1b4b2bc2137970730ab7797d46" exitCode=2
Jul 15 10:57:13.463185 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.463170    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn" event={"ID":"478795bd-bfa8-449e-8ba0-e9d93d900364","Type":"ContainerDied","Data":"3964554c7fa1aa6966209465bae0c6d608367c1b4b2bc2137970730ab7797d46"}
Jul 15 10:57:13.486801 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.486774    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfpsf\" (UniqueName: \"kubernetes.io/projected/bff3e308-3999-4ede-8379-1fdc81f06ae9-kube-api-access-lfpsf\") pod \"error-404-isvc-074d2-predictor-576646885c-d8cfp\" (UID: \"bff3e308-3999-4ede-8379-1fdc81f06ae9\") " pod="kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp"
Jul 15 10:57:13.486897 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.486819    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bff3e308-3999-4ede-8379-1fdc81f06ae9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-074d2-predictor-576646885c-d8cfp\" (UID: \"bff3e308-3999-4ede-8379-1fdc81f06ae9\") " pod="kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp"
Jul 15 10:57:13.486897 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.486848    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bff3e308-3999-4ede-8379-1fdc81f06ae9-proxy-tls\") pod \"error-404-isvc-074d2-predictor-576646885c-d8cfp\" (UID: \"bff3e308-3999-4ede-8379-1fdc81f06ae9\") " pod="kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp"
Jul 15 10:57:13.487798 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.487777    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bff3e308-3999-4ede-8379-1fdc81f06ae9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-074d2-predictor-576646885c-d8cfp\" (UID: \"bff3e308-3999-4ede-8379-1fdc81f06ae9\") " pod="kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp"
Jul 15 10:57:13.489613 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.489589    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bff3e308-3999-4ede-8379-1fdc81f06ae9-proxy-tls\") pod \"error-404-isvc-074d2-predictor-576646885c-d8cfp\" (UID: \"bff3e308-3999-4ede-8379-1fdc81f06ae9\") " pod="kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp"
Jul 15 10:57:13.496587 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.496565    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfpsf\" (UniqueName: \"kubernetes.io/projected/bff3e308-3999-4ede-8379-1fdc81f06ae9-kube-api-access-lfpsf\") pod \"error-404-isvc-074d2-predictor-576646885c-d8cfp\" (UID: \"bff3e308-3999-4ede-8379-1fdc81f06ae9\") " pod="kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp"
Jul 15 10:57:13.530120 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.530093    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp"
Jul 15 10:57:13.537671 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.537653    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp"]
Jul 15 10:57:13.539088 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:57:13.539062    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod582d624f_7f4f_459e_af00_7e5555fe81fd.slice/crio-26496fc170c43758836a0f2ef53b38e45250717a4e60df1af353770336899bf6 WatchSource:0}: Error finding container 26496fc170c43758836a0f2ef53b38e45250717a4e60df1af353770336899bf6: Status 404 returned error can't find the container with id 26496fc170c43758836a0f2ef53b38e45250717a4e60df1af353770336899bf6
Jul 15 10:57:13.668448 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.668423    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp"]
Jul 15 10:57:13.670291 ip-10-0-131-34 kubenswrapper[2563]: W0715 10:57:13.670268    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbff3e308_3999_4ede_8379_1fdc81f06ae9.slice/crio-968fdb5bfd4a87e38028f754cc029e2c3b69e55228d2ebd1a225626d65f78ed2 WatchSource:0}: Error finding container 968fdb5bfd4a87e38028f754cc029e2c3b69e55228d2ebd1a225626d65f78ed2: Status 404 returned error can't find the container with id 968fdb5bfd4a87e38028f754cc029e2c3b69e55228d2ebd1a225626d65f78ed2
Jul 15 10:57:13.674329 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.674306    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh" podUID="da453fef-0f6a-43b1-a04a-e5fabd061460" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 15 10:57:13.934328 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:13.934289    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh" podUID="da453fef-0f6a-43b1-a04a-e5fabd061460" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 15 10:57:14.472860 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:14.472824    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp" event={"ID":"582d624f-7f4f-459e-af00-7e5555fe81fd","Type":"ContainerStarted","Data":"e382a011ab12bf6430b0c41d708b11e64814968f0f23d9232560420692543b0e"}
Jul 15 10:57:14.473329 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:14.472891    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp" event={"ID":"582d624f-7f4f-459e-af00-7e5555fe81fd","Type":"ContainerStarted","Data":"e27f762693c1f319410f9196225ea43be705dd5f1a34c41e98d348d8b1f23867"}
Jul 15 10:57:14.473329 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:14.472924    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp" event={"ID":"582d624f-7f4f-459e-af00-7e5555fe81fd","Type":"ContainerStarted","Data":"26496fc170c43758836a0f2ef53b38e45250717a4e60df1af353770336899bf6"}
Jul 15 10:57:14.473329 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:14.472961    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp"
Jul 15 10:57:14.474907 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:14.474885    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp" event={"ID":"bff3e308-3999-4ede-8379-1fdc81f06ae9","Type":"ContainerStarted","Data":"1b6c20c929c4d41ea66d2bb5752ca565b4862a00655122f4e7f0535b8043475f"}
Jul 15 10:57:14.474982 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:14.474936    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp" event={"ID":"bff3e308-3999-4ede-8379-1fdc81f06ae9","Type":"ContainerStarted","Data":"c3b1c3650fcb4280c8d319f20ee146eca936facea72da0e0bc3e46436d834908"}
Jul 15 10:57:14.474982 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:14.474962    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp" event={"ID":"bff3e308-3999-4ede-8379-1fdc81f06ae9","Type":"ContainerStarted","Data":"968fdb5bfd4a87e38028f754cc029e2c3b69e55228d2ebd1a225626d65f78ed2"}
Jul 15 10:57:14.475048 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:14.474984    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp"
Jul 15 10:57:14.475048 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:14.475022    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp"
Jul 15 10:57:14.476346 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:14.476322    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp" podUID="bff3e308-3999-4ede-8379-1fdc81f06ae9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 15 10:57:14.494660 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:14.494616    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp" podStartSLOduration=1.494603567 podStartE2EDuration="1.494603567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:57:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:57:14.493092184 +0000 UTC m=+653.322607590" watchObservedRunningTime="2026-07-15 10:57:14.494603567 +0000 UTC m=+653.324118970"
Jul 15 10:57:14.510360 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:14.510320    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp" podStartSLOduration=1.510309092 podStartE2EDuration="1.510309092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:57:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:57:14.50900385 +0000 UTC m=+653.338519255" watchObservedRunningTime="2026-07-15 10:57:14.510309092 +0000 UTC m=+653.339824495"
Jul 15 10:57:14.939361 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:14.939267    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb" podUID="0d952e58-6e36-4634-8acd-fdc7dd140fc3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Jul 15 10:57:14.944828 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:14.944797    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb" podUID="0d952e58-6e36-4634-8acd-fdc7dd140fc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 15 10:57:15.008261 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:15.008215    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn" podUID="478795bd-bfa8-449e-8ba0-e9d93d900364" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Jul 15 10:57:15.480313 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:15.480274    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp"
Jul 15 10:57:15.480313 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:15.480265    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp" podUID="bff3e308-3999-4ede-8379-1fdc81f06ae9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 15 10:57:15.481397 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:15.481373    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp" podUID="582d624f-7f4f-459e-af00-7e5555fe81fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 15 10:57:16.489473 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:16.489432    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp" podUID="582d624f-7f4f-459e-af00-7e5555fe81fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 15 10:57:16.888991 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:16.888967    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn"
Jul 15 10:57:16.914590 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:16.914561    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/478795bd-bfa8-449e-8ba0-e9d93d900364-proxy-tls\") pod \"478795bd-bfa8-449e-8ba0-e9d93d900364\" (UID: \"478795bd-bfa8-449e-8ba0-e9d93d900364\") "
Jul 15 10:57:16.914711 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:16.914679    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnbth\" (UniqueName: \"kubernetes.io/projected/478795bd-bfa8-449e-8ba0-e9d93d900364-kube-api-access-gnbth\") pod \"478795bd-bfa8-449e-8ba0-e9d93d900364\" (UID: \"478795bd-bfa8-449e-8ba0-e9d93d900364\") "
Jul 15 10:57:16.914767 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:16.914751    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/478795bd-bfa8-449e-8ba0-e9d93d900364-kserve-provision-location\") pod \"478795bd-bfa8-449e-8ba0-e9d93d900364\" (UID: \"478795bd-bfa8-449e-8ba0-e9d93d900364\") "
Jul 15 10:57:16.914825 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:16.914793    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/478795bd-bfa8-449e-8ba0-e9d93d900364-kube-rbac-proxy-sar-config\") pod \"478795bd-bfa8-449e-8ba0-e9d93d900364\" (UID: \"478795bd-bfa8-449e-8ba0-e9d93d900364\") "
Jul 15 10:57:16.915172 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:16.915124    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/478795bd-bfa8-449e-8ba0-e9d93d900364-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "478795bd-bfa8-449e-8ba0-e9d93d900364" (UID: "478795bd-bfa8-449e-8ba0-e9d93d900364"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:57:16.915275 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:16.915222    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/478795bd-bfa8-449e-8ba0-e9d93d900364-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "478795bd-bfa8-449e-8ba0-e9d93d900364" (UID: "478795bd-bfa8-449e-8ba0-e9d93d900364"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:57:16.916652 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:16.916629    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/478795bd-bfa8-449e-8ba0-e9d93d900364-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "478795bd-bfa8-449e-8ba0-e9d93d900364" (UID: "478795bd-bfa8-449e-8ba0-e9d93d900364"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 10:57:16.916756 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:16.916668    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/478795bd-bfa8-449e-8ba0-e9d93d900364-kube-api-access-gnbth" (OuterVolumeSpecName: "kube-api-access-gnbth") pod "478795bd-bfa8-449e-8ba0-e9d93d900364" (UID: "478795bd-bfa8-449e-8ba0-e9d93d900364"). InnerVolumeSpecName "kube-api-access-gnbth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 10:57:17.015686 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:17.015596    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/478795bd-bfa8-449e-8ba0-e9d93d900364-cabundle-cert\") pod \"478795bd-bfa8-449e-8ba0-e9d93d900364\" (UID: \"478795bd-bfa8-449e-8ba0-e9d93d900364\") "
Jul 15 10:57:17.015880 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:17.015863    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/478795bd-bfa8-449e-8ba0-e9d93d900364-proxy-tls\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:57:17.015946 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:17.015887    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnbth\" (UniqueName: \"kubernetes.io/projected/478795bd-bfa8-449e-8ba0-e9d93d900364-kube-api-access-gnbth\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:57:17.015946 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:17.015902    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/478795bd-bfa8-449e-8ba0-e9d93d900364-kserve-provision-location\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:57:17.015946 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:17.015915    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/478795bd-bfa8-449e-8ba0-e9d93d900364-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:57:17.015946 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:17.015915    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/478795bd-bfa8-449e-8ba0-e9d93d900364-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "478795bd-bfa8-449e-8ba0-e9d93d900364" (UID: "478795bd-bfa8-449e-8ba0-e9d93d900364"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:57:17.116934 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:17.116896    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/478795bd-bfa8-449e-8ba0-e9d93d900364-cabundle-cert\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:57:17.491779 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:17.491743    2563 generic.go:356] "Generic (PLEG): container finished" podID="478795bd-bfa8-449e-8ba0-e9d93d900364" containerID="6bdccddd93b43bd16b99efa59394000efbf2369d74e4ee4e5369f5be5767f2f3" exitCode=0
Jul 15 10:57:17.492227 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:17.491799    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn"
Jul 15 10:57:17.492227 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:17.491830    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn" event={"ID":"478795bd-bfa8-449e-8ba0-e9d93d900364","Type":"ContainerDied","Data":"6bdccddd93b43bd16b99efa59394000efbf2369d74e4ee4e5369f5be5767f2f3"}
Jul 15 10:57:17.492227 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:17.491945    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn" event={"ID":"478795bd-bfa8-449e-8ba0-e9d93d900364","Type":"ContainerDied","Data":"0b208f2bc2233c2f2781bc894b38eedafd07d87ea18bdeae1716a766c6f64247"}
Jul 15 10:57:17.492227 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:17.491983    2563 scope.go:117] "RemoveContainer" containerID="3964554c7fa1aa6966209465bae0c6d608367c1b4b2bc2137970730ab7797d46"
Jul 15 10:57:17.500721 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:17.500705    2563 scope.go:117] "RemoveContainer" containerID="6bdccddd93b43bd16b99efa59394000efbf2369d74e4ee4e5369f5be5767f2f3"
Jul 15 10:57:17.508175 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:17.508157    2563 scope.go:117] "RemoveContainer" containerID="475459a60b80016148d07ab53b820a50988036888948a0d80e7d2b7ae4ad2742"
Jul 15 10:57:17.514775 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:17.514757    2563 scope.go:117] "RemoveContainer" containerID="3964554c7fa1aa6966209465bae0c6d608367c1b4b2bc2137970730ab7797d46"
Jul 15 10:57:17.515033 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:57:17.515017    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3964554c7fa1aa6966209465bae0c6d608367c1b4b2bc2137970730ab7797d46\": container with ID starting with 3964554c7fa1aa6966209465bae0c6d608367c1b4b2bc2137970730ab7797d46 not found: ID does not exist" containerID="3964554c7fa1aa6966209465bae0c6d608367c1b4b2bc2137970730ab7797d46"
Jul 15 10:57:17.515086 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:17.515041    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3964554c7fa1aa6966209465bae0c6d608367c1b4b2bc2137970730ab7797d46"} err="failed to get container status \"3964554c7fa1aa6966209465bae0c6d608367c1b4b2bc2137970730ab7797d46\": rpc error: code = NotFound desc = could not find container \"3964554c7fa1aa6966209465bae0c6d608367c1b4b2bc2137970730ab7797d46\": container with ID starting with 3964554c7fa1aa6966209465bae0c6d608367c1b4b2bc2137970730ab7797d46 not found: ID does not exist"
Jul 15 10:57:17.515086 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:17.515057    2563 scope.go:117] "RemoveContainer" containerID="6bdccddd93b43bd16b99efa59394000efbf2369d74e4ee4e5369f5be5767f2f3"
Jul 15 10:57:17.515317 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:57:17.515289    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bdccddd93b43bd16b99efa59394000efbf2369d74e4ee4e5369f5be5767f2f3\": container with ID starting with 6bdccddd93b43bd16b99efa59394000efbf2369d74e4ee4e5369f5be5767f2f3 not found: ID does not exist" containerID="6bdccddd93b43bd16b99efa59394000efbf2369d74e4ee4e5369f5be5767f2f3"
Jul 15 10:57:17.515415 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:17.515334    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bdccddd93b43bd16b99efa59394000efbf2369d74e4ee4e5369f5be5767f2f3"} err="failed to get container status \"6bdccddd93b43bd16b99efa59394000efbf2369d74e4ee4e5369f5be5767f2f3\": rpc error: code = NotFound desc = could not find container \"6bdccddd93b43bd16b99efa59394000efbf2369d74e4ee4e5369f5be5767f2f3\": container with ID starting with 6bdccddd93b43bd16b99efa59394000efbf2369d74e4ee4e5369f5be5767f2f3 not found: ID does not exist"
Jul 15 10:57:17.515415 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:17.515358    2563 scope.go:117] "RemoveContainer" containerID="475459a60b80016148d07ab53b820a50988036888948a0d80e7d2b7ae4ad2742"
Jul 15 10:57:17.515613 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:57:17.515596    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"475459a60b80016148d07ab53b820a50988036888948a0d80e7d2b7ae4ad2742\": container with ID starting with 475459a60b80016148d07ab53b820a50988036888948a0d80e7d2b7ae4ad2742 not found: ID does not exist" containerID="475459a60b80016148d07ab53b820a50988036888948a0d80e7d2b7ae4ad2742"
Jul 15 10:57:17.515689 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:17.515617    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"475459a60b80016148d07ab53b820a50988036888948a0d80e7d2b7ae4ad2742"} err="failed to get container status \"475459a60b80016148d07ab53b820a50988036888948a0d80e7d2b7ae4ad2742\": rpc error: code = NotFound desc = could not find container \"475459a60b80016148d07ab53b820a50988036888948a0d80e7d2b7ae4ad2742\": container with ID starting with 475459a60b80016148d07ab53b820a50988036888948a0d80e7d2b7ae4ad2742 not found: ID does not exist"
Jul 15 10:57:17.518582 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:17.518563    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn"]
Jul 15 10:57:17.522706 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:17.522689    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn"]
Jul 15 10:57:17.677825 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:17.677790    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="478795bd-bfa8-449e-8ba0-e9d93d900364" path="/var/lib/kubelet/pods/478795bd-bfa8-449e-8ba0-e9d93d900364/volumes"
Jul 15 10:57:18.912291 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:18.912269    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb"
Jul 15 10:57:18.930663 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:18.930643    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkcml\" (UniqueName: \"kubernetes.io/projected/0d952e58-6e36-4634-8acd-fdc7dd140fc3-kube-api-access-qkcml\") pod \"0d952e58-6e36-4634-8acd-fdc7dd140fc3\" (UID: \"0d952e58-6e36-4634-8acd-fdc7dd140fc3\") "
Jul 15 10:57:18.930732 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:18.930672    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d952e58-6e36-4634-8acd-fdc7dd140fc3-kube-rbac-proxy-sar-config\") pod \"0d952e58-6e36-4634-8acd-fdc7dd140fc3\" (UID: \"0d952e58-6e36-4634-8acd-fdc7dd140fc3\") "
Jul 15 10:57:18.930732 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:18.930710    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d952e58-6e36-4634-8acd-fdc7dd140fc3-proxy-tls\") pod \"0d952e58-6e36-4634-8acd-fdc7dd140fc3\" (UID: \"0d952e58-6e36-4634-8acd-fdc7dd140fc3\") "
Jul 15 10:57:18.930796 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:18.930742    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d952e58-6e36-4634-8acd-fdc7dd140fc3-kserve-provision-location\") pod \"0d952e58-6e36-4634-8acd-fdc7dd140fc3\" (UID: \"0d952e58-6e36-4634-8acd-fdc7dd140fc3\") "
Jul 15 10:57:18.930796 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:18.930760    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d952e58-6e36-4634-8acd-fdc7dd140fc3-cabundle-cert\") pod \"0d952e58-6e36-4634-8acd-fdc7dd140fc3\" (UID: \"0d952e58-6e36-4634-8acd-fdc7dd140fc3\") "
Jul 15 10:57:18.930987 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:18.930960    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d952e58-6e36-4634-8acd-fdc7dd140fc3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0d952e58-6e36-4634-8acd-fdc7dd140fc3" (UID: "0d952e58-6e36-4634-8acd-fdc7dd140fc3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:57:18.931093 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:18.931066    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d952e58-6e36-4634-8acd-fdc7dd140fc3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0d952e58-6e36-4634-8acd-fdc7dd140fc3" (UID: "0d952e58-6e36-4634-8acd-fdc7dd140fc3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:57:18.931225 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:18.931181    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d952e58-6e36-4634-8acd-fdc7dd140fc3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0d952e58-6e36-4634-8acd-fdc7dd140fc3" (UID: "0d952e58-6e36-4634-8acd-fdc7dd140fc3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:57:18.933046 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:18.932974    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d952e58-6e36-4634-8acd-fdc7dd140fc3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0d952e58-6e36-4634-8acd-fdc7dd140fc3" (UID: "0d952e58-6e36-4634-8acd-fdc7dd140fc3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 10:57:18.933377 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:18.932894    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d952e58-6e36-4634-8acd-fdc7dd140fc3-kube-api-access-qkcml" (OuterVolumeSpecName: "kube-api-access-qkcml") pod "0d952e58-6e36-4634-8acd-fdc7dd140fc3" (UID: "0d952e58-6e36-4634-8acd-fdc7dd140fc3"). InnerVolumeSpecName "kube-api-access-qkcml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 10:57:18.934077 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:18.934043    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh" podUID="da453fef-0f6a-43b1-a04a-e5fabd061460" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 15 10:57:19.032189 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:19.032099    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkcml\" (UniqueName: \"kubernetes.io/projected/0d952e58-6e36-4634-8acd-fdc7dd140fc3-kube-api-access-qkcml\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:57:19.032189 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:19.032124    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d952e58-6e36-4634-8acd-fdc7dd140fc3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:57:19.032189 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:19.032133    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d952e58-6e36-4634-8acd-fdc7dd140fc3-proxy-tls\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:57:19.032189 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:19.032169    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d952e58-6e36-4634-8acd-fdc7dd140fc3-kserve-provision-location\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:57:19.032189 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:19.032181    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d952e58-6e36-4634-8acd-fdc7dd140fc3-cabundle-cert\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:57:19.504923 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:19.504890    2563 generic.go:356] "Generic (PLEG): container finished" podID="0d952e58-6e36-4634-8acd-fdc7dd140fc3" containerID="178050ef87da42773905f493e7b2eafb5fb250775ae3f192e5fea9efc0b0c941" exitCode=0
Jul 15 10:57:19.505093 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:19.504945    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb"
Jul 15 10:57:19.505093 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:19.504942    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb" event={"ID":"0d952e58-6e36-4634-8acd-fdc7dd140fc3","Type":"ContainerDied","Data":"178050ef87da42773905f493e7b2eafb5fb250775ae3f192e5fea9efc0b0c941"}
Jul 15 10:57:19.505093 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:19.505003    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb" event={"ID":"0d952e58-6e36-4634-8acd-fdc7dd140fc3","Type":"ContainerDied","Data":"fbf0918df8d2769093cb2ffcbb13440ee01266301e89ce95c79653241c8eefa1"}
Jul 15 10:57:19.505093 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:19.505029    2563 scope.go:117] "RemoveContainer" containerID="e87708530f6a5f64d691f250d2a1099cdac496591a689e20e635c5ec5fa3d990"
Jul 15 10:57:19.513404 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:19.513387    2563 scope.go:117] "RemoveContainer" containerID="178050ef87da42773905f493e7b2eafb5fb250775ae3f192e5fea9efc0b0c941"
Jul 15 10:57:19.520373 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:19.520357    2563 scope.go:117] "RemoveContainer" containerID="214ad0f6f8b57a505c276f285068921693817cb4e3b4eecad43a8800443fce1e"
Jul 15 10:57:19.527227 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:19.527211    2563 scope.go:117] "RemoveContainer" containerID="e87708530f6a5f64d691f250d2a1099cdac496591a689e20e635c5ec5fa3d990"
Jul 15 10:57:19.527477 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:57:19.527460    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e87708530f6a5f64d691f250d2a1099cdac496591a689e20e635c5ec5fa3d990\": container with ID starting with e87708530f6a5f64d691f250d2a1099cdac496591a689e20e635c5ec5fa3d990 not found: ID does not exist" containerID="e87708530f6a5f64d691f250d2a1099cdac496591a689e20e635c5ec5fa3d990"
Jul 15 10:57:19.527518 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:19.527487    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e87708530f6a5f64d691f250d2a1099cdac496591a689e20e635c5ec5fa3d990"} err="failed to get container status \"e87708530f6a5f64d691f250d2a1099cdac496591a689e20e635c5ec5fa3d990\": rpc error: code = NotFound desc = could not find container \"e87708530f6a5f64d691f250d2a1099cdac496591a689e20e635c5ec5fa3d990\": container with ID starting with e87708530f6a5f64d691f250d2a1099cdac496591a689e20e635c5ec5fa3d990 not found: ID does not exist"
Jul 15 10:57:19.527518 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:19.527503    2563 scope.go:117] "RemoveContainer" containerID="178050ef87da42773905f493e7b2eafb5fb250775ae3f192e5fea9efc0b0c941"
Jul 15 10:57:19.527714 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:57:19.527700    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"178050ef87da42773905f493e7b2eafb5fb250775ae3f192e5fea9efc0b0c941\": container with ID starting with 178050ef87da42773905f493e7b2eafb5fb250775ae3f192e5fea9efc0b0c941 not found: ID does not exist" containerID="178050ef87da42773905f493e7b2eafb5fb250775ae3f192e5fea9efc0b0c941"
Jul 15 10:57:19.527753 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:19.527718    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"178050ef87da42773905f493e7b2eafb5fb250775ae3f192e5fea9efc0b0c941"} err="failed to get container status \"178050ef87da42773905f493e7b2eafb5fb250775ae3f192e5fea9efc0b0c941\": rpc error: code = NotFound desc = could not find container \"178050ef87da42773905f493e7b2eafb5fb250775ae3f192e5fea9efc0b0c941\": container with ID starting with 178050ef87da42773905f493e7b2eafb5fb250775ae3f192e5fea9efc0b0c941 not found: ID does not exist"
Jul 15 10:57:19.527753 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:19.527731    2563 scope.go:117] "RemoveContainer" containerID="214ad0f6f8b57a505c276f285068921693817cb4e3b4eecad43a8800443fce1e"
Jul 15 10:57:19.527970 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:57:19.527953    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"214ad0f6f8b57a505c276f285068921693817cb4e3b4eecad43a8800443fce1e\": container with ID starting with 214ad0f6f8b57a505c276f285068921693817cb4e3b4eecad43a8800443fce1e not found: ID does not exist" containerID="214ad0f6f8b57a505c276f285068921693817cb4e3b4eecad43a8800443fce1e"
Jul 15 10:57:19.528025 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:19.527982    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"214ad0f6f8b57a505c276f285068921693817cb4e3b4eecad43a8800443fce1e"} err="failed to get container status \"214ad0f6f8b57a505c276f285068921693817cb4e3b4eecad43a8800443fce1e\": rpc error: code = NotFound desc = could not find container \"214ad0f6f8b57a505c276f285068921693817cb4e3b4eecad43a8800443fce1e\": container with ID starting with 214ad0f6f8b57a505c276f285068921693817cb4e3b4eecad43a8800443fce1e not found: ID does not exist"
Jul 15 10:57:19.528224 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:19.528125    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb"]
Jul 15 10:57:19.533784 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:19.533767    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb"]
Jul 15 10:57:19.678073 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:19.678040    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d952e58-6e36-4634-8acd-fdc7dd140fc3" path="/var/lib/kubelet/pods/0d952e58-6e36-4634-8acd-fdc7dd140fc3/volumes"
Jul 15 10:57:20.297274 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:20.297245    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg"
Jul 15 10:57:20.484871 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:20.484845    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp"
Jul 15 10:57:20.485333 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:20.485309    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp" podUID="bff3e308-3999-4ede-8379-1fdc81f06ae9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 15 10:57:20.843600 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:20.843576    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh"
Jul 15 10:57:20.947604 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:20.947585    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da453fef-0f6a-43b1-a04a-e5fabd061460-proxy-tls\") pod \"da453fef-0f6a-43b1-a04a-e5fabd061460\" (UID: \"da453fef-0f6a-43b1-a04a-e5fabd061460\") "
Jul 15 10:57:20.947738 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:20.947642    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sng9b\" (UniqueName: \"kubernetes.io/projected/da453fef-0f6a-43b1-a04a-e5fabd061460-kube-api-access-sng9b\") pod \"da453fef-0f6a-43b1-a04a-e5fabd061460\" (UID: \"da453fef-0f6a-43b1-a04a-e5fabd061460\") "
Jul 15 10:57:20.947738 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:20.947662    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da453fef-0f6a-43b1-a04a-e5fabd061460-kube-rbac-proxy-sar-config\") pod \"da453fef-0f6a-43b1-a04a-e5fabd061460\" (UID: \"da453fef-0f6a-43b1-a04a-e5fabd061460\") "
Jul 15 10:57:20.947738 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:20.947683    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da453fef-0f6a-43b1-a04a-e5fabd061460-cabundle-cert\") pod \"da453fef-0f6a-43b1-a04a-e5fabd061460\" (UID: \"da453fef-0f6a-43b1-a04a-e5fabd061460\") "
Jul 15 10:57:20.947738 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:20.947711    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da453fef-0f6a-43b1-a04a-e5fabd061460-kserve-provision-location\") pod \"da453fef-0f6a-43b1-a04a-e5fabd061460\" (UID: \"da453fef-0f6a-43b1-a04a-e5fabd061460\") "
Jul 15 10:57:20.948043 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:20.948013    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da453fef-0f6a-43b1-a04a-e5fabd061460-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "da453fef-0f6a-43b1-a04a-e5fabd061460" (UID: "da453fef-0f6a-43b1-a04a-e5fabd061460"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:57:20.948043 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:20.948036    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da453fef-0f6a-43b1-a04a-e5fabd061460-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "da453fef-0f6a-43b1-a04a-e5fabd061460" (UID: "da453fef-0f6a-43b1-a04a-e5fabd061460"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:57:20.948180 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:20.948019    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da453fef-0f6a-43b1-a04a-e5fabd061460-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "da453fef-0f6a-43b1-a04a-e5fabd061460" (UID: "da453fef-0f6a-43b1-a04a-e5fabd061460"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:57:20.949481 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:20.949456    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da453fef-0f6a-43b1-a04a-e5fabd061460-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "da453fef-0f6a-43b1-a04a-e5fabd061460" (UID: "da453fef-0f6a-43b1-a04a-e5fabd061460"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 10:57:20.949555 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:20.949539    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da453fef-0f6a-43b1-a04a-e5fabd061460-kube-api-access-sng9b" (OuterVolumeSpecName: "kube-api-access-sng9b") pod "da453fef-0f6a-43b1-a04a-e5fabd061460" (UID: "da453fef-0f6a-43b1-a04a-e5fabd061460"). InnerVolumeSpecName "kube-api-access-sng9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 10:57:21.048454 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:21.048421    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sng9b\" (UniqueName: \"kubernetes.io/projected/da453fef-0f6a-43b1-a04a-e5fabd061460-kube-api-access-sng9b\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:57:21.048454 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:21.048449    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da453fef-0f6a-43b1-a04a-e5fabd061460-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:57:21.048615 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:21.048463    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da453fef-0f6a-43b1-a04a-e5fabd061460-cabundle-cert\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:57:21.048615 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:21.048475    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da453fef-0f6a-43b1-a04a-e5fabd061460-kserve-provision-location\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:57:21.048615 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:21.048487    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da453fef-0f6a-43b1-a04a-e5fabd061460-proxy-tls\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 10:57:21.304996 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:21.304920    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9"
Jul 15 10:57:21.491273 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:21.491247    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp"
Jul 15 10:57:21.491702 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:21.491676    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp" podUID="582d624f-7f4f-459e-af00-7e5555fe81fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 15 10:57:21.514983 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:21.514957    2563 generic.go:356] "Generic (PLEG): container finished" podID="da453fef-0f6a-43b1-a04a-e5fabd061460" containerID="58deac8f5693b704c7a4e9fb0e5fc14adc6eb34dfd95ed17ad19532e9d406caf" exitCode=0
Jul 15 10:57:21.515112 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:21.515014    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh"
Jul 15 10:57:21.515112 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:21.515015    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh" event={"ID":"da453fef-0f6a-43b1-a04a-e5fabd061460","Type":"ContainerDied","Data":"58deac8f5693b704c7a4e9fb0e5fc14adc6eb34dfd95ed17ad19532e9d406caf"}
Jul 15 10:57:21.515232 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:21.515168    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh" event={"ID":"da453fef-0f6a-43b1-a04a-e5fabd061460","Type":"ContainerDied","Data":"716d7a916b796d4dd5b46a1cc9ab23f00ebdbaf0878f388fbd12890d4bbc8d83"}
Jul 15 10:57:21.515232 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:21.515198    2563 scope.go:117] "RemoveContainer" containerID="2b24e8de1cb7c670e751f63bd3601a3ef2a65293665492f056af3cdf7c740175"
Jul 15 10:57:21.523893 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:21.523875    2563 scope.go:117] "RemoveContainer" containerID="58deac8f5693b704c7a4e9fb0e5fc14adc6eb34dfd95ed17ad19532e9d406caf"
Jul 15 10:57:21.531879 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:21.531864    2563 scope.go:117] "RemoveContainer" containerID="dba88f6821ea153fbb60afe2d6d067fcf6c533a466b2fd392eb27354c4422547"
Jul 15 10:57:21.537159 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:21.537121    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh"]
Jul 15 10:57:21.539397 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:21.539384    2563 scope.go:117] "RemoveContainer" containerID="2b24e8de1cb7c670e751f63bd3601a3ef2a65293665492f056af3cdf7c740175"
Jul 15 10:57:21.539646 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:57:21.539626    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b24e8de1cb7c670e751f63bd3601a3ef2a65293665492f056af3cdf7c740175\": container with ID starting with 2b24e8de1cb7c670e751f63bd3601a3ef2a65293665492f056af3cdf7c740175 not found: ID does not exist" containerID="2b24e8de1cb7c670e751f63bd3601a3ef2a65293665492f056af3cdf7c740175"
Jul 15 10:57:21.539703 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:21.539654    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b24e8de1cb7c670e751f63bd3601a3ef2a65293665492f056af3cdf7c740175"} err="failed to get container status \"2b24e8de1cb7c670e751f63bd3601a3ef2a65293665492f056af3cdf7c740175\": rpc error: code = NotFound desc = could not find container \"2b24e8de1cb7c670e751f63bd3601a3ef2a65293665492f056af3cdf7c740175\": container with ID starting with 2b24e8de1cb7c670e751f63bd3601a3ef2a65293665492f056af3cdf7c740175 not found: ID does not exist"
Jul 15 10:57:21.539703 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:21.539671    2563 scope.go:117] "RemoveContainer" containerID="58deac8f5693b704c7a4e9fb0e5fc14adc6eb34dfd95ed17ad19532e9d406caf"
Jul 15 10:57:21.539906 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:57:21.539889    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58deac8f5693b704c7a4e9fb0e5fc14adc6eb34dfd95ed17ad19532e9d406caf\": container with ID starting with 58deac8f5693b704c7a4e9fb0e5fc14adc6eb34dfd95ed17ad19532e9d406caf not found: ID does not exist" containerID="58deac8f5693b704c7a4e9fb0e5fc14adc6eb34dfd95ed17ad19532e9d406caf"
Jul 15 10:57:21.539943 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:21.539911    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58deac8f5693b704c7a4e9fb0e5fc14adc6eb34dfd95ed17ad19532e9d406caf"} err="failed to get container status \"58deac8f5693b704c7a4e9fb0e5fc14adc6eb34dfd95ed17ad19532e9d406caf\": rpc error: code = NotFound desc = could not find container \"58deac8f5693b704c7a4e9fb0e5fc14adc6eb34dfd95ed17ad19532e9d406caf\": container with ID starting with 58deac8f5693b704c7a4e9fb0e5fc14adc6eb34dfd95ed17ad19532e9d406caf not found: ID does not exist"
Jul 15 10:57:21.539943 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:21.539925    2563 scope.go:117] "RemoveContainer" containerID="dba88f6821ea153fbb60afe2d6d067fcf6c533a466b2fd392eb27354c4422547"
Jul 15 10:57:21.540174 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:57:21.540155    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dba88f6821ea153fbb60afe2d6d067fcf6c533a466b2fd392eb27354c4422547\": container with ID starting with dba88f6821ea153fbb60afe2d6d067fcf6c533a466b2fd392eb27354c4422547 not found: ID does not exist" containerID="dba88f6821ea153fbb60afe2d6d067fcf6c533a466b2fd392eb27354c4422547"
Jul 15 10:57:21.540218 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:21.540178    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dba88f6821ea153fbb60afe2d6d067fcf6c533a466b2fd392eb27354c4422547"} err="failed to get container status \"dba88f6821ea153fbb60afe2d6d067fcf6c533a466b2fd392eb27354c4422547\": rpc error: code = NotFound desc = could not find container \"dba88f6821ea153fbb60afe2d6d067fcf6c533a466b2fd392eb27354c4422547\": container with ID starting with dba88f6821ea153fbb60afe2d6d067fcf6c533a466b2fd392eb27354c4422547 not found: ID does not exist"
Jul 15 10:57:21.542176 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:21.542157    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh"]
Jul 15 10:57:21.677041 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:21.677010    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da453fef-0f6a-43b1-a04a-e5fabd061460" path="/var/lib/kubelet/pods/da453fef-0f6a-43b1-a04a-e5fabd061460/volumes"
Jul 15 10:57:22.953264 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:57:22.953233    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda453fef_0f6a_43b1_a04a_e5fabd061460.slice\": RecentStats: unable to find data in memory cache]"
Jul 15 10:57:22.953617 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:57:22.953232    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda453fef_0f6a_43b1_a04a_e5fabd061460.slice\": RecentStats: unable to find data in memory cache]"
Jul 15 10:57:30.485476 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:30.485440    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp" podUID="bff3e308-3999-4ede-8379-1fdc81f06ae9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 15 10:57:31.492040 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:31.491999    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp" podUID="582d624f-7f4f-459e-af00-7e5555fe81fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 15 10:57:32.961745 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:57:32.961710    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda453fef_0f6a_43b1_a04a_e5fabd061460.slice\": RecentStats: unable to find data in memory cache]"
Jul 15 10:57:37.980991 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:57:37.980958    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda453fef_0f6a_43b1_a04a_e5fabd061460.slice\": RecentStats: unable to find data in memory cache]"
Jul 15 10:57:40.485353 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:40.485320    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp" podUID="bff3e308-3999-4ede-8379-1fdc81f06ae9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 15 10:57:41.492255 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:41.492217    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp" podUID="582d624f-7f4f-459e-af00-7e5555fe81fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 15 10:57:42.993470 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:57:42.993438    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda453fef_0f6a_43b1_a04a_e5fabd061460.slice\": RecentStats: unable to find data in memory cache]"
Jul 15 10:57:50.486076 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:50.486034    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp" podUID="bff3e308-3999-4ede-8379-1fdc81f06ae9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 15 10:57:51.492538 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:57:51.492499    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp" podUID="582d624f-7f4f-459e-af00-7e5555fe81fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 15 10:57:52.964637 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:57:52.964606    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda453fef_0f6a_43b1_a04a_e5fabd061460.slice\": RecentStats: unable to find data in memory cache]"
Jul 15 10:57:53.001391 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:57:53.001365    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda453fef_0f6a_43b1_a04a_e5fabd061460.slice\": RecentStats: unable to find data in memory cache]"
Jul 15 10:58:00.486283 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:58:00.486250    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp"
Jul 15 10:58:01.492285 ip-10-0-131-34 kubenswrapper[2563]: I0715 10:58:01.492257    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp"
Jul 15 10:58:03.036471 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:58:03.036436    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda453fef_0f6a_43b1_a04a_e5fabd061460.slice\": RecentStats: unable to find data in memory cache]"
Jul 15 10:58:07.937862 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:58:07.937827    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda453fef_0f6a_43b1_a04a_e5fabd061460.slice\": RecentStats: unable to find data in memory cache]"
Jul 15 10:58:13.071173 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:58:13.071132    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda453fef_0f6a_43b1_a04a_e5fabd061460.slice\": RecentStats: unable to find data in memory cache]"
Jul 15 10:58:13.756150 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:58:13.756120    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda453fef_0f6a_43b1_a04a_e5fabd061460.slice\": RecentStats: unable to find data in memory cache]"
Jul 15 10:58:13.756743 ip-10-0-131-34 kubenswrapper[2563]: E0715 10:58:13.756720    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda453fef_0f6a_43b1_a04a_e5fabd061460.slice\": RecentStats: unable to find data in memory cache]"
Jul 15 11:01:21.568455 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:01:21.568380    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk49n_beb68899-ffb3-436f-8e17-43597aad4030/ovn-acl-logging/0.log"
Jul 15 11:01:21.569667 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:01:21.569645    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk49n_beb68899-ffb3-436f-8e17-43597aad4030/ovn-acl-logging/0.log"
Jul 15 11:05:47.895428 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:47.895391    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9"]
Jul 15 11:05:47.895998 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:47.895614    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9" podUID="c456ffb4-785e-468a-9715-dd367745b6af" containerName="kserve-container" containerID="cri-o://d4dd31a414a9293b8576f79468cbb2e3c9243af0672f8a49ac96f81091a54a1c" gracePeriod=30
Jul 15 11:05:47.895998 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:47.895651    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9" podUID="c456ffb4-785e-468a-9715-dd367745b6af" containerName="kube-rbac-proxy" containerID="cri-o://0d0ca43c47e66a865b1d29a4a7ee5fe77dd69c9862fc7c60a961bf4a6f27a363" gracePeriod=30
Jul 15 11:05:47.983063 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:47.983028    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj"]
Jul 15 11:05:47.983399 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:47.983383    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d952e58-6e36-4634-8acd-fdc7dd140fc3" containerName="kube-rbac-proxy"
Jul 15 11:05:47.983470 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:47.983400    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d952e58-6e36-4634-8acd-fdc7dd140fc3" containerName="kube-rbac-proxy"
Jul 15 11:05:47.983470 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:47.983410    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da453fef-0f6a-43b1-a04a-e5fabd061460" containerName="storage-initializer"
Jul 15 11:05:47.983470 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:47.983415    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="da453fef-0f6a-43b1-a04a-e5fabd061460" containerName="storage-initializer"
Jul 15 11:05:47.983470 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:47.983423    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da453fef-0f6a-43b1-a04a-e5fabd061460" containerName="kube-rbac-proxy"
Jul 15 11:05:47.983470 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:47.983428    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="da453fef-0f6a-43b1-a04a-e5fabd061460" containerName="kube-rbac-proxy"
Jul 15 11:05:47.983470 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:47.983438    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="478795bd-bfa8-449e-8ba0-e9d93d900364" containerName="kserve-container"
Jul 15 11:05:47.983470 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:47.983443    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="478795bd-bfa8-449e-8ba0-e9d93d900364" containerName="kserve-container"
Jul 15 11:05:47.983470 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:47.983451    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d952e58-6e36-4634-8acd-fdc7dd140fc3" containerName="kserve-container"
Jul 15 11:05:47.983470 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:47.983457    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d952e58-6e36-4634-8acd-fdc7dd140fc3" containerName="kserve-container"
Jul 15 11:05:47.983470 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:47.983464    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d952e58-6e36-4634-8acd-fdc7dd140fc3" containerName="storage-initializer"
Jul 15 11:05:47.983470 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:47.983470    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d952e58-6e36-4634-8acd-fdc7dd140fc3" containerName="storage-initializer"
Jul 15 11:05:47.983792 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:47.983478    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da453fef-0f6a-43b1-a04a-e5fabd061460" containerName="kserve-container"
Jul 15 11:05:47.983792 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:47.983484    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="da453fef-0f6a-43b1-a04a-e5fabd061460" containerName="kserve-container"
Jul 15 11:05:47.983792 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:47.983494    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="478795bd-bfa8-449e-8ba0-e9d93d900364" containerName="kube-rbac-proxy"
Jul 15 11:05:47.983792 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:47.983500    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="478795bd-bfa8-449e-8ba0-e9d93d900364" containerName="kube-rbac-proxy"
Jul 15 11:05:47.983792 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:47.983508    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="478795bd-bfa8-449e-8ba0-e9d93d900364" containerName="storage-initializer"
Jul 15 11:05:47.983792 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:47.983513    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="478795bd-bfa8-449e-8ba0-e9d93d900364" containerName="storage-initializer"
Jul 15 11:05:47.983792 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:47.983560    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="478795bd-bfa8-449e-8ba0-e9d93d900364" containerName="kserve-container"
Jul 15 11:05:47.983792 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:47.983566    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d952e58-6e36-4634-8acd-fdc7dd140fc3" containerName="kube-rbac-proxy"
Jul 15 11:05:47.983792 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:47.983573    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="478795bd-bfa8-449e-8ba0-e9d93d900364" containerName="kube-rbac-proxy"
Jul 15 11:05:47.983792 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:47.983579    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="da453fef-0f6a-43b1-a04a-e5fabd061460" containerName="kube-rbac-proxy"
Jul 15 11:05:47.983792 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:47.983585    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d952e58-6e36-4634-8acd-fdc7dd140fc3" containerName="kserve-container"
Jul 15 11:05:47.983792 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:47.983592    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="da453fef-0f6a-43b1-a04a-e5fabd061460" containerName="kserve-container"
Jul 15 11:05:47.986672 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:47.986652    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj"
Jul 15 11:05:47.991836 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:47.991811    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-cf436-kube-rbac-proxy-sar-config\""
Jul 15 11:05:47.991836 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:47.991825    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-cf436-predictor-serving-cert\""
Jul 15 11:05:48.004757 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:48.004734    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj"]
Jul 15 11:05:48.013199 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:48.013176    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg"]
Jul 15 11:05:48.013393 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:48.013372    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg" podUID="ae6f7d2a-dde4-4d1a-8931-9f1434e352f7" containerName="kserve-container" containerID="cri-o://7687d00931774a2b9531d1bd71147f94790b247bf54e2c5ecfa87ec89f2e0d40" gracePeriod=30
Jul 15 11:05:48.013510 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:48.013481    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg" podUID="ae6f7d2a-dde4-4d1a-8931-9f1434e352f7" containerName="kube-rbac-proxy" containerID="cri-o://a57a0c822df0eeac9d6daae008bd498dd64c6934d106490a8e14ad3526bdf464" gracePeriod=30
Jul 15 11:05:48.054675 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:48.054650    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6"]
Jul 15 11:05:48.057963 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:48.057946    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6"
Jul 15 11:05:48.060662 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:48.060642    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-cf436-kube-rbac-proxy-sar-config\""
Jul 15 11:05:48.060977 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:48.060962    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-cf436-predictor-serving-cert\""
Jul 15 11:05:48.070721 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:48.070702    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6"]
Jul 15 11:05:48.083574 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:48.083555    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv9dc\" (UniqueName: \"kubernetes.io/projected/4e971415-7cf0-42c0-88b8-0caac82999b6-kube-api-access-tv9dc\") pod \"success-200-isvc-cf436-predictor-69cbcd7974-z8klj\" (UID: \"4e971415-7cf0-42c0-88b8-0caac82999b6\") " pod="kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj"
Jul 15 11:05:48.083663 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:48.083594    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e971415-7cf0-42c0-88b8-0caac82999b6-proxy-tls\") pod \"success-200-isvc-cf436-predictor-69cbcd7974-z8klj\" (UID: \"4e971415-7cf0-42c0-88b8-0caac82999b6\") " pod="kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj"
Jul 15 11:05:48.083727 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:48.083666    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e971415-7cf0-42c0-88b8-0caac82999b6-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-cf436-predictor-69cbcd7974-z8klj\" (UID: \"4e971415-7cf0-42c0-88b8-0caac82999b6\") " pod="kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj"
Jul 15 11:05:48.184324 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:48.184241    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f62dc8da-e540-4422-af99-61e6d8791537-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-cf436-predictor-777f9bff5f-j9df6\" (UID: \"f62dc8da-e540-4422-af99-61e6d8791537\") " pod="kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6"
Jul 15 11:05:48.184324 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:48.184277    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9mcp\" (UniqueName: \"kubernetes.io/projected/f62dc8da-e540-4422-af99-61e6d8791537-kube-api-access-k9mcp\") pod \"error-404-isvc-cf436-predictor-777f9bff5f-j9df6\" (UID: \"f62dc8da-e540-4422-af99-61e6d8791537\") " pod="kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6"
Jul 15 11:05:48.184324 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:48.184301    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e971415-7cf0-42c0-88b8-0caac82999b6-proxy-tls\") pod \"success-200-isvc-cf436-predictor-69cbcd7974-z8klj\" (UID: \"4e971415-7cf0-42c0-88b8-0caac82999b6\") " pod="kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj"
Jul 15 11:05:48.184602 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:48.184421    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e971415-7cf0-42c0-88b8-0caac82999b6-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-cf436-predictor-69cbcd7974-z8klj\" (UID: \"4e971415-7cf0-42c0-88b8-0caac82999b6\") " pod="kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj"
Jul 15 11:05:48.184602 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:48.184485    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f62dc8da-e540-4422-af99-61e6d8791537-proxy-tls\") pod \"error-404-isvc-cf436-predictor-777f9bff5f-j9df6\" (UID: \"f62dc8da-e540-4422-af99-61e6d8791537\") " pod="kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6"
Jul 15 11:05:48.184602 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:48.184530    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv9dc\" (UniqueName: \"kubernetes.io/projected/4e971415-7cf0-42c0-88b8-0caac82999b6-kube-api-access-tv9dc\") pod \"success-200-isvc-cf436-predictor-69cbcd7974-z8klj\" (UID: \"4e971415-7cf0-42c0-88b8-0caac82999b6\") " pod="kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj"
Jul 15 11:05:48.185006 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:48.184979    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e971415-7cf0-42c0-88b8-0caac82999b6-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-cf436-predictor-69cbcd7974-z8klj\" (UID: \"4e971415-7cf0-42c0-88b8-0caac82999b6\") " pod="kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj"
Jul 15 11:05:48.186766 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:48.186734    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e971415-7cf0-42c0-88b8-0caac82999b6-proxy-tls\") pod \"success-200-isvc-cf436-predictor-69cbcd7974-z8klj\" (UID: \"4e971415-7cf0-42c0-88b8-0caac82999b6\") " pod="kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj"
Jul 15 11:05:48.193155 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:48.193120    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv9dc\" (UniqueName: \"kubernetes.io/projected/4e971415-7cf0-42c0-88b8-0caac82999b6-kube-api-access-tv9dc\") pod \"success-200-isvc-cf436-predictor-69cbcd7974-z8klj\" (UID: \"4e971415-7cf0-42c0-88b8-0caac82999b6\") " pod="kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj"
Jul 15 11:05:48.285085 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:48.285048    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f62dc8da-e540-4422-af99-61e6d8791537-proxy-tls\") pod \"error-404-isvc-cf436-predictor-777f9bff5f-j9df6\" (UID: \"f62dc8da-e540-4422-af99-61e6d8791537\") " pod="kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6"
Jul 15 11:05:48.285281 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:48.285108    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f62dc8da-e540-4422-af99-61e6d8791537-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-cf436-predictor-777f9bff5f-j9df6\" (UID: \"f62dc8da-e540-4422-af99-61e6d8791537\") " pod="kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6"
Jul 15 11:05:48.285281 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:48.285135    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9mcp\" (UniqueName: \"kubernetes.io/projected/f62dc8da-e540-4422-af99-61e6d8791537-kube-api-access-k9mcp\") pod \"error-404-isvc-cf436-predictor-777f9bff5f-j9df6\" (UID: \"f62dc8da-e540-4422-af99-61e6d8791537\") " pod="kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6"
Jul 15 11:05:48.285766 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:48.285740    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f62dc8da-e540-4422-af99-61e6d8791537-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-cf436-predictor-777f9bff5f-j9df6\" (UID: \"f62dc8da-e540-4422-af99-61e6d8791537\") " pod="kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6"
Jul 15 11:05:48.287508 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:48.287484    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f62dc8da-e540-4422-af99-61e6d8791537-proxy-tls\") pod \"error-404-isvc-cf436-predictor-777f9bff5f-j9df6\" (UID: \"f62dc8da-e540-4422-af99-61e6d8791537\") " pod="kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6"
Jul 15 11:05:48.294444 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:48.294425    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9mcp\" (UniqueName: \"kubernetes.io/projected/f62dc8da-e540-4422-af99-61e6d8791537-kube-api-access-k9mcp\") pod \"error-404-isvc-cf436-predictor-777f9bff5f-j9df6\" (UID: \"f62dc8da-e540-4422-af99-61e6d8791537\") " pod="kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6"
Jul 15 11:05:48.297257 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:48.297241    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj"
Jul 15 11:05:48.368301 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:48.368272    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6"
Jul 15 11:05:48.436227 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:48.436181    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj"]
Jul 15 11:05:48.438923 ip-10-0-131-34 kubenswrapper[2563]: W0715 11:05:48.438891    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e971415_7cf0_42c0_88b8_0caac82999b6.slice/crio-37073868de37c33fb1f6f680a8f2dcf649e5d3a6b8ff24d2ae9d75ada7109d6a WatchSource:0}: Error finding container 37073868de37c33fb1f6f680a8f2dcf649e5d3a6b8ff24d2ae9d75ada7109d6a: Status 404 returned error can't find the container with id 37073868de37c33fb1f6f680a8f2dcf649e5d3a6b8ff24d2ae9d75ada7109d6a
Jul 15 11:05:48.468525 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:48.468501    2563 generic.go:356] "Generic (PLEG): container finished" podID="c456ffb4-785e-468a-9715-dd367745b6af" containerID="0d0ca43c47e66a865b1d29a4a7ee5fe77dd69c9862fc7c60a961bf4a6f27a363" exitCode=2
Jul 15 11:05:48.468635 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:48.468571    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9" event={"ID":"c456ffb4-785e-468a-9715-dd367745b6af","Type":"ContainerDied","Data":"0d0ca43c47e66a865b1d29a4a7ee5fe77dd69c9862fc7c60a961bf4a6f27a363"}
Jul 15 11:05:48.469914 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:48.469890    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj" event={"ID":"4e971415-7cf0-42c0-88b8-0caac82999b6","Type":"ContainerStarted","Data":"37073868de37c33fb1f6f680a8f2dcf649e5d3a6b8ff24d2ae9d75ada7109d6a"}
Jul 15 11:05:48.471719 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:48.471694    2563 generic.go:356] "Generic (PLEG): container finished" podID="ae6f7d2a-dde4-4d1a-8931-9f1434e352f7" containerID="a57a0c822df0eeac9d6daae008bd498dd64c6934d106490a8e14ad3526bdf464" exitCode=2
Jul 15 11:05:48.471719 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:48.471722    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg" event={"ID":"ae6f7d2a-dde4-4d1a-8931-9f1434e352f7","Type":"ContainerDied","Data":"a57a0c822df0eeac9d6daae008bd498dd64c6934d106490a8e14ad3526bdf464"}
Jul 15 11:05:48.512188 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:48.512169    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6"]
Jul 15 11:05:48.514265 ip-10-0-131-34 kubenswrapper[2563]: W0715 11:05:48.514244    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf62dc8da_e540_4422_af99_61e6d8791537.slice/crio-3f222c8e07156c9fbd91debd1bb9b97a5d9b072ffa424921b7aa8c15df7d3f23 WatchSource:0}: Error finding container 3f222c8e07156c9fbd91debd1bb9b97a5d9b072ffa424921b7aa8c15df7d3f23: Status 404 returned error can't find the container with id 3f222c8e07156c9fbd91debd1bb9b97a5d9b072ffa424921b7aa8c15df7d3f23
Jul 15 11:05:49.477893 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:49.477856    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj" event={"ID":"4e971415-7cf0-42c0-88b8-0caac82999b6","Type":"ContainerStarted","Data":"2f9db10d4e4853f4f9ded9509feab42f0a6a6391bd309bb55965a82bae91003f"}
Jul 15 11:05:49.478347 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:49.477928    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj" event={"ID":"4e971415-7cf0-42c0-88b8-0caac82999b6","Type":"ContainerStarted","Data":"9066f5d4a7fed0a918664740ae875306cae8f76658435af1364793e9f1ce32f7"}
Jul 15 11:05:49.478347 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:49.477962    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj"
Jul 15 11:05:49.478347 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:49.477993    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj"
Jul 15 11:05:49.479264 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:49.479210    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj" podUID="4e971415-7cf0-42c0-88b8-0caac82999b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 15 11:05:49.479775 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:49.479748    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6" event={"ID":"f62dc8da-e540-4422-af99-61e6d8791537","Type":"ContainerStarted","Data":"724a80d9a54e6ad6718acf5a0604fa63125bec6b89697f55f91a6d8ba7242188"}
Jul 15 11:05:49.479880 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:49.479793    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6" event={"ID":"f62dc8da-e540-4422-af99-61e6d8791537","Type":"ContainerStarted","Data":"54502f96b4a6f8ab4eb207d9a84c737b6493beefcead932b8d072cf3b04b9466"}
Jul 15 11:05:49.479880 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:49.479814    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6" event={"ID":"f62dc8da-e540-4422-af99-61e6d8791537","Type":"ContainerStarted","Data":"3f222c8e07156c9fbd91debd1bb9b97a5d9b072ffa424921b7aa8c15df7d3f23"}
Jul 15 11:05:49.479880 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:49.479845    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6"
Jul 15 11:05:49.479880 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:49.479882    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6"
Jul 15 11:05:49.480891 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:49.480867    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6" podUID="f62dc8da-e540-4422-af99-61e6d8791537" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 15 11:05:49.498911 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:49.498873    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj" podStartSLOduration=2.498861327 podStartE2EDuration="2.498861327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:05:49.498130584 +0000 UTC m=+1168.327645989" watchObservedRunningTime="2026-07-15 11:05:49.498861327 +0000 UTC m=+1168.328376730"
Jul 15 11:05:49.517925 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:49.517887    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6" podStartSLOduration=1.517876161 podStartE2EDuration="1.517876161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:05:49.516313336 +0000 UTC m=+1168.345828742" watchObservedRunningTime="2026-07-15 11:05:49.517876161 +0000 UTC m=+1168.347391564"
Jul 15 11:05:50.291522 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:50.291476    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg" podUID="ae6f7d2a-dde4-4d1a-8931-9f1434e352f7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 15 11:05:50.296911 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:50.296882    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg" podUID="ae6f7d2a-dde4-4d1a-8931-9f1434e352f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 15 11:05:50.484444 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:50.484402    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6" podUID="f62dc8da-e540-4422-af99-61e6d8791537" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 15 11:05:50.484444 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:50.484426    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj" podUID="4e971415-7cf0-42c0-88b8-0caac82999b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 15 11:05:50.844954 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:50.844932    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9"
Jul 15 11:05:51.008580 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.008548    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c456ffb4-785e-468a-9715-dd367745b6af-kube-rbac-proxy-sar-config\") pod \"c456ffb4-785e-468a-9715-dd367745b6af\" (UID: \"c456ffb4-785e-468a-9715-dd367745b6af\") "
Jul 15 11:05:51.008749 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.008637    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c456ffb4-785e-468a-9715-dd367745b6af-proxy-tls\") pod \"c456ffb4-785e-468a-9715-dd367745b6af\" (UID: \"c456ffb4-785e-468a-9715-dd367745b6af\") "
Jul 15 11:05:51.008749 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.008661    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b58zv\" (UniqueName: \"kubernetes.io/projected/c456ffb4-785e-468a-9715-dd367745b6af-kube-api-access-b58zv\") pod \"c456ffb4-785e-468a-9715-dd367745b6af\" (UID: \"c456ffb4-785e-468a-9715-dd367745b6af\") "
Jul 15 11:05:51.008926 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.008898    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c456ffb4-785e-468a-9715-dd367745b6af-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c456ffb4-785e-468a-9715-dd367745b6af" (UID: "c456ffb4-785e-468a-9715-dd367745b6af"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 11:05:51.010646 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.010620    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c456ffb4-785e-468a-9715-dd367745b6af-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c456ffb4-785e-468a-9715-dd367745b6af" (UID: "c456ffb4-785e-468a-9715-dd367745b6af"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:05:51.010743 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.010650    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c456ffb4-785e-468a-9715-dd367745b6af-kube-api-access-b58zv" (OuterVolumeSpecName: "kube-api-access-b58zv") pod "c456ffb4-785e-468a-9715-dd367745b6af" (UID: "c456ffb4-785e-468a-9715-dd367745b6af"). InnerVolumeSpecName "kube-api-access-b58zv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:05:51.109685 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.109658    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c456ffb4-785e-468a-9715-dd367745b6af-proxy-tls\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:05:51.109685 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.109684    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b58zv\" (UniqueName: \"kubernetes.io/projected/c456ffb4-785e-468a-9715-dd367745b6af-kube-api-access-b58zv\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:05:51.109885 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.109696    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c456ffb4-785e-468a-9715-dd367745b6af-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:05:51.166800 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.166776    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg"
Jul 15 11:05:51.311801 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.311714    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae6f7d2a-dde4-4d1a-8931-9f1434e352f7-proxy-tls\") pod \"ae6f7d2a-dde4-4d1a-8931-9f1434e352f7\" (UID: \"ae6f7d2a-dde4-4d1a-8931-9f1434e352f7\") "
Jul 15 11:05:51.311801 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.311775    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7m9z7\" (UniqueName: \"kubernetes.io/projected/ae6f7d2a-dde4-4d1a-8931-9f1434e352f7-kube-api-access-7m9z7\") pod \"ae6f7d2a-dde4-4d1a-8931-9f1434e352f7\" (UID: \"ae6f7d2a-dde4-4d1a-8931-9f1434e352f7\") "
Jul 15 11:05:51.311997 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.311820    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae6f7d2a-dde4-4d1a-8931-9f1434e352f7-kube-rbac-proxy-sar-config\") pod \"ae6f7d2a-dde4-4d1a-8931-9f1434e352f7\" (UID: \"ae6f7d2a-dde4-4d1a-8931-9f1434e352f7\") "
Jul 15 11:05:51.312232 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.312205    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae6f7d2a-dde4-4d1a-8931-9f1434e352f7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ae6f7d2a-dde4-4d1a-8931-9f1434e352f7" (UID: "ae6f7d2a-dde4-4d1a-8931-9f1434e352f7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 11:05:51.313711 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.313687    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae6f7d2a-dde4-4d1a-8931-9f1434e352f7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ae6f7d2a-dde4-4d1a-8931-9f1434e352f7" (UID: "ae6f7d2a-dde4-4d1a-8931-9f1434e352f7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:05:51.313788 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.313712    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae6f7d2a-dde4-4d1a-8931-9f1434e352f7-kube-api-access-7m9z7" (OuterVolumeSpecName: "kube-api-access-7m9z7") pod "ae6f7d2a-dde4-4d1a-8931-9f1434e352f7" (UID: "ae6f7d2a-dde4-4d1a-8931-9f1434e352f7"). InnerVolumeSpecName "kube-api-access-7m9z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:05:51.413093 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.413066    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m9z7\" (UniqueName: \"kubernetes.io/projected/ae6f7d2a-dde4-4d1a-8931-9f1434e352f7-kube-api-access-7m9z7\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:05:51.413093 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.413089    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae6f7d2a-dde4-4d1a-8931-9f1434e352f7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:05:51.413249 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.413098    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae6f7d2a-dde4-4d1a-8931-9f1434e352f7-proxy-tls\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:05:51.489943 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.489914    2563 generic.go:356] "Generic (PLEG): container finished" podID="ae6f7d2a-dde4-4d1a-8931-9f1434e352f7" containerID="7687d00931774a2b9531d1bd71147f94790b247bf54e2c5ecfa87ec89f2e0d40" exitCode=0
Jul 15 11:05:51.490638 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.489970    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg"
Jul 15 11:05:51.490638 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.489986    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg" event={"ID":"ae6f7d2a-dde4-4d1a-8931-9f1434e352f7","Type":"ContainerDied","Data":"7687d00931774a2b9531d1bd71147f94790b247bf54e2c5ecfa87ec89f2e0d40"}
Jul 15 11:05:51.490638 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.490087    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg" event={"ID":"ae6f7d2a-dde4-4d1a-8931-9f1434e352f7","Type":"ContainerDied","Data":"48a8c520cbebb0042089670be010741d232e5171bce75ededc8f9f885ee2950f"}
Jul 15 11:05:51.490638 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.490115    2563 scope.go:117] "RemoveContainer" containerID="a57a0c822df0eeac9d6daae008bd498dd64c6934d106490a8e14ad3526bdf464"
Jul 15 11:05:51.491581 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.491556    2563 generic.go:356] "Generic (PLEG): container finished" podID="c456ffb4-785e-468a-9715-dd367745b6af" containerID="d4dd31a414a9293b8576f79468cbb2e3c9243af0672f8a49ac96f81091a54a1c" exitCode=0
Jul 15 11:05:51.492245 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.491605    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9"
Jul 15 11:05:51.492245 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.491646    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9" event={"ID":"c456ffb4-785e-468a-9715-dd367745b6af","Type":"ContainerDied","Data":"d4dd31a414a9293b8576f79468cbb2e3c9243af0672f8a49ac96f81091a54a1c"}
Jul 15 11:05:51.492245 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.491707    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9" event={"ID":"c456ffb4-785e-468a-9715-dd367745b6af","Type":"ContainerDied","Data":"9e01f44bf8af27bec12adc813fd307df7a2d573f3ee14e751784888f3b15a6a0"}
Jul 15 11:05:51.499305 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.499176    2563 scope.go:117] "RemoveContainer" containerID="7687d00931774a2b9531d1bd71147f94790b247bf54e2c5ecfa87ec89f2e0d40"
Jul 15 11:05:51.507254 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.507231    2563 scope.go:117] "RemoveContainer" containerID="a57a0c822df0eeac9d6daae008bd498dd64c6934d106490a8e14ad3526bdf464"
Jul 15 11:05:51.507554 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:05:51.507529    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a57a0c822df0eeac9d6daae008bd498dd64c6934d106490a8e14ad3526bdf464\": container with ID starting with a57a0c822df0eeac9d6daae008bd498dd64c6934d106490a8e14ad3526bdf464 not found: ID does not exist" containerID="a57a0c822df0eeac9d6daae008bd498dd64c6934d106490a8e14ad3526bdf464"
Jul 15 11:05:51.507629 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.507561    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a57a0c822df0eeac9d6daae008bd498dd64c6934d106490a8e14ad3526bdf464"} err="failed to get container status \"a57a0c822df0eeac9d6daae008bd498dd64c6934d106490a8e14ad3526bdf464\": rpc error: code = NotFound desc = could not find container \"a57a0c822df0eeac9d6daae008bd498dd64c6934d106490a8e14ad3526bdf464\": container with ID starting with a57a0c822df0eeac9d6daae008bd498dd64c6934d106490a8e14ad3526bdf464 not found: ID does not exist"
Jul 15 11:05:51.507629 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.507578    2563 scope.go:117] "RemoveContainer" containerID="7687d00931774a2b9531d1bd71147f94790b247bf54e2c5ecfa87ec89f2e0d40"
Jul 15 11:05:51.507853 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:05:51.507828    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7687d00931774a2b9531d1bd71147f94790b247bf54e2c5ecfa87ec89f2e0d40\": container with ID starting with 7687d00931774a2b9531d1bd71147f94790b247bf54e2c5ecfa87ec89f2e0d40 not found: ID does not exist" containerID="7687d00931774a2b9531d1bd71147f94790b247bf54e2c5ecfa87ec89f2e0d40"
Jul 15 11:05:51.507946 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.507850    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7687d00931774a2b9531d1bd71147f94790b247bf54e2c5ecfa87ec89f2e0d40"} err="failed to get container status \"7687d00931774a2b9531d1bd71147f94790b247bf54e2c5ecfa87ec89f2e0d40\": rpc error: code = NotFound desc = could not find container \"7687d00931774a2b9531d1bd71147f94790b247bf54e2c5ecfa87ec89f2e0d40\": container with ID starting with 7687d00931774a2b9531d1bd71147f94790b247bf54e2c5ecfa87ec89f2e0d40 not found: ID does not exist"
Jul 15 11:05:51.507946 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.507870    2563 scope.go:117] "RemoveContainer" containerID="0d0ca43c47e66a865b1d29a4a7ee5fe77dd69c9862fc7c60a961bf4a6f27a363"
Jul 15 11:05:51.513798 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.513775    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg"]
Jul 15 11:05:51.515321 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.515306    2563 scope.go:117] "RemoveContainer" containerID="d4dd31a414a9293b8576f79468cbb2e3c9243af0672f8a49ac96f81091a54a1c"
Jul 15 11:05:51.518667 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.518648    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg"]
Jul 15 11:05:51.522515 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.522497    2563 scope.go:117] "RemoveContainer" containerID="0d0ca43c47e66a865b1d29a4a7ee5fe77dd69c9862fc7c60a961bf4a6f27a363"
Jul 15 11:05:51.522770 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:05:51.522749    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d0ca43c47e66a865b1d29a4a7ee5fe77dd69c9862fc7c60a961bf4a6f27a363\": container with ID starting with 0d0ca43c47e66a865b1d29a4a7ee5fe77dd69c9862fc7c60a961bf4a6f27a363 not found: ID does not exist" containerID="0d0ca43c47e66a865b1d29a4a7ee5fe77dd69c9862fc7c60a961bf4a6f27a363"
Jul 15 11:05:51.522847 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.522775    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d0ca43c47e66a865b1d29a4a7ee5fe77dd69c9862fc7c60a961bf4a6f27a363"} err="failed to get container status \"0d0ca43c47e66a865b1d29a4a7ee5fe77dd69c9862fc7c60a961bf4a6f27a363\": rpc error: code = NotFound desc = could not find container \"0d0ca43c47e66a865b1d29a4a7ee5fe77dd69c9862fc7c60a961bf4a6f27a363\": container with ID starting with 0d0ca43c47e66a865b1d29a4a7ee5fe77dd69c9862fc7c60a961bf4a6f27a363 not found: ID does not exist"
Jul 15 11:05:51.522847 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.522791    2563 scope.go:117] "RemoveContainer" containerID="d4dd31a414a9293b8576f79468cbb2e3c9243af0672f8a49ac96f81091a54a1c"
Jul 15 11:05:51.522996 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:05:51.522983    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4dd31a414a9293b8576f79468cbb2e3c9243af0672f8a49ac96f81091a54a1c\": container with ID starting with d4dd31a414a9293b8576f79468cbb2e3c9243af0672f8a49ac96f81091a54a1c not found: ID does not exist" containerID="d4dd31a414a9293b8576f79468cbb2e3c9243af0672f8a49ac96f81091a54a1c"
Jul 15 11:05:51.523034 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.522999    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4dd31a414a9293b8576f79468cbb2e3c9243af0672f8a49ac96f81091a54a1c"} err="failed to get container status \"d4dd31a414a9293b8576f79468cbb2e3c9243af0672f8a49ac96f81091a54a1c\": rpc error: code = NotFound desc = could not find container \"d4dd31a414a9293b8576f79468cbb2e3c9243af0672f8a49ac96f81091a54a1c\": container with ID starting with d4dd31a414a9293b8576f79468cbb2e3c9243af0672f8a49ac96f81091a54a1c not found: ID does not exist"
Jul 15 11:05:51.531787 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.531768    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9"]
Jul 15 11:05:51.539473 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.539452    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9"]
Jul 15 11:05:51.677622 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.677601    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae6f7d2a-dde4-4d1a-8931-9f1434e352f7" path="/var/lib/kubelet/pods/ae6f7d2a-dde4-4d1a-8931-9f1434e352f7/volumes"
Jul 15 11:05:51.678020 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:51.678007    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c456ffb4-785e-468a-9715-dd367745b6af" path="/var/lib/kubelet/pods/c456ffb4-785e-468a-9715-dd367745b6af/volumes"
Jul 15 11:05:55.488709 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:55.488681    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6"
Jul 15 11:05:55.489288 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:55.489255    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6" podUID="f62dc8da-e540-4422-af99-61e6d8791537" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 15 11:05:55.489664 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:55.489645    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj"
Jul 15 11:05:55.490114 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:05:55.490080    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj" podUID="4e971415-7cf0-42c0-88b8-0caac82999b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 15 11:06:05.489180 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:05.489117    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6" podUID="f62dc8da-e540-4422-af99-61e6d8791537" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 15 11:06:05.490415 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:05.490391    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj" podUID="4e971415-7cf0-42c0-88b8-0caac82999b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 15 11:06:15.489740 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:15.489702    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6" podUID="f62dc8da-e540-4422-af99-61e6d8791537" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 15 11:06:15.490114 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:15.489976    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj" podUID="4e971415-7cf0-42c0-88b8-0caac82999b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 15 11:06:21.592844 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:21.592816    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk49n_beb68899-ffb3-436f-8e17-43597aad4030/ovn-acl-logging/0.log"
Jul 15 11:06:21.595104 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:21.595082    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk49n_beb68899-ffb3-436f-8e17-43597aad4030/ovn-acl-logging/0.log"
Jul 15 11:06:25.489820 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:25.489779    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6" podUID="f62dc8da-e540-4422-af99-61e6d8791537" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 15 11:06:25.490308 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:25.490045    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj" podUID="4e971415-7cf0-42c0-88b8-0caac82999b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 15 11:06:28.145317 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.145280    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp"]
Jul 15 11:06:28.145765 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.145487    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp" podUID="582d624f-7f4f-459e-af00-7e5555fe81fd" containerName="kserve-container" containerID="cri-o://e27f762693c1f319410f9196225ea43be705dd5f1a34c41e98d348d8b1f23867" gracePeriod=30
Jul 15 11:06:28.145765 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.145529    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp" podUID="582d624f-7f4f-459e-af00-7e5555fe81fd" containerName="kube-rbac-proxy" containerID="cri-o://e382a011ab12bf6430b0c41d708b11e64814968f0f23d9232560420692543b0e" gracePeriod=30
Jul 15 11:06:28.155809 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.155680    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc"]
Jul 15 11:06:28.156043 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.156027    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c456ffb4-785e-468a-9715-dd367745b6af" containerName="kserve-container"
Jul 15 11:06:28.156043 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.156043    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c456ffb4-785e-468a-9715-dd367745b6af" containerName="kserve-container"
Jul 15 11:06:28.156247 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.156055    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae6f7d2a-dde4-4d1a-8931-9f1434e352f7" containerName="kube-rbac-proxy"
Jul 15 11:06:28.156247 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.156060    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae6f7d2a-dde4-4d1a-8931-9f1434e352f7" containerName="kube-rbac-proxy"
Jul 15 11:06:28.156247 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.156070    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae6f7d2a-dde4-4d1a-8931-9f1434e352f7" containerName="kserve-container"
Jul 15 11:06:28.156247 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.156075    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae6f7d2a-dde4-4d1a-8931-9f1434e352f7" containerName="kserve-container"
Jul 15 11:06:28.156247 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.156083    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c456ffb4-785e-468a-9715-dd367745b6af" containerName="kube-rbac-proxy"
Jul 15 11:06:28.156247 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.156088    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c456ffb4-785e-468a-9715-dd367745b6af" containerName="kube-rbac-proxy"
Jul 15 11:06:28.156247 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.156155    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae6f7d2a-dde4-4d1a-8931-9f1434e352f7" containerName="kserve-container"
Jul 15 11:06:28.156247 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.156168    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c456ffb4-785e-468a-9715-dd367745b6af" containerName="kserve-container"
Jul 15 11:06:28.156247 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.156179    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c456ffb4-785e-468a-9715-dd367745b6af" containerName="kube-rbac-proxy"
Jul 15 11:06:28.156247 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.156189    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae6f7d2a-dde4-4d1a-8931-9f1434e352f7" containerName="kube-rbac-proxy"
Jul 15 11:06:28.160775 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.160755    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc"
Jul 15 11:06:28.163646 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.163628    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-30120-kube-rbac-proxy-sar-config\""
Jul 15 11:06:28.163853 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.163841    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-30120-predictor-serving-cert\""
Jul 15 11:06:28.177492 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.177467    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc"]
Jul 15 11:06:28.216873 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.216840    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89161cbe-9df8-4661-a724-8cef68e2fe35-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-30120-predictor-74997b5b5d-6hgnc\" (UID: \"89161cbe-9df8-4661-a724-8cef68e2fe35\") " pod="kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc"
Jul 15 11:06:28.217043 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.216900    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89161cbe-9df8-4661-a724-8cef68e2fe35-proxy-tls\") pod \"success-200-isvc-30120-predictor-74997b5b5d-6hgnc\" (UID: \"89161cbe-9df8-4661-a724-8cef68e2fe35\") " pod="kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc"
Jul 15 11:06:28.217043 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.216971    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccqt7\" (UniqueName: \"kubernetes.io/projected/89161cbe-9df8-4661-a724-8cef68e2fe35-kube-api-access-ccqt7\") pod \"success-200-isvc-30120-predictor-74997b5b5d-6hgnc\" (UID: \"89161cbe-9df8-4661-a724-8cef68e2fe35\") " pod="kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc"
Jul 15 11:06:28.317660 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.317623    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89161cbe-9df8-4661-a724-8cef68e2fe35-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-30120-predictor-74997b5b5d-6hgnc\" (UID: \"89161cbe-9df8-4661-a724-8cef68e2fe35\") " pod="kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc"
Jul 15 11:06:28.317842 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.317695    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89161cbe-9df8-4661-a724-8cef68e2fe35-proxy-tls\") pod \"success-200-isvc-30120-predictor-74997b5b5d-6hgnc\" (UID: \"89161cbe-9df8-4661-a724-8cef68e2fe35\") " pod="kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc"
Jul 15 11:06:28.317842 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.317723    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccqt7\" (UniqueName: \"kubernetes.io/projected/89161cbe-9df8-4661-a724-8cef68e2fe35-kube-api-access-ccqt7\") pod \"success-200-isvc-30120-predictor-74997b5b5d-6hgnc\" (UID: \"89161cbe-9df8-4661-a724-8cef68e2fe35\") " pod="kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc"
Jul 15 11:06:28.317965 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:06:28.317867    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-30120-predictor-serving-cert: secret "success-200-isvc-30120-predictor-serving-cert" not found
Jul 15 11:06:28.317965 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:06:28.317933    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89161cbe-9df8-4661-a724-8cef68e2fe35-proxy-tls podName:89161cbe-9df8-4661-a724-8cef68e2fe35 nodeName:}" failed. No retries permitted until 2026-07-15 11:06:28.817911689 +0000 UTC m=+1207.647427072 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/89161cbe-9df8-4661-a724-8cef68e2fe35-proxy-tls") pod "success-200-isvc-30120-predictor-74997b5b5d-6hgnc" (UID: "89161cbe-9df8-4661-a724-8cef68e2fe35") : secret "success-200-isvc-30120-predictor-serving-cert" not found
Jul 15 11:06:28.318309 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.318292    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89161cbe-9df8-4661-a724-8cef68e2fe35-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-30120-predictor-74997b5b5d-6hgnc\" (UID: \"89161cbe-9df8-4661-a724-8cef68e2fe35\") " pod="kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc"
Jul 15 11:06:28.325247 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.325221    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6"]
Jul 15 11:06:28.328601 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.328585    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6"
Jul 15 11:06:28.333543 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.333523    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-30120-kube-rbac-proxy-sar-config\""
Jul 15 11:06:28.333762 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.333740    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-30120-predictor-serving-cert\""
Jul 15 11:06:28.337363 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.337341    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccqt7\" (UniqueName: \"kubernetes.io/projected/89161cbe-9df8-4661-a724-8cef68e2fe35-kube-api-access-ccqt7\") pod \"success-200-isvc-30120-predictor-74997b5b5d-6hgnc\" (UID: \"89161cbe-9df8-4661-a724-8cef68e2fe35\") " pod="kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc"
Jul 15 11:06:28.357700 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.357165    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6"]
Jul 15 11:06:28.406104 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.406030    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp"]
Jul 15 11:06:28.406280 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.406255    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp" podUID="bff3e308-3999-4ede-8379-1fdc81f06ae9" containerName="kserve-container" containerID="cri-o://c3b1c3650fcb4280c8d319f20ee146eca936facea72da0e0bc3e46436d834908" gracePeriod=30
Jul 15 11:06:28.406337 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.406280    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp" podUID="bff3e308-3999-4ede-8379-1fdc81f06ae9" containerName="kube-rbac-proxy" containerID="cri-o://1b6c20c929c4d41ea66d2bb5752ca565b4862a00655122f4e7f0535b8043475f" gracePeriod=30
Jul 15 11:06:28.418351 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.418329    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa498659-a6ab-4e10-ace6-0400dbd05f68-proxy-tls\") pod \"error-404-isvc-30120-predictor-6c4f57456c-6h8b6\" (UID: \"fa498659-a6ab-4e10-ace6-0400dbd05f68\") " pod="kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6"
Jul 15 11:06:28.418434 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.418362    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfvtx\" (UniqueName: \"kubernetes.io/projected/fa498659-a6ab-4e10-ace6-0400dbd05f68-kube-api-access-zfvtx\") pod \"error-404-isvc-30120-predictor-6c4f57456c-6h8b6\" (UID: \"fa498659-a6ab-4e10-ace6-0400dbd05f68\") " pod="kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6"
Jul 15 11:06:28.418510 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.418493    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa498659-a6ab-4e10-ace6-0400dbd05f68-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-30120-predictor-6c4f57456c-6h8b6\" (UID: \"fa498659-a6ab-4e10-ace6-0400dbd05f68\") " pod="kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6"
Jul 15 11:06:28.519763 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.519734    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa498659-a6ab-4e10-ace6-0400dbd05f68-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-30120-predictor-6c4f57456c-6h8b6\" (UID: \"fa498659-a6ab-4e10-ace6-0400dbd05f68\") " pod="kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6"
Jul 15 11:06:28.519950 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.519785    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa498659-a6ab-4e10-ace6-0400dbd05f68-proxy-tls\") pod \"error-404-isvc-30120-predictor-6c4f57456c-6h8b6\" (UID: \"fa498659-a6ab-4e10-ace6-0400dbd05f68\") " pod="kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6"
Jul 15 11:06:28.519950 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.519904    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfvtx\" (UniqueName: \"kubernetes.io/projected/fa498659-a6ab-4e10-ace6-0400dbd05f68-kube-api-access-zfvtx\") pod \"error-404-isvc-30120-predictor-6c4f57456c-6h8b6\" (UID: \"fa498659-a6ab-4e10-ace6-0400dbd05f68\") " pod="kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6"
Jul 15 11:06:28.520610 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.520586    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa498659-a6ab-4e10-ace6-0400dbd05f68-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-30120-predictor-6c4f57456c-6h8b6\" (UID: \"fa498659-a6ab-4e10-ace6-0400dbd05f68\") " pod="kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6"
Jul 15 11:06:28.522133 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.522108    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa498659-a6ab-4e10-ace6-0400dbd05f68-proxy-tls\") pod \"error-404-isvc-30120-predictor-6c4f57456c-6h8b6\" (UID: \"fa498659-a6ab-4e10-ace6-0400dbd05f68\") " pod="kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6"
Jul 15 11:06:28.538359 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.538338    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfvtx\" (UniqueName: \"kubernetes.io/projected/fa498659-a6ab-4e10-ace6-0400dbd05f68-kube-api-access-zfvtx\") pod \"error-404-isvc-30120-predictor-6c4f57456c-6h8b6\" (UID: \"fa498659-a6ab-4e10-ace6-0400dbd05f68\") " pod="kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6"
Jul 15 11:06:28.638170 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.638109    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6"
Jul 15 11:06:28.642537 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.642514    2563 generic.go:356] "Generic (PLEG): container finished" podID="bff3e308-3999-4ede-8379-1fdc81f06ae9" containerID="1b6c20c929c4d41ea66d2bb5752ca565b4862a00655122f4e7f0535b8043475f" exitCode=2
Jul 15 11:06:28.642641 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.642589    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp" event={"ID":"bff3e308-3999-4ede-8379-1fdc81f06ae9","Type":"ContainerDied","Data":"1b6c20c929c4d41ea66d2bb5752ca565b4862a00655122f4e7f0535b8043475f"}
Jul 15 11:06:28.644417 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.644399    2563 generic.go:356] "Generic (PLEG): container finished" podID="582d624f-7f4f-459e-af00-7e5555fe81fd" containerID="e382a011ab12bf6430b0c41d708b11e64814968f0f23d9232560420692543b0e" exitCode=2
Jul 15 11:06:28.644517 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.644425    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp" event={"ID":"582d624f-7f4f-459e-af00-7e5555fe81fd","Type":"ContainerDied","Data":"e382a011ab12bf6430b0c41d708b11e64814968f0f23d9232560420692543b0e"}
Jul 15 11:06:28.785963 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.785940    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6"]
Jul 15 11:06:28.788233 ip-10-0-131-34 kubenswrapper[2563]: W0715 11:06:28.788206    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa498659_a6ab_4e10_ace6_0400dbd05f68.slice/crio-6dc9ecef2f013f5fdc26f5d2653164d68f5dd61f133ccee7604b4bcc68d899c9 WatchSource:0}: Error finding container 6dc9ecef2f013f5fdc26f5d2653164d68f5dd61f133ccee7604b4bcc68d899c9: Status 404 returned error can't find the container with id 6dc9ecef2f013f5fdc26f5d2653164d68f5dd61f133ccee7604b4bcc68d899c9
Jul 15 11:06:28.823077 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.823042    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89161cbe-9df8-4661-a724-8cef68e2fe35-proxy-tls\") pod \"success-200-isvc-30120-predictor-74997b5b5d-6hgnc\" (UID: \"89161cbe-9df8-4661-a724-8cef68e2fe35\") " pod="kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc"
Jul 15 11:06:28.825256 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:28.825238    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89161cbe-9df8-4661-a724-8cef68e2fe35-proxy-tls\") pod \"success-200-isvc-30120-predictor-74997b5b5d-6hgnc\" (UID: \"89161cbe-9df8-4661-a724-8cef68e2fe35\") " pod="kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc"
Jul 15 11:06:29.069947 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:29.069862    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc"
Jul 15 11:06:29.226064 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:29.225827    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc"]
Jul 15 11:06:29.228365 ip-10-0-131-34 kubenswrapper[2563]: W0715 11:06:29.228341    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89161cbe_9df8_4661_a724_8cef68e2fe35.slice/crio-85798561e0a8b42a0c00a598cd42fa5a061a2ff54fbc44324115bf15dec1c34c WatchSource:0}: Error finding container 85798561e0a8b42a0c00a598cd42fa5a061a2ff54fbc44324115bf15dec1c34c: Status 404 returned error can't find the container with id 85798561e0a8b42a0c00a598cd42fa5a061a2ff54fbc44324115bf15dec1c34c
Jul 15 11:06:29.649855 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:29.649769    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc" event={"ID":"89161cbe-9df8-4661-a724-8cef68e2fe35","Type":"ContainerStarted","Data":"2e6894af55706d77ee129f25dabea9f1326cfc35f1b2e37594b12b0a5807243f"}
Jul 15 11:06:29.649855 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:29.649824    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc" event={"ID":"89161cbe-9df8-4661-a724-8cef68e2fe35","Type":"ContainerStarted","Data":"127f444a6216211469a47bd9a20f1d8c6cc4ad0ec4f30aa421734e11c0134a03"}
Jul 15 11:06:29.649855 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:29.649845    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc" event={"ID":"89161cbe-9df8-4661-a724-8cef68e2fe35","Type":"ContainerStarted","Data":"85798561e0a8b42a0c00a598cd42fa5a061a2ff54fbc44324115bf15dec1c34c"}
Jul 15 11:06:29.650116 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:29.649940    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc"
Jul 15 11:06:29.652971 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:29.652208    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6" event={"ID":"fa498659-a6ab-4e10-ace6-0400dbd05f68","Type":"ContainerStarted","Data":"56e43b0f5d06ab237e4966f426e2b06d6ef98f78e5da782fcd4350cd22b02a22"}
Jul 15 11:06:29.652971 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:29.652271    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6" event={"ID":"fa498659-a6ab-4e10-ace6-0400dbd05f68","Type":"ContainerStarted","Data":"9782b20ac2c597226ff37b84fad82ea49b9aada866846be55a3135acea420a82"}
Jul 15 11:06:29.652971 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:29.652303    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6" event={"ID":"fa498659-a6ab-4e10-ace6-0400dbd05f68","Type":"ContainerStarted","Data":"6dc9ecef2f013f5fdc26f5d2653164d68f5dd61f133ccee7604b4bcc68d899c9"}
Jul 15 11:06:29.652971 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:29.652870    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6"
Jul 15 11:06:29.652971 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:29.652922    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6"
Jul 15 11:06:29.654858 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:29.654634    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6" podUID="fa498659-a6ab-4e10-ace6-0400dbd05f68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 15 11:06:29.681168 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:29.679782    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc" podStartSLOduration=1.679762967 podStartE2EDuration="1.679762967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:06:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:06:29.679171878 +0000 UTC m=+1208.508687286" watchObservedRunningTime="2026-07-15 11:06:29.679762967 +0000 UTC m=+1208.509278373"
Jul 15 11:06:29.717572 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:29.717512    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6" podStartSLOduration=1.7174950679999998 podStartE2EDuration="1.717495068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:06:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:06:29.711234042 +0000 UTC m=+1208.540749448" watchObservedRunningTime="2026-07-15 11:06:29.717495068 +0000 UTC m=+1208.547010473"
Jul 15 11:06:30.481117 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:30.481070    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp" podUID="bff3e308-3999-4ede-8379-1fdc81f06ae9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 15 11:06:30.485647 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:30.485613    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp" podUID="bff3e308-3999-4ede-8379-1fdc81f06ae9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 15 11:06:30.659837 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:30.659773    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6" podUID="fa498659-a6ab-4e10-ace6-0400dbd05f68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 15 11:06:30.660427 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:30.660400    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc"
Jul 15 11:06:30.661531 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:30.661505    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc" podUID="89161cbe-9df8-4661-a724-8cef68e2fe35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 15 11:06:31.487203 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:31.487155    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp" podUID="582d624f-7f4f-459e-af00-7e5555fe81fd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 15 11:06:31.492549 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:31.492479    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp" podUID="582d624f-7f4f-459e-af00-7e5555fe81fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 15 11:06:31.611263 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:31.611205    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp"
Jul 15 11:06:31.667800 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:31.667761    2563 generic.go:356] "Generic (PLEG): container finished" podID="582d624f-7f4f-459e-af00-7e5555fe81fd" containerID="e27f762693c1f319410f9196225ea43be705dd5f1a34c41e98d348d8b1f23867" exitCode=0
Jul 15 11:06:31.667958 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:31.667812    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp"
Jul 15 11:06:31.667958 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:31.667884    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp" event={"ID":"582d624f-7f4f-459e-af00-7e5555fe81fd","Type":"ContainerDied","Data":"e27f762693c1f319410f9196225ea43be705dd5f1a34c41e98d348d8b1f23867"}
Jul 15 11:06:31.667958 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:31.667949    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp" event={"ID":"582d624f-7f4f-459e-af00-7e5555fe81fd","Type":"ContainerDied","Data":"26496fc170c43758836a0f2ef53b38e45250717a4e60df1af353770336899bf6"}
Jul 15 11:06:31.668079 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:31.667984    2563 scope.go:117] "RemoveContainer" containerID="e382a011ab12bf6430b0c41d708b11e64814968f0f23d9232560420692543b0e"
Jul 15 11:06:31.668417 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:31.668370    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6" podUID="fa498659-a6ab-4e10-ace6-0400dbd05f68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 15 11:06:31.668631 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:31.668610    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc" podUID="89161cbe-9df8-4661-a724-8cef68e2fe35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 15 11:06:31.675752 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:31.675731    2563 scope.go:117] "RemoveContainer" containerID="e27f762693c1f319410f9196225ea43be705dd5f1a34c41e98d348d8b1f23867"
Jul 15 11:06:31.683614 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:31.683598    2563 scope.go:117] "RemoveContainer" containerID="e382a011ab12bf6430b0c41d708b11e64814968f0f23d9232560420692543b0e"
Jul 15 11:06:31.683835 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:06:31.683818    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e382a011ab12bf6430b0c41d708b11e64814968f0f23d9232560420692543b0e\": container with ID starting with e382a011ab12bf6430b0c41d708b11e64814968f0f23d9232560420692543b0e not found: ID does not exist" containerID="e382a011ab12bf6430b0c41d708b11e64814968f0f23d9232560420692543b0e"
Jul 15 11:06:31.683888 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:31.683843    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e382a011ab12bf6430b0c41d708b11e64814968f0f23d9232560420692543b0e"} err="failed to get container status \"e382a011ab12bf6430b0c41d708b11e64814968f0f23d9232560420692543b0e\": rpc error: code = NotFound desc = could not find container \"e382a011ab12bf6430b0c41d708b11e64814968f0f23d9232560420692543b0e\": container with ID starting with e382a011ab12bf6430b0c41d708b11e64814968f0f23d9232560420692543b0e not found: ID does not exist"
Jul 15 11:06:31.683888 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:31.683859    2563 scope.go:117] "RemoveContainer" containerID="e27f762693c1f319410f9196225ea43be705dd5f1a34c41e98d348d8b1f23867"
Jul 15 11:06:31.684123 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:06:31.684107    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e27f762693c1f319410f9196225ea43be705dd5f1a34c41e98d348d8b1f23867\": container with ID starting with e27f762693c1f319410f9196225ea43be705dd5f1a34c41e98d348d8b1f23867 not found: ID does not exist" containerID="e27f762693c1f319410f9196225ea43be705dd5f1a34c41e98d348d8b1f23867"
Jul 15 11:06:31.684182 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:31.684130    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e27f762693c1f319410f9196225ea43be705dd5f1a34c41e98d348d8b1f23867"} err="failed to get container status \"e27f762693c1f319410f9196225ea43be705dd5f1a34c41e98d348d8b1f23867\": rpc error: code = NotFound desc = could not find container \"e27f762693c1f319410f9196225ea43be705dd5f1a34c41e98d348d8b1f23867\": container with ID starting with e27f762693c1f319410f9196225ea43be705dd5f1a34c41e98d348d8b1f23867 not found: ID does not exist"
Jul 15 11:06:31.749916 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:31.749889    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dl7lk\" (UniqueName: \"kubernetes.io/projected/582d624f-7f4f-459e-af00-7e5555fe81fd-kube-api-access-dl7lk\") pod \"582d624f-7f4f-459e-af00-7e5555fe81fd\" (UID: \"582d624f-7f4f-459e-af00-7e5555fe81fd\") "
Jul 15 11:06:31.750048 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:31.749924    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/582d624f-7f4f-459e-af00-7e5555fe81fd-proxy-tls\") pod \"582d624f-7f4f-459e-af00-7e5555fe81fd\" (UID: \"582d624f-7f4f-459e-af00-7e5555fe81fd\") "
Jul 15 11:06:31.750103 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:31.750059    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/582d624f-7f4f-459e-af00-7e5555fe81fd-kube-rbac-proxy-sar-config\") pod \"582d624f-7f4f-459e-af00-7e5555fe81fd\" (UID: \"582d624f-7f4f-459e-af00-7e5555fe81fd\") "
Jul 15 11:06:31.751041 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:31.750441    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/582d624f-7f4f-459e-af00-7e5555fe81fd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "582d624f-7f4f-459e-af00-7e5555fe81fd" (UID: "582d624f-7f4f-459e-af00-7e5555fe81fd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 11:06:31.759874 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:31.759581    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/582d624f-7f4f-459e-af00-7e5555fe81fd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "582d624f-7f4f-459e-af00-7e5555fe81fd" (UID: "582d624f-7f4f-459e-af00-7e5555fe81fd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:06:31.759874 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:31.759663    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/582d624f-7f4f-459e-af00-7e5555fe81fd-kube-api-access-dl7lk" (OuterVolumeSpecName: "kube-api-access-dl7lk") pod "582d624f-7f4f-459e-af00-7e5555fe81fd" (UID: "582d624f-7f4f-459e-af00-7e5555fe81fd"). InnerVolumeSpecName "kube-api-access-dl7lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:06:31.850890 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:31.850856    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/582d624f-7f4f-459e-af00-7e5555fe81fd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:06:31.850890 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:31.850888    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl7lk\" (UniqueName: \"kubernetes.io/projected/582d624f-7f4f-459e-af00-7e5555fe81fd-kube-api-access-dl7lk\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:06:31.851169 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:31.850903    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/582d624f-7f4f-459e-af00-7e5555fe81fd-proxy-tls\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:06:31.995708 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:31.995679    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp"]
Jul 15 11:06:32.002386 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:32.002363    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp"]
Jul 15 11:06:32.246892 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:32.246869    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp"
Jul 15 11:06:32.355013 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:32.354976    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfpsf\" (UniqueName: \"kubernetes.io/projected/bff3e308-3999-4ede-8379-1fdc81f06ae9-kube-api-access-lfpsf\") pod \"bff3e308-3999-4ede-8379-1fdc81f06ae9\" (UID: \"bff3e308-3999-4ede-8379-1fdc81f06ae9\") "
Jul 15 11:06:32.355215 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:32.355062    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bff3e308-3999-4ede-8379-1fdc81f06ae9-proxy-tls\") pod \"bff3e308-3999-4ede-8379-1fdc81f06ae9\" (UID: \"bff3e308-3999-4ede-8379-1fdc81f06ae9\") "
Jul 15 11:06:32.355215 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:32.355152    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bff3e308-3999-4ede-8379-1fdc81f06ae9-kube-rbac-proxy-sar-config\") pod \"bff3e308-3999-4ede-8379-1fdc81f06ae9\" (UID: \"bff3e308-3999-4ede-8379-1fdc81f06ae9\") "
Jul 15 11:06:32.355778 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:32.355735    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bff3e308-3999-4ede-8379-1fdc81f06ae9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bff3e308-3999-4ede-8379-1fdc81f06ae9" (UID: "bff3e308-3999-4ede-8379-1fdc81f06ae9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 11:06:32.357196 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:32.357171    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bff3e308-3999-4ede-8379-1fdc81f06ae9-kube-api-access-lfpsf" (OuterVolumeSpecName: "kube-api-access-lfpsf") pod "bff3e308-3999-4ede-8379-1fdc81f06ae9" (UID: "bff3e308-3999-4ede-8379-1fdc81f06ae9"). InnerVolumeSpecName "kube-api-access-lfpsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:06:32.357589 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:32.357572    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bff3e308-3999-4ede-8379-1fdc81f06ae9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bff3e308-3999-4ede-8379-1fdc81f06ae9" (UID: "bff3e308-3999-4ede-8379-1fdc81f06ae9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:06:32.456624 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:32.456578    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfpsf\" (UniqueName: \"kubernetes.io/projected/bff3e308-3999-4ede-8379-1fdc81f06ae9-kube-api-access-lfpsf\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:06:32.456624 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:32.456617    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bff3e308-3999-4ede-8379-1fdc81f06ae9-proxy-tls\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:06:32.456864 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:32.456637    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bff3e308-3999-4ede-8379-1fdc81f06ae9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:06:32.674074 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:32.673981    2563 generic.go:356] "Generic (PLEG): container finished" podID="bff3e308-3999-4ede-8379-1fdc81f06ae9" containerID="c3b1c3650fcb4280c8d319f20ee146eca936facea72da0e0bc3e46436d834908" exitCode=0
Jul 15 11:06:32.674074 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:32.674031    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp"
Jul 15 11:06:32.674595 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:32.674067    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp" event={"ID":"bff3e308-3999-4ede-8379-1fdc81f06ae9","Type":"ContainerDied","Data":"c3b1c3650fcb4280c8d319f20ee146eca936facea72da0e0bc3e46436d834908"}
Jul 15 11:06:32.674595 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:32.674156    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp" event={"ID":"bff3e308-3999-4ede-8379-1fdc81f06ae9","Type":"ContainerDied","Data":"968fdb5bfd4a87e38028f754cc029e2c3b69e55228d2ebd1a225626d65f78ed2"}
Jul 15 11:06:32.674595 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:32.674204    2563 scope.go:117] "RemoveContainer" containerID="1b6c20c929c4d41ea66d2bb5752ca565b4862a00655122f4e7f0535b8043475f"
Jul 15 11:06:32.682286 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:32.682266    2563 scope.go:117] "RemoveContainer" containerID="c3b1c3650fcb4280c8d319f20ee146eca936facea72da0e0bc3e46436d834908"
Jul 15 11:06:32.689269 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:32.689252    2563 scope.go:117] "RemoveContainer" containerID="1b6c20c929c4d41ea66d2bb5752ca565b4862a00655122f4e7f0535b8043475f"
Jul 15 11:06:32.689543 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:06:32.689526    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b6c20c929c4d41ea66d2bb5752ca565b4862a00655122f4e7f0535b8043475f\": container with ID starting with 1b6c20c929c4d41ea66d2bb5752ca565b4862a00655122f4e7f0535b8043475f not found: ID does not exist" containerID="1b6c20c929c4d41ea66d2bb5752ca565b4862a00655122f4e7f0535b8043475f"
Jul 15 11:06:32.689587 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:32.689551    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b6c20c929c4d41ea66d2bb5752ca565b4862a00655122f4e7f0535b8043475f"} err="failed to get container status \"1b6c20c929c4d41ea66d2bb5752ca565b4862a00655122f4e7f0535b8043475f\": rpc error: code = NotFound desc = could not find container \"1b6c20c929c4d41ea66d2bb5752ca565b4862a00655122f4e7f0535b8043475f\": container with ID starting with 1b6c20c929c4d41ea66d2bb5752ca565b4862a00655122f4e7f0535b8043475f not found: ID does not exist"
Jul 15 11:06:32.689587 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:32.689569    2563 scope.go:117] "RemoveContainer" containerID="c3b1c3650fcb4280c8d319f20ee146eca936facea72da0e0bc3e46436d834908"
Jul 15 11:06:32.689793 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:06:32.689775    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3b1c3650fcb4280c8d319f20ee146eca936facea72da0e0bc3e46436d834908\": container with ID starting with c3b1c3650fcb4280c8d319f20ee146eca936facea72da0e0bc3e46436d834908 not found: ID does not exist" containerID="c3b1c3650fcb4280c8d319f20ee146eca936facea72da0e0bc3e46436d834908"
Jul 15 11:06:32.689850 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:32.689802    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3b1c3650fcb4280c8d319f20ee146eca936facea72da0e0bc3e46436d834908"} err="failed to get container status \"c3b1c3650fcb4280c8d319f20ee146eca936facea72da0e0bc3e46436d834908\": rpc error: code = NotFound desc = could not find container \"c3b1c3650fcb4280c8d319f20ee146eca936facea72da0e0bc3e46436d834908\": container with ID starting with c3b1c3650fcb4280c8d319f20ee146eca936facea72da0e0bc3e46436d834908 not found: ID does not exist"
Jul 15 11:06:32.699680 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:32.699654    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp"]
Jul 15 11:06:32.702157 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:32.702108    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp"]
Jul 15 11:06:33.678567 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:33.678533    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="582d624f-7f4f-459e-af00-7e5555fe81fd" path="/var/lib/kubelet/pods/582d624f-7f4f-459e-af00-7e5555fe81fd/volumes"
Jul 15 11:06:33.679100 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:33.679083    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bff3e308-3999-4ede-8379-1fdc81f06ae9" path="/var/lib/kubelet/pods/bff3e308-3999-4ede-8379-1fdc81f06ae9/volumes"
Jul 15 11:06:35.490038 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:35.490000    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj" podUID="4e971415-7cf0-42c0-88b8-0caac82999b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 15 11:06:35.490579 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:35.490562    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6"
Jul 15 11:06:36.672501 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:36.672470    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6"
Jul 15 11:06:36.672968 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:36.672556    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc"
Jul 15 11:06:36.672968 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:36.672920    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc" podUID="89161cbe-9df8-4661-a724-8cef68e2fe35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 15 11:06:36.673075 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:36.672957    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6" podUID="fa498659-a6ab-4e10-ace6-0400dbd05f68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 15 11:06:45.490946 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:45.490914    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj"
Jul 15 11:06:46.673183 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:46.673130    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6" podUID="fa498659-a6ab-4e10-ace6-0400dbd05f68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 15 11:06:46.673550 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:46.673126    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc" podUID="89161cbe-9df8-4661-a724-8cef68e2fe35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 15 11:06:56.672923 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:56.672883    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6" podUID="fa498659-a6ab-4e10-ace6-0400dbd05f68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 15 11:06:56.673424 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:06:56.672970    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc" podUID="89161cbe-9df8-4661-a724-8cef68e2fe35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 15 11:07:06.672945 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:06.672901    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc" podUID="89161cbe-9df8-4661-a724-8cef68e2fe35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 15 11:07:06.674990 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:06.672901    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6" podUID="fa498659-a6ab-4e10-ace6-0400dbd05f68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 15 11:07:08.202395 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.202363    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6"]
Jul 15 11:07:08.202791 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.202632    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6" podUID="f62dc8da-e540-4422-af99-61e6d8791537" containerName="kserve-container" containerID="cri-o://54502f96b4a6f8ab4eb207d9a84c737b6493beefcead932b8d072cf3b04b9466" gracePeriod=30
Jul 15 11:07:08.202791 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.202657    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6" podUID="f62dc8da-e540-4422-af99-61e6d8791537" containerName="kube-rbac-proxy" containerID="cri-o://724a80d9a54e6ad6718acf5a0604fa63125bec6b89697f55f91a6d8ba7242188" gracePeriod=30
Jul 15 11:07:08.263623 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.263589    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj"]
Jul 15 11:07:08.263826 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.263801    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj" podUID="4e971415-7cf0-42c0-88b8-0caac82999b6" containerName="kserve-container" containerID="cri-o://9066f5d4a7fed0a918664740ae875306cae8f76658435af1364793e9f1ce32f7" gracePeriod=30
Jul 15 11:07:08.263928 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.263875    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj" podUID="4e971415-7cf0-42c0-88b8-0caac82999b6" containerName="kube-rbac-proxy" containerID="cri-o://2f9db10d4e4853f4f9ded9509feab42f0a6a6391bd309bb55965a82bae91003f" gracePeriod=30
Jul 15 11:07:08.364029 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.363996    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp"]
Jul 15 11:07:08.364351 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.364339    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="582d624f-7f4f-459e-af00-7e5555fe81fd" containerName="kube-rbac-proxy"
Jul 15 11:07:08.364402 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.364353    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="582d624f-7f4f-459e-af00-7e5555fe81fd" containerName="kube-rbac-proxy"
Jul 15 11:07:08.364402 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.364363    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bff3e308-3999-4ede-8379-1fdc81f06ae9" containerName="kserve-container"
Jul 15 11:07:08.364402 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.364369    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bff3e308-3999-4ede-8379-1fdc81f06ae9" containerName="kserve-container"
Jul 15 11:07:08.364402 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.364379    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="582d624f-7f4f-459e-af00-7e5555fe81fd" containerName="kserve-container"
Jul 15 11:07:08.364402 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.364388    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="582d624f-7f4f-459e-af00-7e5555fe81fd" containerName="kserve-container"
Jul 15 11:07:08.364555 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.364411    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bff3e308-3999-4ede-8379-1fdc81f06ae9" containerName="kube-rbac-proxy"
Jul 15 11:07:08.364555 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.364417    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bff3e308-3999-4ede-8379-1fdc81f06ae9" containerName="kube-rbac-proxy"
Jul 15 11:07:08.364555 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.364461    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bff3e308-3999-4ede-8379-1fdc81f06ae9" containerName="kube-rbac-proxy"
Jul 15 11:07:08.364555 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.364473    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="582d624f-7f4f-459e-af00-7e5555fe81fd" containerName="kube-rbac-proxy"
Jul 15 11:07:08.364555 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.364483    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="582d624f-7f4f-459e-af00-7e5555fe81fd" containerName="kserve-container"
Jul 15 11:07:08.364555 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.364488    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bff3e308-3999-4ede-8379-1fdc81f06ae9" containerName="kserve-container"
Jul 15 11:07:08.367626 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.367605    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp"
Jul 15 11:07:08.370544 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.370524    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d6666-kube-rbac-proxy-sar-config\""
Jul 15 11:07:08.370754 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.370740    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d6666-predictor-serving-cert\""
Jul 15 11:07:08.381515 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.381490    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp"]
Jul 15 11:07:08.458466 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.458374    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84640095-4359-4999-b3c9-7d7602b8daca-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d6666-predictor-8674b65f5d-8hxmp\" (UID: \"84640095-4359-4999-b3c9-7d7602b8daca\") " pod="kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp"
Jul 15 11:07:08.458466 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.458415    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf4cl\" (UniqueName: \"kubernetes.io/projected/84640095-4359-4999-b3c9-7d7602b8daca-kube-api-access-qf4cl\") pod \"success-200-isvc-d6666-predictor-8674b65f5d-8hxmp\" (UID: \"84640095-4359-4999-b3c9-7d7602b8daca\") " pod="kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp"
Jul 15 11:07:08.458662 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.458500    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84640095-4359-4999-b3c9-7d7602b8daca-proxy-tls\") pod \"success-200-isvc-d6666-predictor-8674b65f5d-8hxmp\" (UID: \"84640095-4359-4999-b3c9-7d7602b8daca\") " pod="kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp"
Jul 15 11:07:08.506074 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.506045    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4"]
Jul 15 11:07:08.509403 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.509385    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4"
Jul 15 11:07:08.513190 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.513169    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d6666-kube-rbac-proxy-sar-config\""
Jul 15 11:07:08.513318 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.513301    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d6666-predictor-serving-cert\""
Jul 15 11:07:08.523287 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.523264    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4"]
Jul 15 11:07:08.559730 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.559681    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92cf56c4-3843-4b76-9352-acc8bf60b6a8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d6666-predictor-774b85585f-t7hr4\" (UID: \"92cf56c4-3843-4b76-9352-acc8bf60b6a8\") " pod="kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4"
Jul 15 11:07:08.559730 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.559732    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkj6n\" (UniqueName: \"kubernetes.io/projected/92cf56c4-3843-4b76-9352-acc8bf60b6a8-kube-api-access-xkj6n\") pod \"error-404-isvc-d6666-predictor-774b85585f-t7hr4\" (UID: \"92cf56c4-3843-4b76-9352-acc8bf60b6a8\") " pod="kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4"
Jul 15 11:07:08.559959 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.559809    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92cf56c4-3843-4b76-9352-acc8bf60b6a8-proxy-tls\") pod \"error-404-isvc-d6666-predictor-774b85585f-t7hr4\" (UID: \"92cf56c4-3843-4b76-9352-acc8bf60b6a8\") " pod="kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4"
Jul 15 11:07:08.559959 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.559884    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84640095-4359-4999-b3c9-7d7602b8daca-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d6666-predictor-8674b65f5d-8hxmp\" (UID: \"84640095-4359-4999-b3c9-7d7602b8daca\") " pod="kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp"
Jul 15 11:07:08.559959 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.559919    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf4cl\" (UniqueName: \"kubernetes.io/projected/84640095-4359-4999-b3c9-7d7602b8daca-kube-api-access-qf4cl\") pod \"success-200-isvc-d6666-predictor-8674b65f5d-8hxmp\" (UID: \"84640095-4359-4999-b3c9-7d7602b8daca\") " pod="kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp"
Jul 15 11:07:08.560113 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.559996    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84640095-4359-4999-b3c9-7d7602b8daca-proxy-tls\") pod \"success-200-isvc-d6666-predictor-8674b65f5d-8hxmp\" (UID: \"84640095-4359-4999-b3c9-7d7602b8daca\") " pod="kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp"
Jul 15 11:07:08.560732 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.560710    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84640095-4359-4999-b3c9-7d7602b8daca-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d6666-predictor-8674b65f5d-8hxmp\" (UID: \"84640095-4359-4999-b3c9-7d7602b8daca\") " pod="kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp"
Jul 15 11:07:08.562540 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.562517    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84640095-4359-4999-b3c9-7d7602b8daca-proxy-tls\") pod \"success-200-isvc-d6666-predictor-8674b65f5d-8hxmp\" (UID: \"84640095-4359-4999-b3c9-7d7602b8daca\") " pod="kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp"
Jul 15 11:07:08.569183 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.569165    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf4cl\" (UniqueName: \"kubernetes.io/projected/84640095-4359-4999-b3c9-7d7602b8daca-kube-api-access-qf4cl\") pod \"success-200-isvc-d6666-predictor-8674b65f5d-8hxmp\" (UID: \"84640095-4359-4999-b3c9-7d7602b8daca\") " pod="kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp"
Jul 15 11:07:08.660780 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.660749    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92cf56c4-3843-4b76-9352-acc8bf60b6a8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d6666-predictor-774b85585f-t7hr4\" (UID: \"92cf56c4-3843-4b76-9352-acc8bf60b6a8\") " pod="kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4"
Jul 15 11:07:08.660780 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.660784    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkj6n\" (UniqueName: \"kubernetes.io/projected/92cf56c4-3843-4b76-9352-acc8bf60b6a8-kube-api-access-xkj6n\") pod \"error-404-isvc-d6666-predictor-774b85585f-t7hr4\" (UID: \"92cf56c4-3843-4b76-9352-acc8bf60b6a8\") " pod="kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4"
Jul 15 11:07:08.660987 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.660805    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92cf56c4-3843-4b76-9352-acc8bf60b6a8-proxy-tls\") pod \"error-404-isvc-d6666-predictor-774b85585f-t7hr4\" (UID: \"92cf56c4-3843-4b76-9352-acc8bf60b6a8\") " pod="kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4"
Jul 15 11:07:08.660987 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:07:08.660913    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-d6666-predictor-serving-cert: secret "error-404-isvc-d6666-predictor-serving-cert" not found
Jul 15 11:07:08.660987 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:07:08.660971    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92cf56c4-3843-4b76-9352-acc8bf60b6a8-proxy-tls podName:92cf56c4-3843-4b76-9352-acc8bf60b6a8 nodeName:}" failed. No retries permitted until 2026-07-15 11:07:09.160951173 +0000 UTC m=+1247.990466574 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/92cf56c4-3843-4b76-9352-acc8bf60b6a8-proxy-tls") pod "error-404-isvc-d6666-predictor-774b85585f-t7hr4" (UID: "92cf56c4-3843-4b76-9352-acc8bf60b6a8") : secret "error-404-isvc-d6666-predictor-serving-cert" not found
Jul 15 11:07:08.661460 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.661442    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92cf56c4-3843-4b76-9352-acc8bf60b6a8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d6666-predictor-774b85585f-t7hr4\" (UID: \"92cf56c4-3843-4b76-9352-acc8bf60b6a8\") " pod="kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4"
Jul 15 11:07:08.675681 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.675656    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkj6n\" (UniqueName: \"kubernetes.io/projected/92cf56c4-3843-4b76-9352-acc8bf60b6a8-kube-api-access-xkj6n\") pod \"error-404-isvc-d6666-predictor-774b85585f-t7hr4\" (UID: \"92cf56c4-3843-4b76-9352-acc8bf60b6a8\") " pod="kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4"
Jul 15 11:07:08.678309 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.678290    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp"
Jul 15 11:07:08.813873 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.813848    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp"]
Jul 15 11:07:08.816307 ip-10-0-131-34 kubenswrapper[2563]: W0715 11:07:08.816275    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84640095_4359_4999_b3c9_7d7602b8daca.slice/crio-a5b72f2b5c3405903dc04ac9645c07b4e95afce8e0b951784222ea9f2ea22c40 WatchSource:0}: Error finding container a5b72f2b5c3405903dc04ac9645c07b4e95afce8e0b951784222ea9f2ea22c40: Status 404 returned error can't find the container with id a5b72f2b5c3405903dc04ac9645c07b4e95afce8e0b951784222ea9f2ea22c40
Jul 15 11:07:08.824187 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.824164    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp" event={"ID":"84640095-4359-4999-b3c9-7d7602b8daca","Type":"ContainerStarted","Data":"a5b72f2b5c3405903dc04ac9645c07b4e95afce8e0b951784222ea9f2ea22c40"}
Jul 15 11:07:08.825936 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.825908    2563 generic.go:356] "Generic (PLEG): container finished" podID="4e971415-7cf0-42c0-88b8-0caac82999b6" containerID="2f9db10d4e4853f4f9ded9509feab42f0a6a6391bd309bb55965a82bae91003f" exitCode=2
Jul 15 11:07:08.826040 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.825976    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj" event={"ID":"4e971415-7cf0-42c0-88b8-0caac82999b6","Type":"ContainerDied","Data":"2f9db10d4e4853f4f9ded9509feab42f0a6a6391bd309bb55965a82bae91003f"}
Jul 15 11:07:08.827645 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.827627    2563 generic.go:356] "Generic (PLEG): container finished" podID="f62dc8da-e540-4422-af99-61e6d8791537" containerID="724a80d9a54e6ad6718acf5a0604fa63125bec6b89697f55f91a6d8ba7242188" exitCode=2
Jul 15 11:07:08.827729 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:08.827650    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6" event={"ID":"f62dc8da-e540-4422-af99-61e6d8791537","Type":"ContainerDied","Data":"724a80d9a54e6ad6718acf5a0604fa63125bec6b89697f55f91a6d8ba7242188"}
Jul 15 11:07:09.165328 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:09.165241    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92cf56c4-3843-4b76-9352-acc8bf60b6a8-proxy-tls\") pod \"error-404-isvc-d6666-predictor-774b85585f-t7hr4\" (UID: \"92cf56c4-3843-4b76-9352-acc8bf60b6a8\") " pod="kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4"
Jul 15 11:07:09.167616 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:09.167587    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92cf56c4-3843-4b76-9352-acc8bf60b6a8-proxy-tls\") pod \"error-404-isvc-d6666-predictor-774b85585f-t7hr4\" (UID: \"92cf56c4-3843-4b76-9352-acc8bf60b6a8\") " pod="kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4"
Jul 15 11:07:09.419034 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:09.418952    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4"
Jul 15 11:07:09.554842 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:09.554793    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4"]
Jul 15 11:07:09.560197 ip-10-0-131-34 kubenswrapper[2563]: W0715 11:07:09.559462    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92cf56c4_3843_4b76_9352_acc8bf60b6a8.slice/crio-476aa45c2c6cde9d78c2e1b7a9070c473492d9e1570c1348f152653dd85b6bf0 WatchSource:0}: Error finding container 476aa45c2c6cde9d78c2e1b7a9070c473492d9e1570c1348f152653dd85b6bf0: Status 404 returned error can't find the container with id 476aa45c2c6cde9d78c2e1b7a9070c473492d9e1570c1348f152653dd85b6bf0
Jul 15 11:07:09.834292 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:09.834256    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4" event={"ID":"92cf56c4-3843-4b76-9352-acc8bf60b6a8","Type":"ContainerStarted","Data":"9aef39621580411e7ceca1ff1dce68ef5f15c49a69463a113ed28dcbfb81cafc"}
Jul 15 11:07:09.834484 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:09.834320    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4" event={"ID":"92cf56c4-3843-4b76-9352-acc8bf60b6a8","Type":"ContainerStarted","Data":"abbcb180eb57de7d21d5df80117c534dedb8f3f7bbbfa239091f8070db1ae06b"}
Jul 15 11:07:09.834484 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:09.834353    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4" event={"ID":"92cf56c4-3843-4b76-9352-acc8bf60b6a8","Type":"ContainerStarted","Data":"476aa45c2c6cde9d78c2e1b7a9070c473492d9e1570c1348f152653dd85b6bf0"}
Jul 15 11:07:09.834614 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:09.834484    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4"
Jul 15 11:07:09.834614 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:09.834526    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4"
Jul 15 11:07:09.836023 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:09.835996    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4" podUID="92cf56c4-3843-4b76-9352-acc8bf60b6a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 15 11:07:09.836187 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:09.836166    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp" event={"ID":"84640095-4359-4999-b3c9-7d7602b8daca","Type":"ContainerStarted","Data":"d6b6ecf471a9b35ef4311eb2ac7c8f9143f602a44ce075f30ff916764938ef3d"}
Jul 15 11:07:09.836261 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:09.836220    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp" event={"ID":"84640095-4359-4999-b3c9-7d7602b8daca","Type":"ContainerStarted","Data":"ab43859a6b02667a1ea2a1cff9cdb6a8711a09398d14cec0daf55ff2127bb740"}
Jul 15 11:07:09.836319 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:09.836304    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp"
Jul 15 11:07:09.855238 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:09.855181    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4" podStartSLOduration=1.855122664 podStartE2EDuration="1.855122664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:07:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:07:09.852781421 +0000 UTC m=+1248.682296840" watchObservedRunningTime="2026-07-15 11:07:09.855122664 +0000 UTC m=+1248.684638071"
Jul 15 11:07:09.872240 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:09.872083    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp" podStartSLOduration=1.872066998 podStartE2EDuration="1.872066998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:07:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:07:09.871197402 +0000 UTC m=+1248.700712807" watchObservedRunningTime="2026-07-15 11:07:09.872066998 +0000 UTC m=+1248.701582403"
Jul 15 11:07:10.484822 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:10.484785    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6" podUID="f62dc8da-e540-4422-af99-61e6d8791537" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 15 11:07:10.485202 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:10.484787    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj" podUID="4e971415-7cf0-42c0-88b8-0caac82999b6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 15 11:07:10.840704 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:10.840595    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4" podUID="92cf56c4-3843-4b76-9352-acc8bf60b6a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 15 11:07:10.840704 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:10.840619    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp"
Jul 15 11:07:10.841859 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:10.841834    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp" podUID="84640095-4359-4999-b3c9-7d7602b8daca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 15 11:07:11.765058 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.765031    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj"
Jul 15 11:07:11.767742 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.767722    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6"
Jul 15 11:07:11.787799 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.787778    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tv9dc\" (UniqueName: \"kubernetes.io/projected/4e971415-7cf0-42c0-88b8-0caac82999b6-kube-api-access-tv9dc\") pod \"4e971415-7cf0-42c0-88b8-0caac82999b6\" (UID: \"4e971415-7cf0-42c0-88b8-0caac82999b6\") "
Jul 15 11:07:11.787909 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.787837    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e971415-7cf0-42c0-88b8-0caac82999b6-kube-rbac-proxy-sar-config\") pod \"4e971415-7cf0-42c0-88b8-0caac82999b6\" (UID: \"4e971415-7cf0-42c0-88b8-0caac82999b6\") "
Jul 15 11:07:11.787909 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.787885    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f62dc8da-e540-4422-af99-61e6d8791537-kube-rbac-proxy-sar-config\") pod \"f62dc8da-e540-4422-af99-61e6d8791537\" (UID: \"f62dc8da-e540-4422-af99-61e6d8791537\") "
Jul 15 11:07:11.788024 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.787945    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e971415-7cf0-42c0-88b8-0caac82999b6-proxy-tls\") pod \"4e971415-7cf0-42c0-88b8-0caac82999b6\" (UID: \"4e971415-7cf0-42c0-88b8-0caac82999b6\") "
Jul 15 11:07:11.788024 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.787973    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f62dc8da-e540-4422-af99-61e6d8791537-proxy-tls\") pod \"f62dc8da-e540-4422-af99-61e6d8791537\" (UID: \"f62dc8da-e540-4422-af99-61e6d8791537\") "
Jul 15 11:07:11.788024 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.788003    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9mcp\" (UniqueName: \"kubernetes.io/projected/f62dc8da-e540-4422-af99-61e6d8791537-kube-api-access-k9mcp\") pod \"f62dc8da-e540-4422-af99-61e6d8791537\" (UID: \"f62dc8da-e540-4422-af99-61e6d8791537\") "
Jul 15 11:07:11.788385 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.788257    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e971415-7cf0-42c0-88b8-0caac82999b6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4e971415-7cf0-42c0-88b8-0caac82999b6" (UID: "4e971415-7cf0-42c0-88b8-0caac82999b6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 11:07:11.788385 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.788311    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f62dc8da-e540-4422-af99-61e6d8791537-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f62dc8da-e540-4422-af99-61e6d8791537" (UID: "f62dc8da-e540-4422-af99-61e6d8791537"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 11:07:11.788385 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.788350    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e971415-7cf0-42c0-88b8-0caac82999b6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:07:11.790433 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.790402    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f62dc8da-e540-4422-af99-61e6d8791537-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f62dc8da-e540-4422-af99-61e6d8791537" (UID: "f62dc8da-e540-4422-af99-61e6d8791537"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:07:11.790433 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.790417    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e971415-7cf0-42c0-88b8-0caac82999b6-kube-api-access-tv9dc" (OuterVolumeSpecName: "kube-api-access-tv9dc") pod "4e971415-7cf0-42c0-88b8-0caac82999b6" (UID: "4e971415-7cf0-42c0-88b8-0caac82999b6"). InnerVolumeSpecName "kube-api-access-tv9dc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:07:11.790555 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.790446    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e971415-7cf0-42c0-88b8-0caac82999b6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4e971415-7cf0-42c0-88b8-0caac82999b6" (UID: "4e971415-7cf0-42c0-88b8-0caac82999b6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:07:11.790555 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.790471    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f62dc8da-e540-4422-af99-61e6d8791537-kube-api-access-k9mcp" (OuterVolumeSpecName: "kube-api-access-k9mcp") pod "f62dc8da-e540-4422-af99-61e6d8791537" (UID: "f62dc8da-e540-4422-af99-61e6d8791537"). InnerVolumeSpecName "kube-api-access-k9mcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:07:11.847175 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.847085    2563 generic.go:356] "Generic (PLEG): container finished" podID="f62dc8da-e540-4422-af99-61e6d8791537" containerID="54502f96b4a6f8ab4eb207d9a84c737b6493beefcead932b8d072cf3b04b9466" exitCode=0
Jul 15 11:07:11.847175 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.847132    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6"
Jul 15 11:07:11.847375 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.847180    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6" event={"ID":"f62dc8da-e540-4422-af99-61e6d8791537","Type":"ContainerDied","Data":"54502f96b4a6f8ab4eb207d9a84c737b6493beefcead932b8d072cf3b04b9466"}
Jul 15 11:07:11.847375 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.847274    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6" event={"ID":"f62dc8da-e540-4422-af99-61e6d8791537","Type":"ContainerDied","Data":"3f222c8e07156c9fbd91debd1bb9b97a5d9b072ffa424921b7aa8c15df7d3f23"}
Jul 15 11:07:11.847375 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.847309    2563 scope.go:117] "RemoveContainer" containerID="724a80d9a54e6ad6718acf5a0604fa63125bec6b89697f55f91a6d8ba7242188"
Jul 15 11:07:11.848968 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.848941    2563 generic.go:356] "Generic (PLEG): container finished" podID="4e971415-7cf0-42c0-88b8-0caac82999b6" containerID="9066f5d4a7fed0a918664740ae875306cae8f76658435af1364793e9f1ce32f7" exitCode=0
Jul 15 11:07:11.849072 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.848971    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj" event={"ID":"4e971415-7cf0-42c0-88b8-0caac82999b6","Type":"ContainerDied","Data":"9066f5d4a7fed0a918664740ae875306cae8f76658435af1364793e9f1ce32f7"}
Jul 15 11:07:11.849072 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.848988    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj"
Jul 15 11:07:11.849072 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.849028    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj" event={"ID":"4e971415-7cf0-42c0-88b8-0caac82999b6","Type":"ContainerDied","Data":"37073868de37c33fb1f6f680a8f2dcf649e5d3a6b8ff24d2ae9d75ada7109d6a"}
Jul 15 11:07:11.849439 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.849400    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp" podUID="84640095-4359-4999-b3c9-7d7602b8daca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 15 11:07:11.857984 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.857919    2563 scope.go:117] "RemoveContainer" containerID="54502f96b4a6f8ab4eb207d9a84c737b6493beefcead932b8d072cf3b04b9466"
Jul 15 11:07:11.868378 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.868359    2563 scope.go:117] "RemoveContainer" containerID="724a80d9a54e6ad6718acf5a0604fa63125bec6b89697f55f91a6d8ba7242188"
Jul 15 11:07:11.868674 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:07:11.868651    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"724a80d9a54e6ad6718acf5a0604fa63125bec6b89697f55f91a6d8ba7242188\": container with ID starting with 724a80d9a54e6ad6718acf5a0604fa63125bec6b89697f55f91a6d8ba7242188 not found: ID does not exist" containerID="724a80d9a54e6ad6718acf5a0604fa63125bec6b89697f55f91a6d8ba7242188"
Jul 15 11:07:11.868792 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.868683    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"724a80d9a54e6ad6718acf5a0604fa63125bec6b89697f55f91a6d8ba7242188"} err="failed to get container status \"724a80d9a54e6ad6718acf5a0604fa63125bec6b89697f55f91a6d8ba7242188\": rpc error: code = NotFound desc = could not find container \"724a80d9a54e6ad6718acf5a0604fa63125bec6b89697f55f91a6d8ba7242188\": container with ID starting with 724a80d9a54e6ad6718acf5a0604fa63125bec6b89697f55f91a6d8ba7242188 not found: ID does not exist"
Jul 15 11:07:11.868792 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.868705    2563 scope.go:117] "RemoveContainer" containerID="54502f96b4a6f8ab4eb207d9a84c737b6493beefcead932b8d072cf3b04b9466"
Jul 15 11:07:11.868941 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:07:11.868924    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54502f96b4a6f8ab4eb207d9a84c737b6493beefcead932b8d072cf3b04b9466\": container with ID starting with 54502f96b4a6f8ab4eb207d9a84c737b6493beefcead932b8d072cf3b04b9466 not found: ID does not exist" containerID="54502f96b4a6f8ab4eb207d9a84c737b6493beefcead932b8d072cf3b04b9466"
Jul 15 11:07:11.869006 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.868952    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54502f96b4a6f8ab4eb207d9a84c737b6493beefcead932b8d072cf3b04b9466"} err="failed to get container status \"54502f96b4a6f8ab4eb207d9a84c737b6493beefcead932b8d072cf3b04b9466\": rpc error: code = NotFound desc = could not find container \"54502f96b4a6f8ab4eb207d9a84c737b6493beefcead932b8d072cf3b04b9466\": container with ID starting with 54502f96b4a6f8ab4eb207d9a84c737b6493beefcead932b8d072cf3b04b9466 not found: ID does not exist"
Jul 15 11:07:11.869006 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.868976    2563 scope.go:117] "RemoveContainer" containerID="2f9db10d4e4853f4f9ded9509feab42f0a6a6391bd309bb55965a82bae91003f"
Jul 15 11:07:11.876624 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.876608    2563 scope.go:117] "RemoveContainer" containerID="9066f5d4a7fed0a918664740ae875306cae8f76658435af1364793e9f1ce32f7"
Jul 15 11:07:11.879708 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.879687    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj"]
Jul 15 11:07:11.884127 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.884105    2563 scope.go:117] "RemoveContainer" containerID="2f9db10d4e4853f4f9ded9509feab42f0a6a6391bd309bb55965a82bae91003f"
Jul 15 11:07:11.884415 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:07:11.884398    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f9db10d4e4853f4f9ded9509feab42f0a6a6391bd309bb55965a82bae91003f\": container with ID starting with 2f9db10d4e4853f4f9ded9509feab42f0a6a6391bd309bb55965a82bae91003f not found: ID does not exist" containerID="2f9db10d4e4853f4f9ded9509feab42f0a6a6391bd309bb55965a82bae91003f"
Jul 15 11:07:11.884469 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.884421    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f9db10d4e4853f4f9ded9509feab42f0a6a6391bd309bb55965a82bae91003f"} err="failed to get container status \"2f9db10d4e4853f4f9ded9509feab42f0a6a6391bd309bb55965a82bae91003f\": rpc error: code = NotFound desc = could not find container \"2f9db10d4e4853f4f9ded9509feab42f0a6a6391bd309bb55965a82bae91003f\": container with ID starting with 2f9db10d4e4853f4f9ded9509feab42f0a6a6391bd309bb55965a82bae91003f not found: ID does not exist"
Jul 15 11:07:11.884469 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.884437    2563 scope.go:117] "RemoveContainer" containerID="9066f5d4a7fed0a918664740ae875306cae8f76658435af1364793e9f1ce32f7"
Jul 15 11:07:11.884546 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.884532    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj"]
Jul 15 11:07:11.884669 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:07:11.884652    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9066f5d4a7fed0a918664740ae875306cae8f76658435af1364793e9f1ce32f7\": container with ID starting with 9066f5d4a7fed0a918664740ae875306cae8f76658435af1364793e9f1ce32f7 not found: ID does not exist" containerID="9066f5d4a7fed0a918664740ae875306cae8f76658435af1364793e9f1ce32f7"
Jul 15 11:07:11.884712 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.884678    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9066f5d4a7fed0a918664740ae875306cae8f76658435af1364793e9f1ce32f7"} err="failed to get container status \"9066f5d4a7fed0a918664740ae875306cae8f76658435af1364793e9f1ce32f7\": rpc error: code = NotFound desc = could not find container \"9066f5d4a7fed0a918664740ae875306cae8f76658435af1364793e9f1ce32f7\": container with ID starting with 9066f5d4a7fed0a918664740ae875306cae8f76658435af1364793e9f1ce32f7 not found: ID does not exist"
Jul 15 11:07:11.889196 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.889178    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f62dc8da-e540-4422-af99-61e6d8791537-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:07:11.889281 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.889199    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e971415-7cf0-42c0-88b8-0caac82999b6-proxy-tls\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:07:11.889281 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.889245    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f62dc8da-e540-4422-af99-61e6d8791537-proxy-tls\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:07:11.889281 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.889258    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9mcp\" (UniqueName: \"kubernetes.io/projected/f62dc8da-e540-4422-af99-61e6d8791537-kube-api-access-k9mcp\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:07:11.889281 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.889272    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv9dc\" (UniqueName: \"kubernetes.io/projected/4e971415-7cf0-42c0-88b8-0caac82999b6-kube-api-access-tv9dc\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:07:11.907254 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.907225    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6"]
Jul 15 11:07:11.908399 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:11.908382    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6"]
Jul 15 11:07:13.678436 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:13.678348    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e971415-7cf0-42c0-88b8-0caac82999b6" path="/var/lib/kubelet/pods/4e971415-7cf0-42c0-88b8-0caac82999b6/volumes"
Jul 15 11:07:13.678878 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:13.678786    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f62dc8da-e540-4422-af99-61e6d8791537" path="/var/lib/kubelet/pods/f62dc8da-e540-4422-af99-61e6d8791537/volumes"
Jul 15 11:07:15.845656 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:15.845623    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4"
Jul 15 11:07:15.846263 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:15.846236    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4" podUID="92cf56c4-3843-4b76-9352-acc8bf60b6a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 15 11:07:16.673623 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:16.673587    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc" podUID="89161cbe-9df8-4661-a724-8cef68e2fe35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 15 11:07:16.674175 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:16.674133    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6"
Jul 15 11:07:16.854802 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:16.854772    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp"
Jul 15 11:07:16.855518 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:16.855486    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp" podUID="84640095-4359-4999-b3c9-7d7602b8daca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 15 11:07:25.847135 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:25.847093    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4" podUID="92cf56c4-3843-4b76-9352-acc8bf60b6a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 15 11:07:26.673823 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:26.673794    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc"
Jul 15 11:07:26.856134 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:26.856089    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp" podUID="84640095-4359-4999-b3c9-7d7602b8daca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 15 11:07:35.846365 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:35.846324    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4" podUID="92cf56c4-3843-4b76-9352-acc8bf60b6a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 15 11:07:36.856401 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:36.856355    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp" podUID="84640095-4359-4999-b3c9-7d7602b8daca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 15 11:07:45.846990 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:45.846947    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4" podUID="92cf56c4-3843-4b76-9352-acc8bf60b6a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 15 11:07:46.855977 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:46.855937    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp" podUID="84640095-4359-4999-b3c9-7d7602b8daca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 15 11:07:48.282778 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.282739    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc"]
Jul 15 11:07:48.283215 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.282970    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc" podUID="89161cbe-9df8-4661-a724-8cef68e2fe35" containerName="kserve-container" containerID="cri-o://127f444a6216211469a47bd9a20f1d8c6cc4ad0ec4f30aa421734e11c0134a03" gracePeriod=30
Jul 15 11:07:48.283215 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.283003    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc" podUID="89161cbe-9df8-4661-a724-8cef68e2fe35" containerName="kube-rbac-proxy" containerID="cri-o://2e6894af55706d77ee129f25dabea9f1326cfc35f1b2e37594b12b0a5807243f" gracePeriod=30
Jul 15 11:07:48.365550 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.365521    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6"]
Jul 15 11:07:48.365758 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.365732    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6" podUID="fa498659-a6ab-4e10-ace6-0400dbd05f68" containerName="kserve-container" containerID="cri-o://9782b20ac2c597226ff37b84fad82ea49b9aada866846be55a3135acea420a82" gracePeriod=30
Jul 15 11:07:48.365880 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.365775    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6" podUID="fa498659-a6ab-4e10-ace6-0400dbd05f68" containerName="kube-rbac-proxy" containerID="cri-o://56e43b0f5d06ab237e4966f426e2b06d6ef98f78e5da782fcd4350cd22b02a22" gracePeriod=30
Jul 15 11:07:48.369933 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.369906    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt"]
Jul 15 11:07:48.370531 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.370380    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e971415-7cf0-42c0-88b8-0caac82999b6" containerName="kserve-container"
Jul 15 11:07:48.370531 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.370396    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e971415-7cf0-42c0-88b8-0caac82999b6" containerName="kserve-container"
Jul 15 11:07:48.370531 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.370411    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f62dc8da-e540-4422-af99-61e6d8791537" containerName="kube-rbac-proxy"
Jul 15 11:07:48.370531 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.370419    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f62dc8da-e540-4422-af99-61e6d8791537" containerName="kube-rbac-proxy"
Jul 15 11:07:48.370531 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.370443    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f62dc8da-e540-4422-af99-61e6d8791537" containerName="kserve-container"
Jul 15 11:07:48.370531 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.370454    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f62dc8da-e540-4422-af99-61e6d8791537" containerName="kserve-container"
Jul 15 11:07:48.370531 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.370477    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e971415-7cf0-42c0-88b8-0caac82999b6" containerName="kube-rbac-proxy"
Jul 15 11:07:48.370531 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.370485    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e971415-7cf0-42c0-88b8-0caac82999b6" containerName="kube-rbac-proxy"
Jul 15 11:07:48.371033 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.370554    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f62dc8da-e540-4422-af99-61e6d8791537" containerName="kserve-container"
Jul 15 11:07:48.371033 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.370565    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f62dc8da-e540-4422-af99-61e6d8791537" containerName="kube-rbac-proxy"
Jul 15 11:07:48.371033 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.370579    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e971415-7cf0-42c0-88b8-0caac82999b6" containerName="kube-rbac-proxy"
Jul 15 11:07:48.371033 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.370589    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e971415-7cf0-42c0-88b8-0caac82999b6" containerName="kserve-container"
Jul 15 11:07:48.375765 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.375662    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt"
Jul 15 11:07:48.378228 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.378208    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-70137-predictor-serving-cert\""
Jul 15 11:07:48.378437 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.378420    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-70137-kube-rbac-proxy-sar-config\""
Jul 15 11:07:48.395977 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.395952    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt"]
Jul 15 11:07:48.475425 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.475398    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv"]
Jul 15 11:07:48.480511 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.480489    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv"
Jul 15 11:07:48.484865 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.484843    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-70137-predictor-serving-cert\""
Jul 15 11:07:48.485283 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.485268    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-70137-kube-rbac-proxy-sar-config\""
Jul 15 11:07:48.491765 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.491722    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwn9x\" (UniqueName: \"kubernetes.io/projected/0d80ad20-8011-4e76-a4d9-2d8335b4e25c-kube-api-access-dwn9x\") pod \"success-200-isvc-70137-predictor-77585cb866-kcdrt\" (UID: \"0d80ad20-8011-4e76-a4d9-2d8335b4e25c\") " pod="kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt"
Jul 15 11:07:48.491912 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.491813    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d80ad20-8011-4e76-a4d9-2d8335b4e25c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-70137-predictor-77585cb866-kcdrt\" (UID: \"0d80ad20-8011-4e76-a4d9-2d8335b4e25c\") " pod="kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt"
Jul 15 11:07:48.491912 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.491870    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d80ad20-8011-4e76-a4d9-2d8335b4e25c-proxy-tls\") pod \"success-200-isvc-70137-predictor-77585cb866-kcdrt\" (UID: \"0d80ad20-8011-4e76-a4d9-2d8335b4e25c\") " pod="kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt"
Jul 15 11:07:48.493416 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.493397    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv"]
Jul 15 11:07:48.593096 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.593011    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwn9x\" (UniqueName: \"kubernetes.io/projected/0d80ad20-8011-4e76-a4d9-2d8335b4e25c-kube-api-access-dwn9x\") pod \"success-200-isvc-70137-predictor-77585cb866-kcdrt\" (UID: \"0d80ad20-8011-4e76-a4d9-2d8335b4e25c\") " pod="kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt"
Jul 15 11:07:48.593096 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.593056    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d80ad20-8011-4e76-a4d9-2d8335b4e25c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-70137-predictor-77585cb866-kcdrt\" (UID: \"0d80ad20-8011-4e76-a4d9-2d8335b4e25c\") " pod="kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt"
Jul 15 11:07:48.593096 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.593086    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/742c19fa-753e-49c9-9b71-ca36ef4dd9bc-proxy-tls\") pod \"error-404-isvc-70137-predictor-544d7497cf-k6qvv\" (UID: \"742c19fa-753e-49c9-9b71-ca36ef4dd9bc\") " pod="kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv"
Jul 15 11:07:48.593424 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.593103    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d80ad20-8011-4e76-a4d9-2d8335b4e25c-proxy-tls\") pod \"success-200-isvc-70137-predictor-77585cb866-kcdrt\" (UID: \"0d80ad20-8011-4e76-a4d9-2d8335b4e25c\") " pod="kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt"
Jul 15 11:07:48.593424 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.593187    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/742c19fa-753e-49c9-9b71-ca36ef4dd9bc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-70137-predictor-544d7497cf-k6qvv\" (UID: \"742c19fa-753e-49c9-9b71-ca36ef4dd9bc\") " pod="kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv"
Jul 15 11:07:48.593424 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.593228    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltk2t\" (UniqueName: \"kubernetes.io/projected/742c19fa-753e-49c9-9b71-ca36ef4dd9bc-kube-api-access-ltk2t\") pod \"error-404-isvc-70137-predictor-544d7497cf-k6qvv\" (UID: \"742c19fa-753e-49c9-9b71-ca36ef4dd9bc\") " pod="kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv"
Jul 15 11:07:48.593424 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:07:48.593196    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-70137-predictor-serving-cert: secret "success-200-isvc-70137-predictor-serving-cert" not found
Jul 15 11:07:48.593424 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:07:48.593333    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d80ad20-8011-4e76-a4d9-2d8335b4e25c-proxy-tls podName:0d80ad20-8011-4e76-a4d9-2d8335b4e25c nodeName:}" failed. No retries permitted until 2026-07-15 11:07:49.093315751 +0000 UTC m=+1287.922831138 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0d80ad20-8011-4e76-a4d9-2d8335b4e25c-proxy-tls") pod "success-200-isvc-70137-predictor-77585cb866-kcdrt" (UID: "0d80ad20-8011-4e76-a4d9-2d8335b4e25c") : secret "success-200-isvc-70137-predictor-serving-cert" not found
Jul 15 11:07:48.593832 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.593810    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d80ad20-8011-4e76-a4d9-2d8335b4e25c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-70137-predictor-77585cb866-kcdrt\" (UID: \"0d80ad20-8011-4e76-a4d9-2d8335b4e25c\") " pod="kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt"
Jul 15 11:07:48.606043 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.606014    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwn9x\" (UniqueName: \"kubernetes.io/projected/0d80ad20-8011-4e76-a4d9-2d8335b4e25c-kube-api-access-dwn9x\") pod \"success-200-isvc-70137-predictor-77585cb866-kcdrt\" (UID: \"0d80ad20-8011-4e76-a4d9-2d8335b4e25c\") " pod="kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt"
Jul 15 11:07:48.693718 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.693688    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltk2t\" (UniqueName: \"kubernetes.io/projected/742c19fa-753e-49c9-9b71-ca36ef4dd9bc-kube-api-access-ltk2t\") pod \"error-404-isvc-70137-predictor-544d7497cf-k6qvv\" (UID: \"742c19fa-753e-49c9-9b71-ca36ef4dd9bc\") " pod="kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv"
Jul 15 11:07:48.693889 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.693755    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/742c19fa-753e-49c9-9b71-ca36ef4dd9bc-proxy-tls\") pod \"error-404-isvc-70137-predictor-544d7497cf-k6qvv\" (UID: \"742c19fa-753e-49c9-9b71-ca36ef4dd9bc\") " pod="kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv"
Jul 15 11:07:48.693889 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.693809    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/742c19fa-753e-49c9-9b71-ca36ef4dd9bc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-70137-predictor-544d7497cf-k6qvv\" (UID: \"742c19fa-753e-49c9-9b71-ca36ef4dd9bc\") " pod="kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv"
Jul 15 11:07:48.694511 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.694490    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/742c19fa-753e-49c9-9b71-ca36ef4dd9bc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-70137-predictor-544d7497cf-k6qvv\" (UID: \"742c19fa-753e-49c9-9b71-ca36ef4dd9bc\") " pod="kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv"
Jul 15 11:07:48.696228 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.696207    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/742c19fa-753e-49c9-9b71-ca36ef4dd9bc-proxy-tls\") pod \"error-404-isvc-70137-predictor-544d7497cf-k6qvv\" (UID: \"742c19fa-753e-49c9-9b71-ca36ef4dd9bc\") " pod="kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv"
Jul 15 11:07:48.703901 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.703881    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltk2t\" (UniqueName: \"kubernetes.io/projected/742c19fa-753e-49c9-9b71-ca36ef4dd9bc-kube-api-access-ltk2t\") pod \"error-404-isvc-70137-predictor-544d7497cf-k6qvv\" (UID: \"742c19fa-753e-49c9-9b71-ca36ef4dd9bc\") " pod="kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv"
Jul 15 11:07:48.790583 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.790553    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv"
Jul 15 11:07:48.924531 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:48.924502    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv"]
Jul 15 11:07:48.926865 ip-10-0-131-34 kubenswrapper[2563]: W0715 11:07:48.926837    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod742c19fa_753e_49c9_9b71_ca36ef4dd9bc.slice/crio-f55815ad3c27eb926672bdd87ec4d58c9fb588b8ea01a1d2adb249bb026ad707 WatchSource:0}: Error finding container f55815ad3c27eb926672bdd87ec4d58c9fb588b8ea01a1d2adb249bb026ad707: Status 404 returned error can't find the container with id f55815ad3c27eb926672bdd87ec4d58c9fb588b8ea01a1d2adb249bb026ad707
Jul 15 11:07:49.001735 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:49.001709    2563 generic.go:356] "Generic (PLEG): container finished" podID="89161cbe-9df8-4661-a724-8cef68e2fe35" containerID="2e6894af55706d77ee129f25dabea9f1326cfc35f1b2e37594b12b0a5807243f" exitCode=2
Jul 15 11:07:49.001839 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:49.001783    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc" event={"ID":"89161cbe-9df8-4661-a724-8cef68e2fe35","Type":"ContainerDied","Data":"2e6894af55706d77ee129f25dabea9f1326cfc35f1b2e37594b12b0a5807243f"}
Jul 15 11:07:49.003526 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:49.003506    2563 generic.go:356] "Generic (PLEG): container finished" podID="fa498659-a6ab-4e10-ace6-0400dbd05f68" containerID="56e43b0f5d06ab237e4966f426e2b06d6ef98f78e5da782fcd4350cd22b02a22" exitCode=2
Jul 15 11:07:49.003632 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:49.003578    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6" event={"ID":"fa498659-a6ab-4e10-ace6-0400dbd05f68","Type":"ContainerDied","Data":"56e43b0f5d06ab237e4966f426e2b06d6ef98f78e5da782fcd4350cd22b02a22"}
Jul 15 11:07:49.004781 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:49.004763    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv" event={"ID":"742c19fa-753e-49c9-9b71-ca36ef4dd9bc","Type":"ContainerStarted","Data":"5c99d86138b9491b436533f542830db6bc45f0e201f999b636d96678f5efa0ec"}
Jul 15 11:07:49.004887 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:49.004806    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv" event={"ID":"742c19fa-753e-49c9-9b71-ca36ef4dd9bc","Type":"ContainerStarted","Data":"f55815ad3c27eb926672bdd87ec4d58c9fb588b8ea01a1d2adb249bb026ad707"}
Jul 15 11:07:49.097042 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:49.097008    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d80ad20-8011-4e76-a4d9-2d8335b4e25c-proxy-tls\") pod \"success-200-isvc-70137-predictor-77585cb866-kcdrt\" (UID: \"0d80ad20-8011-4e76-a4d9-2d8335b4e25c\") " pod="kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt"
Jul 15 11:07:49.099346 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:49.099328    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d80ad20-8011-4e76-a4d9-2d8335b4e25c-proxy-tls\") pod \"success-200-isvc-70137-predictor-77585cb866-kcdrt\" (UID: \"0d80ad20-8011-4e76-a4d9-2d8335b4e25c\") " pod="kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt"
Jul 15 11:07:49.287103 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:49.287065    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt"
Jul 15 11:07:49.423341 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:49.423313    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt"]
Jul 15 11:07:49.425664 ip-10-0-131-34 kubenswrapper[2563]: W0715 11:07:49.425640    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d80ad20_8011_4e76_a4d9_2d8335b4e25c.slice/crio-f323b317cd9fbd8da142f41bd41a98c0e49365039993158051c612e544d88d04 WatchSource:0}: Error finding container f323b317cd9fbd8da142f41bd41a98c0e49365039993158051c612e544d88d04: Status 404 returned error can't find the container with id f323b317cd9fbd8da142f41bd41a98c0e49365039993158051c612e544d88d04
Jul 15 11:07:50.010363 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:50.010323    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt" event={"ID":"0d80ad20-8011-4e76-a4d9-2d8335b4e25c","Type":"ContainerStarted","Data":"118d116e7a2d09df1a4179bf54dec9034076b9de957d5062cc8adc07b9128e77"}
Jul 15 11:07:50.010566 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:50.010496    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt"
Jul 15 11:07:50.010566 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:50.010554    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt" event={"ID":"0d80ad20-8011-4e76-a4d9-2d8335b4e25c","Type":"ContainerStarted","Data":"e6f051f0dd5355ab091149a886793fb6a34a8c8f3d3280dd88c6d0185aa687c2"}
Jul 15 11:07:50.010680 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:50.010582    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt" event={"ID":"0d80ad20-8011-4e76-a4d9-2d8335b4e25c","Type":"ContainerStarted","Data":"f323b317cd9fbd8da142f41bd41a98c0e49365039993158051c612e544d88d04"}
Jul 15 11:07:50.012130 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:50.012102    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv" event={"ID":"742c19fa-753e-49c9-9b71-ca36ef4dd9bc","Type":"ContainerStarted","Data":"29272cf85b34b75ed1a22d45366a3f98772ccb4e0ccb280c081509567fc6326d"}
Jul 15 11:07:50.012275 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:50.012259    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv"
Jul 15 11:07:50.031478 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:50.031435    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt" podStartSLOduration=2.031421874 podStartE2EDuration="2.031421874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:07:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:07:50.030132209 +0000 UTC m=+1288.859647614" watchObservedRunningTime="2026-07-15 11:07:50.031421874 +0000 UTC m=+1288.860937282"
Jul 15 11:07:50.048241 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:50.048198    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv" podStartSLOduration=2.048186584 podStartE2EDuration="2.048186584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:07:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:07:50.047108275 +0000 UTC m=+1288.876623681" watchObservedRunningTime="2026-07-15 11:07:50.048186584 +0000 UTC m=+1288.877702033"
Jul 15 11:07:51.016610 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:51.016574    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt"
Jul 15 11:07:51.017027 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:51.016640    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv"
Jul 15 11:07:51.017562 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:51.017529    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt" podUID="0d80ad20-8011-4e76-a4d9-2d8335b4e25c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 15 11:07:51.017562 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:51.017545    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv" podUID="742c19fa-753e-49c9-9b71-ca36ef4dd9bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 15 11:07:51.616806 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:51.616784    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6"
Jul 15 11:07:51.668616 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:51.668579    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc" podUID="89161cbe-9df8-4661-a724-8cef68e2fe35" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 15 11:07:51.717860 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:51.717790    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfvtx\" (UniqueName: \"kubernetes.io/projected/fa498659-a6ab-4e10-ace6-0400dbd05f68-kube-api-access-zfvtx\") pod \"fa498659-a6ab-4e10-ace6-0400dbd05f68\" (UID: \"fa498659-a6ab-4e10-ace6-0400dbd05f68\") "
Jul 15 11:07:51.717860 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:51.717823    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa498659-a6ab-4e10-ace6-0400dbd05f68-proxy-tls\") pod \"fa498659-a6ab-4e10-ace6-0400dbd05f68\" (UID: \"fa498659-a6ab-4e10-ace6-0400dbd05f68\") "
Jul 15 11:07:51.718086 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:51.717866    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa498659-a6ab-4e10-ace6-0400dbd05f68-kube-rbac-proxy-sar-config\") pod \"fa498659-a6ab-4e10-ace6-0400dbd05f68\" (UID: \"fa498659-a6ab-4e10-ace6-0400dbd05f68\") "
Jul 15 11:07:51.718232 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:51.718208    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa498659-a6ab-4e10-ace6-0400dbd05f68-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fa498659-a6ab-4e10-ace6-0400dbd05f68" (UID: "fa498659-a6ab-4e10-ace6-0400dbd05f68"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 11:07:51.719909 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:51.719884    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa498659-a6ab-4e10-ace6-0400dbd05f68-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fa498659-a6ab-4e10-ace6-0400dbd05f68" (UID: "fa498659-a6ab-4e10-ace6-0400dbd05f68"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:07:51.719909 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:51.719895    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa498659-a6ab-4e10-ace6-0400dbd05f68-kube-api-access-zfvtx" (OuterVolumeSpecName: "kube-api-access-zfvtx") pod "fa498659-a6ab-4e10-ace6-0400dbd05f68" (UID: "fa498659-a6ab-4e10-ace6-0400dbd05f68"). InnerVolumeSpecName "kube-api-access-zfvtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:07:51.819190 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:51.819160    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfvtx\" (UniqueName: \"kubernetes.io/projected/fa498659-a6ab-4e10-ace6-0400dbd05f68-kube-api-access-zfvtx\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:07:51.819190 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:51.819187    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa498659-a6ab-4e10-ace6-0400dbd05f68-proxy-tls\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:07:51.819190 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:51.819196    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa498659-a6ab-4e10-ace6-0400dbd05f68-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:07:52.013566 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:52.013540    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc"
Jul 15 11:07:52.021941 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:52.021916    2563 generic.go:356] "Generic (PLEG): container finished" podID="89161cbe-9df8-4661-a724-8cef68e2fe35" containerID="127f444a6216211469a47bd9a20f1d8c6cc4ad0ec4f30aa421734e11c0134a03" exitCode=0
Jul 15 11:07:52.022298 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:52.021955    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc"
Jul 15 11:07:52.022298 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:52.021998    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc" event={"ID":"89161cbe-9df8-4661-a724-8cef68e2fe35","Type":"ContainerDied","Data":"127f444a6216211469a47bd9a20f1d8c6cc4ad0ec4f30aa421734e11c0134a03"}
Jul 15 11:07:52.022298 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:52.022105    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc" event={"ID":"89161cbe-9df8-4661-a724-8cef68e2fe35","Type":"ContainerDied","Data":"85798561e0a8b42a0c00a598cd42fa5a061a2ff54fbc44324115bf15dec1c34c"}
Jul 15 11:07:52.022298 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:52.022125    2563 scope.go:117] "RemoveContainer" containerID="2e6894af55706d77ee129f25dabea9f1326cfc35f1b2e37594b12b0a5807243f"
Jul 15 11:07:52.023563 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:52.023544    2563 generic.go:356] "Generic (PLEG): container finished" podID="fa498659-a6ab-4e10-ace6-0400dbd05f68" containerID="9782b20ac2c597226ff37b84fad82ea49b9aada866846be55a3135acea420a82" exitCode=0
Jul 15 11:07:52.023650 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:52.023609    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6"
Jul 15 11:07:52.023650 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:52.023632    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6" event={"ID":"fa498659-a6ab-4e10-ace6-0400dbd05f68","Type":"ContainerDied","Data":"9782b20ac2c597226ff37b84fad82ea49b9aada866846be55a3135acea420a82"}
Jul 15 11:07:52.023750 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:52.023696    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6" event={"ID":"fa498659-a6ab-4e10-ace6-0400dbd05f68","Type":"ContainerDied","Data":"6dc9ecef2f013f5fdc26f5d2653164d68f5dd61f133ccee7604b4bcc68d899c9"}
Jul 15 11:07:52.024002 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:52.023970    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt" podUID="0d80ad20-8011-4e76-a4d9-2d8335b4e25c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 15 11:07:52.024344 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:52.024320    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv" podUID="742c19fa-753e-49c9-9b71-ca36ef4dd9bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 15 11:07:52.030078 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:52.030062    2563 scope.go:117] "RemoveContainer" containerID="127f444a6216211469a47bd9a20f1d8c6cc4ad0ec4f30aa421734e11c0134a03"
Jul 15 11:07:52.039406 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:52.039388    2563 scope.go:117] "RemoveContainer" containerID="2e6894af55706d77ee129f25dabea9f1326cfc35f1b2e37594b12b0a5807243f"
Jul 15 11:07:52.039676 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:07:52.039649    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e6894af55706d77ee129f25dabea9f1326cfc35f1b2e37594b12b0a5807243f\": container with ID starting with 2e6894af55706d77ee129f25dabea9f1326cfc35f1b2e37594b12b0a5807243f not found: ID does not exist" containerID="2e6894af55706d77ee129f25dabea9f1326cfc35f1b2e37594b12b0a5807243f"
Jul 15 11:07:52.039782 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:52.039680    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e6894af55706d77ee129f25dabea9f1326cfc35f1b2e37594b12b0a5807243f"} err="failed to get container status \"2e6894af55706d77ee129f25dabea9f1326cfc35f1b2e37594b12b0a5807243f\": rpc error: code = NotFound desc = could not find container \"2e6894af55706d77ee129f25dabea9f1326cfc35f1b2e37594b12b0a5807243f\": container with ID starting with 2e6894af55706d77ee129f25dabea9f1326cfc35f1b2e37594b12b0a5807243f not found: ID does not exist"
Jul 15 11:07:52.039782 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:52.039702    2563 scope.go:117] "RemoveContainer" containerID="127f444a6216211469a47bd9a20f1d8c6cc4ad0ec4f30aa421734e11c0134a03"
Jul 15 11:07:52.039931 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:07:52.039913    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"127f444a6216211469a47bd9a20f1d8c6cc4ad0ec4f30aa421734e11c0134a03\": container with ID starting with 127f444a6216211469a47bd9a20f1d8c6cc4ad0ec4f30aa421734e11c0134a03 not found: ID does not exist" containerID="127f444a6216211469a47bd9a20f1d8c6cc4ad0ec4f30aa421734e11c0134a03"
Jul 15 11:07:52.039984 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:52.039933    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"127f444a6216211469a47bd9a20f1d8c6cc4ad0ec4f30aa421734e11c0134a03"} err="failed to get container status \"127f444a6216211469a47bd9a20f1d8c6cc4ad0ec4f30aa421734e11c0134a03\": rpc error: code = NotFound desc = could not find container \"127f444a6216211469a47bd9a20f1d8c6cc4ad0ec4f30aa421734e11c0134a03\": container with ID starting with 127f444a6216211469a47bd9a20f1d8c6cc4ad0ec4f30aa421734e11c0134a03 not found: ID does not exist"
Jul 15 11:07:52.039984 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:52.039946    2563 scope.go:117] "RemoveContainer" containerID="56e43b0f5d06ab237e4966f426e2b06d6ef98f78e5da782fcd4350cd22b02a22"
Jul 15 11:07:52.048379 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:52.048362    2563 scope.go:117] "RemoveContainer" containerID="9782b20ac2c597226ff37b84fad82ea49b9aada866846be55a3135acea420a82"
Jul 15 11:07:52.052627 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:52.052604    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6"]
Jul 15 11:07:52.057275 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:52.057252    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6"]
Jul 15 11:07:52.057397 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:52.057382    2563 scope.go:117] "RemoveContainer" containerID="56e43b0f5d06ab237e4966f426e2b06d6ef98f78e5da782fcd4350cd22b02a22"
Jul 15 11:07:52.057756 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:07:52.057733    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56e43b0f5d06ab237e4966f426e2b06d6ef98f78e5da782fcd4350cd22b02a22\": container with ID starting with 56e43b0f5d06ab237e4966f426e2b06d6ef98f78e5da782fcd4350cd22b02a22 not found: ID does not exist" containerID="56e43b0f5d06ab237e4966f426e2b06d6ef98f78e5da782fcd4350cd22b02a22"
Jul 15 11:07:52.057858 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:52.057761    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56e43b0f5d06ab237e4966f426e2b06d6ef98f78e5da782fcd4350cd22b02a22"} err="failed to get container status \"56e43b0f5d06ab237e4966f426e2b06d6ef98f78e5da782fcd4350cd22b02a22\": rpc error: code = NotFound desc = could not find container \"56e43b0f5d06ab237e4966f426e2b06d6ef98f78e5da782fcd4350cd22b02a22\": container with ID starting with 56e43b0f5d06ab237e4966f426e2b06d6ef98f78e5da782fcd4350cd22b02a22 not found: ID does not exist"
Jul 15 11:07:52.057858 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:52.057778    2563 scope.go:117] "RemoveContainer" containerID="9782b20ac2c597226ff37b84fad82ea49b9aada866846be55a3135acea420a82"
Jul 15 11:07:52.058082 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:07:52.058052    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9782b20ac2c597226ff37b84fad82ea49b9aada866846be55a3135acea420a82\": container with ID starting with 9782b20ac2c597226ff37b84fad82ea49b9aada866846be55a3135acea420a82 not found: ID does not exist" containerID="9782b20ac2c597226ff37b84fad82ea49b9aada866846be55a3135acea420a82"
Jul 15 11:07:52.058171 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:52.058075    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9782b20ac2c597226ff37b84fad82ea49b9aada866846be55a3135acea420a82"} err="failed to get container status \"9782b20ac2c597226ff37b84fad82ea49b9aada866846be55a3135acea420a82\": rpc error: code = NotFound desc = could not find container \"9782b20ac2c597226ff37b84fad82ea49b9aada866846be55a3135acea420a82\": container with ID starting with 9782b20ac2c597226ff37b84fad82ea49b9aada866846be55a3135acea420a82 not found: ID does not exist"
Jul 15 11:07:52.121456 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:52.121423    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89161cbe-9df8-4661-a724-8cef68e2fe35-proxy-tls\") pod \"89161cbe-9df8-4661-a724-8cef68e2fe35\" (UID: \"89161cbe-9df8-4661-a724-8cef68e2fe35\") "
Jul 15 11:07:52.121649 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:52.121488    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccqt7\" (UniqueName: \"kubernetes.io/projected/89161cbe-9df8-4661-a724-8cef68e2fe35-kube-api-access-ccqt7\") pod \"89161cbe-9df8-4661-a724-8cef68e2fe35\" (UID: \"89161cbe-9df8-4661-a724-8cef68e2fe35\") "
Jul 15 11:07:52.121649 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:52.121528    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89161cbe-9df8-4661-a724-8cef68e2fe35-kube-rbac-proxy-sar-config\") pod \"89161cbe-9df8-4661-a724-8cef68e2fe35\" (UID: \"89161cbe-9df8-4661-a724-8cef68e2fe35\") "
Jul 15 11:07:52.121991 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:52.121959    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89161cbe-9df8-4661-a724-8cef68e2fe35-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "89161cbe-9df8-4661-a724-8cef68e2fe35" (UID: "89161cbe-9df8-4661-a724-8cef68e2fe35"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 11:07:52.123645 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:52.123625    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89161cbe-9df8-4661-a724-8cef68e2fe35-kube-api-access-ccqt7" (OuterVolumeSpecName: "kube-api-access-ccqt7") pod "89161cbe-9df8-4661-a724-8cef68e2fe35" (UID: "89161cbe-9df8-4661-a724-8cef68e2fe35"). InnerVolumeSpecName "kube-api-access-ccqt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:07:52.123709 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:52.123647    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89161cbe-9df8-4661-a724-8cef68e2fe35-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "89161cbe-9df8-4661-a724-8cef68e2fe35" (UID: "89161cbe-9df8-4661-a724-8cef68e2fe35"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:07:52.223248 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:52.223216    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89161cbe-9df8-4661-a724-8cef68e2fe35-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:07:52.223248 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:52.223243    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89161cbe-9df8-4661-a724-8cef68e2fe35-proxy-tls\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:07:52.223248 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:52.223256    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccqt7\" (UniqueName: \"kubernetes.io/projected/89161cbe-9df8-4661-a724-8cef68e2fe35-kube-api-access-ccqt7\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:07:52.353277 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:52.353232    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc"]
Jul 15 11:07:52.357095 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:52.357062    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc"]
Jul 15 11:07:53.679989 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:53.679945    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89161cbe-9df8-4661-a724-8cef68e2fe35" path="/var/lib/kubelet/pods/89161cbe-9df8-4661-a724-8cef68e2fe35/volumes"
Jul 15 11:07:53.680574 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:53.680552    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa498659-a6ab-4e10-ace6-0400dbd05f68" path="/var/lib/kubelet/pods/fa498659-a6ab-4e10-ace6-0400dbd05f68/volumes"
Jul 15 11:07:55.847224 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:55.847199    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4"
Jul 15 11:07:56.856351 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:56.856319    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp"
Jul 15 11:07:57.028445 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:57.028415    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv"
Jul 15 11:07:57.028733 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:57.028715    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt"
Jul 15 11:07:57.028827 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:57.028800    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv" podUID="742c19fa-753e-49c9-9b71-ca36ef4dd9bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 15 11:07:57.029136 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:07:57.029113    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt" podUID="0d80ad20-8011-4e76-a4d9-2d8335b4e25c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 15 11:08:07.029275 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:08:07.029234    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt" podUID="0d80ad20-8011-4e76-a4d9-2d8335b4e25c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 15 11:08:07.029735 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:08:07.029234    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv" podUID="742c19fa-753e-49c9-9b71-ca36ef4dd9bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 15 11:08:17.028906 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:08:17.028867    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv" podUID="742c19fa-753e-49c9-9b71-ca36ef4dd9bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 15 11:08:17.029384 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:08:17.029208    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt" podUID="0d80ad20-8011-4e76-a4d9-2d8335b4e25c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 15 11:08:27.029786 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:08:27.029737    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv" podUID="742c19fa-753e-49c9-9b71-ca36ef4dd9bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 15 11:08:27.030286 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:08:27.029746    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt" podUID="0d80ad20-8011-4e76-a4d9-2d8335b4e25c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 15 11:08:37.030102 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:08:37.030076    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt"
Jul 15 11:08:37.030503 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:08:37.030169    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv"
Jul 15 11:11:21.616173 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:11:21.616123    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk49n_beb68899-ffb3-436f-8e17-43597aad4030/ovn-acl-logging/0.log"
Jul 15 11:11:21.619420 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:11:21.619393    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk49n_beb68899-ffb3-436f-8e17-43597aad4030/ovn-acl-logging/0.log"
Jul 15 11:16:21.642775 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:21.642735    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk49n_beb68899-ffb3-436f-8e17-43597aad4030/ovn-acl-logging/0.log"
Jul 15 11:16:21.647676 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:21.647655    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk49n_beb68899-ffb3-436f-8e17-43597aad4030/ovn-acl-logging/0.log"
Jul 15 11:16:23.055269 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.055231    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp"]
Jul 15 11:16:23.055710 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.055503    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp" podUID="84640095-4359-4999-b3c9-7d7602b8daca" containerName="kserve-container" containerID="cri-o://ab43859a6b02667a1ea2a1cff9cdb6a8711a09398d14cec0daf55ff2127bb740" gracePeriod=30
Jul 15 11:16:23.055710 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.055571    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp" podUID="84640095-4359-4999-b3c9-7d7602b8daca" containerName="kube-rbac-proxy" containerID="cri-o://d6b6ecf471a9b35ef4311eb2ac7c8f9143f602a44ce075f30ff916764938ef3d" gracePeriod=30
Jul 15 11:16:23.132424 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.132385    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd"]
Jul 15 11:16:23.132785 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.132768    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89161cbe-9df8-4661-a724-8cef68e2fe35" containerName="kube-rbac-proxy"
Jul 15 11:16:23.132864 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.132787    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="89161cbe-9df8-4661-a724-8cef68e2fe35" containerName="kube-rbac-proxy"
Jul 15 11:16:23.132864 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.132802    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa498659-a6ab-4e10-ace6-0400dbd05f68" containerName="kserve-container"
Jul 15 11:16:23.132864 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.132809    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa498659-a6ab-4e10-ace6-0400dbd05f68" containerName="kserve-container"
Jul 15 11:16:23.132864 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.132817    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89161cbe-9df8-4661-a724-8cef68e2fe35" containerName="kserve-container"
Jul 15 11:16:23.132864 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.132823    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="89161cbe-9df8-4661-a724-8cef68e2fe35" containerName="kserve-container"
Jul 15 11:16:23.132864 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.132834    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa498659-a6ab-4e10-ace6-0400dbd05f68" containerName="kube-rbac-proxy"
Jul 15 11:16:23.132864 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.132839    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa498659-a6ab-4e10-ace6-0400dbd05f68" containerName="kube-rbac-proxy"
Jul 15 11:16:23.133228 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.132897    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="89161cbe-9df8-4661-a724-8cef68e2fe35" containerName="kube-rbac-proxy"
Jul 15 11:16:23.133228 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.132904    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa498659-a6ab-4e10-ace6-0400dbd05f68" containerName="kserve-container"
Jul 15 11:16:23.133228 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.132910    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="89161cbe-9df8-4661-a724-8cef68e2fe35" containerName="kserve-container"
Jul 15 11:16:23.133228 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.132917    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa498659-a6ab-4e10-ace6-0400dbd05f68" containerName="kube-rbac-proxy"
Jul 15 11:16:23.136218 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.136195    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd"
Jul 15 11:16:23.138885 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.138859    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-cdb7a-predictor-serving-cert\""
Jul 15 11:16:23.140089 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.140046    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-cdb7a-kube-rbac-proxy-sar-config\""
Jul 15 11:16:23.145936 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.145906    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd"]
Jul 15 11:16:23.165668 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.165638    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4"]
Jul 15 11:16:23.165892 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.165850    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4" podUID="92cf56c4-3843-4b76-9352-acc8bf60b6a8" containerName="kserve-container" containerID="cri-o://abbcb180eb57de7d21d5df80117c534dedb8f3f7bbbfa239091f8070db1ae06b" gracePeriod=30
Jul 15 11:16:23.165976 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.165886    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4" podUID="92cf56c4-3843-4b76-9352-acc8bf60b6a8" containerName="kube-rbac-proxy" containerID="cri-o://9aef39621580411e7ceca1ff1dce68ef5f15c49a69463a113ed28dcbfb81cafc" gracePeriod=30
Jul 15 11:16:23.168859 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.168829    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99dbd1d6-1f44-478c-a821-f786760dedd8-proxy-tls\") pod \"success-200-isvc-cdb7a-predictor-54d956f49b-dzphd\" (UID: \"99dbd1d6-1f44-478c-a821-f786760dedd8\") " pod="kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd"
Jul 15 11:16:23.168967 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.168897    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg7bd\" (UniqueName: \"kubernetes.io/projected/99dbd1d6-1f44-478c-a821-f786760dedd8-kube-api-access-gg7bd\") pod \"success-200-isvc-cdb7a-predictor-54d956f49b-dzphd\" (UID: \"99dbd1d6-1f44-478c-a821-f786760dedd8\") " pod="kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd"
Jul 15 11:16:23.168967 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.168957    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99dbd1d6-1f44-478c-a821-f786760dedd8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-cdb7a-predictor-54d956f49b-dzphd\" (UID: \"99dbd1d6-1f44-478c-a821-f786760dedd8\") " pod="kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd"
Jul 15 11:16:23.192861 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.192834    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs"]
Jul 15 11:16:23.195971 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.195955    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs"
Jul 15 11:16:23.200107 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.200082    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-cdb7a-predictor-serving-cert\""
Jul 15 11:16:23.200239 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.200189    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-cdb7a-kube-rbac-proxy-sar-config\""
Jul 15 11:16:23.203731 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.203708    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs"]
Jul 15 11:16:23.270220 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.270191    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99dbd1d6-1f44-478c-a821-f786760dedd8-proxy-tls\") pod \"success-200-isvc-cdb7a-predictor-54d956f49b-dzphd\" (UID: \"99dbd1d6-1f44-478c-a821-f786760dedd8\") " pod="kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd"
Jul 15 11:16:23.270384 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.270234    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg7bd\" (UniqueName: \"kubernetes.io/projected/99dbd1d6-1f44-478c-a821-f786760dedd8-kube-api-access-gg7bd\") pod \"success-200-isvc-cdb7a-predictor-54d956f49b-dzphd\" (UID: \"99dbd1d6-1f44-478c-a821-f786760dedd8\") " pod="kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd"
Jul 15 11:16:23.270384 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.270261    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjfl8\" (UniqueName: \"kubernetes.io/projected/148f5d1c-e482-441e-8420-cdc14896673b-kube-api-access-rjfl8\") pod \"error-404-isvc-cdb7a-predictor-66db674dcc-48hvs\" (UID: \"148f5d1c-e482-441e-8420-cdc14896673b\") " pod="kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs"
Jul 15 11:16:23.270384 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.270291    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/148f5d1c-e482-441e-8420-cdc14896673b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-cdb7a-predictor-66db674dcc-48hvs\" (UID: \"148f5d1c-e482-441e-8420-cdc14896673b\") " pod="kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs"
Jul 15 11:16:23.270384 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.270342    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99dbd1d6-1f44-478c-a821-f786760dedd8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-cdb7a-predictor-54d956f49b-dzphd\" (UID: \"99dbd1d6-1f44-478c-a821-f786760dedd8\") " pod="kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd"
Jul 15 11:16:23.270590 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.270466    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/148f5d1c-e482-441e-8420-cdc14896673b-proxy-tls\") pod \"error-404-isvc-cdb7a-predictor-66db674dcc-48hvs\" (UID: \"148f5d1c-e482-441e-8420-cdc14896673b\") " pod="kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs"
Jul 15 11:16:23.271003 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.270979    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99dbd1d6-1f44-478c-a821-f786760dedd8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-cdb7a-predictor-54d956f49b-dzphd\" (UID: \"99dbd1d6-1f44-478c-a821-f786760dedd8\") " pod="kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd"
Jul 15 11:16:23.272633 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.272611    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99dbd1d6-1f44-478c-a821-f786760dedd8-proxy-tls\") pod \"success-200-isvc-cdb7a-predictor-54d956f49b-dzphd\" (UID: \"99dbd1d6-1f44-478c-a821-f786760dedd8\") " pod="kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd"
Jul 15 11:16:23.279396 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.279374    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg7bd\" (UniqueName: \"kubernetes.io/projected/99dbd1d6-1f44-478c-a821-f786760dedd8-kube-api-access-gg7bd\") pod \"success-200-isvc-cdb7a-predictor-54d956f49b-dzphd\" (UID: \"99dbd1d6-1f44-478c-a821-f786760dedd8\") " pod="kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd"
Jul 15 11:16:23.371167 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.371056    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjfl8\" (UniqueName: \"kubernetes.io/projected/148f5d1c-e482-441e-8420-cdc14896673b-kube-api-access-rjfl8\") pod \"error-404-isvc-cdb7a-predictor-66db674dcc-48hvs\" (UID: \"148f5d1c-e482-441e-8420-cdc14896673b\") " pod="kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs"
Jul 15 11:16:23.371167 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.371099    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/148f5d1c-e482-441e-8420-cdc14896673b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-cdb7a-predictor-66db674dcc-48hvs\" (UID: \"148f5d1c-e482-441e-8420-cdc14896673b\") " pod="kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs"
Jul 15 11:16:23.371399 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.371182    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/148f5d1c-e482-441e-8420-cdc14896673b-proxy-tls\") pod \"error-404-isvc-cdb7a-predictor-66db674dcc-48hvs\" (UID: \"148f5d1c-e482-441e-8420-cdc14896673b\") " pod="kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs"
Jul 15 11:16:23.371740 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.371722    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/148f5d1c-e482-441e-8420-cdc14896673b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-cdb7a-predictor-66db674dcc-48hvs\" (UID: \"148f5d1c-e482-441e-8420-cdc14896673b\") " pod="kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs"
Jul 15 11:16:23.373448 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.373430    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/148f5d1c-e482-441e-8420-cdc14896673b-proxy-tls\") pod \"error-404-isvc-cdb7a-predictor-66db674dcc-48hvs\" (UID: \"148f5d1c-e482-441e-8420-cdc14896673b\") " pod="kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs"
Jul 15 11:16:23.381355 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.381329    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjfl8\" (UniqueName: \"kubernetes.io/projected/148f5d1c-e482-441e-8420-cdc14896673b-kube-api-access-rjfl8\") pod \"error-404-isvc-cdb7a-predictor-66db674dcc-48hvs\" (UID: \"148f5d1c-e482-441e-8420-cdc14896673b\") " pod="kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs"
Jul 15 11:16:23.447963 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.447932    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd"
Jul 15 11:16:23.506768 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.506488    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs"
Jul 15 11:16:23.594975 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.594949    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd"]
Jul 15 11:16:23.597898 ip-10-0-131-34 kubenswrapper[2563]: W0715 11:16:23.597764    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99dbd1d6_1f44_478c_a821_f786760dedd8.slice/crio-99327dc06980449f971fe9fc77d0421194c40d139be21d879a55b08f2959ad27 WatchSource:0}: Error finding container 99327dc06980449f971fe9fc77d0421194c40d139be21d879a55b08f2959ad27: Status 404 returned error can't find the container with id 99327dc06980449f971fe9fc77d0421194c40d139be21d879a55b08f2959ad27
Jul 15 11:16:23.654252 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.654192    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs"]
Jul 15 11:16:23.657580 ip-10-0-131-34 kubenswrapper[2563]: W0715 11:16:23.657544    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod148f5d1c_e482_441e_8420_cdc14896673b.slice/crio-9b88b7d7b111720f31bb23a51e92d22db73f39b020da06666afd3265ac42e343 WatchSource:0}: Error finding container 9b88b7d7b111720f31bb23a51e92d22db73f39b020da06666afd3265ac42e343: Status 404 returned error can't find the container with id 9b88b7d7b111720f31bb23a51e92d22db73f39b020da06666afd3265ac42e343
Jul 15 11:16:23.993128 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.993095    2563 generic.go:356] "Generic (PLEG): container finished" podID="92cf56c4-3843-4b76-9352-acc8bf60b6a8" containerID="9aef39621580411e7ceca1ff1dce68ef5f15c49a69463a113ed28dcbfb81cafc" exitCode=2
Jul 15 11:16:23.993308 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.993185    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4" event={"ID":"92cf56c4-3843-4b76-9352-acc8bf60b6a8","Type":"ContainerDied","Data":"9aef39621580411e7ceca1ff1dce68ef5f15c49a69463a113ed28dcbfb81cafc"}
Jul 15 11:16:23.994859 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.994834    2563 generic.go:356] "Generic (PLEG): container finished" podID="84640095-4359-4999-b3c9-7d7602b8daca" containerID="d6b6ecf471a9b35ef4311eb2ac7c8f9143f602a44ce075f30ff916764938ef3d" exitCode=2
Jul 15 11:16:23.994977 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.994902    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp" event={"ID":"84640095-4359-4999-b3c9-7d7602b8daca","Type":"ContainerDied","Data":"d6b6ecf471a9b35ef4311eb2ac7c8f9143f602a44ce075f30ff916764938ef3d"}
Jul 15 11:16:23.996593 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.996568    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs" event={"ID":"148f5d1c-e482-441e-8420-cdc14896673b","Type":"ContainerStarted","Data":"0282d38312f5fa3db48231b514b0a4e89e1f9550f3167d7ba6c7d0200fd81f11"}
Jul 15 11:16:23.996689 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.996620    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs" event={"ID":"148f5d1c-e482-441e-8420-cdc14896673b","Type":"ContainerStarted","Data":"41f5b948fe28a158e5f45d598b24e77ae70245e73deeaeb7ebfe54072d17c642"}
Jul 15 11:16:23.996689 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.996651    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs" event={"ID":"148f5d1c-e482-441e-8420-cdc14896673b","Type":"ContainerStarted","Data":"9b88b7d7b111720f31bb23a51e92d22db73f39b020da06666afd3265ac42e343"}
Jul 15 11:16:23.996806 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.996765    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs"
Jul 15 11:16:23.998260 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.998240    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd" event={"ID":"99dbd1d6-1f44-478c-a821-f786760dedd8","Type":"ContainerStarted","Data":"c08c81cfde7adc2769db1de0b543cdea84f0ab79853dd4a9e13650e6d6448c61"}
Jul 15 11:16:23.998359 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.998290    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd" event={"ID":"99dbd1d6-1f44-478c-a821-f786760dedd8","Type":"ContainerStarted","Data":"7f3c8738ffa162f3bd235c791af2f6839d31043525181f22af2bb501c30f02c5"}
Jul 15 11:16:23.998359 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.998322    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd" event={"ID":"99dbd1d6-1f44-478c-a821-f786760dedd8","Type":"ContainerStarted","Data":"99327dc06980449f971fe9fc77d0421194c40d139be21d879a55b08f2959ad27"}
Jul 15 11:16:23.998470 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:23.998358    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd"
Jul 15 11:16:24.019545 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:24.019504    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs" podStartSLOduration=1.019491378 podStartE2EDuration="1.019491378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:16:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:16:24.019044197 +0000 UTC m=+1802.848559601" watchObservedRunningTime="2026-07-15 11:16:24.019491378 +0000 UTC m=+1802.849006783"
Jul 15 11:16:24.040721 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:24.040681    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd" podStartSLOduration=1.040667399 podStartE2EDuration="1.040667399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:16:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:16:24.039128902 +0000 UTC m=+1802.868644309" watchObservedRunningTime="2026-07-15 11:16:24.040667399 +0000 UTC m=+1802.870182804"
Jul 15 11:16:25.003048 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:25.003017    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs"
Jul 15 11:16:25.003567 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:25.003083    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd"
Jul 15 11:16:25.004000 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:25.003972    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs" podUID="148f5d1c-e482-441e-8420-cdc14896673b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 15 11:16:25.004110 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:25.004036    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd" podUID="99dbd1d6-1f44-478c-a821-f786760dedd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 15 11:16:25.841449 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:25.841411    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4" podUID="92cf56c4-3843-4b76-9352-acc8bf60b6a8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Jul 15 11:16:25.846854 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:25.846824    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4" podUID="92cf56c4-3843-4b76-9352-acc8bf60b6a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 15 11:16:26.012883 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:26.012838    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs" podUID="148f5d1c-e482-441e-8420-cdc14896673b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 15 11:16:26.013299 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:26.012910    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd" podUID="99dbd1d6-1f44-478c-a821-f786760dedd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 15 11:16:26.414642 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:26.414617    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp"
Jul 15 11:16:26.494987 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:26.494957    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84640095-4359-4999-b3c9-7d7602b8daca-kube-rbac-proxy-sar-config\") pod \"84640095-4359-4999-b3c9-7d7602b8daca\" (UID: \"84640095-4359-4999-b3c9-7d7602b8daca\") "
Jul 15 11:16:26.495186 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:26.495016    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84640095-4359-4999-b3c9-7d7602b8daca-proxy-tls\") pod \"84640095-4359-4999-b3c9-7d7602b8daca\" (UID: \"84640095-4359-4999-b3c9-7d7602b8daca\") "
Jul 15 11:16:26.495186 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:26.495038    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qf4cl\" (UniqueName: \"kubernetes.io/projected/84640095-4359-4999-b3c9-7d7602b8daca-kube-api-access-qf4cl\") pod \"84640095-4359-4999-b3c9-7d7602b8daca\" (UID: \"84640095-4359-4999-b3c9-7d7602b8daca\") "
Jul 15 11:16:26.495378 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:26.495351    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84640095-4359-4999-b3c9-7d7602b8daca-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "84640095-4359-4999-b3c9-7d7602b8daca" (UID: "84640095-4359-4999-b3c9-7d7602b8daca"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 11:16:26.497125 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:26.497099    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84640095-4359-4999-b3c9-7d7602b8daca-kube-api-access-qf4cl" (OuterVolumeSpecName: "kube-api-access-qf4cl") pod "84640095-4359-4999-b3c9-7d7602b8daca" (UID: "84640095-4359-4999-b3c9-7d7602b8daca"). InnerVolumeSpecName "kube-api-access-qf4cl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:16:26.497125 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:26.497107    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84640095-4359-4999-b3c9-7d7602b8daca-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "84640095-4359-4999-b3c9-7d7602b8daca" (UID: "84640095-4359-4999-b3c9-7d7602b8daca"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:16:26.595885 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:26.595792    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84640095-4359-4999-b3c9-7d7602b8daca-proxy-tls\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:16:26.595885 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:26.595827    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf4cl\" (UniqueName: \"kubernetes.io/projected/84640095-4359-4999-b3c9-7d7602b8daca-kube-api-access-qf4cl\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:16:26.595885 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:26.595837    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84640095-4359-4999-b3c9-7d7602b8daca-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:16:27.018004 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:27.017971    2563 generic.go:356] "Generic (PLEG): container finished" podID="84640095-4359-4999-b3c9-7d7602b8daca" containerID="ab43859a6b02667a1ea2a1cff9cdb6a8711a09398d14cec0daf55ff2127bb740" exitCode=0
Jul 15 11:16:27.018427 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:27.018021    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp"
Jul 15 11:16:27.018427 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:27.018062    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp" event={"ID":"84640095-4359-4999-b3c9-7d7602b8daca","Type":"ContainerDied","Data":"ab43859a6b02667a1ea2a1cff9cdb6a8711a09398d14cec0daf55ff2127bb740"}
Jul 15 11:16:27.018427 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:27.018116    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp" event={"ID":"84640095-4359-4999-b3c9-7d7602b8daca","Type":"ContainerDied","Data":"a5b72f2b5c3405903dc04ac9645c07b4e95afce8e0b951784222ea9f2ea22c40"}
Jul 15 11:16:27.018427 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:27.018192    2563 scope.go:117] "RemoveContainer" containerID="d6b6ecf471a9b35ef4311eb2ac7c8f9143f602a44ce075f30ff916764938ef3d"
Jul 15 11:16:27.027782 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:27.027755    2563 scope.go:117] "RemoveContainer" containerID="ab43859a6b02667a1ea2a1cff9cdb6a8711a09398d14cec0daf55ff2127bb740"
Jul 15 11:16:27.035292 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:27.035274    2563 scope.go:117] "RemoveContainer" containerID="d6b6ecf471a9b35ef4311eb2ac7c8f9143f602a44ce075f30ff916764938ef3d"
Jul 15 11:16:27.035529 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:16:27.035508    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6b6ecf471a9b35ef4311eb2ac7c8f9143f602a44ce075f30ff916764938ef3d\": container with ID starting with d6b6ecf471a9b35ef4311eb2ac7c8f9143f602a44ce075f30ff916764938ef3d not found: ID does not exist" containerID="d6b6ecf471a9b35ef4311eb2ac7c8f9143f602a44ce075f30ff916764938ef3d"
Jul 15 11:16:27.035577 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:27.035538    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6b6ecf471a9b35ef4311eb2ac7c8f9143f602a44ce075f30ff916764938ef3d"} err="failed to get container status \"d6b6ecf471a9b35ef4311eb2ac7c8f9143f602a44ce075f30ff916764938ef3d\": rpc error: code = NotFound desc = could not find container \"d6b6ecf471a9b35ef4311eb2ac7c8f9143f602a44ce075f30ff916764938ef3d\": container with ID starting with d6b6ecf471a9b35ef4311eb2ac7c8f9143f602a44ce075f30ff916764938ef3d not found: ID does not exist"
Jul 15 11:16:27.035577 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:27.035555    2563 scope.go:117] "RemoveContainer" containerID="ab43859a6b02667a1ea2a1cff9cdb6a8711a09398d14cec0daf55ff2127bb740"
Jul 15 11:16:27.035797 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:16:27.035779    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab43859a6b02667a1ea2a1cff9cdb6a8711a09398d14cec0daf55ff2127bb740\": container with ID starting with ab43859a6b02667a1ea2a1cff9cdb6a8711a09398d14cec0daf55ff2127bb740 not found: ID does not exist" containerID="ab43859a6b02667a1ea2a1cff9cdb6a8711a09398d14cec0daf55ff2127bb740"
Jul 15 11:16:27.035836 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:27.035803    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab43859a6b02667a1ea2a1cff9cdb6a8711a09398d14cec0daf55ff2127bb740"} err="failed to get container status \"ab43859a6b02667a1ea2a1cff9cdb6a8711a09398d14cec0daf55ff2127bb740\": rpc error: code = NotFound desc = could not find container \"ab43859a6b02667a1ea2a1cff9cdb6a8711a09398d14cec0daf55ff2127bb740\": container with ID starting with ab43859a6b02667a1ea2a1cff9cdb6a8711a09398d14cec0daf55ff2127bb740 not found: ID does not exist"
Jul 15 11:16:27.044444 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:27.044422    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp"]
Jul 15 11:16:27.050211 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:27.050186    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp"]
Jul 15 11:16:27.200057 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:27.200036    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4"
Jul 15 11:16:27.301392 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:27.301313    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92cf56c4-3843-4b76-9352-acc8bf60b6a8-proxy-tls\") pod \"92cf56c4-3843-4b76-9352-acc8bf60b6a8\" (UID: \"92cf56c4-3843-4b76-9352-acc8bf60b6a8\") "
Jul 15 11:16:27.301392 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:27.301362    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92cf56c4-3843-4b76-9352-acc8bf60b6a8-kube-rbac-proxy-sar-config\") pod \"92cf56c4-3843-4b76-9352-acc8bf60b6a8\" (UID: \"92cf56c4-3843-4b76-9352-acc8bf60b6a8\") "
Jul 15 11:16:27.301392 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:27.301380    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkj6n\" (UniqueName: \"kubernetes.io/projected/92cf56c4-3843-4b76-9352-acc8bf60b6a8-kube-api-access-xkj6n\") pod \"92cf56c4-3843-4b76-9352-acc8bf60b6a8\" (UID: \"92cf56c4-3843-4b76-9352-acc8bf60b6a8\") "
Jul 15 11:16:27.301788 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:27.301764    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92cf56c4-3843-4b76-9352-acc8bf60b6a8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "92cf56c4-3843-4b76-9352-acc8bf60b6a8" (UID: "92cf56c4-3843-4b76-9352-acc8bf60b6a8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 11:16:27.303477 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:27.303450    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92cf56c4-3843-4b76-9352-acc8bf60b6a8-kube-api-access-xkj6n" (OuterVolumeSpecName: "kube-api-access-xkj6n") pod "92cf56c4-3843-4b76-9352-acc8bf60b6a8" (UID: "92cf56c4-3843-4b76-9352-acc8bf60b6a8"). InnerVolumeSpecName "kube-api-access-xkj6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:16:27.303567 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:27.303484    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92cf56c4-3843-4b76-9352-acc8bf60b6a8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "92cf56c4-3843-4b76-9352-acc8bf60b6a8" (UID: "92cf56c4-3843-4b76-9352-acc8bf60b6a8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:16:27.402086 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:27.402046    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92cf56c4-3843-4b76-9352-acc8bf60b6a8-proxy-tls\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:16:27.402086 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:27.402078    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92cf56c4-3843-4b76-9352-acc8bf60b6a8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:16:27.402086 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:27.402091    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkj6n\" (UniqueName: \"kubernetes.io/projected/92cf56c4-3843-4b76-9352-acc8bf60b6a8-kube-api-access-xkj6n\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:16:27.678125 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:27.678025    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84640095-4359-4999-b3c9-7d7602b8daca" path="/var/lib/kubelet/pods/84640095-4359-4999-b3c9-7d7602b8daca/volumes"
Jul 15 11:16:28.023848 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:28.023813    2563 generic.go:356] "Generic (PLEG): container finished" podID="92cf56c4-3843-4b76-9352-acc8bf60b6a8" containerID="abbcb180eb57de7d21d5df80117c534dedb8f3f7bbbfa239091f8070db1ae06b" exitCode=0
Jul 15 11:16:28.024354 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:28.023865    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4"
Jul 15 11:16:28.024354 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:28.023894    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4" event={"ID":"92cf56c4-3843-4b76-9352-acc8bf60b6a8","Type":"ContainerDied","Data":"abbcb180eb57de7d21d5df80117c534dedb8f3f7bbbfa239091f8070db1ae06b"}
Jul 15 11:16:28.024354 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:28.023969    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4" event={"ID":"92cf56c4-3843-4b76-9352-acc8bf60b6a8","Type":"ContainerDied","Data":"476aa45c2c6cde9d78c2e1b7a9070c473492d9e1570c1348f152653dd85b6bf0"}
Jul 15 11:16:28.024354 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:28.024012    2563 scope.go:117] "RemoveContainer" containerID="9aef39621580411e7ceca1ff1dce68ef5f15c49a69463a113ed28dcbfb81cafc"
Jul 15 11:16:28.032292 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:28.032266    2563 scope.go:117] "RemoveContainer" containerID="abbcb180eb57de7d21d5df80117c534dedb8f3f7bbbfa239091f8070db1ae06b"
Jul 15 11:16:28.041261 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:28.041246    2563 scope.go:117] "RemoveContainer" containerID="9aef39621580411e7ceca1ff1dce68ef5f15c49a69463a113ed28dcbfb81cafc"
Jul 15 11:16:28.041512 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:16:28.041496    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aef39621580411e7ceca1ff1dce68ef5f15c49a69463a113ed28dcbfb81cafc\": container with ID starting with 9aef39621580411e7ceca1ff1dce68ef5f15c49a69463a113ed28dcbfb81cafc not found: ID does not exist" containerID="9aef39621580411e7ceca1ff1dce68ef5f15c49a69463a113ed28dcbfb81cafc"
Jul 15 11:16:28.041580 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:28.041519    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aef39621580411e7ceca1ff1dce68ef5f15c49a69463a113ed28dcbfb81cafc"} err="failed to get container status \"9aef39621580411e7ceca1ff1dce68ef5f15c49a69463a113ed28dcbfb81cafc\": rpc error: code = NotFound desc = could not find container \"9aef39621580411e7ceca1ff1dce68ef5f15c49a69463a113ed28dcbfb81cafc\": container with ID starting with 9aef39621580411e7ceca1ff1dce68ef5f15c49a69463a113ed28dcbfb81cafc not found: ID does not exist"
Jul 15 11:16:28.041580 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:28.041537    2563 scope.go:117] "RemoveContainer" containerID="abbcb180eb57de7d21d5df80117c534dedb8f3f7bbbfa239091f8070db1ae06b"
Jul 15 11:16:28.041738 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:16:28.041719    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abbcb180eb57de7d21d5df80117c534dedb8f3f7bbbfa239091f8070db1ae06b\": container with ID starting with abbcb180eb57de7d21d5df80117c534dedb8f3f7bbbfa239091f8070db1ae06b not found: ID does not exist" containerID="abbcb180eb57de7d21d5df80117c534dedb8f3f7bbbfa239091f8070db1ae06b"
Jul 15 11:16:28.041778 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:28.041741    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abbcb180eb57de7d21d5df80117c534dedb8f3f7bbbfa239091f8070db1ae06b"} err="failed to get container status \"abbcb180eb57de7d21d5df80117c534dedb8f3f7bbbfa239091f8070db1ae06b\": rpc error: code = NotFound desc = could not find container \"abbcb180eb57de7d21d5df80117c534dedb8f3f7bbbfa239091f8070db1ae06b\": container with ID starting with abbcb180eb57de7d21d5df80117c534dedb8f3f7bbbfa239091f8070db1ae06b not found: ID does not exist"
Jul 15 11:16:28.047580 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:28.047557    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4"]
Jul 15 11:16:28.052685 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:28.052661    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4"]
Jul 15 11:16:29.677352 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:29.677269    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92cf56c4-3843-4b76-9352-acc8bf60b6a8" path="/var/lib/kubelet/pods/92cf56c4-3843-4b76-9352-acc8bf60b6a8/volumes"
Jul 15 11:16:31.017577 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:31.017548    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs"
Jul 15 11:16:31.018015 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:31.017852    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd"
Jul 15 11:16:31.018015 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:31.017946    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs" podUID="148f5d1c-e482-441e-8420-cdc14896673b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 15 11:16:31.018350 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:31.018320    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd" podUID="99dbd1d6-1f44-478c-a821-f786760dedd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 15 11:16:41.018607 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:41.018565    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd" podUID="99dbd1d6-1f44-478c-a821-f786760dedd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 15 11:16:41.018607 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:41.018565    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs" podUID="148f5d1c-e482-441e-8420-cdc14896673b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 15 11:16:51.018344 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:51.018306    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs" podUID="148f5d1c-e482-441e-8420-cdc14896673b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 15 11:16:51.018792 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:16:51.018306    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd" podUID="99dbd1d6-1f44-478c-a821-f786760dedd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 15 11:17:01.018130 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:01.018083    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs" podUID="148f5d1c-e482-441e-8420-cdc14896673b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 15 11:17:01.018650 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:01.018262    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd" podUID="99dbd1d6-1f44-478c-a821-f786760dedd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 15 11:17:03.133739 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.133703    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv"]
Jul 15 11:17:03.134131 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.133924    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv" podUID="742c19fa-753e-49c9-9b71-ca36ef4dd9bc" containerName="kserve-container" containerID="cri-o://5c99d86138b9491b436533f542830db6bc45f0e201f999b636d96678f5efa0ec" gracePeriod=30
Jul 15 11:17:03.134131 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.134014    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv" podUID="742c19fa-753e-49c9-9b71-ca36ef4dd9bc" containerName="kube-rbac-proxy" containerID="cri-o://29272cf85b34b75ed1a22d45366a3f98772ccb4e0ccb280c081509567fc6326d" gracePeriod=30
Jul 15 11:17:03.180509 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.180473    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt"]
Jul 15 11:17:03.180767 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.180739    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt" podUID="0d80ad20-8011-4e76-a4d9-2d8335b4e25c" containerName="kserve-container" containerID="cri-o://e6f051f0dd5355ab091149a886793fb6a34a8c8f3d3280dd88c6d0185aa687c2" gracePeriod=30
Jul 15 11:17:03.180856 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.180784    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt" podUID="0d80ad20-8011-4e76-a4d9-2d8335b4e25c" containerName="kube-rbac-proxy" containerID="cri-o://118d116e7a2d09df1a4179bf54dec9034076b9de957d5062cc8adc07b9128e77" gracePeriod=30
Jul 15 11:17:03.245541 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.245509    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw"]
Jul 15 11:17:03.245838 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.245827    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92cf56c4-3843-4b76-9352-acc8bf60b6a8" containerName="kserve-container"
Jul 15 11:17:03.245882 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.245840    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="92cf56c4-3843-4b76-9352-acc8bf60b6a8" containerName="kserve-container"
Jul 15 11:17:03.245882 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.245848    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92cf56c4-3843-4b76-9352-acc8bf60b6a8" containerName="kube-rbac-proxy"
Jul 15 11:17:03.245882 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.245853    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="92cf56c4-3843-4b76-9352-acc8bf60b6a8" containerName="kube-rbac-proxy"
Jul 15 11:17:03.245882 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.245872    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84640095-4359-4999-b3c9-7d7602b8daca" containerName="kserve-container"
Jul 15 11:17:03.245882 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.245878    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="84640095-4359-4999-b3c9-7d7602b8daca" containerName="kserve-container"
Jul 15 11:17:03.246037 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.245887    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84640095-4359-4999-b3c9-7d7602b8daca" containerName="kube-rbac-proxy"
Jul 15 11:17:03.246037 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.245892    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="84640095-4359-4999-b3c9-7d7602b8daca" containerName="kube-rbac-proxy"
Jul 15 11:17:03.246037 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.245935    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="92cf56c4-3843-4b76-9352-acc8bf60b6a8" containerName="kube-rbac-proxy"
Jul 15 11:17:03.246037 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.245944    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="84640095-4359-4999-b3c9-7d7602b8daca" containerName="kube-rbac-proxy"
Jul 15 11:17:03.246037 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.245951    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="84640095-4359-4999-b3c9-7d7602b8daca" containerName="kserve-container"
Jul 15 11:17:03.246037 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.245959    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="92cf56c4-3843-4b76-9352-acc8bf60b6a8" containerName="kserve-container"
Jul 15 11:17:03.248749 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.248733    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw"
Jul 15 11:17:03.251276 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.251254    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-bc774-predictor-serving-cert\""
Jul 15 11:17:03.251436 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.251411    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-bc774-kube-rbac-proxy-sar-config\""
Jul 15 11:17:03.259685 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.259659    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw"]
Jul 15 11:17:03.350342 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.350307    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb"]
Jul 15 11:17:03.361458 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.361424    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb"
Jul 15 11:17:03.362966 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.362933    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb"]
Jul 15 11:17:03.364235 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.364213    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-bc774-kube-rbac-proxy-sar-config\""
Jul 15 11:17:03.364378 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.364213    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-bc774-predictor-serving-cert\""
Jul 15 11:17:03.400239 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.400128    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58d5e7d5-59a8-4d71-a937-c32ac57ab1aa-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bc774-predictor-5b6b775cdd-29slw\" (UID: \"58d5e7d5-59a8-4d71-a937-c32ac57ab1aa\") " pod="kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw"
Jul 15 11:17:03.400239 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.400209    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58d5e7d5-59a8-4d71-a937-c32ac57ab1aa-proxy-tls\") pod \"success-200-isvc-bc774-predictor-5b6b775cdd-29slw\" (UID: \"58d5e7d5-59a8-4d71-a937-c32ac57ab1aa\") " pod="kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw"
Jul 15 11:17:03.400438 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.400270    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9ds6\" (UniqueName: \"kubernetes.io/projected/58d5e7d5-59a8-4d71-a937-c32ac57ab1aa-kube-api-access-l9ds6\") pod \"success-200-isvc-bc774-predictor-5b6b775cdd-29slw\" (UID: \"58d5e7d5-59a8-4d71-a937-c32ac57ab1aa\") " pod="kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw"
Jul 15 11:17:03.501391 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.501351    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9ds6\" (UniqueName: \"kubernetes.io/projected/58d5e7d5-59a8-4d71-a937-c32ac57ab1aa-kube-api-access-l9ds6\") pod \"success-200-isvc-bc774-predictor-5b6b775cdd-29slw\" (UID: \"58d5e7d5-59a8-4d71-a937-c32ac57ab1aa\") " pod="kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw"
Jul 15 11:17:03.501391 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.501399    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bc774-predictor-8547545bdf-ktmgb\" (UID: \"e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7\") " pod="kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb"
Jul 15 11:17:03.501623 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.501424    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbbkc\" (UniqueName: \"kubernetes.io/projected/e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7-kube-api-access-kbbkc\") pod \"error-404-isvc-bc774-predictor-8547545bdf-ktmgb\" (UID: \"e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7\") " pod="kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb"
Jul 15 11:17:03.501623 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.501453    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58d5e7d5-59a8-4d71-a937-c32ac57ab1aa-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bc774-predictor-5b6b775cdd-29slw\" (UID: \"58d5e7d5-59a8-4d71-a937-c32ac57ab1aa\") " pod="kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw"
Jul 15 11:17:03.501623 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.501546    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58d5e7d5-59a8-4d71-a937-c32ac57ab1aa-proxy-tls\") pod \"success-200-isvc-bc774-predictor-5b6b775cdd-29slw\" (UID: \"58d5e7d5-59a8-4d71-a937-c32ac57ab1aa\") " pod="kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw"
Jul 15 11:17:03.501623 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.501586    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7-proxy-tls\") pod \"error-404-isvc-bc774-predictor-8547545bdf-ktmgb\" (UID: \"e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7\") " pod="kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb"
Jul 15 11:17:03.502068 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.502048    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58d5e7d5-59a8-4d71-a937-c32ac57ab1aa-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bc774-predictor-5b6b775cdd-29slw\" (UID: \"58d5e7d5-59a8-4d71-a937-c32ac57ab1aa\") " pod="kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw"
Jul 15 11:17:03.503965 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.503949    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58d5e7d5-59a8-4d71-a937-c32ac57ab1aa-proxy-tls\") pod \"success-200-isvc-bc774-predictor-5b6b775cdd-29slw\" (UID: \"58d5e7d5-59a8-4d71-a937-c32ac57ab1aa\") " pod="kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw"
Jul 15 11:17:03.512524 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.512500    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9ds6\" (UniqueName: \"kubernetes.io/projected/58d5e7d5-59a8-4d71-a937-c32ac57ab1aa-kube-api-access-l9ds6\") pod \"success-200-isvc-bc774-predictor-5b6b775cdd-29slw\" (UID: \"58d5e7d5-59a8-4d71-a937-c32ac57ab1aa\") " pod="kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw"
Jul 15 11:17:03.561032 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.560990    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw"
Jul 15 11:17:03.602980 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.602334    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bc774-predictor-8547545bdf-ktmgb\" (UID: \"e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7\") " pod="kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb"
Jul 15 11:17:03.602980 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.602395    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbbkc\" (UniqueName: \"kubernetes.io/projected/e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7-kube-api-access-kbbkc\") pod \"error-404-isvc-bc774-predictor-8547545bdf-ktmgb\" (UID: \"e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7\") " pod="kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb"
Jul 15 11:17:03.602980 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.602477    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7-proxy-tls\") pod \"error-404-isvc-bc774-predictor-8547545bdf-ktmgb\" (UID: \"e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7\") " pod="kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb"
Jul 15 11:17:03.603448 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.603422    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bc774-predictor-8547545bdf-ktmgb\" (UID: \"e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7\") " pod="kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb"
Jul 15 11:17:03.605363 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.605340    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7-proxy-tls\") pod \"error-404-isvc-bc774-predictor-8547545bdf-ktmgb\" (UID: \"e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7\") " pod="kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb"
Jul 15 11:17:03.614317 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.614022    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbbkc\" (UniqueName: \"kubernetes.io/projected/e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7-kube-api-access-kbbkc\") pod \"error-404-isvc-bc774-predictor-8547545bdf-ktmgb\" (UID: \"e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7\") " pod="kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb"
Jul 15 11:17:03.672617 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.672530    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb"
Jul 15 11:17:03.701538 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.701507    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw"]
Jul 15 11:17:03.704756 ip-10-0-131-34 kubenswrapper[2563]: W0715 11:17:03.704723    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58d5e7d5_59a8_4d71_a937_c32ac57ab1aa.slice/crio-a5060027c15816c4808489419de719c29c11705c13955b290530f0deab124947 WatchSource:0}: Error finding container a5060027c15816c4808489419de719c29c11705c13955b290530f0deab124947: Status 404 returned error can't find the container with id a5060027c15816c4808489419de719c29c11705c13955b290530f0deab124947
Jul 15 11:17:03.818754 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:03.818725    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb"]
Jul 15 11:17:03.822232 ip-10-0-131-34 kubenswrapper[2563]: W0715 11:17:03.822208    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4a430db_cad0_4b4b_b3a1_ba9a562b6dc7.slice/crio-60beb3a626d6a0305c4b3c52cb33862e812b58607163d48a2c48c7f4a724c360 WatchSource:0}: Error finding container 60beb3a626d6a0305c4b3c52cb33862e812b58607163d48a2c48c7f4a724c360: Status 404 returned error can't find the container with id 60beb3a626d6a0305c4b3c52cb33862e812b58607163d48a2c48c7f4a724c360
Jul 15 11:17:04.171433 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:04.171399    2563 generic.go:356] "Generic (PLEG): container finished" podID="0d80ad20-8011-4e76-a4d9-2d8335b4e25c" containerID="118d116e7a2d09df1a4179bf54dec9034076b9de957d5062cc8adc07b9128e77" exitCode=2
Jul 15 11:17:04.171834 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:04.171465    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt" event={"ID":"0d80ad20-8011-4e76-a4d9-2d8335b4e25c","Type":"ContainerDied","Data":"118d116e7a2d09df1a4179bf54dec9034076b9de957d5062cc8adc07b9128e77"}
Jul 15 11:17:04.175724 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:04.173502    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb" event={"ID":"e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7","Type":"ContainerStarted","Data":"830630dacbf6de2b32fc5517d3e1f9c877c463a4bf85daa9b4a1d81600dd51d9"}
Jul 15 11:17:04.175724 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:04.173559    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb" event={"ID":"e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7","Type":"ContainerStarted","Data":"5d06730eb1209c5226a30ee888a29ec434bcdd7fb7e3ca13eb22c1c3ce9bf523"}
Jul 15 11:17:04.175724 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:04.173588    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb" event={"ID":"e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7","Type":"ContainerStarted","Data":"60beb3a626d6a0305c4b3c52cb33862e812b58607163d48a2c48c7f4a724c360"}
Jul 15 11:17:04.175724 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:04.174567    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb"
Jul 15 11:17:04.175724 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:04.174614    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb"
Jul 15 11:17:04.176000 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:04.175956    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb" podUID="e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 15 11:17:04.177405 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:04.177383    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw" event={"ID":"58d5e7d5-59a8-4d71-a937-c32ac57ab1aa","Type":"ContainerStarted","Data":"67a49f6f806995b6cd162f3c64a65041901285792766aec56551a07c1647a8a3"}
Jul 15 11:17:04.177525 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:04.177423    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw" event={"ID":"58d5e7d5-59a8-4d71-a937-c32ac57ab1aa","Type":"ContainerStarted","Data":"3485486a9bb2ebaecfd8c21dd9477dbb9ab663ba3bf96417502a35a90b250b69"}
Jul 15 11:17:04.177525 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:04.177447    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw" event={"ID":"58d5e7d5-59a8-4d71-a937-c32ac57ab1aa","Type":"ContainerStarted","Data":"a5060027c15816c4808489419de719c29c11705c13955b290530f0deab124947"}
Jul 15 11:17:04.177525 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:04.177481    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw"
Jul 15 11:17:04.179345 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:04.179324    2563 generic.go:356] "Generic (PLEG): container finished" podID="742c19fa-753e-49c9-9b71-ca36ef4dd9bc" containerID="29272cf85b34b75ed1a22d45366a3f98772ccb4e0ccb280c081509567fc6326d" exitCode=2
Jul 15 11:17:04.179450 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:04.179360    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv" event={"ID":"742c19fa-753e-49c9-9b71-ca36ef4dd9bc","Type":"ContainerDied","Data":"29272cf85b34b75ed1a22d45366a3f98772ccb4e0ccb280c081509567fc6326d"}
Jul 15 11:17:04.193433 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:04.193388    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb" podStartSLOduration=1.193373565 podStartE2EDuration="1.193373565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:17:04.19196091 +0000 UTC m=+1843.021476317" watchObservedRunningTime="2026-07-15 11:17:04.193373565 +0000 UTC m=+1843.022888969"
Jul 15 11:17:04.211047 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:04.211005    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw" podStartSLOduration=1.210990906 podStartE2EDuration="1.210990906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:17:04.209336348 +0000 UTC m=+1843.038851754" watchObservedRunningTime="2026-07-15 11:17:04.210990906 +0000 UTC m=+1843.040506310"
Jul 15 11:17:05.183695 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:05.183664    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw"
Jul 15 11:17:05.184109 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:05.183687    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb" podUID="e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 15 11:17:05.184821 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:05.184795    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw" podUID="58d5e7d5-59a8-4d71-a937-c32ac57ab1aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 15 11:17:06.188153 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:06.188107    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw" podUID="58d5e7d5-59a8-4d71-a937-c32ac57ab1aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 15 11:17:06.188640 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:06.188207    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb" podUID="e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 15 11:17:07.024955 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:07.024908    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt" podUID="0d80ad20-8011-4e76-a4d9-2d8335b4e25c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jul 15 11:17:07.025170 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:07.024908    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv" podUID="742c19fa-753e-49c9-9b71-ca36ef4dd9bc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Jul 15 11:17:07.029428 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:07.029398    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv" podUID="742c19fa-753e-49c9-9b71-ca36ef4dd9bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 15 11:17:07.029522 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:07.029404    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt" podUID="0d80ad20-8011-4e76-a4d9-2d8335b4e25c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 15 11:17:07.195915 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:07.195879    2563 generic.go:356] "Generic (PLEG): container finished" podID="0d80ad20-8011-4e76-a4d9-2d8335b4e25c" containerID="e6f051f0dd5355ab091149a886793fb6a34a8c8f3d3280dd88c6d0185aa687c2" exitCode=0
Jul 15 11:17:07.196394 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:07.195949    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt" event={"ID":"0d80ad20-8011-4e76-a4d9-2d8335b4e25c","Type":"ContainerDied","Data":"e6f051f0dd5355ab091149a886793fb6a34a8c8f3d3280dd88c6d0185aa687c2"}
Jul 15 11:17:07.257020 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:07.256992    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt"
Jul 15 11:17:07.374679 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:07.374655    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv"
Jul 15 11:17:07.433829 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:07.433787    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d80ad20-8011-4e76-a4d9-2d8335b4e25c-proxy-tls\") pod \"0d80ad20-8011-4e76-a4d9-2d8335b4e25c\" (UID: \"0d80ad20-8011-4e76-a4d9-2d8335b4e25c\") "
Jul 15 11:17:07.433829 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:07.433839    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwn9x\" (UniqueName: \"kubernetes.io/projected/0d80ad20-8011-4e76-a4d9-2d8335b4e25c-kube-api-access-dwn9x\") pod \"0d80ad20-8011-4e76-a4d9-2d8335b4e25c\" (UID: \"0d80ad20-8011-4e76-a4d9-2d8335b4e25c\") "
Jul 15 11:17:07.434089 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:07.433882    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d80ad20-8011-4e76-a4d9-2d8335b4e25c-kube-rbac-proxy-sar-config\") pod \"0d80ad20-8011-4e76-a4d9-2d8335b4e25c\" (UID: \"0d80ad20-8011-4e76-a4d9-2d8335b4e25c\") "
Jul 15 11:17:07.434342 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:07.434308    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d80ad20-8011-4e76-a4d9-2d8335b4e25c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0d80ad20-8011-4e76-a4d9-2d8335b4e25c" (UID: "0d80ad20-8011-4e76-a4d9-2d8335b4e25c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 11:17:07.435994 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:07.435968    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d80ad20-8011-4e76-a4d9-2d8335b4e25c-kube-api-access-dwn9x" (OuterVolumeSpecName: "kube-api-access-dwn9x") pod "0d80ad20-8011-4e76-a4d9-2d8335b4e25c" (UID: "0d80ad20-8011-4e76-a4d9-2d8335b4e25c"). InnerVolumeSpecName "kube-api-access-dwn9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:17:07.436091 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:07.435990    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d80ad20-8011-4e76-a4d9-2d8335b4e25c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0d80ad20-8011-4e76-a4d9-2d8335b4e25c" (UID: "0d80ad20-8011-4e76-a4d9-2d8335b4e25c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:17:07.534691 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:07.534653    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/742c19fa-753e-49c9-9b71-ca36ef4dd9bc-proxy-tls\") pod \"742c19fa-753e-49c9-9b71-ca36ef4dd9bc\" (UID: \"742c19fa-753e-49c9-9b71-ca36ef4dd9bc\") "
Jul 15 11:17:07.534890 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:07.534700    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltk2t\" (UniqueName: \"kubernetes.io/projected/742c19fa-753e-49c9-9b71-ca36ef4dd9bc-kube-api-access-ltk2t\") pod \"742c19fa-753e-49c9-9b71-ca36ef4dd9bc\" (UID: \"742c19fa-753e-49c9-9b71-ca36ef4dd9bc\") "
Jul 15 11:17:07.534890 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:07.534754    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/742c19fa-753e-49c9-9b71-ca36ef4dd9bc-kube-rbac-proxy-sar-config\") pod \"742c19fa-753e-49c9-9b71-ca36ef4dd9bc\" (UID: \"742c19fa-753e-49c9-9b71-ca36ef4dd9bc\") "
Jul 15 11:17:07.535028 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:07.535011    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d80ad20-8011-4e76-a4d9-2d8335b4e25c-proxy-tls\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:17:07.535102 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:07.535035    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwn9x\" (UniqueName: \"kubernetes.io/projected/0d80ad20-8011-4e76-a4d9-2d8335b4e25c-kube-api-access-dwn9x\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:17:07.535102 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:07.535050    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d80ad20-8011-4e76-a4d9-2d8335b4e25c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:17:07.535279 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:07.535250    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/742c19fa-753e-49c9-9b71-ca36ef4dd9bc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "742c19fa-753e-49c9-9b71-ca36ef4dd9bc" (UID: "742c19fa-753e-49c9-9b71-ca36ef4dd9bc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 11:17:07.536804 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:07.536779    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/742c19fa-753e-49c9-9b71-ca36ef4dd9bc-kube-api-access-ltk2t" (OuterVolumeSpecName: "kube-api-access-ltk2t") pod "742c19fa-753e-49c9-9b71-ca36ef4dd9bc" (UID: "742c19fa-753e-49c9-9b71-ca36ef4dd9bc"). InnerVolumeSpecName "kube-api-access-ltk2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:17:07.536804 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:07.536790    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/742c19fa-753e-49c9-9b71-ca36ef4dd9bc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "742c19fa-753e-49c9-9b71-ca36ef4dd9bc" (UID: "742c19fa-753e-49c9-9b71-ca36ef4dd9bc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:17:07.636388 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:07.636291    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/742c19fa-753e-49c9-9b71-ca36ef4dd9bc-proxy-tls\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:17:07.636388 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:07.636324    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltk2t\" (UniqueName: \"kubernetes.io/projected/742c19fa-753e-49c9-9b71-ca36ef4dd9bc-kube-api-access-ltk2t\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:17:07.636388 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:07.636337    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/742c19fa-753e-49c9-9b71-ca36ef4dd9bc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:17:08.202194 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:08.202163    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt"
Jul 15 11:17:08.202194 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:08.202176    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt" event={"ID":"0d80ad20-8011-4e76-a4d9-2d8335b4e25c","Type":"ContainerDied","Data":"f323b317cd9fbd8da142f41bd41a98c0e49365039993158051c612e544d88d04"}
Jul 15 11:17:08.202730 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:08.202238    2563 scope.go:117] "RemoveContainer" containerID="118d116e7a2d09df1a4179bf54dec9034076b9de957d5062cc8adc07b9128e77"
Jul 15 11:17:08.203955 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:08.203931    2563 generic.go:356] "Generic (PLEG): container finished" podID="742c19fa-753e-49c9-9b71-ca36ef4dd9bc" containerID="5c99d86138b9491b436533f542830db6bc45f0e201f999b636d96678f5efa0ec" exitCode=0
Jul 15 11:17:08.204071 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:08.203990    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv"
Jul 15 11:17:08.204071 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:08.204002    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv" event={"ID":"742c19fa-753e-49c9-9b71-ca36ef4dd9bc","Type":"ContainerDied","Data":"5c99d86138b9491b436533f542830db6bc45f0e201f999b636d96678f5efa0ec"}
Jul 15 11:17:08.204071 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:08.204047    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv" event={"ID":"742c19fa-753e-49c9-9b71-ca36ef4dd9bc","Type":"ContainerDied","Data":"f55815ad3c27eb926672bdd87ec4d58c9fb588b8ea01a1d2adb249bb026ad707"}
Jul 15 11:17:08.210862 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:08.210842    2563 scope.go:117] "RemoveContainer" containerID="e6f051f0dd5355ab091149a886793fb6a34a8c8f3d3280dd88c6d0185aa687c2"
Jul 15 11:17:08.218319 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:08.218290    2563 scope.go:117] "RemoveContainer" containerID="29272cf85b34b75ed1a22d45366a3f98772ccb4e0ccb280c081509567fc6326d"
Jul 15 11:17:08.225134 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:08.225108    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv"]
Jul 15 11:17:08.226773 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:08.226756    2563 scope.go:117] "RemoveContainer" containerID="5c99d86138b9491b436533f542830db6bc45f0e201f999b636d96678f5efa0ec"
Jul 15 11:17:08.228161 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:08.228115    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv"]
Jul 15 11:17:08.234289 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:08.234270    2563 scope.go:117] "RemoveContainer" containerID="29272cf85b34b75ed1a22d45366a3f98772ccb4e0ccb280c081509567fc6326d"
Jul 15 11:17:08.234596 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:17:08.234575    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29272cf85b34b75ed1a22d45366a3f98772ccb4e0ccb280c081509567fc6326d\": container with ID starting with 29272cf85b34b75ed1a22d45366a3f98772ccb4e0ccb280c081509567fc6326d not found: ID does not exist" containerID="29272cf85b34b75ed1a22d45366a3f98772ccb4e0ccb280c081509567fc6326d"
Jul 15 11:17:08.234646 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:08.234605    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29272cf85b34b75ed1a22d45366a3f98772ccb4e0ccb280c081509567fc6326d"} err="failed to get container status \"29272cf85b34b75ed1a22d45366a3f98772ccb4e0ccb280c081509567fc6326d\": rpc error: code = NotFound desc = could not find container \"29272cf85b34b75ed1a22d45366a3f98772ccb4e0ccb280c081509567fc6326d\": container with ID starting with 29272cf85b34b75ed1a22d45366a3f98772ccb4e0ccb280c081509567fc6326d not found: ID does not exist"
Jul 15 11:17:08.234646 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:08.234625    2563 scope.go:117] "RemoveContainer" containerID="5c99d86138b9491b436533f542830db6bc45f0e201f999b636d96678f5efa0ec"
Jul 15 11:17:08.234863 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:17:08.234846    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c99d86138b9491b436533f542830db6bc45f0e201f999b636d96678f5efa0ec\": container with ID starting with 5c99d86138b9491b436533f542830db6bc45f0e201f999b636d96678f5efa0ec not found: ID does not exist" containerID="5c99d86138b9491b436533f542830db6bc45f0e201f999b636d96678f5efa0ec"
Jul 15 11:17:08.234911 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:08.234869    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c99d86138b9491b436533f542830db6bc45f0e201f999b636d96678f5efa0ec"} err="failed to get container status \"5c99d86138b9491b436533f542830db6bc45f0e201f999b636d96678f5efa0ec\": rpc error: code = NotFound desc = could not find container \"5c99d86138b9491b436533f542830db6bc45f0e201f999b636d96678f5efa0ec\": container with ID starting with 5c99d86138b9491b436533f542830db6bc45f0e201f999b636d96678f5efa0ec not found: ID does not exist"
Jul 15 11:17:08.242322 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:08.242295    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt"]
Jul 15 11:17:08.247561 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:08.247526    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt"]
Jul 15 11:17:09.677392 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:09.677357    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d80ad20-8011-4e76-a4d9-2d8335b4e25c" path="/var/lib/kubelet/pods/0d80ad20-8011-4e76-a4d9-2d8335b4e25c/volumes"
Jul 15 11:17:09.677801 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:09.677787    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="742c19fa-753e-49c9-9b71-ca36ef4dd9bc" path="/var/lib/kubelet/pods/742c19fa-753e-49c9-9b71-ca36ef4dd9bc/volumes"
Jul 15 11:17:11.018915 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:11.018891    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd"
Jul 15 11:17:11.019318 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:11.018965    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs"
Jul 15 11:17:11.192476 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:11.192446    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb"
Jul 15 11:17:11.192630 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:11.192561    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw"
Jul 15 11:17:11.192985 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:11.192956    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb" podUID="e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 15 11:17:11.193042 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:11.192977    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw" podUID="58d5e7d5-59a8-4d71-a937-c32ac57ab1aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 15 11:17:21.193440 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:21.193395    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw" podUID="58d5e7d5-59a8-4d71-a937-c32ac57ab1aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 15 11:17:21.193440 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:21.193395    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb" podUID="e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 15 11:17:31.193123 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:31.193077    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw" podUID="58d5e7d5-59a8-4d71-a937-c32ac57ab1aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 15 11:17:31.193613 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:31.193078    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb" podUID="e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 15 11:17:33.371018 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.370987    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs"]
Jul 15 11:17:33.371487 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.371243    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs" podUID="148f5d1c-e482-441e-8420-cdc14896673b" containerName="kserve-container" containerID="cri-o://41f5b948fe28a158e5f45d598b24e77ae70245e73deeaeb7ebfe54072d17c642" gracePeriod=30
Jul 15 11:17:33.371487 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.371270    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs" podUID="148f5d1c-e482-441e-8420-cdc14896673b" containerName="kube-rbac-proxy" containerID="cri-o://0282d38312f5fa3db48231b514b0a4e89e1f9550f3167d7ba6c7d0200fd81f11" gracePeriod=30
Jul 15 11:17:33.446797 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.446767    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd"]
Jul 15 11:17:33.447037 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.446991    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd" podUID="99dbd1d6-1f44-478c-a821-f786760dedd8" containerName="kserve-container" containerID="cri-o://7f3c8738ffa162f3bd235c791af2f6839d31043525181f22af2bb501c30f02c5" gracePeriod=30
Jul 15 11:17:33.447122 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.447029    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd" podUID="99dbd1d6-1f44-478c-a821-f786760dedd8" containerName="kube-rbac-proxy" containerID="cri-o://c08c81cfde7adc2769db1de0b543cdea84f0ab79853dd4a9e13650e6d6448c61" gracePeriod=30
Jul 15 11:17:33.480789 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.480759    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf"]
Jul 15 11:17:33.481100 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.481087    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d80ad20-8011-4e76-a4d9-2d8335b4e25c" containerName="kube-rbac-proxy"
Jul 15 11:17:33.481164 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.481101    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d80ad20-8011-4e76-a4d9-2d8335b4e25c" containerName="kube-rbac-proxy"
Jul 15 11:17:33.481164 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.481112    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d80ad20-8011-4e76-a4d9-2d8335b4e25c" containerName="kserve-container"
Jul 15 11:17:33.481164 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.481119    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d80ad20-8011-4e76-a4d9-2d8335b4e25c" containerName="kserve-container"
Jul 15 11:17:33.481164 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.481133    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="742c19fa-753e-49c9-9b71-ca36ef4dd9bc" containerName="kube-rbac-proxy"
Jul 15 11:17:33.481164 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.481153    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="742c19fa-753e-49c9-9b71-ca36ef4dd9bc" containerName="kube-rbac-proxy"
Jul 15 11:17:33.481318 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.481170    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="742c19fa-753e-49c9-9b71-ca36ef4dd9bc" containerName="kserve-container"
Jul 15 11:17:33.481318 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.481176    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="742c19fa-753e-49c9-9b71-ca36ef4dd9bc" containerName="kserve-container"
Jul 15 11:17:33.481318 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.481228    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d80ad20-8011-4e76-a4d9-2d8335b4e25c" containerName="kube-rbac-proxy"
Jul 15 11:17:33.481318 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.481241    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="742c19fa-753e-49c9-9b71-ca36ef4dd9bc" containerName="kube-rbac-proxy"
Jul 15 11:17:33.481318 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.481250    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="742c19fa-753e-49c9-9b71-ca36ef4dd9bc" containerName="kserve-container"
Jul 15 11:17:33.481318 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.481259    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d80ad20-8011-4e76-a4d9-2d8335b4e25c" containerName="kserve-container"
Jul 15 11:17:33.484439 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.484420    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf"
Jul 15 11:17:33.487152 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.487118    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d1ffd-kube-rbac-proxy-sar-config\""
Jul 15 11:17:33.487254 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.487117    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d1ffd-predictor-serving-cert\""
Jul 15 11:17:33.499816 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.499796    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf"]
Jul 15 11:17:33.549158 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.549119    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97287f11-af35-42b1-9806-315ef67e4740-proxy-tls\") pod \"success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf\" (UID: \"97287f11-af35-42b1-9806-315ef67e4740\") " pod="kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf"
Jul 15 11:17:33.549300 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.549244    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97287f11-af35-42b1-9806-315ef67e4740-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf\" (UID: \"97287f11-af35-42b1-9806-315ef67e4740\") " pod="kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf"
Jul 15 11:17:33.549300 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.549284    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx4mx\" (UniqueName: \"kubernetes.io/projected/97287f11-af35-42b1-9806-315ef67e4740-kube-api-access-rx4mx\") pod \"success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf\" (UID: \"97287f11-af35-42b1-9806-315ef67e4740\") " pod="kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf"
Jul 15 11:17:33.607388 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.607356    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2"]
Jul 15 11:17:33.610634 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.610616    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2"
Jul 15 11:17:33.613169 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.613131    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d1ffd-predictor-serving-cert\""
Jul 15 11:17:33.613257 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.613226    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d1ffd-kube-rbac-proxy-sar-config\""
Jul 15 11:17:33.621341 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.621277    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2"]
Jul 15 11:17:33.650473 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.650438    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0830e31-83c6-47ee-993b-9ee52899112f-proxy-tls\") pod \"error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2\" (UID: \"b0830e31-83c6-47ee-993b-9ee52899112f\") " pod="kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2"
Jul 15 11:17:33.650642 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.650494    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97287f11-af35-42b1-9806-315ef67e4740-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf\" (UID: \"97287f11-af35-42b1-9806-315ef67e4740\") " pod="kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf"
Jul 15 11:17:33.650642 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.650534    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx4mx\" (UniqueName: \"kubernetes.io/projected/97287f11-af35-42b1-9806-315ef67e4740-kube-api-access-rx4mx\") pod \"success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf\" (UID: \"97287f11-af35-42b1-9806-315ef67e4740\") " pod="kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf"
Jul 15 11:17:33.650642 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.650564    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0830e31-83c6-47ee-993b-9ee52899112f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2\" (UID: \"b0830e31-83c6-47ee-993b-9ee52899112f\") " pod="kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2"
Jul 15 11:17:33.650642 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.650628    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7746\" (UniqueName: \"kubernetes.io/projected/b0830e31-83c6-47ee-993b-9ee52899112f-kube-api-access-z7746\") pod \"error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2\" (UID: \"b0830e31-83c6-47ee-993b-9ee52899112f\") " pod="kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2"
Jul 15 11:17:33.650851 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.650670    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97287f11-af35-42b1-9806-315ef67e4740-proxy-tls\") pod \"success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf\" (UID: \"97287f11-af35-42b1-9806-315ef67e4740\") " pod="kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf"
Jul 15 11:17:33.651343 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.651313    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97287f11-af35-42b1-9806-315ef67e4740-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf\" (UID: \"97287f11-af35-42b1-9806-315ef67e4740\") " pod="kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf"
Jul 15 11:17:33.653107 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.653074    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97287f11-af35-42b1-9806-315ef67e4740-proxy-tls\") pod \"success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf\" (UID: \"97287f11-af35-42b1-9806-315ef67e4740\") " pod="kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf"
Jul 15 11:17:33.658885 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.658856    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx4mx\" (UniqueName: \"kubernetes.io/projected/97287f11-af35-42b1-9806-315ef67e4740-kube-api-access-rx4mx\") pod \"success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf\" (UID: \"97287f11-af35-42b1-9806-315ef67e4740\") " pod="kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf"
Jul 15 11:17:33.751838 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.751805    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0830e31-83c6-47ee-993b-9ee52899112f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2\" (UID: \"b0830e31-83c6-47ee-993b-9ee52899112f\") " pod="kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2"
Jul 15 11:17:33.752050 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.751853    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7746\" (UniqueName: \"kubernetes.io/projected/b0830e31-83c6-47ee-993b-9ee52899112f-kube-api-access-z7746\") pod \"error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2\" (UID: \"b0830e31-83c6-47ee-993b-9ee52899112f\") " pod="kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2"
Jul 15 11:17:33.752050 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.751923    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0830e31-83c6-47ee-993b-9ee52899112f-proxy-tls\") pod \"error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2\" (UID: \"b0830e31-83c6-47ee-993b-9ee52899112f\") " pod="kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2"
Jul 15 11:17:33.752203 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:17:33.752055    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-serving-cert: secret "error-404-isvc-d1ffd-predictor-serving-cert" not found
Jul 15 11:17:33.752203 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:17:33.752129    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0830e31-83c6-47ee-993b-9ee52899112f-proxy-tls podName:b0830e31-83c6-47ee-993b-9ee52899112f nodeName:}" failed. No retries permitted until 2026-07-15 11:17:34.252108983 +0000 UTC m=+1873.081624383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b0830e31-83c6-47ee-993b-9ee52899112f-proxy-tls") pod "error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2" (UID: "b0830e31-83c6-47ee-993b-9ee52899112f") : secret "error-404-isvc-d1ffd-predictor-serving-cert" not found
Jul 15 11:17:33.752651 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.752626    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0830e31-83c6-47ee-993b-9ee52899112f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2\" (UID: \"b0830e31-83c6-47ee-993b-9ee52899112f\") " pod="kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2"
Jul 15 11:17:33.761380 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.761347    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7746\" (UniqueName: \"kubernetes.io/projected/b0830e31-83c6-47ee-993b-9ee52899112f-kube-api-access-z7746\") pod \"error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2\" (UID: \"b0830e31-83c6-47ee-993b-9ee52899112f\") " pod="kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2"
Jul 15 11:17:33.794285 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.794255    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf"
Jul 15 11:17:33.931515 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:33.931482    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf"]
Jul 15 11:17:33.934025 ip-10-0-131-34 kubenswrapper[2563]: W0715 11:17:33.933990    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97287f11_af35_42b1_9806_315ef67e4740.slice/crio-dbec08901f14bb134cf0248a9235f5fc8d7d6eae8ab8dbc261497d8d3a22b271 WatchSource:0}: Error finding container dbec08901f14bb134cf0248a9235f5fc8d7d6eae8ab8dbc261497d8d3a22b271: Status 404 returned error can't find the container with id dbec08901f14bb134cf0248a9235f5fc8d7d6eae8ab8dbc261497d8d3a22b271
Jul 15 11:17:34.256933 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:34.256884    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0830e31-83c6-47ee-993b-9ee52899112f-proxy-tls\") pod \"error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2\" (UID: \"b0830e31-83c6-47ee-993b-9ee52899112f\") " pod="kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2"
Jul 15 11:17:34.260297 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:34.260266    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0830e31-83c6-47ee-993b-9ee52899112f-proxy-tls\") pod \"error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2\" (UID: \"b0830e31-83c6-47ee-993b-9ee52899112f\") " pod="kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2"
Jul 15 11:17:34.319576 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:34.319543    2563 generic.go:356] "Generic (PLEG): container finished" podID="148f5d1c-e482-441e-8420-cdc14896673b" containerID="0282d38312f5fa3db48231b514b0a4e89e1f9550f3167d7ba6c7d0200fd81f11" exitCode=2
Jul 15 11:17:34.319741 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:34.319616    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs" event={"ID":"148f5d1c-e482-441e-8420-cdc14896673b","Type":"ContainerDied","Data":"0282d38312f5fa3db48231b514b0a4e89e1f9550f3167d7ba6c7d0200fd81f11"}
Jul 15 11:17:34.321338 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:34.321311    2563 generic.go:356] "Generic (PLEG): container finished" podID="99dbd1d6-1f44-478c-a821-f786760dedd8" containerID="c08c81cfde7adc2769db1de0b543cdea84f0ab79853dd4a9e13650e6d6448c61" exitCode=2
Jul 15 11:17:34.321468 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:34.321381    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd" event={"ID":"99dbd1d6-1f44-478c-a821-f786760dedd8","Type":"ContainerDied","Data":"c08c81cfde7adc2769db1de0b543cdea84f0ab79853dd4a9e13650e6d6448c61"}
Jul 15 11:17:34.323042 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:34.323015    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf" event={"ID":"97287f11-af35-42b1-9806-315ef67e4740","Type":"ContainerStarted","Data":"36cf230c0359560b3e9d662d016bd399f3b4e4deb7e5262fe7b0cd07e7710002"}
Jul 15 11:17:34.323133 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:34.323061    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf" event={"ID":"97287f11-af35-42b1-9806-315ef67e4740","Type":"ContainerStarted","Data":"c5714656c76a20adf4ec0d42d1cfba3ab1ca3148056534d74c3e367d20f70737"}
Jul 15 11:17:34.323133 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:34.323089    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf" event={"ID":"97287f11-af35-42b1-9806-315ef67e4740","Type":"ContainerStarted","Data":"dbec08901f14bb134cf0248a9235f5fc8d7d6eae8ab8dbc261497d8d3a22b271"}
Jul 15 11:17:34.323224 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:34.323136    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf"
Jul 15 11:17:34.342363 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:34.342318    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf" podStartSLOduration=1.342303392 podStartE2EDuration="1.342303392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:17:34.341889745 +0000 UTC m=+1873.171405151" watchObservedRunningTime="2026-07-15 11:17:34.342303392 +0000 UTC m=+1873.171818849"
Jul 15 11:17:34.520938 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:34.520843    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2"
Jul 15 11:17:34.660721 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:34.660687    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2"]
Jul 15 11:17:35.328491 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:35.328447    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2" event={"ID":"b0830e31-83c6-47ee-993b-9ee52899112f","Type":"ContainerStarted","Data":"3ecb7aca81ddd30b687fce0e4deb5b5063e0dbc96c0b3672e0a13656b0192ac2"}
Jul 15 11:17:35.328750 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:35.328730    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2"
Jul 15 11:17:35.328828 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:35.328787    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2"
Jul 15 11:17:35.328828 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:35.328810    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2" event={"ID":"b0830e31-83c6-47ee-993b-9ee52899112f","Type":"ContainerStarted","Data":"2e8e70dd1d6a7c8dda4699856462f39026958c4a6ed807cd4b54b767af5bb0ea"}
Jul 15 11:17:35.328922 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:35.328833    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf"
Jul 15 11:17:35.328922 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:35.328855    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2" event={"ID":"b0830e31-83c6-47ee-993b-9ee52899112f","Type":"ContainerStarted","Data":"ecaf6d4e05b51b29ed72f617a99202f857d3933746c92e0020688cf7aa61ff18"}
Jul 15 11:17:35.329827 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:35.329802    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf" podUID="97287f11-af35-42b1-9806-315ef67e4740" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 15 11:17:35.329910 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:35.329804    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2" podUID="b0830e31-83c6-47ee-993b-9ee52899112f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 15 11:17:35.350535 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:35.350477    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2" podStartSLOduration=2.350459077 podStartE2EDuration="2.350459077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:17:35.347737433 +0000 UTC m=+1874.177252849" watchObservedRunningTime="2026-07-15 11:17:35.350459077 +0000 UTC m=+1874.179974483"
Jul 15 11:17:36.013522 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:36.013475    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs" podUID="148f5d1c-e482-441e-8420-cdc14896673b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Jul 15 11:17:36.013900 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:36.013480    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd" podUID="99dbd1d6-1f44-478c-a821-f786760dedd8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Jul 15 11:17:36.337758 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:36.337674    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2" podUID="b0830e31-83c6-47ee-993b-9ee52899112f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 15 11:17:36.337911 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:36.337679    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf" podUID="97287f11-af35-42b1-9806-315ef67e4740" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 15 11:17:37.340250 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:37.340135    2563 generic.go:356] "Generic (PLEG): container finished" podID="148f5d1c-e482-441e-8420-cdc14896673b" containerID="41f5b948fe28a158e5f45d598b24e77ae70245e73deeaeb7ebfe54072d17c642" exitCode=0
Jul 15 11:17:37.340250 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:37.340212    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs" event={"ID":"148f5d1c-e482-441e-8420-cdc14896673b","Type":"ContainerDied","Data":"41f5b948fe28a158e5f45d598b24e77ae70245e73deeaeb7ebfe54072d17c642"}
Jul 15 11:17:37.598842 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:37.598778    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd"
Jul 15 11:17:37.685797 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:37.685766    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99dbd1d6-1f44-478c-a821-f786760dedd8-proxy-tls\") pod \"99dbd1d6-1f44-478c-a821-f786760dedd8\" (UID: \"99dbd1d6-1f44-478c-a821-f786760dedd8\") "
Jul 15 11:17:37.685958 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:37.685803    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99dbd1d6-1f44-478c-a821-f786760dedd8-kube-rbac-proxy-sar-config\") pod \"99dbd1d6-1f44-478c-a821-f786760dedd8\" (UID: \"99dbd1d6-1f44-478c-a821-f786760dedd8\") "
Jul 15 11:17:37.685958 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:37.685878    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gg7bd\" (UniqueName: \"kubernetes.io/projected/99dbd1d6-1f44-478c-a821-f786760dedd8-kube-api-access-gg7bd\") pod \"99dbd1d6-1f44-478c-a821-f786760dedd8\" (UID: \"99dbd1d6-1f44-478c-a821-f786760dedd8\") "
Jul 15 11:17:37.686292 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:37.686265    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99dbd1d6-1f44-478c-a821-f786760dedd8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "99dbd1d6-1f44-478c-a821-f786760dedd8" (UID: "99dbd1d6-1f44-478c-a821-f786760dedd8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 11:17:37.689949 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:37.689915    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99dbd1d6-1f44-478c-a821-f786760dedd8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "99dbd1d6-1f44-478c-a821-f786760dedd8" (UID: "99dbd1d6-1f44-478c-a821-f786760dedd8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:17:37.690064 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:37.689952    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99dbd1d6-1f44-478c-a821-f786760dedd8-kube-api-access-gg7bd" (OuterVolumeSpecName: "kube-api-access-gg7bd") pod "99dbd1d6-1f44-478c-a821-f786760dedd8" (UID: "99dbd1d6-1f44-478c-a821-f786760dedd8"). InnerVolumeSpecName "kube-api-access-gg7bd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:17:37.707999 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:37.707978    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs"
Jul 15 11:17:37.787022 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:37.786989    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjfl8\" (UniqueName: \"kubernetes.io/projected/148f5d1c-e482-441e-8420-cdc14896673b-kube-api-access-rjfl8\") pod \"148f5d1c-e482-441e-8420-cdc14896673b\" (UID: \"148f5d1c-e482-441e-8420-cdc14896673b\") "
Jul 15 11:17:37.787212 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:37.787043    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/148f5d1c-e482-441e-8420-cdc14896673b-proxy-tls\") pod \"148f5d1c-e482-441e-8420-cdc14896673b\" (UID: \"148f5d1c-e482-441e-8420-cdc14896673b\") "
Jul 15 11:17:37.787212 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:37.787080    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/148f5d1c-e482-441e-8420-cdc14896673b-kube-rbac-proxy-sar-config\") pod \"148f5d1c-e482-441e-8420-cdc14896673b\" (UID: \"148f5d1c-e482-441e-8420-cdc14896673b\") "
Jul 15 11:17:37.787348 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:37.787257    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99dbd1d6-1f44-478c-a821-f786760dedd8-proxy-tls\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:17:37.787348 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:37.787274    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99dbd1d6-1f44-478c-a821-f786760dedd8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:17:37.787348 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:37.787286    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg7bd\" (UniqueName: \"kubernetes.io/projected/99dbd1d6-1f44-478c-a821-f786760dedd8-kube-api-access-gg7bd\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:17:37.787545 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:37.787509    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/148f5d1c-e482-441e-8420-cdc14896673b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "148f5d1c-e482-441e-8420-cdc14896673b" (UID: "148f5d1c-e482-441e-8420-cdc14896673b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 11:17:37.789195 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:37.789173    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/148f5d1c-e482-441e-8420-cdc14896673b-kube-api-access-rjfl8" (OuterVolumeSpecName: "kube-api-access-rjfl8") pod "148f5d1c-e482-441e-8420-cdc14896673b" (UID: "148f5d1c-e482-441e-8420-cdc14896673b"). InnerVolumeSpecName "kube-api-access-rjfl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:17:37.789270 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:37.789191    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/148f5d1c-e482-441e-8420-cdc14896673b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "148f5d1c-e482-441e-8420-cdc14896673b" (UID: "148f5d1c-e482-441e-8420-cdc14896673b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:17:37.888486 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:37.888394    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/148f5d1c-e482-441e-8420-cdc14896673b-proxy-tls\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:17:37.888486 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:37.888426    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/148f5d1c-e482-441e-8420-cdc14896673b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:17:37.888486 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:37.888440    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjfl8\" (UniqueName: \"kubernetes.io/projected/148f5d1c-e482-441e-8420-cdc14896673b-kube-api-access-rjfl8\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:17:38.345347 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:38.345312    2563 generic.go:356] "Generic (PLEG): container finished" podID="99dbd1d6-1f44-478c-a821-f786760dedd8" containerID="7f3c8738ffa162f3bd235c791af2f6839d31043525181f22af2bb501c30f02c5" exitCode=0
Jul 15 11:17:38.345776 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:38.345364    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd"
Jul 15 11:17:38.345776 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:38.345388    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd" event={"ID":"99dbd1d6-1f44-478c-a821-f786760dedd8","Type":"ContainerDied","Data":"7f3c8738ffa162f3bd235c791af2f6839d31043525181f22af2bb501c30f02c5"}
Jul 15 11:17:38.345776 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:38.345442    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd" event={"ID":"99dbd1d6-1f44-478c-a821-f786760dedd8","Type":"ContainerDied","Data":"99327dc06980449f971fe9fc77d0421194c40d139be21d879a55b08f2959ad27"}
Jul 15 11:17:38.345776 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:38.345467    2563 scope.go:117] "RemoveContainer" containerID="c08c81cfde7adc2769db1de0b543cdea84f0ab79853dd4a9e13650e6d6448c61"
Jul 15 11:17:38.347136 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:38.347115    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs" event={"ID":"148f5d1c-e482-441e-8420-cdc14896673b","Type":"ContainerDied","Data":"9b88b7d7b111720f31bb23a51e92d22db73f39b020da06666afd3265ac42e343"}
Jul 15 11:17:38.347136 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:38.347132    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs"
Jul 15 11:17:38.356374 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:38.356339    2563 scope.go:117] "RemoveContainer" containerID="7f3c8738ffa162f3bd235c791af2f6839d31043525181f22af2bb501c30f02c5"
Jul 15 11:17:38.365838 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:38.365820    2563 scope.go:117] "RemoveContainer" containerID="c08c81cfde7adc2769db1de0b543cdea84f0ab79853dd4a9e13650e6d6448c61"
Jul 15 11:17:38.366156 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:17:38.366123    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c08c81cfde7adc2769db1de0b543cdea84f0ab79853dd4a9e13650e6d6448c61\": container with ID starting with c08c81cfde7adc2769db1de0b543cdea84f0ab79853dd4a9e13650e6d6448c61 not found: ID does not exist" containerID="c08c81cfde7adc2769db1de0b543cdea84f0ab79853dd4a9e13650e6d6448c61"
Jul 15 11:17:38.366278 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:38.366258    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c08c81cfde7adc2769db1de0b543cdea84f0ab79853dd4a9e13650e6d6448c61"} err="failed to get container status \"c08c81cfde7adc2769db1de0b543cdea84f0ab79853dd4a9e13650e6d6448c61\": rpc error: code = NotFound desc = could not find container \"c08c81cfde7adc2769db1de0b543cdea84f0ab79853dd4a9e13650e6d6448c61\": container with ID starting with c08c81cfde7adc2769db1de0b543cdea84f0ab79853dd4a9e13650e6d6448c61 not found: ID does not exist"
Jul 15 11:17:38.366346 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:38.366280    2563 scope.go:117] "RemoveContainer" containerID="7f3c8738ffa162f3bd235c791af2f6839d31043525181f22af2bb501c30f02c5"
Jul 15 11:17:38.366561 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:17:38.366540    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f3c8738ffa162f3bd235c791af2f6839d31043525181f22af2bb501c30f02c5\": container with ID starting with 7f3c8738ffa162f3bd235c791af2f6839d31043525181f22af2bb501c30f02c5 not found: ID does not exist" containerID="7f3c8738ffa162f3bd235c791af2f6839d31043525181f22af2bb501c30f02c5"
Jul 15 11:17:38.366642 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:38.366569    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f3c8738ffa162f3bd235c791af2f6839d31043525181f22af2bb501c30f02c5"} err="failed to get container status \"7f3c8738ffa162f3bd235c791af2f6839d31043525181f22af2bb501c30f02c5\": rpc error: code = NotFound desc = could not find container \"7f3c8738ffa162f3bd235c791af2f6839d31043525181f22af2bb501c30f02c5\": container with ID starting with 7f3c8738ffa162f3bd235c791af2f6839d31043525181f22af2bb501c30f02c5 not found: ID does not exist"
Jul 15 11:17:38.366642 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:38.366590    2563 scope.go:117] "RemoveContainer" containerID="0282d38312f5fa3db48231b514b0a4e89e1f9550f3167d7ba6c7d0200fd81f11"
Jul 15 11:17:38.370854 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:38.370831    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd"]
Jul 15 11:17:38.373899 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:38.373875    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd"]
Jul 15 11:17:38.375176 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:38.375157    2563 scope.go:117] "RemoveContainer" containerID="41f5b948fe28a158e5f45d598b24e77ae70245e73deeaeb7ebfe54072d17c642"
Jul 15 11:17:38.385260 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:38.385243    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs"]
Jul 15 11:17:38.388839 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:38.388817    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs"]
Jul 15 11:17:39.677388 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:39.677355    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="148f5d1c-e482-441e-8420-cdc14896673b" path="/var/lib/kubelet/pods/148f5d1c-e482-441e-8420-cdc14896673b/volumes"
Jul 15 11:17:39.677785 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:39.677772    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99dbd1d6-1f44-478c-a821-f786760dedd8" path="/var/lib/kubelet/pods/99dbd1d6-1f44-478c-a821-f786760dedd8/volumes"
Jul 15 11:17:41.193258 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:41.193192    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw" podUID="58d5e7d5-59a8-4d71-a937-c32ac57ab1aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 15 11:17:41.193635 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:41.193205    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb" podUID="e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 15 11:17:41.339823 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:41.339789    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf"
Jul 15 11:17:41.340220 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:41.340199    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2"
Jul 15 11:17:41.340337 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:41.340249    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf" podUID="97287f11-af35-42b1-9806-315ef67e4740" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 15 11:17:41.340532 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:41.340509    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2" podUID="b0830e31-83c6-47ee-993b-9ee52899112f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 15 11:17:51.193308 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:51.193260    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb" podUID="e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 15 11:17:51.193847 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:51.193829    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw"
Jul 15 11:17:51.340939 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:51.340903    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf" podUID="97287f11-af35-42b1-9806-315ef67e4740" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 15 11:17:51.340939 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:17:51.340932    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2" podUID="b0830e31-83c6-47ee-993b-9ee52899112f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 15 11:18:01.193282 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:18:01.193247    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb"
Jul 15 11:18:01.340332 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:18:01.340292    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf" podUID="97287f11-af35-42b1-9806-315ef67e4740" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 15 11:18:01.340713 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:18:01.340689    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2" podUID="b0830e31-83c6-47ee-993b-9ee52899112f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 15 11:18:11.340758 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:18:11.340712    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2" podUID="b0830e31-83c6-47ee-993b-9ee52899112f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 15 11:18:11.341261 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:18:11.340712    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf" podUID="97287f11-af35-42b1-9806-315ef67e4740" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 15 11:18:21.341123 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:18:21.341092    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf"
Jul 15 11:18:21.341730 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:18:21.341211    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2"
Jul 15 11:21:21.669322 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:21:21.669289    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk49n_beb68899-ffb3-436f-8e17-43597aad4030/ovn-acl-logging/0.log"
Jul 15 11:21:21.675090 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:21:21.675069    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk49n_beb68899-ffb3-436f-8e17-43597aad4030/ovn-acl-logging/0.log"
Jul 15 11:26:21.704264 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:21.704234    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk49n_beb68899-ffb3-436f-8e17-43597aad4030/ovn-acl-logging/0.log"
Jul 15 11:26:21.706672 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:21.704270    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk49n_beb68899-ffb3-436f-8e17-43597aad4030/ovn-acl-logging/0.log"
Jul 15 11:26:48.504393 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:48.504357    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf"]
Jul 15 11:26:48.505349 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:48.504659    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf" podUID="97287f11-af35-42b1-9806-315ef67e4740" containerName="kserve-container" containerID="cri-o://c5714656c76a20adf4ec0d42d1cfba3ab1ca3148056534d74c3e367d20f70737" gracePeriod=30
Jul 15 11:26:48.505349 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:48.504749    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf" podUID="97287f11-af35-42b1-9806-315ef67e4740" containerName="kube-rbac-proxy" containerID="cri-o://36cf230c0359560b3e9d662d016bd399f3b4e4deb7e5262fe7b0cd07e7710002" gracePeriod=30
Jul 15 11:26:48.658469 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:48.658428    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2"]
Jul 15 11:26:48.658700 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:48.658674    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2" podUID="b0830e31-83c6-47ee-993b-9ee52899112f" containerName="kserve-container" containerID="cri-o://2e8e70dd1d6a7c8dda4699856462f39026958c4a6ed807cd4b54b767af5bb0ea" gracePeriod=30
Jul 15 11:26:48.658777 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:48.658744    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2" podUID="b0830e31-83c6-47ee-993b-9ee52899112f" containerName="kube-rbac-proxy" containerID="cri-o://3ecb7aca81ddd30b687fce0e4deb5b5063e0dbc96c0b3672e0a13656b0192ac2" gracePeriod=30
Jul 15 11:26:49.529761 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:49.529723    2563 generic.go:356] "Generic (PLEG): container finished" podID="b0830e31-83c6-47ee-993b-9ee52899112f" containerID="3ecb7aca81ddd30b687fce0e4deb5b5063e0dbc96c0b3672e0a13656b0192ac2" exitCode=2
Jul 15 11:26:49.530249 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:49.529795    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2" event={"ID":"b0830e31-83c6-47ee-993b-9ee52899112f","Type":"ContainerDied","Data":"3ecb7aca81ddd30b687fce0e4deb5b5063e0dbc96c0b3672e0a13656b0192ac2"}
Jul 15 11:26:49.531526 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:49.531504    2563 generic.go:356] "Generic (PLEG): container finished" podID="97287f11-af35-42b1-9806-315ef67e4740" containerID="36cf230c0359560b3e9d662d016bd399f3b4e4deb7e5262fe7b0cd07e7710002" exitCode=2
Jul 15 11:26:49.531650 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:49.531533    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf" event={"ID":"97287f11-af35-42b1-9806-315ef67e4740","Type":"ContainerDied","Data":"36cf230c0359560b3e9d662d016bd399f3b4e4deb7e5262fe7b0cd07e7710002"}
Jul 15 11:26:51.336821 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:51.336784    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf" podUID="97287f11-af35-42b1-9806-315ef67e4740" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Jul 15 11:26:51.337237 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:51.336784    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2" podUID="b0830e31-83c6-47ee-993b-9ee52899112f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Jul 15 11:26:51.346322 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:51.346261    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2" podUID="b0830e31-83c6-47ee-993b-9ee52899112f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 15 11:26:51.346486 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:51.346250    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf" podUID="97287f11-af35-42b1-9806-315ef67e4740" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 15 11:26:51.852206 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:51.852179    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf"
Jul 15 11:26:51.939990 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:51.939892    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97287f11-af35-42b1-9806-315ef67e4740-proxy-tls\") pod \"97287f11-af35-42b1-9806-315ef67e4740\" (UID: \"97287f11-af35-42b1-9806-315ef67e4740\") "
Jul 15 11:26:51.939990 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:51.939946    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rx4mx\" (UniqueName: \"kubernetes.io/projected/97287f11-af35-42b1-9806-315ef67e4740-kube-api-access-rx4mx\") pod \"97287f11-af35-42b1-9806-315ef67e4740\" (UID: \"97287f11-af35-42b1-9806-315ef67e4740\") "
Jul 15 11:26:51.940287 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:51.940014    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97287f11-af35-42b1-9806-315ef67e4740-kube-rbac-proxy-sar-config\") pod \"97287f11-af35-42b1-9806-315ef67e4740\" (UID: \"97287f11-af35-42b1-9806-315ef67e4740\") "
Jul 15 11:26:51.940480 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:51.940456    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97287f11-af35-42b1-9806-315ef67e4740-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "97287f11-af35-42b1-9806-315ef67e4740" (UID: "97287f11-af35-42b1-9806-315ef67e4740"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 11:26:51.950656 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:51.950620    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97287f11-af35-42b1-9806-315ef67e4740-kube-api-access-rx4mx" (OuterVolumeSpecName: "kube-api-access-rx4mx") pod "97287f11-af35-42b1-9806-315ef67e4740" (UID: "97287f11-af35-42b1-9806-315ef67e4740"). InnerVolumeSpecName "kube-api-access-rx4mx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:26:51.950774 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:51.950669    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97287f11-af35-42b1-9806-315ef67e4740-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "97287f11-af35-42b1-9806-315ef67e4740" (UID: "97287f11-af35-42b1-9806-315ef67e4740"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:26:52.040646 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.040615    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97287f11-af35-42b1-9806-315ef67e4740-proxy-tls\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:26:52.040646 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.040645    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx4mx\" (UniqueName: \"kubernetes.io/projected/97287f11-af35-42b1-9806-315ef67e4740-kube-api-access-rx4mx\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:26:52.040836 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.040654    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97287f11-af35-42b1-9806-315ef67e4740-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:26:52.098676 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.098644    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2"
Jul 15 11:26:52.242542 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.242506    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7746\" (UniqueName: \"kubernetes.io/projected/b0830e31-83c6-47ee-993b-9ee52899112f-kube-api-access-z7746\") pod \"b0830e31-83c6-47ee-993b-9ee52899112f\" (UID: \"b0830e31-83c6-47ee-993b-9ee52899112f\") "
Jul 15 11:26:52.242701 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.242571    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0830e31-83c6-47ee-993b-9ee52899112f-proxy-tls\") pod \"b0830e31-83c6-47ee-993b-9ee52899112f\" (UID: \"b0830e31-83c6-47ee-993b-9ee52899112f\") "
Jul 15 11:26:52.242701 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.242672    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0830e31-83c6-47ee-993b-9ee52899112f-kube-rbac-proxy-sar-config\") pod \"b0830e31-83c6-47ee-993b-9ee52899112f\" (UID: \"b0830e31-83c6-47ee-993b-9ee52899112f\") "
Jul 15 11:26:52.243011 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.242980    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0830e31-83c6-47ee-993b-9ee52899112f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b0830e31-83c6-47ee-993b-9ee52899112f" (UID: "b0830e31-83c6-47ee-993b-9ee52899112f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 11:26:52.244698 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.244671    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0830e31-83c6-47ee-993b-9ee52899112f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b0830e31-83c6-47ee-993b-9ee52899112f" (UID: "b0830e31-83c6-47ee-993b-9ee52899112f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:26:52.244698 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.244684    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0830e31-83c6-47ee-993b-9ee52899112f-kube-api-access-z7746" (OuterVolumeSpecName: "kube-api-access-z7746") pod "b0830e31-83c6-47ee-993b-9ee52899112f" (UID: "b0830e31-83c6-47ee-993b-9ee52899112f"). InnerVolumeSpecName "kube-api-access-z7746". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:26:52.344127 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.344086    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0830e31-83c6-47ee-993b-9ee52899112f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:26:52.344127 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.344120    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7746\" (UniqueName: \"kubernetes.io/projected/b0830e31-83c6-47ee-993b-9ee52899112f-kube-api-access-z7746\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:26:52.344127 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.344129    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0830e31-83c6-47ee-993b-9ee52899112f-proxy-tls\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:26:52.545080 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.544991    2563 generic.go:356] "Generic (PLEG): container finished" podID="b0830e31-83c6-47ee-993b-9ee52899112f" containerID="2e8e70dd1d6a7c8dda4699856462f39026958c4a6ed807cd4b54b767af5bb0ea" exitCode=0
Jul 15 11:26:52.545080 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.545053    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2"
Jul 15 11:26:52.545327 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.545081    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2" event={"ID":"b0830e31-83c6-47ee-993b-9ee52899112f","Type":"ContainerDied","Data":"2e8e70dd1d6a7c8dda4699856462f39026958c4a6ed807cd4b54b767af5bb0ea"}
Jul 15 11:26:52.545327 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.545172    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2" event={"ID":"b0830e31-83c6-47ee-993b-9ee52899112f","Type":"ContainerDied","Data":"ecaf6d4e05b51b29ed72f617a99202f857d3933746c92e0020688cf7aa61ff18"}
Jul 15 11:26:52.545327 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.545212    2563 scope.go:117] "RemoveContainer" containerID="3ecb7aca81ddd30b687fce0e4deb5b5063e0dbc96c0b3672e0a13656b0192ac2"
Jul 15 11:26:52.546776 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.546758    2563 generic.go:356] "Generic (PLEG): container finished" podID="97287f11-af35-42b1-9806-315ef67e4740" containerID="c5714656c76a20adf4ec0d42d1cfba3ab1ca3148056534d74c3e367d20f70737" exitCode=0
Jul 15 11:26:52.546848 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.546785    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf" event={"ID":"97287f11-af35-42b1-9806-315ef67e4740","Type":"ContainerDied","Data":"c5714656c76a20adf4ec0d42d1cfba3ab1ca3148056534d74c3e367d20f70737"}
Jul 15 11:26:52.546848 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.546825    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf"
Jul 15 11:26:52.546983 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.546827    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf" event={"ID":"97287f11-af35-42b1-9806-315ef67e4740","Type":"ContainerDied","Data":"dbec08901f14bb134cf0248a9235f5fc8d7d6eae8ab8dbc261497d8d3a22b271"}
Jul 15 11:26:52.556282 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.556260    2563 scope.go:117] "RemoveContainer" containerID="2e8e70dd1d6a7c8dda4699856462f39026958c4a6ed807cd4b54b767af5bb0ea"
Jul 15 11:26:52.564531 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.564509    2563 scope.go:117] "RemoveContainer" containerID="3ecb7aca81ddd30b687fce0e4deb5b5063e0dbc96c0b3672e0a13656b0192ac2"
Jul 15 11:26:52.564824 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:26:52.564803    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ecb7aca81ddd30b687fce0e4deb5b5063e0dbc96c0b3672e0a13656b0192ac2\": container with ID starting with 3ecb7aca81ddd30b687fce0e4deb5b5063e0dbc96c0b3672e0a13656b0192ac2 not found: ID does not exist" containerID="3ecb7aca81ddd30b687fce0e4deb5b5063e0dbc96c0b3672e0a13656b0192ac2"
Jul 15 11:26:52.564876 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.564833    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ecb7aca81ddd30b687fce0e4deb5b5063e0dbc96c0b3672e0a13656b0192ac2"} err="failed to get container status \"3ecb7aca81ddd30b687fce0e4deb5b5063e0dbc96c0b3672e0a13656b0192ac2\": rpc error: code = NotFound desc = could not find container \"3ecb7aca81ddd30b687fce0e4deb5b5063e0dbc96c0b3672e0a13656b0192ac2\": container with ID starting with 3ecb7aca81ddd30b687fce0e4deb5b5063e0dbc96c0b3672e0a13656b0192ac2 not found: ID does not exist"
Jul 15 11:26:52.564876 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.564851    2563 scope.go:117] "RemoveContainer" containerID="2e8e70dd1d6a7c8dda4699856462f39026958c4a6ed807cd4b54b767af5bb0ea"
Jul 15 11:26:52.565073 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:26:52.565058    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e8e70dd1d6a7c8dda4699856462f39026958c4a6ed807cd4b54b767af5bb0ea\": container with ID starting with 2e8e70dd1d6a7c8dda4699856462f39026958c4a6ed807cd4b54b767af5bb0ea not found: ID does not exist" containerID="2e8e70dd1d6a7c8dda4699856462f39026958c4a6ed807cd4b54b767af5bb0ea"
Jul 15 11:26:52.565114 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.565077    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e8e70dd1d6a7c8dda4699856462f39026958c4a6ed807cd4b54b767af5bb0ea"} err="failed to get container status \"2e8e70dd1d6a7c8dda4699856462f39026958c4a6ed807cd4b54b767af5bb0ea\": rpc error: code = NotFound desc = could not find container \"2e8e70dd1d6a7c8dda4699856462f39026958c4a6ed807cd4b54b767af5bb0ea\": container with ID starting with 2e8e70dd1d6a7c8dda4699856462f39026958c4a6ed807cd4b54b767af5bb0ea not found: ID does not exist"
Jul 15 11:26:52.565114 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.565090    2563 scope.go:117] "RemoveContainer" containerID="36cf230c0359560b3e9d662d016bd399f3b4e4deb7e5262fe7b0cd07e7710002"
Jul 15 11:26:52.572244 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.572226    2563 scope.go:117] "RemoveContainer" containerID="c5714656c76a20adf4ec0d42d1cfba3ab1ca3148056534d74c3e367d20f70737"
Jul 15 11:26:52.575667 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.575642    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf"]
Jul 15 11:26:52.580258 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.580235    2563 scope.go:117] "RemoveContainer" containerID="36cf230c0359560b3e9d662d016bd399f3b4e4deb7e5262fe7b0cd07e7710002"
Jul 15 11:26:52.580615 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:26:52.580591    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36cf230c0359560b3e9d662d016bd399f3b4e4deb7e5262fe7b0cd07e7710002\": container with ID starting with 36cf230c0359560b3e9d662d016bd399f3b4e4deb7e5262fe7b0cd07e7710002 not found: ID does not exist" containerID="36cf230c0359560b3e9d662d016bd399f3b4e4deb7e5262fe7b0cd07e7710002"
Jul 15 11:26:52.580726 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.580625    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36cf230c0359560b3e9d662d016bd399f3b4e4deb7e5262fe7b0cd07e7710002"} err="failed to get container status \"36cf230c0359560b3e9d662d016bd399f3b4e4deb7e5262fe7b0cd07e7710002\": rpc error: code = NotFound desc = could not find container \"36cf230c0359560b3e9d662d016bd399f3b4e4deb7e5262fe7b0cd07e7710002\": container with ID starting with 36cf230c0359560b3e9d662d016bd399f3b4e4deb7e5262fe7b0cd07e7710002 not found: ID does not exist"
Jul 15 11:26:52.580726 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.580651    2563 scope.go:117] "RemoveContainer" containerID="c5714656c76a20adf4ec0d42d1cfba3ab1ca3148056534d74c3e367d20f70737"
Jul 15 11:26:52.581025 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:26:52.581005    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5714656c76a20adf4ec0d42d1cfba3ab1ca3148056534d74c3e367d20f70737\": container with ID starting with c5714656c76a20adf4ec0d42d1cfba3ab1ca3148056534d74c3e367d20f70737 not found: ID does not exist" containerID="c5714656c76a20adf4ec0d42d1cfba3ab1ca3148056534d74c3e367d20f70737"
Jul 15 11:26:52.581109 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.581032    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5714656c76a20adf4ec0d42d1cfba3ab1ca3148056534d74c3e367d20f70737"} err="failed to get container status \"c5714656c76a20adf4ec0d42d1cfba3ab1ca3148056534d74c3e367d20f70737\": rpc error: code = NotFound desc = could not find container \"c5714656c76a20adf4ec0d42d1cfba3ab1ca3148056534d74c3e367d20f70737\": container with ID starting with c5714656c76a20adf4ec0d42d1cfba3ab1ca3148056534d74c3e367d20f70737 not found: ID does not exist"
Jul 15 11:26:52.581729 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.581711    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf"]
Jul 15 11:26:52.596283 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.596253    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2"]
Jul 15 11:26:52.599470 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:52.599443    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2"]
Jul 15 11:26:53.678424 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:53.678384    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97287f11-af35-42b1-9806-315ef67e4740" path="/var/lib/kubelet/pods/97287f11-af35-42b1-9806-315ef67e4740/volumes"
Jul 15 11:26:53.678991 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:26:53.678968    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0830e31-83c6-47ee-993b-9ee52899112f" path="/var/lib/kubelet/pods/b0830e31-83c6-47ee-993b-9ee52899112f/volumes"
Jul 15 11:31:21.727442 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:31:21.727317    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk49n_beb68899-ffb3-436f-8e17-43597aad4030/ovn-acl-logging/0.log"
Jul 15 11:31:21.730633 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:31:21.728023    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk49n_beb68899-ffb3-436f-8e17-43597aad4030/ovn-acl-logging/0.log"
Jul 15 11:34:22.552380 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:22.552299    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw"]
Jul 15 11:34:22.552923 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:22.552533    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw" podUID="58d5e7d5-59a8-4d71-a937-c32ac57ab1aa" containerName="kserve-container" containerID="cri-o://3485486a9bb2ebaecfd8c21dd9477dbb9ab663ba3bf96417502a35a90b250b69" gracePeriod=30
Jul 15 11:34:22.552923 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:22.552575    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw" podUID="58d5e7d5-59a8-4d71-a937-c32ac57ab1aa" containerName="kube-rbac-proxy" containerID="cri-o://67a49f6f806995b6cd162f3c64a65041901285792766aec56551a07c1647a8a3" gracePeriod=30
Jul 15 11:34:22.601044 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:22.601010    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb"]
Jul 15 11:34:22.601306 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:22.601280    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb" podUID="e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7" containerName="kserve-container" containerID="cri-o://5d06730eb1209c5226a30ee888a29ec434bcdd7fb7e3ca13eb22c1c3ce9bf523" gracePeriod=30
Jul 15 11:34:22.601405 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:22.601319    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb" podUID="e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7" containerName="kube-rbac-proxy" containerID="cri-o://830630dacbf6de2b32fc5517d3e1f9c877c463a4bf85daa9b4a1d81600dd51d9" gracePeriod=30
Jul 15 11:34:23.227628 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:23.227589    2563 generic.go:356] "Generic (PLEG): container finished" podID="e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7" containerID="830630dacbf6de2b32fc5517d3e1f9c877c463a4bf85daa9b4a1d81600dd51d9" exitCode=2
Jul 15 11:34:23.227808 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:23.227662    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb" event={"ID":"e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7","Type":"ContainerDied","Data":"830630dacbf6de2b32fc5517d3e1f9c877c463a4bf85daa9b4a1d81600dd51d9"}
Jul 15 11:34:23.229321 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:23.229295    2563 generic.go:356] "Generic (PLEG): container finished" podID="58d5e7d5-59a8-4d71-a937-c32ac57ab1aa" containerID="67a49f6f806995b6cd162f3c64a65041901285792766aec56551a07c1647a8a3" exitCode=2
Jul 15 11:34:23.229448 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:23.229359    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw" event={"ID":"58d5e7d5-59a8-4d71-a937-c32ac57ab1aa","Type":"ContainerDied","Data":"67a49f6f806995b6cd162f3c64a65041901285792766aec56551a07c1647a8a3"}
Jul 15 11:34:24.176892 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.176861    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-d6qjm/must-gather-4lpnh"]
Jul 15 11:34:24.177301 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.177158    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="148f5d1c-e482-441e-8420-cdc14896673b" containerName="kube-rbac-proxy"
Jul 15 11:34:24.177301 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.177169    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="148f5d1c-e482-441e-8420-cdc14896673b" containerName="kube-rbac-proxy"
Jul 15 11:34:24.177301 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.177183    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0830e31-83c6-47ee-993b-9ee52899112f" containerName="kube-rbac-proxy"
Jul 15 11:34:24.177301 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.177189    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0830e31-83c6-47ee-993b-9ee52899112f" containerName="kube-rbac-proxy"
Jul 15 11:34:24.177301 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.177196    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97287f11-af35-42b1-9806-315ef67e4740" containerName="kube-rbac-proxy"
Jul 15 11:34:24.177301 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.177202    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="97287f11-af35-42b1-9806-315ef67e4740" containerName="kube-rbac-proxy"
Jul 15 11:34:24.177301 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.177208    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0830e31-83c6-47ee-993b-9ee52899112f" containerName="kserve-container"
Jul 15 11:34:24.177301 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.177214    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0830e31-83c6-47ee-993b-9ee52899112f" containerName="kserve-container"
Jul 15 11:34:24.177301 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.177223    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99dbd1d6-1f44-478c-a821-f786760dedd8" containerName="kube-rbac-proxy"
Jul 15 11:34:24.177301 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.177229    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="99dbd1d6-1f44-478c-a821-f786760dedd8" containerName="kube-rbac-proxy"
Jul 15 11:34:24.177301 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.177236    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99dbd1d6-1f44-478c-a821-f786760dedd8" containerName="kserve-container"
Jul 15 11:34:24.177301 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.177242    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="99dbd1d6-1f44-478c-a821-f786760dedd8" containerName="kserve-container"
Jul 15 11:34:24.177301 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.177251    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="148f5d1c-e482-441e-8420-cdc14896673b" containerName="kserve-container"
Jul 15 11:34:24.177301 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.177257    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="148f5d1c-e482-441e-8420-cdc14896673b" containerName="kserve-container"
Jul 15 11:34:24.177301 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.177262    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97287f11-af35-42b1-9806-315ef67e4740" containerName="kserve-container"
Jul 15 11:34:24.177301 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.177267    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="97287f11-af35-42b1-9806-315ef67e4740" containerName="kserve-container"
Jul 15 11:34:24.177301 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.177311    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0830e31-83c6-47ee-993b-9ee52899112f" containerName="kserve-container"
Jul 15 11:34:24.177799 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.177318    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="97287f11-af35-42b1-9806-315ef67e4740" containerName="kube-rbac-proxy"
Jul 15 11:34:24.177799 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.177325    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="97287f11-af35-42b1-9806-315ef67e4740" containerName="kserve-container"
Jul 15 11:34:24.177799 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.177332    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0830e31-83c6-47ee-993b-9ee52899112f" containerName="kube-rbac-proxy"
Jul 15 11:34:24.177799 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.177338    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="99dbd1d6-1f44-478c-a821-f786760dedd8" containerName="kserve-container"
Jul 15 11:34:24.177799 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.177344    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="99dbd1d6-1f44-478c-a821-f786760dedd8" containerName="kube-rbac-proxy"
Jul 15 11:34:24.177799 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.177349    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="148f5d1c-e482-441e-8420-cdc14896673b" containerName="kube-rbac-proxy"
Jul 15 11:34:24.177799 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.177354    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="148f5d1c-e482-441e-8420-cdc14896673b" containerName="kserve-container"
Jul 15 11:34:24.180706 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.180680    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d6qjm/must-gather-4lpnh"
Jul 15 11:34:24.183415 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.183391    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-d6qjm\"/\"kube-root-ca.crt\""
Jul 15 11:34:24.183522 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.183417    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-d6qjm\"/\"openshift-service-ca.crt\""
Jul 15 11:34:24.184627 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.184610    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-d6qjm\"/\"default-dockercfg-jh8hd\""
Jul 15 11:34:24.188426 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.188406    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d6qjm/must-gather-4lpnh"]
Jul 15 11:34:24.299731 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.299695    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j24s7\" (UniqueName: \"kubernetes.io/projected/4072bab9-7867-4451-91f4-20dd1383ec9e-kube-api-access-j24s7\") pod \"must-gather-4lpnh\" (UID: \"4072bab9-7867-4451-91f4-20dd1383ec9e\") " pod="openshift-must-gather-d6qjm/must-gather-4lpnh"
Jul 15 11:34:24.299916 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.299769    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4072bab9-7867-4451-91f4-20dd1383ec9e-must-gather-output\") pod \"must-gather-4lpnh\" (UID: \"4072bab9-7867-4451-91f4-20dd1383ec9e\") " pod="openshift-must-gather-d6qjm/must-gather-4lpnh"
Jul 15 11:34:24.400876 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.400835    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4072bab9-7867-4451-91f4-20dd1383ec9e-must-gather-output\") pod \"must-gather-4lpnh\" (UID: \"4072bab9-7867-4451-91f4-20dd1383ec9e\") " pod="openshift-must-gather-d6qjm/must-gather-4lpnh"
Jul 15 11:34:24.401053 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.400905    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j24s7\" (UniqueName: \"kubernetes.io/projected/4072bab9-7867-4451-91f4-20dd1383ec9e-kube-api-access-j24s7\") pod \"must-gather-4lpnh\" (UID: \"4072bab9-7867-4451-91f4-20dd1383ec9e\") " pod="openshift-must-gather-d6qjm/must-gather-4lpnh"
Jul 15 11:34:24.401224 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.401206    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4072bab9-7867-4451-91f4-20dd1383ec9e-must-gather-output\") pod \"must-gather-4lpnh\" (UID: \"4072bab9-7867-4451-91f4-20dd1383ec9e\") " pod="openshift-must-gather-d6qjm/must-gather-4lpnh"
Jul 15 11:34:24.408511 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.408488    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j24s7\" (UniqueName: \"kubernetes.io/projected/4072bab9-7867-4451-91f4-20dd1383ec9e-kube-api-access-j24s7\") pod \"must-gather-4lpnh\" (UID: \"4072bab9-7867-4451-91f4-20dd1383ec9e\") " pod="openshift-must-gather-d6qjm/must-gather-4lpnh"
Jul 15 11:34:24.501950 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.501912    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d6qjm/must-gather-4lpnh"
Jul 15 11:34:24.631553 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.631500    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d6qjm/must-gather-4lpnh"]
Jul 15 11:34:24.634405 ip-10-0-131-34 kubenswrapper[2563]: W0715 11:34:24.634374    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4072bab9_7867_4451_91f4_20dd1383ec9e.slice/crio-14586ddd5c88b7c48ebd800ff5869e7e4aaede3c0f3778f4d55726766a7d5e94 WatchSource:0}: Error finding container 14586ddd5c88b7c48ebd800ff5869e7e4aaede3c0f3778f4d55726766a7d5e94: Status 404 returned error can't find the container with id 14586ddd5c88b7c48ebd800ff5869e7e4aaede3c0f3778f4d55726766a7d5e94
Jul 15 11:34:24.636474 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:24.636457    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 15 11:34:25.238903 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:25.238867    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d6qjm/must-gather-4lpnh" event={"ID":"4072bab9-7867-4451-91f4-20dd1383ec9e","Type":"ContainerStarted","Data":"14586ddd5c88b7c48ebd800ff5869e7e4aaede3c0f3778f4d55726766a7d5e94"}
Jul 15 11:34:25.936799 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:25.936775    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw"
Jul 15 11:34:26.015887 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.015858    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58d5e7d5-59a8-4d71-a937-c32ac57ab1aa-kube-rbac-proxy-sar-config\") pod \"58d5e7d5-59a8-4d71-a937-c32ac57ab1aa\" (UID: \"58d5e7d5-59a8-4d71-a937-c32ac57ab1aa\") "
Jul 15 11:34:26.016074 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.015952    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9ds6\" (UniqueName: \"kubernetes.io/projected/58d5e7d5-59a8-4d71-a937-c32ac57ab1aa-kube-api-access-l9ds6\") pod \"58d5e7d5-59a8-4d71-a937-c32ac57ab1aa\" (UID: \"58d5e7d5-59a8-4d71-a937-c32ac57ab1aa\") "
Jul 15 11:34:26.016074 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.015996    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58d5e7d5-59a8-4d71-a937-c32ac57ab1aa-proxy-tls\") pod \"58d5e7d5-59a8-4d71-a937-c32ac57ab1aa\" (UID: \"58d5e7d5-59a8-4d71-a937-c32ac57ab1aa\") "
Jul 15 11:34:26.016299 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.016280    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58d5e7d5-59a8-4d71-a937-c32ac57ab1aa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "58d5e7d5-59a8-4d71-a937-c32ac57ab1aa" (UID: "58d5e7d5-59a8-4d71-a937-c32ac57ab1aa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 11:34:26.019713 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.019683    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d5e7d5-59a8-4d71-a937-c32ac57ab1aa-kube-api-access-l9ds6" (OuterVolumeSpecName: "kube-api-access-l9ds6") pod "58d5e7d5-59a8-4d71-a937-c32ac57ab1aa" (UID: "58d5e7d5-59a8-4d71-a937-c32ac57ab1aa"). InnerVolumeSpecName "kube-api-access-l9ds6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:34:26.019809 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.019733    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58d5e7d5-59a8-4d71-a937-c32ac57ab1aa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "58d5e7d5-59a8-4d71-a937-c32ac57ab1aa" (UID: "58d5e7d5-59a8-4d71-a937-c32ac57ab1aa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:34:26.043154 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.043120    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb"
Jul 15 11:34:26.116868 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.116824    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7-proxy-tls\") pod \"e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7\" (UID: \"e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7\") "
Jul 15 11:34:26.117041 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.116942    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbbkc\" (UniqueName: \"kubernetes.io/projected/e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7-kube-api-access-kbbkc\") pod \"e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7\" (UID: \"e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7\") "
Jul 15 11:34:26.117041 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.116999    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7-kube-rbac-proxy-sar-config\") pod \"e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7\" (UID: \"e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7\") "
Jul 15 11:34:26.117226 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.117211    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9ds6\" (UniqueName: \"kubernetes.io/projected/58d5e7d5-59a8-4d71-a937-c32ac57ab1aa-kube-api-access-l9ds6\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:34:26.117274 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.117233    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58d5e7d5-59a8-4d71-a937-c32ac57ab1aa-proxy-tls\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:34:26.117274 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.117250    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58d5e7d5-59a8-4d71-a937-c32ac57ab1aa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:34:26.117390 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.117368    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7" (UID: "e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 11:34:26.119058 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.119035    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7" (UID: "e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:34:26.119111 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.119038    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7-kube-api-access-kbbkc" (OuterVolumeSpecName: "kube-api-access-kbbkc") pod "e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7" (UID: "e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7"). InnerVolumeSpecName "kube-api-access-kbbkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:34:26.218029 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.217997    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:34:26.218029 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.218026    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7-proxy-tls\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:34:26.218226 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.218036    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbbkc\" (UniqueName: \"kubernetes.io/projected/e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7-kube-api-access-kbbkc\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:34:26.244378 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.244344    2563 generic.go:356] "Generic (PLEG): container finished" podID="e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7" containerID="5d06730eb1209c5226a30ee888a29ec434bcdd7fb7e3ca13eb22c1c3ce9bf523" exitCode=0
Jul 15 11:34:26.244784 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.244390    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb"
Jul 15 11:34:26.244784 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.244435    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb" event={"ID":"e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7","Type":"ContainerDied","Data":"5d06730eb1209c5226a30ee888a29ec434bcdd7fb7e3ca13eb22c1c3ce9bf523"}
Jul 15 11:34:26.244784 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.244507    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb" event={"ID":"e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7","Type":"ContainerDied","Data":"60beb3a626d6a0305c4b3c52cb33862e812b58607163d48a2c48c7f4a724c360"}
Jul 15 11:34:26.244784 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.244544    2563 scope.go:117] "RemoveContainer" containerID="830630dacbf6de2b32fc5517d3e1f9c877c463a4bf85daa9b4a1d81600dd51d9"
Jul 15 11:34:26.246021 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.245999    2563 generic.go:356] "Generic (PLEG): container finished" podID="58d5e7d5-59a8-4d71-a937-c32ac57ab1aa" containerID="3485486a9bb2ebaecfd8c21dd9477dbb9ab663ba3bf96417502a35a90b250b69" exitCode=0
Jul 15 11:34:26.246129 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.246057    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw"
Jul 15 11:34:26.246301 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.246058    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw" event={"ID":"58d5e7d5-59a8-4d71-a937-c32ac57ab1aa","Type":"ContainerDied","Data":"3485486a9bb2ebaecfd8c21dd9477dbb9ab663ba3bf96417502a35a90b250b69"}
Jul 15 11:34:26.246301 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.246201    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw" event={"ID":"58d5e7d5-59a8-4d71-a937-c32ac57ab1aa","Type":"ContainerDied","Data":"a5060027c15816c4808489419de719c29c11705c13955b290530f0deab124947"}
Jul 15 11:34:26.253376 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.253345    2563 scope.go:117] "RemoveContainer" containerID="5d06730eb1209c5226a30ee888a29ec434bcdd7fb7e3ca13eb22c1c3ce9bf523"
Jul 15 11:34:26.260641 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.260624    2563 scope.go:117] "RemoveContainer" containerID="830630dacbf6de2b32fc5517d3e1f9c877c463a4bf85daa9b4a1d81600dd51d9"
Jul 15 11:34:26.260890 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:34:26.260872    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"830630dacbf6de2b32fc5517d3e1f9c877c463a4bf85daa9b4a1d81600dd51d9\": container with ID starting with 830630dacbf6de2b32fc5517d3e1f9c877c463a4bf85daa9b4a1d81600dd51d9 not found: ID does not exist" containerID="830630dacbf6de2b32fc5517d3e1f9c877c463a4bf85daa9b4a1d81600dd51d9"
Jul 15 11:34:26.260936 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.260901    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"830630dacbf6de2b32fc5517d3e1f9c877c463a4bf85daa9b4a1d81600dd51d9"} err="failed to get container status \"830630dacbf6de2b32fc5517d3e1f9c877c463a4bf85daa9b4a1d81600dd51d9\": rpc error: code = NotFound desc = could not find container \"830630dacbf6de2b32fc5517d3e1f9c877c463a4bf85daa9b4a1d81600dd51d9\": container with ID starting with 830630dacbf6de2b32fc5517d3e1f9c877c463a4bf85daa9b4a1d81600dd51d9 not found: ID does not exist"
Jul 15 11:34:26.260936 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.260920    2563 scope.go:117] "RemoveContainer" containerID="5d06730eb1209c5226a30ee888a29ec434bcdd7fb7e3ca13eb22c1c3ce9bf523"
Jul 15 11:34:26.261180 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:34:26.261154    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d06730eb1209c5226a30ee888a29ec434bcdd7fb7e3ca13eb22c1c3ce9bf523\": container with ID starting with 5d06730eb1209c5226a30ee888a29ec434bcdd7fb7e3ca13eb22c1c3ce9bf523 not found: ID does not exist" containerID="5d06730eb1209c5226a30ee888a29ec434bcdd7fb7e3ca13eb22c1c3ce9bf523"
Jul 15 11:34:26.261275 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.261187    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d06730eb1209c5226a30ee888a29ec434bcdd7fb7e3ca13eb22c1c3ce9bf523"} err="failed to get container status \"5d06730eb1209c5226a30ee888a29ec434bcdd7fb7e3ca13eb22c1c3ce9bf523\": rpc error: code = NotFound desc = could not find container \"5d06730eb1209c5226a30ee888a29ec434bcdd7fb7e3ca13eb22c1c3ce9bf523\": container with ID starting with 5d06730eb1209c5226a30ee888a29ec434bcdd7fb7e3ca13eb22c1c3ce9bf523 not found: ID does not exist"
Jul 15 11:34:26.261275 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.261207    2563 scope.go:117] "RemoveContainer" containerID="67a49f6f806995b6cd162f3c64a65041901285792766aec56551a07c1647a8a3"
Jul 15 11:34:26.267854 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.267831    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb"]
Jul 15 11:34:26.270770 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.270751    2563 scope.go:117] "RemoveContainer" containerID="3485486a9bb2ebaecfd8c21dd9477dbb9ab663ba3bf96417502a35a90b250b69"
Jul 15 11:34:26.272098 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.272077    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb"]
Jul 15 11:34:26.278227 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.278210    2563 scope.go:117] "RemoveContainer" containerID="67a49f6f806995b6cd162f3c64a65041901285792766aec56551a07c1647a8a3"
Jul 15 11:34:26.278482 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:34:26.278466    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67a49f6f806995b6cd162f3c64a65041901285792766aec56551a07c1647a8a3\": container with ID starting with 67a49f6f806995b6cd162f3c64a65041901285792766aec56551a07c1647a8a3 not found: ID does not exist" containerID="67a49f6f806995b6cd162f3c64a65041901285792766aec56551a07c1647a8a3"
Jul 15 11:34:26.278528 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.278489    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67a49f6f806995b6cd162f3c64a65041901285792766aec56551a07c1647a8a3"} err="failed to get container status \"67a49f6f806995b6cd162f3c64a65041901285792766aec56551a07c1647a8a3\": rpc error: code = NotFound desc = could not find container \"67a49f6f806995b6cd162f3c64a65041901285792766aec56551a07c1647a8a3\": container with ID starting with 67a49f6f806995b6cd162f3c64a65041901285792766aec56551a07c1647a8a3 not found: ID does not exist"
Jul 15 11:34:26.278528 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.278504    2563 scope.go:117] "RemoveContainer" containerID="3485486a9bb2ebaecfd8c21dd9477dbb9ab663ba3bf96417502a35a90b250b69"
Jul 15 11:34:26.278745 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:34:26.278729    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3485486a9bb2ebaecfd8c21dd9477dbb9ab663ba3bf96417502a35a90b250b69\": container with ID starting with 3485486a9bb2ebaecfd8c21dd9477dbb9ab663ba3bf96417502a35a90b250b69 not found: ID does not exist" containerID="3485486a9bb2ebaecfd8c21dd9477dbb9ab663ba3bf96417502a35a90b250b69"
Jul 15 11:34:26.278784 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.278751    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3485486a9bb2ebaecfd8c21dd9477dbb9ab663ba3bf96417502a35a90b250b69"} err="failed to get container status \"3485486a9bb2ebaecfd8c21dd9477dbb9ab663ba3bf96417502a35a90b250b69\": rpc error: code = NotFound desc = could not find container \"3485486a9bb2ebaecfd8c21dd9477dbb9ab663ba3bf96417502a35a90b250b69\": container with ID starting with 3485486a9bb2ebaecfd8c21dd9477dbb9ab663ba3bf96417502a35a90b250b69 not found: ID does not exist"
Jul 15 11:34:26.282726 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.282698    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw"]
Jul 15 11:34:26.286449 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:26.286431    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw"]
Jul 15 11:34:27.682865 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:27.682830    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58d5e7d5-59a8-4d71-a937-c32ac57ab1aa" path="/var/lib/kubelet/pods/58d5e7d5-59a8-4d71-a937-c32ac57ab1aa/volumes"
Jul 15 11:34:27.683290 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:27.683274    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7" path="/var/lib/kubelet/pods/e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7/volumes"
Jul 15 11:34:36.290743 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:36.290638    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d6qjm/must-gather-4lpnh" event={"ID":"4072bab9-7867-4451-91f4-20dd1383ec9e","Type":"ContainerStarted","Data":"f824db6b5cacf78a0eaaf94544725a58a215a26fb38e519457d962e4024264b5"}
Jul 15 11:34:36.290743 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:36.290699    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d6qjm/must-gather-4lpnh" event={"ID":"4072bab9-7867-4451-91f4-20dd1383ec9e","Type":"ContainerStarted","Data":"ff811ef6cf397ace52c2dabeb29e658e8356fbdb7f316d3baf9a4f64e10e17c6"}
Jul 15 11:34:36.308291 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:36.308236    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-d6qjm/must-gather-4lpnh" podStartSLOduration=0.931079489 podStartE2EDuration="12.308218604s" totalImagesPullingTime="11.377139115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:34:36.307057527 +0000 UTC m=+2895.136572934" watchObservedRunningTime="2026-07-15 11:34:36.308218604 +0000 UTC m=+2895.137733988"
Jul 15 11:34:57.383741 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:57.383705    2563 generic.go:356] "Generic (PLEG): container finished" podID="4072bab9-7867-4451-91f4-20dd1383ec9e" containerID="ff811ef6cf397ace52c2dabeb29e658e8356fbdb7f316d3baf9a4f64e10e17c6" exitCode=0
Jul 15 11:34:57.384214 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:57.383784    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d6qjm/must-gather-4lpnh" event={"ID":"4072bab9-7867-4451-91f4-20dd1383ec9e","Type":"ContainerDied","Data":"ff811ef6cf397ace52c2dabeb29e658e8356fbdb7f316d3baf9a4f64e10e17c6"}
Jul 15 11:34:57.384214 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:57.384107    2563 scope.go:117] "RemoveContainer" containerID="ff811ef6cf397ace52c2dabeb29e658e8356fbdb7f316d3baf9a4f64e10e17c6"
Jul 15 11:34:57.680039 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:57.679949    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-d6qjm_must-gather-4lpnh_4072bab9-7867-4451-91f4-20dd1383ec9e/gather/0.log"
Jul 15 11:34:58.307820 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:58.307781    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-f872n/must-gather-8bdh8"]
Jul 15 11:34:58.308093 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:58.308080    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58d5e7d5-59a8-4d71-a937-c32ac57ab1aa" containerName="kserve-container"
Jul 15 11:34:58.308158 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:58.308095    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="58d5e7d5-59a8-4d71-a937-c32ac57ab1aa" containerName="kserve-container"
Jul 15 11:34:58.308158 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:58.308112    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7" containerName="kube-rbac-proxy"
Jul 15 11:34:58.308158 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:58.308118    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7" containerName="kube-rbac-proxy"
Jul 15 11:34:58.308158 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:58.308130    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7" containerName="kserve-container"
Jul 15 11:34:58.308158 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:58.308136    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7" containerName="kserve-container"
Jul 15 11:34:58.308319 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:58.308163    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58d5e7d5-59a8-4d71-a937-c32ac57ab1aa" containerName="kube-rbac-proxy"
Jul 15 11:34:58.308319 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:58.308172    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="58d5e7d5-59a8-4d71-a937-c32ac57ab1aa" containerName="kube-rbac-proxy"
Jul 15 11:34:58.308319 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:58.308224    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="58d5e7d5-59a8-4d71-a937-c32ac57ab1aa" containerName="kserve-container"
Jul 15 11:34:58.308319 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:58.308232    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7" containerName="kube-rbac-proxy"
Jul 15 11:34:58.308319 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:58.308238    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4a430db-cad0-4b4b-b3a1-ba9a562b6dc7" containerName="kserve-container"
Jul 15 11:34:58.308319 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:58.308247    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="58d5e7d5-59a8-4d71-a937-c32ac57ab1aa" containerName="kube-rbac-proxy"
Jul 15 11:34:58.310306 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:58.310289    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f872n/must-gather-8bdh8"
Jul 15 11:34:58.313387 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:58.313364    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f872n\"/\"kube-root-ca.crt\""
Jul 15 11:34:58.313546 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:58.313369    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-f872n\"/\"default-dockercfg-tfb7w\""
Jul 15 11:34:58.313546 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:58.313422    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f872n\"/\"openshift-service-ca.crt\""
Jul 15 11:34:58.323702 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:58.323679    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f872n/must-gather-8bdh8"]
Jul 15 11:34:58.379439 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:58.379411    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fde906f1-9c7d-4898-9850-933b897f9f2a-must-gather-output\") pod \"must-gather-8bdh8\" (UID: \"fde906f1-9c7d-4898-9850-933b897f9f2a\") " pod="openshift-must-gather-f872n/must-gather-8bdh8"
Jul 15 11:34:58.379578 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:58.379450    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb2hv\" (UniqueName: \"kubernetes.io/projected/fde906f1-9c7d-4898-9850-933b897f9f2a-kube-api-access-vb2hv\") pod \"must-gather-8bdh8\" (UID: \"fde906f1-9c7d-4898-9850-933b897f9f2a\") " pod="openshift-must-gather-f872n/must-gather-8bdh8"
Jul 15 11:34:58.480339 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:58.480306    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb2hv\" (UniqueName: \"kubernetes.io/projected/fde906f1-9c7d-4898-9850-933b897f9f2a-kube-api-access-vb2hv\") pod \"must-gather-8bdh8\" (UID: \"fde906f1-9c7d-4898-9850-933b897f9f2a\") " pod="openshift-must-gather-f872n/must-gather-8bdh8"
Jul 15 11:34:58.480775 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:58.480401    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fde906f1-9c7d-4898-9850-933b897f9f2a-must-gather-output\") pod \"must-gather-8bdh8\" (UID: \"fde906f1-9c7d-4898-9850-933b897f9f2a\") " pod="openshift-must-gather-f872n/must-gather-8bdh8"
Jul 15 11:34:58.480775 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:58.480688    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fde906f1-9c7d-4898-9850-933b897f9f2a-must-gather-output\") pod \"must-gather-8bdh8\" (UID: \"fde906f1-9c7d-4898-9850-933b897f9f2a\") " pod="openshift-must-gather-f872n/must-gather-8bdh8"
Jul 15 11:34:58.489642 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:58.489617    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb2hv\" (UniqueName: \"kubernetes.io/projected/fde906f1-9c7d-4898-9850-933b897f9f2a-kube-api-access-vb2hv\") pod \"must-gather-8bdh8\" (UID: \"fde906f1-9c7d-4898-9850-933b897f9f2a\") " pod="openshift-must-gather-f872n/must-gather-8bdh8"
Jul 15 11:34:58.619010 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:58.618928    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f872n/must-gather-8bdh8"
Jul 15 11:34:58.753548 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:58.753521    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f872n/must-gather-8bdh8"]
Jul 15 11:34:58.755783 ip-10-0-131-34 kubenswrapper[2563]: W0715 11:34:58.755753    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfde906f1_9c7d_4898_9850_933b897f9f2a.slice/crio-06e7ff3c0d4326c7675bd426235bc7f17bc5638e23422ef8a1dcb2365d6bbf19 WatchSource:0}: Error finding container 06e7ff3c0d4326c7675bd426235bc7f17bc5638e23422ef8a1dcb2365d6bbf19: Status 404 returned error can't find the container with id 06e7ff3c0d4326c7675bd426235bc7f17bc5638e23422ef8a1dcb2365d6bbf19
Jul 15 11:34:59.392564 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:34:59.392525    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f872n/must-gather-8bdh8" event={"ID":"fde906f1-9c7d-4898-9850-933b897f9f2a","Type":"ContainerStarted","Data":"06e7ff3c0d4326c7675bd426235bc7f17bc5638e23422ef8a1dcb2365d6bbf19"}
Jul 15 11:35:00.399876 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:00.399766    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f872n/must-gather-8bdh8" event={"ID":"fde906f1-9c7d-4898-9850-933b897f9f2a","Type":"ContainerStarted","Data":"851217f55927822b2546b574617f349dab6f7ff7962d2c8aebf258d475dd9202"}
Jul 15 11:35:00.399876 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:00.399818    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f872n/must-gather-8bdh8" event={"ID":"fde906f1-9c7d-4898-9850-933b897f9f2a","Type":"ContainerStarted","Data":"8687f86eedfc279eaef0759a3659348bcbc8334542f6b1d910ad19a63813b0b8"}
Jul 15 11:35:00.417497 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:00.417443    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-f872n/must-gather-8bdh8" podStartSLOduration=1.226153362 podStartE2EDuration="2.417425888s" totalImagesPullingTime="1.191272526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:34:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:35:00.416262749 +0000 UTC m=+2919.245778155" watchObservedRunningTime="2026-07-15 11:35:00.417425888 +0000 UTC m=+2919.246941340"
Jul 15 11:35:01.485449 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:01.485398    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rx95w_37ad4cef-91aa-459c-80f3-f67d36e697aa/global-pull-secret-syncer/0.log"
Jul 15 11:35:01.616186 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:01.616129    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wdxpd_eaebac66-19db-44df-85f4-887969b5984e/konnectivity-agent/0.log"
Jul 15 11:35:01.798374 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:01.798293    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-34.ec2.internal_a18a70f1e59122311577155ea3452e36/haproxy/0.log"
Jul 15 11:35:03.146550 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:03.146505    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-d6qjm/must-gather-4lpnh"]
Jul 15 11:35:03.147573 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:03.147537    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-d6qjm/must-gather-4lpnh" podUID="4072bab9-7867-4451-91f4-20dd1383ec9e" containerName="copy" containerID="cri-o://f824db6b5cacf78a0eaaf94544725a58a215a26fb38e519457d962e4024264b5" gracePeriod=2
Jul 15 11:35:03.150296 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:35:03.150262    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-4lpnh\" is forbidden: User \"system:node:ip-10-0-131-34.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-d6qjm\": no relationship found between node 'ip-10-0-131-34.ec2.internal' and this object" podUID="4072bab9-7867-4451-91f4-20dd1383ec9e" pod="openshift-must-gather-d6qjm/must-gather-4lpnh"
Jul 15 11:35:03.150493 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:03.150468    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-d6qjm/must-gather-4lpnh"]
Jul 15 11:35:03.426660 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:03.425952    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-d6qjm_must-gather-4lpnh_4072bab9-7867-4451-91f4-20dd1383ec9e/copy/0.log"
Jul 15 11:35:03.426660 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:03.426372    2563 generic.go:356] "Generic (PLEG): container finished" podID="4072bab9-7867-4451-91f4-20dd1383ec9e" containerID="f824db6b5cacf78a0eaaf94544725a58a215a26fb38e519457d962e4024264b5" exitCode=143
Jul 15 11:35:03.502841 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:03.500966    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-d6qjm_must-gather-4lpnh_4072bab9-7867-4451-91f4-20dd1383ec9e/copy/0.log"
Jul 15 11:35:03.505887 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:03.505694    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d6qjm/must-gather-4lpnh"
Jul 15 11:35:03.523163 ip-10-0-131-34 kubenswrapper[2563]: E0715 11:35:03.514322    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-4lpnh\" is forbidden: User \"system:node:ip-10-0-131-34.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-d6qjm\": no relationship found between node 'ip-10-0-131-34.ec2.internal' and this object" podUID="4072bab9-7867-4451-91f4-20dd1383ec9e" pod="openshift-must-gather-d6qjm/must-gather-4lpnh"
Jul 15 11:35:03.533212 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:03.527986    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j24s7\" (UniqueName: \"kubernetes.io/projected/4072bab9-7867-4451-91f4-20dd1383ec9e-kube-api-access-j24s7\") pod \"4072bab9-7867-4451-91f4-20dd1383ec9e\" (UID: \"4072bab9-7867-4451-91f4-20dd1383ec9e\") "
Jul 15 11:35:03.533212 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:03.528034    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4072bab9-7867-4451-91f4-20dd1383ec9e-must-gather-output\") pod \"4072bab9-7867-4451-91f4-20dd1383ec9e\" (UID: \"4072bab9-7867-4451-91f4-20dd1383ec9e\") "
Jul 15 11:35:03.533212 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:03.529634    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4072bab9-7867-4451-91f4-20dd1383ec9e-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "4072bab9-7867-4451-91f4-20dd1383ec9e" (UID: "4072bab9-7867-4451-91f4-20dd1383ec9e"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:35:03.546162 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:03.539912    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4072bab9-7867-4451-91f4-20dd1383ec9e-kube-api-access-j24s7" (OuterVolumeSpecName: "kube-api-access-j24s7") pod "4072bab9-7867-4451-91f4-20dd1383ec9e" (UID: "4072bab9-7867-4451-91f4-20dd1383ec9e"). InnerVolumeSpecName "kube-api-access-j24s7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:35:03.629528 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:03.629487    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j24s7\" (UniqueName: \"kubernetes.io/projected/4072bab9-7867-4451-91f4-20dd1383ec9e-kube-api-access-j24s7\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:35:03.629727 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:03.629527    2563 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4072bab9-7867-4451-91f4-20dd1383ec9e-must-gather-output\") on node \"ip-10-0-131-34.ec2.internal\" DevicePath \"\""
Jul 15 11:35:03.682116 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:03.682016    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4072bab9-7867-4451-91f4-20dd1383ec9e" path="/var/lib/kubelet/pods/4072bab9-7867-4451-91f4-20dd1383ec9e/volumes"
Jul 15 11:35:04.433951 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:04.433917    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-d6qjm_must-gather-4lpnh_4072bab9-7867-4451-91f4-20dd1383ec9e/copy/0.log"
Jul 15 11:35:04.434565 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:04.434430    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d6qjm/must-gather-4lpnh"
Jul 15 11:35:04.434565 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:04.434429    2563 scope.go:117] "RemoveContainer" containerID="f824db6b5cacf78a0eaaf94544725a58a215a26fb38e519457d962e4024264b5"
Jul 15 11:35:04.445508 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:04.445413    2563 scope.go:117] "RemoveContainer" containerID="ff811ef6cf397ace52c2dabeb29e658e8356fbdb7f316d3baf9a4f64e10e17c6"
Jul 15 11:35:06.051579 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:06.051548    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-zcbpw_96740e9c-b52a-4571-92eb-60c936c00d7c/kube-state-metrics/0.log"
Jul 15 11:35:06.089550 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:06.089515    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-zcbpw_96740e9c-b52a-4571-92eb-60c936c00d7c/kube-rbac-proxy-main/0.log"
Jul 15 11:35:06.122971 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:06.122929    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-zcbpw_96740e9c-b52a-4571-92eb-60c936c00d7c/kube-rbac-proxy-self/0.log"
Jul 15 11:35:06.203899 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:06.203846    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-l6qwd_d4b1598b-43de-43d0-98e9-0ec886523487/monitoring-plugin/0.log"
Jul 15 11:35:06.327654 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:06.327556    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j2hs4_d5bb1c2f-6490-4e67-a722-0c360b3cc058/node-exporter/0.log"
Jul 15 11:35:06.354478 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:06.354429    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j2hs4_d5bb1c2f-6490-4e67-a722-0c360b3cc058/kube-rbac-proxy/0.log"
Jul 15 11:35:06.387786 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:06.387578    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j2hs4_d5bb1c2f-6490-4e67-a722-0c360b3cc058/init-textfile/0.log"
Jul 15 11:35:06.641107 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:06.641009    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4f1d838-0655-4218-8247-4233d2e0ac04/prometheus/0.log"
Jul 15 11:35:06.674835 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:06.674803    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4f1d838-0655-4218-8247-4233d2e0ac04/config-reloader/0.log"
Jul 15 11:35:06.731386 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:06.731360    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4f1d838-0655-4218-8247-4233d2e0ac04/thanos-sidecar/0.log"
Jul 15 11:35:06.788036 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:06.788006    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4f1d838-0655-4218-8247-4233d2e0ac04/kube-rbac-proxy-web/0.log"
Jul 15 11:35:06.857456 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:06.857413    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4f1d838-0655-4218-8247-4233d2e0ac04/kube-rbac-proxy/0.log"
Jul 15 11:35:06.906975 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:06.906869    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4f1d838-0655-4218-8247-4233d2e0ac04/kube-rbac-proxy-thanos/0.log"
Jul 15 11:35:06.935701 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:06.935667    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4f1d838-0655-4218-8247-4233d2e0ac04/init-config-reloader/0.log"
Jul 15 11:35:06.978779 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:06.978749    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-kd448_07d78928-8e15-4a72-93e3-af348e9593b2/prometheus-operator/0.log"
Jul 15 11:35:07.003891 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:07.003847    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-kd448_07d78928-8e15-4a72-93e3-af348e9593b2/kube-rbac-proxy/0.log"
Jul 15 11:35:07.040182 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:07.040134    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-8rwts_cf2605ca-f1b1-4931-8b11-c3badbcdb1cd/prometheus-operator-admission-webhook/0.log"
Jul 15 11:35:09.559485 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:09.559444    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8b6dfdb7d-z5qxq_e27b46b3-6e7b-422b-ba47-9c4df4d9a99d/console/0.log"
Jul 15 11:35:10.009491 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:10.009453    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-f872n/perf-node-gather-daemonset-tqbdj"]
Jul 15 11:35:10.009894 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:10.009870    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4072bab9-7867-4451-91f4-20dd1383ec9e" containerName="copy"
Jul 15 11:35:10.009992 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:10.009897    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4072bab9-7867-4451-91f4-20dd1383ec9e" containerName="copy"
Jul 15 11:35:10.009992 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:10.009913    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4072bab9-7867-4451-91f4-20dd1383ec9e" containerName="gather"
Jul 15 11:35:10.009992 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:10.009922    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4072bab9-7867-4451-91f4-20dd1383ec9e" containerName="gather"
Jul 15 11:35:10.010191 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:10.009993    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4072bab9-7867-4451-91f4-20dd1383ec9e" containerName="copy"
Jul 15 11:35:10.010191 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:10.010008    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4072bab9-7867-4451-91f4-20dd1383ec9e" containerName="gather"
Jul 15 11:35:10.015392 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:10.015368    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f872n/perf-node-gather-daemonset-tqbdj"
Jul 15 11:35:10.025586 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:10.025557    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f872n/perf-node-gather-daemonset-tqbdj"]
Jul 15 11:35:10.095036 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:10.095005    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50b664e4-8e6c-4a92-80e3-ecd18b037c07-lib-modules\") pod \"perf-node-gather-daemonset-tqbdj\" (UID: \"50b664e4-8e6c-4a92-80e3-ecd18b037c07\") " pod="openshift-must-gather-f872n/perf-node-gather-daemonset-tqbdj"
Jul 15 11:35:10.095243 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:10.095058    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxgq6\" (UniqueName: \"kubernetes.io/projected/50b664e4-8e6c-4a92-80e3-ecd18b037c07-kube-api-access-fxgq6\") pod \"perf-node-gather-daemonset-tqbdj\" (UID: \"50b664e4-8e6c-4a92-80e3-ecd18b037c07\") " pod="openshift-must-gather-f872n/perf-node-gather-daemonset-tqbdj"
Jul 15 11:35:10.095243 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:10.095091    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/50b664e4-8e6c-4a92-80e3-ecd18b037c07-podres\") pod \"perf-node-gather-daemonset-tqbdj\" (UID: \"50b664e4-8e6c-4a92-80e3-ecd18b037c07\") " pod="openshift-must-gather-f872n/perf-node-gather-daemonset-tqbdj"
Jul 15 11:35:10.095243 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:10.095119    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50b664e4-8e6c-4a92-80e3-ecd18b037c07-sys\") pod \"perf-node-gather-daemonset-tqbdj\" (UID: \"50b664e4-8e6c-4a92-80e3-ecd18b037c07\") " pod="openshift-must-gather-f872n/perf-node-gather-daemonset-tqbdj"
Jul 15 11:35:10.095243 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:10.095163    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/50b664e4-8e6c-4a92-80e3-ecd18b037c07-proc\") pod \"perf-node-gather-daemonset-tqbdj\" (UID: \"50b664e4-8e6c-4a92-80e3-ecd18b037c07\") " pod="openshift-must-gather-f872n/perf-node-gather-daemonset-tqbdj"
Jul 15 11:35:10.196158 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:10.196111    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxgq6\" (UniqueName: \"kubernetes.io/projected/50b664e4-8e6c-4a92-80e3-ecd18b037c07-kube-api-access-fxgq6\") pod \"perf-node-gather-daemonset-tqbdj\" (UID: \"50b664e4-8e6c-4a92-80e3-ecd18b037c07\") " pod="openshift-must-gather-f872n/perf-node-gather-daemonset-tqbdj"
Jul 15 11:35:10.196340 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:10.196163    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/50b664e4-8e6c-4a92-80e3-ecd18b037c07-podres\") pod \"perf-node-gather-daemonset-tqbdj\" (UID: \"50b664e4-8e6c-4a92-80e3-ecd18b037c07\") " pod="openshift-must-gather-f872n/perf-node-gather-daemonset-tqbdj"
Jul 15 11:35:10.196340 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:10.196208    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50b664e4-8e6c-4a92-80e3-ecd18b037c07-sys\") pod \"perf-node-gather-daemonset-tqbdj\" (UID: \"50b664e4-8e6c-4a92-80e3-ecd18b037c07\") " pod="openshift-must-gather-f872n/perf-node-gather-daemonset-tqbdj"
Jul 15 11:35:10.196340 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:10.196247    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/50b664e4-8e6c-4a92-80e3-ecd18b037c07-proc\") pod \"perf-node-gather-daemonset-tqbdj\" (UID: \"50b664e4-8e6c-4a92-80e3-ecd18b037c07\") " pod="openshift-must-gather-f872n/perf-node-gather-daemonset-tqbdj"
Jul 15 11:35:10.196340 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:10.196295    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50b664e4-8e6c-4a92-80e3-ecd18b037c07-lib-modules\") pod \"perf-node-gather-daemonset-tqbdj\" (UID: \"50b664e4-8e6c-4a92-80e3-ecd18b037c07\") " pod="openshift-must-gather-f872n/perf-node-gather-daemonset-tqbdj"
Jul 15 11:35:10.196531 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:10.196341    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50b664e4-8e6c-4a92-80e3-ecd18b037c07-sys\") pod \"perf-node-gather-daemonset-tqbdj\" (UID: \"50b664e4-8e6c-4a92-80e3-ecd18b037c07\") " pod="openshift-must-gather-f872n/perf-node-gather-daemonset-tqbdj"
Jul 15 11:35:10.196531 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:10.196373    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/50b664e4-8e6c-4a92-80e3-ecd18b037c07-proc\") pod \"perf-node-gather-daemonset-tqbdj\" (UID: \"50b664e4-8e6c-4a92-80e3-ecd18b037c07\") " pod="openshift-must-gather-f872n/perf-node-gather-daemonset-tqbdj"
Jul 15 11:35:10.196531 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:10.196348    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/50b664e4-8e6c-4a92-80e3-ecd18b037c07-podres\") pod \"perf-node-gather-daemonset-tqbdj\" (UID: \"50b664e4-8e6c-4a92-80e3-ecd18b037c07\") " pod="openshift-must-gather-f872n/perf-node-gather-daemonset-tqbdj"
Jul 15 11:35:10.196531 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:10.196435    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50b664e4-8e6c-4a92-80e3-ecd18b037c07-lib-modules\") pod \"perf-node-gather-daemonset-tqbdj\" (UID: \"50b664e4-8e6c-4a92-80e3-ecd18b037c07\") " pod="openshift-must-gather-f872n/perf-node-gather-daemonset-tqbdj"
Jul 15 11:35:10.204730 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:10.204700    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxgq6\" (UniqueName: \"kubernetes.io/projected/50b664e4-8e6c-4a92-80e3-ecd18b037c07-kube-api-access-fxgq6\") pod \"perf-node-gather-daemonset-tqbdj\" (UID: \"50b664e4-8e6c-4a92-80e3-ecd18b037c07\") " pod="openshift-must-gather-f872n/perf-node-gather-daemonset-tqbdj"
Jul 15 11:35:10.330320 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:10.330231    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f872n/perf-node-gather-daemonset-tqbdj"
Jul 15 11:35:10.499177 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:10.497819    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f872n/perf-node-gather-daemonset-tqbdj"]
Jul 15 11:35:11.243745 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:11.243715    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r52qn_b38e6e47-2551-4b35-a480-cda496f5af1f/dns/0.log"
Jul 15 11:35:11.266885 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:11.266860    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r52qn_b38e6e47-2551-4b35-a480-cda496f5af1f/kube-rbac-proxy/0.log"
Jul 15 11:35:11.315387 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:11.315364    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-klpd8_7e4a1af3-5ae7-495e-bd96-25df977fef6f/dns-node-resolver/0.log"
Jul 15 11:35:11.471910 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:11.471871    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f872n/perf-node-gather-daemonset-tqbdj" event={"ID":"50b664e4-8e6c-4a92-80e3-ecd18b037c07","Type":"ContainerStarted","Data":"45dec1654972a38f5e23675006b4bb630912ae17699c11d1eaebfbff95fcde07"}
Jul 15 11:35:11.472069 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:11.471937    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f872n/perf-node-gather-daemonset-tqbdj" event={"ID":"50b664e4-8e6c-4a92-80e3-ecd18b037c07","Type":"ContainerStarted","Data":"92d5e0d6585fbdb89bbad2c0932eb5f36e3e0b0ac923800847fcbbb1207800cd"}
Jul 15 11:35:11.472069 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:11.471979    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-f872n/perf-node-gather-daemonset-tqbdj"
Jul 15 11:35:11.494969 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:11.494864    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-f872n/perf-node-gather-daemonset-tqbdj" podStartSLOduration=2.494847655 podStartE2EDuration="2.494847655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 11:35:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:35:11.493026302 +0000 UTC m=+2930.322541708" watchObservedRunningTime="2026-07-15 11:35:11.494847655 +0000 UTC m=+2930.324363059"
Jul 15 11:35:11.858799 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:11.858724    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-b4gsj_2a7699c1-9758-4c35-8b4c-aed46512560e/node-ca/0.log"
Jul 15 11:35:13.051395 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:13.051368    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7ltzc_92ddad92-166b-43ce-a067-0007c9f8b30d/serve-healthcheck-canary/0.log"
Jul 15 11:35:13.481665 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:13.481633    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f8rzs_8c72e99f-bc5f-4851-a890-2605a51de0e0/kube-rbac-proxy/0.log"
Jul 15 11:35:13.509562 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:13.509535    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f8rzs_8c72e99f-bc5f-4851-a890-2605a51de0e0/exporter/0.log"
Jul 15 11:35:13.539917 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:13.539890    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f8rzs_8c72e99f-bc5f-4851-a890-2605a51de0e0/extractor/0.log"
Jul 15 11:35:16.070881 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:16.070848    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-7zmgw_b1ca34b8-2a49-454b-9dcf-e684a84d9198/s3-init/0.log"
Jul 15 11:35:16.090094 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:16.090070    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-7zmgw_b1ca34b8-2a49-454b-9dcf-e684a84d9198/download-hf-model/0.log"
Jul 15 11:35:17.488435 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:17.488406    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-f872n/perf-node-gather-daemonset-tqbdj"
Jul 15 11:35:22.198631 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:22.198599    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l8v8c_902fd91e-2979-4d2c-b428-ff5e62d8203b/kube-multus-additional-cni-plugins/0.log"
Jul 15 11:35:22.218525 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:22.218501    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l8v8c_902fd91e-2979-4d2c-b428-ff5e62d8203b/egress-router-binary-copy/0.log"
Jul 15 11:35:22.244047 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:22.244026    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l8v8c_902fd91e-2979-4d2c-b428-ff5e62d8203b/cni-plugins/0.log"
Jul 15 11:35:22.264926 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:22.264900    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l8v8c_902fd91e-2979-4d2c-b428-ff5e62d8203b/bond-cni-plugin/0.log"
Jul 15 11:35:22.285629 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:22.285607    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l8v8c_902fd91e-2979-4d2c-b428-ff5e62d8203b/routeoverride-cni/0.log"
Jul 15 11:35:22.305200 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:22.305175    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l8v8c_902fd91e-2979-4d2c-b428-ff5e62d8203b/whereabouts-cni-bincopy/0.log"
Jul 15 11:35:22.327700 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:22.327664    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l8v8c_902fd91e-2979-4d2c-b428-ff5e62d8203b/whereabouts-cni/0.log"
Jul 15 11:35:22.370673 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:22.370644    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fcnjt_cae13509-00de-48df-ae3d-943755c4b2e9/kube-multus/0.log"
Jul 15 11:35:22.448775 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:22.448702    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-84n76_c8d1da87-af98-4f6b-af6a-27a5fbabba60/network-metrics-daemon/0.log"
Jul 15 11:35:22.466715 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:22.466691    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-84n76_c8d1da87-af98-4f6b-af6a-27a5fbabba60/kube-rbac-proxy/0.log"
Jul 15 11:35:23.695582 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:23.695549    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk49n_beb68899-ffb3-436f-8e17-43597aad4030/ovn-controller/0.log"
Jul 15 11:35:23.712697 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:23.712663    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk49n_beb68899-ffb3-436f-8e17-43597aad4030/ovn-acl-logging/1.log"
Jul 15 11:35:23.712881 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:23.712864    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk49n_beb68899-ffb3-436f-8e17-43597aad4030/ovn-acl-logging/0.log"
Jul 15 11:35:23.734722 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:23.734698    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk49n_beb68899-ffb3-436f-8e17-43597aad4030/kube-rbac-proxy-node/0.log"
Jul 15 11:35:23.757248 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:23.757214    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk49n_beb68899-ffb3-436f-8e17-43597aad4030/kube-rbac-proxy-ovn-metrics/0.log"
Jul 15 11:35:23.777784 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:23.777754    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk49n_beb68899-ffb3-436f-8e17-43597aad4030/northd/0.log"
Jul 15 11:35:23.798900 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:23.798874    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk49n_beb68899-ffb3-436f-8e17-43597aad4030/nbdb/0.log"
Jul 15 11:35:23.819925 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:23.819897    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk49n_beb68899-ffb3-436f-8e17-43597aad4030/sbdb/0.log"
Jul 15 11:35:23.893660 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:23.893628    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk49n_beb68899-ffb3-436f-8e17-43597aad4030/ovnkube-controller/0.log"
Jul 15 11:35:24.885403 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:24.885373    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-9mhqc_b5f21679-bed4-4e21-9a3b-22ec261aec90/network-check-target-container/0.log"
Jul 15 11:35:25.793471 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:25.793442    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-4jcc7_2f949cb2-6105-448d-b4df-82ab30888fef/iptables-alerter/0.log"
Jul 15 11:35:26.580991 ip-10-0-131-34 kubenswrapper[2563]: I0715 11:35:26.580952    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-4445h_2456e018-71e0-42bc-8153-64c597b91f17/tuned/0.log"