Jul 02 12:38:51.738556 ip-10-0-137-127 systemd[1]: Starting Kubernetes Kubelet...
Jul 02 12:38:52.190003 ip-10-0-137-127 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 12:38:52.190003 ip-10-0-137-127 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 02 12:38:52.190003 ip-10-0-137-127 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 12:38:52.190003 ip-10-0-137-127 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 02 12:38:52.190003 ip-10-0-137-127 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 12:38:52.191247 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.189987    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 02 12:38:52.197212 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197194    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 12:38:52.197212 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197211    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 12:38:52.197294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197216    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 12:38:52.197294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197219    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 12:38:52.197294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197223    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 12:38:52.197294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197226    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 12:38:52.197294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197229    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 12:38:52.197294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197234    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 12:38:52.197294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197238    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 12:38:52.197294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197241    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 12:38:52.197294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197244    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 12:38:52.197294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197263    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 12:38:52.197294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197268    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 12:38:52.197294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197274    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 12:38:52.197294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197277    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 12:38:52.197294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197280    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 12:38:52.197294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197282    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 12:38:52.197294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197285    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 12:38:52.197294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197288    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 12:38:52.197294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197291    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 12:38:52.197294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197294    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 12:38:52.197294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197297    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 12:38:52.197788 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197301    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 12:38:52.197788 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197304    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 12:38:52.197788 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197307    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 12:38:52.197788 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197310    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 12:38:52.197788 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197313    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 12:38:52.197788 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197315    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 12:38:52.197788 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197318    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 12:38:52.197788 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197321    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 12:38:52.197788 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197325    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 12:38:52.197788 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197328    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 12:38:52.197788 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197331    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 12:38:52.197788 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197334    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 12:38:52.197788 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197336    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 12:38:52.197788 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197339    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 12:38:52.197788 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197342    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 12:38:52.197788 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197344    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 12:38:52.197788 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197347    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 12:38:52.197788 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197350    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 12:38:52.197788 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197353    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 12:38:52.197788 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197355    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 12:38:52.198336 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197358    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 12:38:52.198336 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197361    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 12:38:52.198336 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197364    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 12:38:52.198336 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197367    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 12:38:52.198336 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197369    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 12:38:52.198336 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197372    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 12:38:52.198336 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197375    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 12:38:52.198336 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197378    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 12:38:52.198336 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197382    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 12:38:52.198336 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197386    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 12:38:52.198336 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197389    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 12:38:52.198336 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197393    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 12:38:52.198336 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197396    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 12:38:52.198336 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197401    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 12:38:52.198336 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197404    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 12:38:52.198336 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197407    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 12:38:52.198336 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197410    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 12:38:52.198336 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197413    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 12:38:52.198336 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197415    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 12:38:52.198336 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197419    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 12:38:52.198861 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197422    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 12:38:52.198861 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197425    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 12:38:52.198861 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197428    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 12:38:52.198861 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197431    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 12:38:52.198861 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197434    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 12:38:52.198861 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197438    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 12:38:52.198861 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197441    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 12:38:52.198861 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197444    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 12:38:52.198861 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197447    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 12:38:52.198861 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197450    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 12:38:52.198861 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197454    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 12:38:52.198861 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197457    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 12:38:52.198861 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197460    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 12:38:52.198861 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197463    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 12:38:52.198861 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197466    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 12:38:52.198861 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197469    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 12:38:52.198861 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197472    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 12:38:52.198861 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197475    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 12:38:52.198861 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197477    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 12:38:52.199374 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197480    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 12:38:52.199374 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197483    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 12:38:52.199374 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197486    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 12:38:52.199374 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197488    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 12:38:52.199374 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197491    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 12:38:52.199374 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197494    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 12:38:52.199374 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197498    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 12:38:52.199374 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197501    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 12:38:52.199374 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197504    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 12:38:52.199374 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197507    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 12:38:52.199374 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197509    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 12:38:52.199374 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197512    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 12:38:52.199374 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197514    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 12:38:52.199374 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197517    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 12:38:52.199374 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197520    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 12:38:52.199374 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197523    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 12:38:52.199374 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197525    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 12:38:52.199374 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.197528    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 12:38:52.200390 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200377    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 12:38:52.200390 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200389    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 12:38:52.200452 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200393    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 12:38:52.200452 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200396    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 12:38:52.200452 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200400    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 12:38:52.200452 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200403    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 12:38:52.200452 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200407    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 12:38:52.200452 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200409    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 12:38:52.200452 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200412    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 12:38:52.200452 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200416    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 12:38:52.200452 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200419    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 12:38:52.200452 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200422    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 12:38:52.200452 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200425    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 12:38:52.200452 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200428    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 12:38:52.200452 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200431    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 12:38:52.200452 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200434    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 12:38:52.200452 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200437    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 12:38:52.200452 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200440    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 12:38:52.200452 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200442    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 12:38:52.200452 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200445    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 12:38:52.200452 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200448    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 12:38:52.200452 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200450    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 12:38:52.201000 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200453    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 12:38:52.201000 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200456    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 12:38:52.201000 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200459    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 12:38:52.201000 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200464    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 12:38:52.201000 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200468    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 12:38:52.201000 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200471    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 12:38:52.201000 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200474    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 12:38:52.201000 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200477    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 12:38:52.201000 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200480    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 12:38:52.201000 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200483    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 12:38:52.201000 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200487    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 12:38:52.201000 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200491    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 12:38:52.201000 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200495    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 12:38:52.201000 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200497    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 12:38:52.201000 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200500    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 12:38:52.201000 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200503    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 12:38:52.201000 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200506    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 12:38:52.201000 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200510    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 12:38:52.201000 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200514    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 12:38:52.201549 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200517    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 12:38:52.201549 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200520    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 12:38:52.201549 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200523    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 12:38:52.201549 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200527    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 12:38:52.201549 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200530    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 12:38:52.201549 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200533    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 12:38:52.201549 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200536    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 12:38:52.201549 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200538    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 12:38:52.201549 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200541    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 12:38:52.201549 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200544    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 12:38:52.201549 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200547    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 12:38:52.201549 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200549    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 12:38:52.201549 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200552    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 12:38:52.201549 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200555    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 12:38:52.201549 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200559    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 12:38:52.201549 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200562    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 12:38:52.201549 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200565    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 12:38:52.201549 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200568    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 12:38:52.201549 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200571    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 12:38:52.201549 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200575    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 12:38:52.202060 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200578    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 12:38:52.202060 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200580    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 12:38:52.202060 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200583    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 12:38:52.202060 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200586    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 12:38:52.202060 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200589    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 12:38:52.202060 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200591    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 12:38:52.202060 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200594    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 12:38:52.202060 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200597    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 12:38:52.202060 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200601    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 12:38:52.202060 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200603    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 12:38:52.202060 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200606    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 12:38:52.202060 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200609    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 12:38:52.202060 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200612    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 12:38:52.202060 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200615    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 12:38:52.202060 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200617    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 12:38:52.202060 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200620    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 12:38:52.202060 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200623    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 12:38:52.202060 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200626    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 12:38:52.202060 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200630    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 12:38:52.202060 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200633    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 12:38:52.202608 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200636    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 12:38:52.202608 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200639    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 12:38:52.202608 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200642    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 12:38:52.202608 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200645    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 12:38:52.202608 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200648    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 12:38:52.202608 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200650    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 12:38:52.202608 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200653    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 12:38:52.202608 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200656    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 12:38:52.202608 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200659    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 12:38:52.202608 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200661    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 12:38:52.202608 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200664    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 12:38:52.202608 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200667    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 12:38:52.202608 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200669    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 12:38:52.202608 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200672    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 12:38:52.202608 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200675    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 12:38:52.202608 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200677    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 12:38:52.202608 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200680    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 12:38:52.202608 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.200683    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 12:38:52.202608 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201073    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 12:38:52.202608 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201078    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 12:38:52.203109 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201082    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 12:38:52.203109 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201085    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 12:38:52.203109 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201090    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 12:38:52.203109 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201093    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 12:38:52.203109 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201096    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 12:38:52.203109 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201100    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 12:38:52.203109 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201103    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 12:38:52.203109 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201106    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 12:38:52.203109 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201110    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 12:38:52.203109 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201114    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 12:38:52.203109 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201119    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 12:38:52.203109 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201123    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 12:38:52.203109 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201126    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 12:38:52.203109 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201129    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 12:38:52.203109 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201132    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 12:38:52.203109 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201135    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 12:38:52.203109 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201138    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 12:38:52.203109 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201141    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 12:38:52.203109 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201144    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 12:38:52.203613 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201146    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 12:38:52.203613 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201149    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 12:38:52.203613 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201152    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 12:38:52.203613 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201155    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 12:38:52.203613 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201157    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 12:38:52.203613 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201160    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 12:38:52.203613 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201163    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 12:38:52.203613 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201166    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 12:38:52.203613 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201168    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 12:38:52.203613 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201171    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 12:38:52.203613 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201175    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 12:38:52.203613 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201177    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 12:38:52.203613 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201180    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 12:38:52.203613 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201183    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 12:38:52.203613 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201186    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 12:38:52.203613 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201188    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 12:38:52.203613 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201191    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 12:38:52.203613 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201194    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 12:38:52.203613 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201197    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 12:38:52.203613 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201200    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 12:38:52.204126 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201203    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 12:38:52.204126 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201205    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 12:38:52.204126 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201208    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 12:38:52.204126 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201214    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 12:38:52.204126 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201216    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 12:38:52.204126 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201219    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 12:38:52.204126 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201221    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 12:38:52.204126 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201224    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 12:38:52.204126 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201227    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 12:38:52.204126 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201230    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 12:38:52.204126 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201232    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 12:38:52.204126 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201235    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 12:38:52.204126 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201238    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 12:38:52.204126 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201240    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 12:38:52.204126 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201243    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 12:38:52.204126 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201246    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 12:38:52.204126 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201249    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 12:38:52.204126 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201251    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 12:38:52.204126 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201269    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 12:38:52.204126 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201272    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 12:38:52.204665 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201275    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 12:38:52.204665 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201279    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 12:38:52.204665 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201283    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 12:38:52.204665 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201286    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 12:38:52.204665 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201290    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 12:38:52.204665 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201293    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 12:38:52.204665 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201296    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 12:38:52.204665 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201299    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 12:38:52.204665 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201304    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 12:38:52.204665 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201307    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 12:38:52.204665 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201310    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 12:38:52.204665 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201313    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 12:38:52.204665 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201316    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 12:38:52.204665 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201319    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 12:38:52.204665 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201322    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 12:38:52.204665 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201329    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 12:38:52.204665 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201333    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 12:38:52.204665 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201335    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 12:38:52.204665 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201338    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 12:38:52.205147 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201341    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 12:38:52.205147 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201343    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 12:38:52.205147 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201346    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 12:38:52.205147 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201349    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 12:38:52.205147 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201352    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 12:38:52.205147 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201354    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 12:38:52.205147 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201357    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 12:38:52.205147 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201360    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 12:38:52.205147 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201362    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 12:38:52.205147 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201365    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 12:38:52.205147 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201368    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 12:38:52.205147 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201371    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 12:38:52.205147 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201373    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 12:38:52.205147 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201376    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 12:38:52.205147 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201379    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 12:38:52.205147 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201382    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 12:38:52.205147 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201384    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 12:38:52.205147 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201387    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 12:38:52.205147 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.201389    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 12:38:52.205147 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201466    2578 flags.go:64] FLAG: --address="0.0.0.0"
Jul 02 12:38:52.205676 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201474    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 02 12:38:52.205676 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201481    2578 flags.go:64] FLAG: --anonymous-auth="true"
Jul 02 12:38:52.205676 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201486    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 02 12:38:52.205676 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201490    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 02 12:38:52.205676 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201494    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 02 12:38:52.205676 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201499    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 02 12:38:52.205676 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201506    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 02 12:38:52.205676 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201510    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 02 12:38:52.205676 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201515    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 02 12:38:52.205676 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201520    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 02 12:38:52.205676 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201524    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 02 12:38:52.205676 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201528    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 02 12:38:52.205676 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201531    2578 flags.go:64] FLAG: --cgroup-root=""
Jul 02 12:38:52.205676 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201534    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 02 12:38:52.205676 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201537    2578 flags.go:64] FLAG: --client-ca-file=""
Jul 02 12:38:52.205676 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201540    2578 flags.go:64] FLAG: --cloud-provider="external"
Jul 02 12:38:52.205676 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201544    2578 flags.go:64] FLAG: --cluster-dns="[]"
Jul 02 12:38:52.205676 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201548    2578 flags.go:64] FLAG: --cluster-domain=""
Jul 02 12:38:52.205676 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201551    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 02 12:38:52.205676 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201555    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 02 12:38:52.205676 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201558    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 02 12:38:52.205676 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201562    2578 flags.go:64] FLAG: --container-log-max-files="5"
Jul 02 12:38:52.205676 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201566    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 02 12:38:52.205676 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201570    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 02 12:38:52.206289 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201573    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 02 12:38:52.206289 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201577    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 02 12:38:52.206289 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201580    2578 flags.go:64] FLAG: --contention-profiling="false"
Jul 02 12:38:52.206289 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201583    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 02 12:38:52.206289 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201587    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 02 12:38:52.206289 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201590    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 02 12:38:52.206289 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201593    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 02 12:38:52.206289 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201598    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 02 12:38:52.206289 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201601    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 02 12:38:52.206289 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201604    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 02 12:38:52.206289 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201608    2578 flags.go:64] FLAG: --enable-load-reader="false"
Jul 02 12:38:52.206289 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201611    2578 flags.go:64] FLAG: --enable-server="true"
Jul 02 12:38:52.206289 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201614    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 02 12:38:52.206289 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201619    2578 flags.go:64] FLAG: --event-burst="100"
Jul 02 12:38:52.206289 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201623    2578 flags.go:64] FLAG: --event-qps="50"
Jul 02 12:38:52.206289 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201626    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 02 12:38:52.206289 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201630    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 02 12:38:52.206289 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201633    2578 flags.go:64] FLAG: --eviction-hard=""
Jul 02 12:38:52.206289 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201638    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 02 12:38:52.206289 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201641    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 02 12:38:52.206289 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201644    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 02 12:38:52.206289 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201648    2578 flags.go:64] FLAG: --eviction-soft=""
Jul 02 12:38:52.206289 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201651    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 02 12:38:52.206289 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201654    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 02 12:38:52.206289 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201657    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 02 12:38:52.207011 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201660    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 02 12:38:52.207011 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201663    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 02 12:38:52.207011 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201667    2578 flags.go:64] FLAG: --fail-swap-on="true"
Jul 02 12:38:52.207011 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201670    2578 flags.go:64] FLAG: --feature-gates=""
Jul 02 12:38:52.207011 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201673    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 02 12:38:52.207011 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201677    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 02 12:38:52.207011 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201680    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 02 12:38:52.207011 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201683    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 02 12:38:52.207011 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201687    2578 flags.go:64] FLAG: --healthz-port="10248"
Jul 02 12:38:52.207011 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201690    2578 flags.go:64] FLAG: --help="false"
Jul 02 12:38:52.207011 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201693    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-137-127.ec2.internal"
Jul 02 12:38:52.207011 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201696    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 02 12:38:52.207011 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201700    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 02 12:38:52.207011 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201703    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 02 12:38:52.207011 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201706    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 02 12:38:52.207011 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201711    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 02 12:38:52.207011 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201714    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 02 12:38:52.207011 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201717    2578 flags.go:64] FLAG: --image-service-endpoint=""
Jul 02 12:38:52.207011 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201720    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 02 12:38:52.207011 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201723    2578 flags.go:64] FLAG: --kube-api-burst="100"
Jul 02 12:38:52.207011 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201726    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 02 12:38:52.207011 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201730    2578 flags.go:64] FLAG: --kube-api-qps="50"
Jul 02 12:38:52.207011 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201733    2578 flags.go:64] FLAG: --kube-reserved=""
Jul 02 12:38:52.207011 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201736    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 02 12:38:52.207662 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201739    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 02 12:38:52.207662 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201743    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 02 12:38:52.207662 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201746    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 02 12:38:52.207662 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201749    2578 flags.go:64] FLAG: --lock-file=""
Jul 02 12:38:52.207662 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201753    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 02 12:38:52.207662 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201756    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 02 12:38:52.207662 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201759    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 02 12:38:52.207662 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201765    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 02 12:38:52.207662 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201768    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 02 12:38:52.207662 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201771    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 02 12:38:52.207662 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201774    2578 flags.go:64] FLAG: --logging-format="text"
Jul 02 12:38:52.207662 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201777    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 02 12:38:52.207662 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201780    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 02 12:38:52.207662 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201783    2578 flags.go:64] FLAG: --manifest-url=""
Jul 02 12:38:52.207662 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201787    2578 flags.go:64] FLAG: --manifest-url-header=""
Jul 02 12:38:52.207662 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201791    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 02 12:38:52.207662 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201794    2578 flags.go:64] FLAG: --max-open-files="1000000"
Jul 02 12:38:52.207662 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201798    2578 flags.go:64] FLAG: --max-pods="110"
Jul 02 12:38:52.207662 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201801    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 02 12:38:52.207662 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201805    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 02 12:38:52.207662 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201808    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 02 12:38:52.207662 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201811    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 02 12:38:52.207662 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201814    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 02 12:38:52.207662 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201821    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 02 12:38:52.207662 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201824    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 02 12:38:52.208346 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201832    2578 flags.go:64] FLAG: --node-status-max-images="50"
Jul 02 12:38:52.208346 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201836    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 02 12:38:52.208346 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201839    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 02 12:38:52.208346 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201842    2578 flags.go:64] FLAG: --pod-cidr=""
Jul 02 12:38:52.208346 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201845    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04"
Jul 02 12:38:52.208346 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201851    2578 flags.go:64] FLAG: --pod-manifest-path=""
Jul 02 12:38:52.208346 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201854    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 02 12:38:52.208346 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201857    2578 flags.go:64] FLAG: --pods-per-core="0"
Jul 02 12:38:52.208346 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201861    2578 flags.go:64] FLAG: --port="10250"
Jul 02 12:38:52.208346 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201864    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 02 12:38:52.208346 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201868    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08a76f20815a50505"
Jul 02 12:38:52.208346 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201871    2578 flags.go:64] FLAG: --qos-reserved=""
Jul 02 12:38:52.208346 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201874    2578 flags.go:64] FLAG: --read-only-port="10255"
Jul 02 12:38:52.208346 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201878    2578 flags.go:64] FLAG: --register-node="true"
Jul 02 12:38:52.208346 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201881    2578 flags.go:64] FLAG: --register-with-taints=""
Jul 02 12:38:52.208346 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201884    2578 flags.go:64] FLAG: --registry-burst="10"
Jul 02 12:38:52.208346 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201888    2578 flags.go:64] FLAG: --registry-qps="5"
Jul 02 12:38:52.208346 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201891    2578 flags.go:64] FLAG: --reserved-cpus=""
Jul 02 12:38:52.208346 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201894    2578 flags.go:64] FLAG: --reserved-memory=""
Jul 02 12:38:52.208346 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201898    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 02 12:38:52.208346 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201901    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 02 12:38:52.208346 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201904    2578 flags.go:64] FLAG: --rotate-certificates="false"
Jul 02 12:38:52.208346 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201907    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 02 12:38:52.208346 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201910    2578 flags.go:64] FLAG: --runonce="false"
Jul 02 12:38:52.208346 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201917    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 02 12:38:52.209006 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201920    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 02 12:38:52.209006 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201923    2578 flags.go:64] FLAG: --seccomp-default="false"
Jul 02 12:38:52.209006 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201927    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 02 12:38:52.209006 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201930    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 02 12:38:52.209006 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201933    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 02 12:38:52.209006 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201937    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 02 12:38:52.209006 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201942    2578 flags.go:64] FLAG: --storage-driver-password="root"
Jul 02 12:38:52.209006 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201945    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 02 12:38:52.209006 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201948    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 02 12:38:52.209006 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201951    2578 flags.go:64] FLAG: --storage-driver-user="root"
Jul 02 12:38:52.209006 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201954    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 02 12:38:52.209006 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201958    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 02 12:38:52.209006 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201961    2578 flags.go:64] FLAG: --system-cgroups=""
Jul 02 12:38:52.209006 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201964    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 02 12:38:52.209006 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201970    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 02 12:38:52.209006 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201973    2578 flags.go:64] FLAG: --tls-cert-file=""
Jul 02 12:38:52.209006 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201976    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 02 12:38:52.209006 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201981    2578 flags.go:64] FLAG: --tls-min-version=""
Jul 02 12:38:52.209006 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201984    2578 flags.go:64] FLAG: --tls-private-key-file=""
Jul 02 12:38:52.209006 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201987    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 02 12:38:52.209006 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201991    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 02 12:38:52.209006 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201994    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 02 12:38:52.209006 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.201997    2578 flags.go:64] FLAG: --v="2"
Jul 02 12:38:52.209006 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.202001    2578 flags.go:64] FLAG: --version="false"
Jul 02 12:38:52.209006 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.202005    2578 flags.go:64] FLAG: --vmodule=""
Jul 02 12:38:52.210076 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.202009    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 02 12:38:52.210076 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.202012    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 02 12:38:52.210076 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202147    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 12:38:52.210076 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202152    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 12:38:52.210076 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202155    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 12:38:52.210076 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202158    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 12:38:52.210076 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202161    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 12:38:52.210076 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202164    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 12:38:52.210076 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202167    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 12:38:52.210076 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202170    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 12:38:52.210076 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202173    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 12:38:52.210076 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202176    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 12:38:52.210076 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202180    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 12:38:52.210076 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202186    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 12:38:52.210076 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202189    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 12:38:52.210076 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202191    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 12:38:52.210076 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202194    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 12:38:52.210076 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202198    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 12:38:52.210076 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202201    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 12:38:52.210076 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202203    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 12:38:52.210769 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202206    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 12:38:52.210769 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202209    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 12:38:52.210769 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202212    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 12:38:52.210769 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202215    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 12:38:52.210769 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202218    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 12:38:52.210769 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202221    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 12:38:52.210769 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202225    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 12:38:52.210769 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202228    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 12:38:52.210769 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202231    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 12:38:52.210769 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202233    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 12:38:52.210769 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202236    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 12:38:52.210769 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202238    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 12:38:52.210769 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202241    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 12:38:52.210769 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202244    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 12:38:52.210769 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202246    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 12:38:52.210769 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202249    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 12:38:52.210769 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202265    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 12:38:52.210769 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202268    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 12:38:52.210769 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202271    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 12:38:52.211294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202274    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 12:38:52.211294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202277    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 12:38:52.211294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202280    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 12:38:52.211294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202282    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 12:38:52.211294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202285    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 12:38:52.211294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202290    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 12:38:52.211294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202295    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 12:38:52.211294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202298    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 12:38:52.211294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202301    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 12:38:52.211294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202304    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 12:38:52.211294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202307    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 12:38:52.211294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202310    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 12:38:52.211294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202313    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 12:38:52.211294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202315    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 12:38:52.211294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202318    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 12:38:52.211294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202321    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 12:38:52.211294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202324    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 12:38:52.211294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202327    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 12:38:52.211294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202329    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 12:38:52.211294 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202332    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 12:38:52.211838 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202336    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 12:38:52.211838 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202339    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 12:38:52.211838 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202342    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 12:38:52.211838 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202345    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 12:38:52.211838 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202347    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 12:38:52.211838 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202350    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 12:38:52.211838 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202353    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 12:38:52.211838 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202356    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 12:38:52.211838 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202359    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 12:38:52.211838 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202361    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 12:38:52.211838 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202364    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 12:38:52.211838 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202367    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 12:38:52.211838 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202370    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 12:38:52.211838 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202373    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 12:38:52.211838 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202376    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 12:38:52.211838 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202378    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 12:38:52.211838 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202382    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 12:38:52.211838 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202387    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 12:38:52.211838 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202391    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 12:38:52.211838 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202395    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 12:38:52.212389 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202399    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 12:38:52.212389 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202402    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 12:38:52.212389 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202405    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 12:38:52.212389 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202409    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 12:38:52.212389 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202413    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 12:38:52.212389 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202416    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 12:38:52.212389 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202419    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 12:38:52.212389 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202422    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 12:38:52.212389 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202426    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 12:38:52.212389 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202430    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 12:38:52.212389 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202433    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 12:38:52.212389 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202437    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 12:38:52.212389 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202440    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 12:38:52.212389 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202443    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 12:38:52.212389 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202446    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 12:38:52.212389 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202450    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 12:38:52.212389 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202453    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 12:38:52.212389 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202456    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 12:38:52.212389 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202458    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 12:38:52.212891 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202461    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 12:38:52.212891 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202464    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 12:38:52.212891 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.202466    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 12:38:52.212891 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.203560    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 12:38:52.212891 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.212268    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 02 12:38:52.212891 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.212285    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 02 12:38:52.212891 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.212309    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 02 12:38:52.212891 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.212315    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 02 12:38:52.212891 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212375    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 12:38:52.212891 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212381    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 12:38:52.212891 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212385    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 12:38:52.212891 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212388    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 12:38:52.212891 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212391    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 12:38:52.212891 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212394    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 12:38:52.212891 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212397    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 12:38:52.213316 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212400    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 12:38:52.213316 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212403    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 12:38:52.213316 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212406    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 12:38:52.213316 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212409    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 12:38:52.213316 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212412    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 12:38:52.213316 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212415    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 12:38:52.213316 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212418    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 12:38:52.213316 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212422    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 12:38:52.213316 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212426    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 12:38:52.213316 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212429    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 12:38:52.213316 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212432    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 12:38:52.213316 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212435    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 12:38:52.213316 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212438    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 12:38:52.213316 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212441    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 12:38:52.213316 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212444    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 12:38:52.213316 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212446    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 12:38:52.213316 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212449    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 12:38:52.213316 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212453    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 12:38:52.213316 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212456    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 12:38:52.213316 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212459    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 12:38:52.213819 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212462    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 12:38:52.213819 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212464    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 12:38:52.213819 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212468    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 12:38:52.213819 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212471    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 12:38:52.213819 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212474    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 12:38:52.213819 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212477    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 12:38:52.213819 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212480    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 12:38:52.213819 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212482    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 12:38:52.213819 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212485    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 12:38:52.213819 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212489    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 12:38:52.213819 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212492    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 12:38:52.213819 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212494    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 12:38:52.213819 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212497    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 12:38:52.213819 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212500    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 12:38:52.213819 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212503    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 12:38:52.213819 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212506    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 12:38:52.213819 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212509    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 12:38:52.213819 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212512    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 12:38:52.213819 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212514    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 12:38:52.213819 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212517    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 12:38:52.214455 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212520    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 12:38:52.214455 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212523    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 12:38:52.214455 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212525    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 12:38:52.214455 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212528    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 12:38:52.214455 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212531    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 12:38:52.214455 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212533    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 12:38:52.214455 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212536    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 12:38:52.214455 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212539    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 12:38:52.214455 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212542    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 12:38:52.214455 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212545    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 12:38:52.214455 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212548    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 12:38:52.214455 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212551    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 12:38:52.214455 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212553    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 12:38:52.214455 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212558    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 12:38:52.214455 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212562    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 12:38:52.214455 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212566    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 12:38:52.214455 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212569    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 12:38:52.214455 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212573    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 12:38:52.214455 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212576    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 12:38:52.214953 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212579    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 12:38:52.214953 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212581    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 12:38:52.214953 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212584    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 12:38:52.214953 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212587    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 12:38:52.214953 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212590    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 12:38:52.214953 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212592    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 12:38:52.214953 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212595    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 12:38:52.214953 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212598    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 12:38:52.214953 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212601    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 12:38:52.214953 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212604    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 12:38:52.214953 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212606    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 12:38:52.214953 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212609    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 12:38:52.214953 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212612    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 12:38:52.214953 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212616    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 12:38:52.214953 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212618    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 12:38:52.214953 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212621    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 12:38:52.214953 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212624    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 12:38:52.214953 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212627    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 12:38:52.214953 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212630    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 12:38:52.214953 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212633    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 12:38:52.215498 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212635    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 12:38:52.215498 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212639    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 12:38:52.215498 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212641    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 12:38:52.215498 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212644    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 12:38:52.215498 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212647    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 12:38:52.215498 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212650    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 12:38:52.215498 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212652    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 12:38:52.215498 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212655    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 12:38:52.215498 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212658    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 12:38:52.215498 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212661    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 12:38:52.215498 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212663    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 12:38:52.215498 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212666    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 12:38:52.215498 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212670    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 12:38:52.215498 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.212677    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 12:38:52.216021 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212798    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 12:38:52.216021 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212804    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 12:38:52.216021 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212809    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 12:38:52.216021 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212813    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 12:38:52.216021 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212816    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 12:38:52.216021 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212819    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 12:38:52.216021 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212822    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 12:38:52.216021 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212825    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 12:38:52.216021 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212828    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 12:38:52.216021 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212831    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 12:38:52.216021 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212834    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 12:38:52.216021 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212837    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 12:38:52.216021 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212843    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 12:38:52.216021 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212846    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 12:38:52.216021 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212848    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 12:38:52.216021 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212851    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 12:38:52.216021 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212854    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 12:38:52.216021 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212857    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 12:38:52.216021 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212859    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 12:38:52.216540 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212862    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 12:38:52.216540 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212865    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 12:38:52.216540 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212868    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 12:38:52.216540 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212871    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 12:38:52.216540 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212874    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 12:38:52.216540 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212877    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 12:38:52.216540 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212880    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 12:38:52.216540 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212882    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 12:38:52.216540 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212885    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 12:38:52.216540 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212888    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 12:38:52.216540 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212891    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 12:38:52.216540 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212894    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 12:38:52.216540 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212897    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 12:38:52.216540 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212901    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 12:38:52.216540 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212905    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 12:38:52.216540 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212908    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 12:38:52.216540 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212911    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 12:38:52.216540 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212914    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 12:38:52.216540 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212917    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 12:38:52.217026 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212919    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 12:38:52.217026 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212922    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 12:38:52.217026 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212925    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 12:38:52.217026 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212928    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 12:38:52.217026 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212931    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 12:38:52.217026 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212934    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 12:38:52.217026 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212937    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 12:38:52.217026 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212940    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 12:38:52.217026 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212943    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 12:38:52.217026 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212946    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 12:38:52.217026 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212949    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 12:38:52.217026 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212952    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 12:38:52.217026 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212955    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 12:38:52.217026 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212958    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 12:38:52.217026 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212960    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 12:38:52.217026 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212963    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 12:38:52.217026 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212966    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 12:38:52.217026 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212968    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 12:38:52.217026 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212971    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 12:38:52.217026 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212974    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 12:38:52.217646 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212976    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 12:38:52.217646 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212979    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 12:38:52.217646 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212983    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 12:38:52.217646 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212987    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 12:38:52.217646 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212990    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 12:38:52.217646 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212993    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 12:38:52.217646 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212995    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 12:38:52.217646 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.212998    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 12:38:52.217646 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.213001    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 12:38:52.217646 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.213003    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 12:38:52.217646 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.213006    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 12:38:52.217646 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.213008    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 12:38:52.217646 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.213011    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 12:38:52.217646 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.213014    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 12:38:52.217646 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.213017    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 12:38:52.217646 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.213019    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 12:38:52.217646 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.213022    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 12:38:52.217646 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.213025    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 12:38:52.217646 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.213027    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 12:38:52.218128 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.213030    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 12:38:52.218128 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.213035    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 12:38:52.218128 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.213038    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 12:38:52.218128 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.213042    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 12:38:52.218128 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.213044    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 12:38:52.218128 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.213047    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 12:38:52.218128 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.213050    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 12:38:52.218128 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.213052    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 12:38:52.218128 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.213055    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 12:38:52.218128 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.213058    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 12:38:52.218128 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.213060    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 12:38:52.218128 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.213063    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 12:38:52.218128 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.213066    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 12:38:52.218128 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.213068    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 12:38:52.218128 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.213071    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 12:38:52.218128 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.213073    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 12:38:52.218128 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.213076    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 12:38:52.218128 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.213079    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 12:38:52.218128 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.213081    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 12:38:52.218128 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.213084    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 12:38:52.218668 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.213087    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 12:38:52.218668 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:52.213089    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 12:38:52.218668 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.213095    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 12:38:52.218668 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.213216    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 02 12:38:52.218668 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.215517    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 02 12:38:52.218668 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.217104    2578 server.go:1019] "Starting client certificate rotation"
Jul 02 12:38:52.218668 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.217293    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 12:38:52.218668 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.217332    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 12:38:52.220451 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.220437    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 12:38:52.221712 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.221688    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 12:38:52.238740 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.238721    2578 log.go:25] "Validated CRI v1 runtime API"
Jul 02 12:38:52.241748 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.241731    2578 log.go:25] "Validated CRI v1 image API"
Jul 02 12:38:52.242635 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.242617    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 02 12:38:52.246150 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.246128    2578 fs.go:135] Filesystem UUIDs: map[082d545c-46b4-444f-b444-844be0a6128c:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 edacb2e3-1a30-4dbe-a66f-c706cf444d9a:/dev/nvme0n1p3]
Jul 02 12:38:52.246219 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.246148    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 02 12:38:52.251312 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.251293    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 12:38:52.252583 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.252465    2578 manager.go:216] Machine: {Timestamp:2026-07-02 12:38:52.250832848 +0000 UTC m=+0.383658710 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102447 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22ebb4f4ac35e5ad62e16a71417033 SystemUUID:ec22ebb4-f4ac-35e5-ad62-e16a71417033 BootID:c8cf34b3-1e39-460c-906c-6201371d16d3 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:78:57:12:6f:99 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:78:57:12:6f:99 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:37:48:7e:d5:e6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 02 12:38:52.252583 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.252582    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 02 12:38:52.252708 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.252676    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 02 12:38:52.256554 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.256526    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 02 12:38:52.256701 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.256557    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-127.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 02 12:38:52.257538 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.257527    2578 topology_manager.go:138] "Creating topology manager with none policy"
Jul 02 12:38:52.257574 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.257541    2578 container_manager_linux.go:309] "Creating device plugin manager"
Jul 02 12:38:52.257574 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.257550    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 12:38:52.258696 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.258684    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 12:38:52.259799 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.259788    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 02 12:38:52.263207 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.263194    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 12:38:52.263361 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.263351    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 02 12:38:52.263395 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.263383    2578 kubelet.go:486] "Attempting to sync node with API server"
Jul 02 12:38:52.263395 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.263392    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 02 12:38:52.263448 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.263408    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 02 12:38:52.263448 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.263418    2578 kubelet.go:398] "Adding apiserver pod source"
Jul 02 12:38:52.263448 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.263428    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 02 12:38:52.263830 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.263816    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 12:38:52.263876 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.263835    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 12:38:52.266133 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.266116    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 02 12:38:52.267893 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.267874    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 02 12:38:52.267966 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.267923    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 02 12:38:52.269451 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.269435    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 02 12:38:52.269523 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.269459    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 02 12:38:52.269523 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.269472    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 02 12:38:52.269523 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.269481    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 02 12:38:52.269523 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.269495    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 02 12:38:52.269523 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.269507    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 02 12:38:52.269713 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.269525    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 02 12:38:52.269713 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.269538    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 02 12:38:52.269713 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.269552    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 02 12:38:52.269713 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.269563    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 02 12:38:52.269713 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.269581    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 02 12:38:52.269713 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.269591    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 02 12:38:52.269713 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.269621    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-whdzd"
Jul 02 12:38:52.270488 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.270476    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 02 12:38:52.270553 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.270490    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 02 12:38:52.274194 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.274173    2578 server.go:1268] "Started kubelet"
Jul 02 12:38:52.274335 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.274291    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 02 12:38:52.274590 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.274314    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 02 12:38:52.274698 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.274623    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 02 12:38:52.275041 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.274433    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-127.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 02 12:38:52.275088 ip-10-0-137-127 systemd[1]: Started Kubernetes Kubelet.
Jul 02 12:38:52.275340 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:52.275159    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 02 12:38:52.275340 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:52.275175    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-127.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 02 12:38:52.276317 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.276296    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 02 12:38:52.276705 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.276693    2578 server.go:310] "Adding debug handlers to kubelet server"
Jul 02 12:38:52.278453 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.278422    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-whdzd"
Jul 02 12:38:52.281130 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:52.281110    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 02 12:38:52.281686 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:52.280500    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-127.ec2.internal.18be795a8b2793a8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-127.ec2.internal,UID:ip-10-0-137-127.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-127.ec2.internal,},FirstTimestamp:2026-07-02 12:38:52.273939368 +0000 UTC m=+0.406765207,LastTimestamp:2026-07-02 12:38:52.273939368 +0000 UTC m=+0.406765207,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-127.ec2.internal,}"
Jul 02 12:38:52.286113 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.286090    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 02 12:38:52.286113 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.286101    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 02 12:38:52.286307 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:52.286285    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-127.ec2.internal\" not found"
Jul 02 12:38:52.287090 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.287057    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 02 12:38:52.287090 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.287059    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 02 12:38:52.287234 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.287100    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 02 12:38:52.287234 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.287210    2578 reconstruct.go:97] "Volume reconstruction finished"
Jul 02 12:38:52.287234 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.287221    2578 reconciler.go:29] "Reconciler: start to sync state"
Jul 02 12:38:52.287422 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.287244    2578 factory.go:153] Registering CRI-O factory
Jul 02 12:38:52.287422 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.287259    2578 factory.go:223] Registration of the crio container factory successfully
Jul 02 12:38:52.287422 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.287314    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 02 12:38:52.287422 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.287325    2578 factory.go:55] Registering systemd factory
Jul 02 12:38:52.287422 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.287333    2578 factory.go:223] Registration of the systemd container factory successfully
Jul 02 12:38:52.287422 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.287362    2578 factory.go:103] Registering Raw factory
Jul 02 12:38:52.287422 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.287373    2578 manager.go:1196] Started watching for new ooms in manager
Jul 02 12:38:52.287728 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.287674    2578 manager.go:349] Starting recovery of all containers
Jul 02 12:38:52.288164 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.288136    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 12:38:52.291595 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:52.291571    2578 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-137-127.ec2.internal\" not found" node="ip-10-0-137-127.ec2.internal"
Jul 02 12:38:52.298833 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.298807    2578 manager.go:354] Recovery completed
Jul 02 12:38:52.300576 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:52.300556    2578 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": readdirent /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 02 12:38:52.303735 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.303718    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:52.308720 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.308702    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-127.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:52.308775 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.308737    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-127.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:52.308775 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.308750    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-127.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:52.309336 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.309320    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 02 12:38:52.309336 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.309334    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 02 12:38:52.309484 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.309355    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 12:38:52.309484 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.309379    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:52.310269 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.310240    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-127.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:52.310351 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.310286    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-127.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:52.310351 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.310301    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-127.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:52.310661 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.310647    2578 policy_none.go:49] "None policy: Start"
Jul 02 12:38:52.310732 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.310665    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 02 12:38:52.310732 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.310680    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 02 12:38:52.311799 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.311786    2578 policy_none.go:47] "Start"
Jul 02 12:38:52.348949 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.348935    2578 manager.go:337] "Starting Device Plugin manager"
Jul 02 12:38:52.360614 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:52.348966    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 02 12:38:52.360614 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.348977    2578 server.go:87] "Starting device plugin registration server"
Jul 02 12:38:52.360614 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.349537    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 02 12:38:52.360614 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.349558    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 02 12:38:52.360614 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.349790    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 02 12:38:52.360614 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.350342    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 02 12:38:52.360614 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.350356    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 02 12:38:52.360614 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:52.350378    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 02 12:38:52.360614 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:52.350432    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-127.ec2.internal\" not found"
Jul 02 12:38:52.360614 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.354994    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 02 12:38:52.360614 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.356166    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 02 12:38:52.360614 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.356200    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 02 12:38:52.360614 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.356220    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 02 12:38:52.360614 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:52.356276    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 02 12:38:52.360614 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.357863    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 12:38:52.453976 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.453875    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:52.455101 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.455079    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-127.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:52.455204 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.455115    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-127.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:52.455204 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.455134    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-127.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:52.455204 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.455163    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-127.ec2.internal"
Jul 02 12:38:52.457098 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.457078    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-127.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-127.ec2.internal"]
Jul 02 12:38:52.457147 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.457142    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:52.458638 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.458623    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-127.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:52.458700 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.458652    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-127.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:52.458700 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.458662    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-127.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:52.460009 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.459997    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:52.460159 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.460142    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-127.ec2.internal"
Jul 02 12:38:52.460292 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.460208    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:52.460805 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.460788    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-127.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:52.460883 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.460817    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-127.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:52.460883 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.460828    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-127.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:52.460883 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.460793    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-127.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:52.461037 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.460902    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-127.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:52.461037 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.460924    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-127.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:52.462154 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.462136    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-127.ec2.internal"
Jul 02 12:38:52.462206 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.462171    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:52.462882 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.462865    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-127.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:52.462967 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.462892    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-127.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:52.462967 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.462903    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-127.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:52.463515 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.463500    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-127.ec2.internal"
Jul 02 12:38:52.463561 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:52.463522    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-127.ec2.internal\": node \"ip-10-0-137-127.ec2.internal\" not found"
Jul 02 12:38:52.473472 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:52.473453    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-127.ec2.internal\" not found"
Jul 02 12:38:52.476814 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:52.476800    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-127.ec2.internal\" not found" node="ip-10-0-137-127.ec2.internal"
Jul 02 12:38:52.481017 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:52.481000    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-127.ec2.internal\" not found" node="ip-10-0-137-127.ec2.internal"
Jul 02 12:38:52.488370 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.488351    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4dcb23c7e9d915b5863a6ffa788e26c0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-127.ec2.internal\" (UID: \"4dcb23c7e9d915b5863a6ffa788e26c0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-127.ec2.internal"
Jul 02 12:38:52.488439 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.488379    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4dcb23c7e9d915b5863a6ffa788e26c0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-127.ec2.internal\" (UID: \"4dcb23c7e9d915b5863a6ffa788e26c0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-127.ec2.internal"
Jul 02 12:38:52.488439 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.488399    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/59308479622ecc528e66f3883871894d-config\") pod \"kube-apiserver-proxy-ip-10-0-137-127.ec2.internal\" (UID: \"59308479622ecc528e66f3883871894d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-127.ec2.internal"
Jul 02 12:38:52.574390 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:52.574347    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-127.ec2.internal\" not found"
Jul 02 12:38:52.589404 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.589382    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4dcb23c7e9d915b5863a6ffa788e26c0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-127.ec2.internal\" (UID: \"4dcb23c7e9d915b5863a6ffa788e26c0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-127.ec2.internal"
Jul 02 12:38:52.589488 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.589411    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4dcb23c7e9d915b5863a6ffa788e26c0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-127.ec2.internal\" (UID: \"4dcb23c7e9d915b5863a6ffa788e26c0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-127.ec2.internal"
Jul 02 12:38:52.589488 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.589429    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/59308479622ecc528e66f3883871894d-config\") pod \"kube-apiserver-proxy-ip-10-0-137-127.ec2.internal\" (UID: \"59308479622ecc528e66f3883871894d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-127.ec2.internal"
Jul 02 12:38:52.589488 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.589469    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/59308479622ecc528e66f3883871894d-config\") pod \"kube-apiserver-proxy-ip-10-0-137-127.ec2.internal\" (UID: \"59308479622ecc528e66f3883871894d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-127.ec2.internal"
Jul 02 12:38:52.589589 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.589493    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4dcb23c7e9d915b5863a6ffa788e26c0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-127.ec2.internal\" (UID: \"4dcb23c7e9d915b5863a6ffa788e26c0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-127.ec2.internal"
Jul 02 12:38:52.589589 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.589493    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4dcb23c7e9d915b5863a6ffa788e26c0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-127.ec2.internal\" (UID: \"4dcb23c7e9d915b5863a6ffa788e26c0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-127.ec2.internal"
Jul 02 12:38:52.675027 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:52.674980    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-127.ec2.internal\" not found"
Jul 02 12:38:52.776151 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:52.776075    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-127.ec2.internal\" not found"
Jul 02 12:38:52.780334 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.780316    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-127.ec2.internal"
Jul 02 12:38:52.784392 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:52.784376    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-127.ec2.internal"
Jul 02 12:38:52.877582 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:52.877536    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-127.ec2.internal\" not found"
Jul 02 12:38:52.978479 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:52.978448    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-127.ec2.internal\" not found"
Jul 02 12:38:53.080061 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:53.079983    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-127.ec2.internal\" not found"
Jul 02 12:38:53.180920 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:53.180888    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-127.ec2.internal\" not found"
Jul 02 12:38:53.216527 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:53.216499    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 02 12:38:53.217155 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:53.216677    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 02 12:38:53.217155 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:53.216678    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 02 12:38:53.281301 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:53.281270    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-127.ec2.internal\" not found"
Jul 02 12:38:53.284650 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:53.284618    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-01 12:33:52 +0000 UTC" deadline="2028-03-27 02:26:08.655163416 +0000 UTC"
Jul 02 12:38:53.284747 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:53.284651    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15205h47m15.370516255s"
Jul 02 12:38:53.298044 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:53.298018    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59308479622ecc528e66f3883871894d.slice/crio-6aa46a487a0a11544869e7ae14453556abe1548436f60ccb0b06ab15a6a81566 WatchSource:0}: Error finding container 6aa46a487a0a11544869e7ae14453556abe1548436f60ccb0b06ab15a6a81566: Status 404 returned error can't find the container with id 6aa46a487a0a11544869e7ae14453556abe1548436f60ccb0b06ab15a6a81566
Jul 02 12:38:53.298525 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:53.298507    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dcb23c7e9d915b5863a6ffa788e26c0.slice/crio-68c47fbd5918b5b8942d34b884d89faa22f0cfa11b5679f0f198815e0186ab16 WatchSource:0}: Error finding container 68c47fbd5918b5b8942d34b884d89faa22f0cfa11b5679f0f198815e0186ab16: Status 404 returned error can't find the container with id 68c47fbd5918b5b8942d34b884d89faa22f0cfa11b5679f0f198815e0186ab16
Jul 02 12:38:53.303871 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:53.303854    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 12:38:53.359878 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:53.359793    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-127.ec2.internal" event={"ID":"4dcb23c7e9d915b5863a6ffa788e26c0","Type":"ContainerStarted","Data":"68c47fbd5918b5b8942d34b884d89faa22f0cfa11b5679f0f198815e0186ab16"}
Jul 02 12:38:53.360738 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:53.360713    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-127.ec2.internal" event={"ID":"59308479622ecc528e66f3883871894d","Type":"ContainerStarted","Data":"6aa46a487a0a11544869e7ae14453556abe1548436f60ccb0b06ab15a6a81566"}
Jul 02 12:38:53.382226 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:53.382204    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-127.ec2.internal\" not found"
Jul 02 12:38:53.483231 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:53.483165    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-127.ec2.internal\" not found"
Jul 02 12:38:53.522363 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:53.522340    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 12:38:53.586863 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:53.586843    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-127.ec2.internal"
Jul 02 12:38:53.598836 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:53.598818    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 12:38:53.600749 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:53.600737    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-127.ec2.internal"
Jul 02 12:38:53.608165 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:53.608149    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 12:38:53.724732 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:53.724643    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 12:38:54.264245 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.264218    2578 apiserver.go:52] "Watching apiserver"
Jul 02 12:38:54.278295 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.278264    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 02 12:38:54.279834 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.279801    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-xsf2s","openshift-network-diagnostics/network-check-target-4cw8v","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftctb","openshift-dns/node-resolver-cjwdw","openshift-image-registry/node-ca-cp22n","openshift-multus/multus-kpfkc","openshift-network-operator/iptables-alerter-xplqk","openshift-ovn-kubernetes/ovnkube-node-95kwm","kube-system/global-pull-secret-syncer-tsj7t","kube-system/konnectivity-agent-9f8kj","kube-system/kube-apiserver-proxy-ip-10-0-137-127.ec2.internal","openshift-cluster-node-tuning-operator/tuned-nj6tf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-127.ec2.internal","openshift-multus/multus-additional-cni-plugins-mtbbk"]
Jul 02 12:38:54.281502 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.281484    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xplqk"
Jul 02 12:38:54.283573 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.283549    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.285108 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.285090    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftctb"
Jul 02 12:38:54.286683 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.286660    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 02 12:38:54.286918 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.286892    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 02 12:38:54.287304 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.287283    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 02 12:38:54.287654 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.287637    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ngzpz\""
Jul 02 12:38:54.288293 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.288271    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 02 12:38:54.288391 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.288340    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 02 12:38:54.288458 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.288340    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 02 12:38:54.288523 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.288476    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 02 12:38:54.289331 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.289309    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mbxq9\""
Jul 02 12:38:54.289672 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.289656    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cjwdw"
Jul 02 12:38:54.289852 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.289833    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 02 12:38:54.289934 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.289836    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 02 12:38:54.290422 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.290063    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 02 12:38:54.290422 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.290139    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-66sh9\""
Jul 02 12:38:54.290422 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.290146    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 02 12:38:54.290422 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.290204    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 02 12:38:54.290422 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.290252    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 02 12:38:54.291906 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.291886    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cp22n"
Jul 02 12:38:54.292252 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.292231    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 02 12:38:54.293834 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.293819    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.294435 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.294415    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 02 12:38:54.296062 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.296046    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tsj7t"
Jul 02 12:38:54.296169 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:54.296147    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tsj7t" podUID="2718fba6-5e1c-4c48-a90c-fc26b0ccc385"
Jul 02 12:38:54.297106 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.297088    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kr8v8\""
Jul 02 12:38:54.297462 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.297444    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 02 12:38:54.297529 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.297462    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 02 12:38:54.297691 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.297673    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsf2s"
Jul 02 12:38:54.297759 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:54.297739    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xsf2s" podUID="e2432aa2-93ec-46ce-afcd-e19f9dd03f8e"
Jul 02 12:38:54.298157 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.298139    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 02 12:38:54.298281 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.298263    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 02 12:38:54.298346 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.298267    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 02 12:38:54.298724 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.298708    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 02 12:38:54.298814 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.298739    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 02 12:38:54.298877 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.298836    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5l9w4\""
Jul 02 12:38:54.298963 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.298937    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vxb59\""
Jul 02 12:38:54.299074 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.299059    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-host-kubelet\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.299133 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.299102    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmqlb\" (UniqueName: \"kubernetes.io/projected/be2acf25-23d8-4ce7-bcb1-24aa464b48aa-kube-api-access-mmqlb\") pod \"aws-ebs-csi-driver-node-ftctb\" (UID: \"be2acf25-23d8-4ce7-bcb1-24aa464b48aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftctb"
Jul 02 12:38:54.299209 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.299130    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/495e9fa8-b93f-42e0-95b5-813527df0993-env-overrides\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.299209 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.299153    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/495e9fa8-b93f-42e0-95b5-813527df0993-ovnkube-script-lib\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.299316 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.299211    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/be2acf25-23d8-4ce7-bcb1-24aa464b48aa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ftctb\" (UID: \"be2acf25-23d8-4ce7-bcb1-24aa464b48aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftctb"
Jul 02 12:38:54.299316 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.299244    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/be2acf25-23d8-4ce7-bcb1-24aa464b48aa-device-dir\") pod \"aws-ebs-csi-driver-node-ftctb\" (UID: \"be2acf25-23d8-4ce7-bcb1-24aa464b48aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftctb"
Jul 02 12:38:54.299316 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.299270    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/be2acf25-23d8-4ce7-bcb1-24aa464b48aa-etc-selinux\") pod \"aws-ebs-csi-driver-node-ftctb\" (UID: \"be2acf25-23d8-4ce7-bcb1-24aa464b48aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftctb"
Jul 02 12:38:54.299316 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.299292    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-host-run-netns\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.299515 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.299368    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-log-socket\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.299515 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.299396    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-host-cni-bin\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.299515 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.299434    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/495e9fa8-b93f-42e0-95b5-813527df0993-ovnkube-config\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.299515 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.299458    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/be2acf25-23d8-4ce7-bcb1-24aa464b48aa-registration-dir\") pod \"aws-ebs-csi-driver-node-ftctb\" (UID: \"be2acf25-23d8-4ce7-bcb1-24aa464b48aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftctb"
Jul 02 12:38:54.299515 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.299478    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3a25dc59-8e44-4f73-8275-342792a291b6-hosts-file\") pod \"node-resolver-cjwdw\" (UID: \"3a25dc59-8e44-4f73-8275-342792a291b6\") " pod="openshift-dns/node-resolver-cjwdw"
Jul 02 12:38:54.299749 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.299517    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-systemd-units\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.299749 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.299558    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-host-slash\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.299749 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.299584    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cw8v"
Jul 02 12:38:54.299749 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.299641    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-host-run-ovn-kubernetes\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.299749 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:54.299639    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cw8v" podUID="cd215e6b-c112-412f-a031-30940e656f0d"
Jul 02 12:38:54.299749 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.299680    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-run-systemd\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.299749 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.299708    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-var-lib-openvswitch\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.299749 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.299732    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-host-cni-netd\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.300116 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.299757    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.300116 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.299788    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/be2acf25-23d8-4ce7-bcb1-24aa464b48aa-sys-fs\") pod \"aws-ebs-csi-driver-node-ftctb\" (UID: \"be2acf25-23d8-4ce7-bcb1-24aa464b48aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftctb"
Jul 02 12:38:54.300116 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.299820    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a25dc59-8e44-4f73-8275-342792a291b6-tmp-dir\") pod \"node-resolver-cjwdw\" (UID: \"3a25dc59-8e44-4f73-8275-342792a291b6\") " pod="openshift-dns/node-resolver-cjwdw"
Jul 02 12:38:54.300116 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.299850    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dffv8\" (UniqueName: \"kubernetes.io/projected/495e9fa8-b93f-42e0-95b5-813527df0993-kube-api-access-dffv8\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.300116 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.299877    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/be2acf25-23d8-4ce7-bcb1-24aa464b48aa-socket-dir\") pod \"aws-ebs-csi-driver-node-ftctb\" (UID: \"be2acf25-23d8-4ce7-bcb1-24aa464b48aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftctb"
Jul 02 12:38:54.300116 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.299900    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjn9x\" (UniqueName: \"kubernetes.io/projected/3a25dc59-8e44-4f73-8275-342792a291b6-kube-api-access-hjn9x\") pod \"node-resolver-cjwdw\" (UID: \"3a25dc59-8e44-4f73-8275-342792a291b6\") " pod="openshift-dns/node-resolver-cjwdw"
Jul 02 12:38:54.300116 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.299924    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b1a6f623-3064-4500-be97-92644ef17328-iptables-alerter-script\") pod \"iptables-alerter-xplqk\" (UID: \"b1a6f623-3064-4500-be97-92644ef17328\") " pod="openshift-network-operator/iptables-alerter-xplqk"
Jul 02 12:38:54.300116 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.299959    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-node-log\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.300116 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.300002    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/495e9fa8-b93f-42e0-95b5-813527df0993-ovn-node-metrics-cert\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.300116 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.300017    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1a6f623-3064-4500-be97-92644ef17328-host-slash\") pod \"iptables-alerter-xplqk\" (UID: \"b1a6f623-3064-4500-be97-92644ef17328\") " pod="openshift-network-operator/iptables-alerter-xplqk"
Jul 02 12:38:54.300116 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.300043    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-etc-openvswitch\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.300116 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.300063    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-run-openvswitch\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.300116 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.300092    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-run-ovn\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.300748 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.300144    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcgjx\" (UniqueName: \"kubernetes.io/projected/b1a6f623-3064-4500-be97-92644ef17328-kube-api-access-kcgjx\") pod \"iptables-alerter-xplqk\" (UID: \"b1a6f623-3064-4500-be97-92644ef17328\") " pod="openshift-network-operator/iptables-alerter-xplqk"
Jul 02 12:38:54.301084 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.301065    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9f8kj"
Jul 02 12:38:54.303117 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.303097    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mtbbk"
Jul 02 12:38:54.304344 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.304325    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 02 12:38:54.304473 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.304454    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.304645 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.304629    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rvv7k\""
Jul 02 12:38:54.304881 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.304852    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 02 12:38:54.305988 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.305969    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-d5chc\""
Jul 02 12:38:54.306226 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.306210    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 02 12:38:54.306316 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.306289    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 02 12:38:54.311400 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.311382    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p44xz\""
Jul 02 12:38:54.311597 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.311580    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 02 12:38:54.311673 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.311607    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 12:38:54.312318 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.312300    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 02 12:38:54.323986 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.323970    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 12:38:54.326368 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.326351    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-x9f77"
Jul 02 12:38:54.333267 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.333249    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-x9f77"
Jul 02 12:38:54.388414 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.388389    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 02 12:38:54.401164 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.401118    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-host-kubelet\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.401299 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.401169    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmqlb\" (UniqueName: \"kubernetes.io/projected/be2acf25-23d8-4ce7-bcb1-24aa464b48aa-kube-api-access-mmqlb\") pod \"aws-ebs-csi-driver-node-ftctb\" (UID: \"be2acf25-23d8-4ce7-bcb1-24aa464b48aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftctb"
Jul 02 12:38:54.401299 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.401212    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-host-var-lib-cni-multus\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.401299 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.401237    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9777c828-7acd-4d27-be6a-dbe5c3f6f18b-host\") pod \"node-ca-cp22n\" (UID: \"9777c828-7acd-4d27-be6a-dbe5c3f6f18b\") " pod="openshift-image-registry/node-ca-cp22n"
Jul 02 12:38:54.401299 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.401249    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-host-kubelet\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.401299 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.401255    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2718fba6-5e1c-4c48-a90c-fc26b0ccc385-dbus\") pod \"global-pull-secret-syncer-tsj7t\" (UID: \"2718fba6-5e1c-4c48-a90c-fc26b0ccc385\") " pod="kube-system/global-pull-secret-syncer-tsj7t"
Jul 02 12:38:54.401543 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.401396    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/be2acf25-23d8-4ce7-bcb1-24aa464b48aa-etc-selinux\") pod \"aws-ebs-csi-driver-node-ftctb\" (UID: \"be2acf25-23d8-4ce7-bcb1-24aa464b48aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftctb"
Jul 02 12:38:54.401543 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.401444    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-host-run-netns\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.401543 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.401470    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/176a476f-a437-42c6-a32d-b6d23e4d2a9f-agent-certs\") pod \"konnectivity-agent-9f8kj\" (UID: \"176a476f-a437-42c6-a32d-b6d23e4d2a9f\") " pod="kube-system/konnectivity-agent-9f8kj"
Jul 02 12:38:54.401543 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.401479    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/be2acf25-23d8-4ce7-bcb1-24aa464b48aa-etc-selinux\") pod \"aws-ebs-csi-driver-node-ftctb\" (UID: \"be2acf25-23d8-4ce7-bcb1-24aa464b48aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftctb"
Jul 02 12:38:54.401543 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.401494    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2b74229e-eeba-4f5d-b05a-1d67c83d450c-etc-sysctl-d\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.401543 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.401517    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b74229e-eeba-4f5d-b05a-1d67c83d450c-var-lib-kubelet\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.401792 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.401555    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-log-socket\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.401792 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.401584    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/495e9fa8-b93f-42e0-95b5-813527df0993-ovnkube-config\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.401792 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.401602    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/be2acf25-23d8-4ce7-bcb1-24aa464b48aa-registration-dir\") pod \"aws-ebs-csi-driver-node-ftctb\" (UID: \"be2acf25-23d8-4ce7-bcb1-24aa464b48aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftctb"
Jul 02 12:38:54.401792 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.401629    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-multus-socket-dir-parent\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.401792 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.401634    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-log-socket\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.401792 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.401673    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp5sc\" (UniqueName: \"kubernetes.io/projected/7908d111-8ec6-4177-bd91-38966a57ac13-kube-api-access-gp5sc\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.401792 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.401698    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b74229e-eeba-4f5d-b05a-1d67c83d450c-host\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.401792 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.401716    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/be2acf25-23d8-4ce7-bcb1-24aa464b48aa-registration-dir\") pod \"aws-ebs-csi-driver-node-ftctb\" (UID: \"be2acf25-23d8-4ce7-bcb1-24aa464b48aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftctb"
Jul 02 12:38:54.401792 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.401727    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-systemd-units\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.401792 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.401758    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-host-run-ovn-kubernetes\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.402246 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.401793    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-systemd-units\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.402246 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.401847    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1652685-279d-4a21-a047-736b5076b4e4-system-cni-dir\") pod \"multus-additional-cni-plugins-mtbbk\" (UID: \"b1652685-279d-4a21-a047-736b5076b4e4\") " pod="openshift-multus/multus-additional-cni-plugins-mtbbk"
Jul 02 12:38:54.402246 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.401859    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-host-run-ovn-kubernetes\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.402246 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.401905    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-multus-cni-dir\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.402246 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.401933    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-host-run-multus-certs\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.402246 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.401968    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-run-systemd\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.402246 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.402046    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-run-systemd\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.402246 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.402074    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-host-cni-netd\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.402246 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.402103    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.402246 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.402145    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a25dc59-8e44-4f73-8275-342792a291b6-tmp-dir\") pod \"node-resolver-cjwdw\" (UID: \"3a25dc59-8e44-4f73-8275-342792a291b6\") " pod="openshift-dns/node-resolver-cjwdw"
Jul 02 12:38:54.402246 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.402151    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-host-cni-netd\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.402246 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.402191    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/495e9fa8-b93f-42e0-95b5-813527df0993-ovnkube-config\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.402246 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.402173    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1652685-279d-4a21-a047-736b5076b4e4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mtbbk\" (UID: \"b1652685-279d-4a21-a047-736b5076b4e4\") " pod="openshift-multus/multus-additional-cni-plugins-mtbbk"
Jul 02 12:38:54.402246 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.402237    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.402941 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.402277    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b74229e-eeba-4f5d-b05a-1d67c83d450c-tmp\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.402941 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.402311    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dffv8\" (UniqueName: \"kubernetes.io/projected/495e9fa8-b93f-42e0-95b5-813527df0993-kube-api-access-dffv8\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.402941 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.402335    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjn9x\" (UniqueName: \"kubernetes.io/projected/3a25dc59-8e44-4f73-8275-342792a291b6-kube-api-access-hjn9x\") pod \"node-resolver-cjwdw\" (UID: \"3a25dc59-8e44-4f73-8275-342792a291b6\") " pod="openshift-dns/node-resolver-cjwdw"
Jul 02 12:38:54.402941 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.402414    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a25dc59-8e44-4f73-8275-342792a291b6-tmp-dir\") pod \"node-resolver-cjwdw\" (UID: \"3a25dc59-8e44-4f73-8275-342792a291b6\") " pod="openshift-dns/node-resolver-cjwdw"
Jul 02 12:38:54.402941 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.402492    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1652685-279d-4a21-a047-736b5076b4e4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mtbbk\" (UID: \"b1652685-279d-4a21-a047-736b5076b4e4\") " pod="openshift-multus/multus-additional-cni-plugins-mtbbk"
Jul 02 12:38:54.402941 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.402549    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-os-release\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.402941 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.402590    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7908d111-8ec6-4177-bd91-38966a57ac13-multus-daemon-config\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.402941 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.402614    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/495e9fa8-b93f-42e0-95b5-813527df0993-ovn-node-metrics-cert\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.402941 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.402642    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpvs6\" (UniqueName: \"kubernetes.io/projected/cd215e6b-c112-412f-a031-30940e656f0d-kube-api-access-mpvs6\") pod \"network-check-target-4cw8v\" (UID: \"cd215e6b-c112-412f-a031-30940e656f0d\") " pod="openshift-network-diagnostics/network-check-target-4cw8v"
Jul 02 12:38:54.402941 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.402690    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2b74229e-eeba-4f5d-b05a-1d67c83d450c-etc-modprobe-d\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.402941 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.402714    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4qzp\" (UniqueName: \"kubernetes.io/projected/9777c828-7acd-4d27-be6a-dbe5c3f6f18b-kube-api-access-r4qzp\") pod \"node-ca-cp22n\" (UID: \"9777c828-7acd-4d27-be6a-dbe5c3f6f18b\") " pod="openshift-image-registry/node-ca-cp22n"
Jul 02 12:38:54.402941 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.402750    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7908d111-8ec6-4177-bd91-38966a57ac13-cni-binary-copy\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.402941 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.402773    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-host-run-k8s-cni-cncf-io\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.402941 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.402793    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b74229e-eeba-4f5d-b05a-1d67c83d450c-sys\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.402941 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.402815    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2b74229e-eeba-4f5d-b05a-1d67c83d450c-lib-modules\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.402941 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.402840    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxzvc\" (UniqueName: \"kubernetes.io/projected/2b74229e-eeba-4f5d-b05a-1d67c83d450c-kube-api-access-nxzvc\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.402941 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.402862    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/495e9fa8-b93f-42e0-95b5-813527df0993-env-overrides\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.403725 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.402885    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/495e9fa8-b93f-42e0-95b5-813527df0993-ovnkube-script-lib\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.403725 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.402910    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/be2acf25-23d8-4ce7-bcb1-24aa464b48aa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ftctb\" (UID: \"be2acf25-23d8-4ce7-bcb1-24aa464b48aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftctb"
Jul 02 12:38:54.403725 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.402955    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/be2acf25-23d8-4ce7-bcb1-24aa464b48aa-device-dir\") pod \"aws-ebs-csi-driver-node-ftctb\" (UID: \"be2acf25-23d8-4ce7-bcb1-24aa464b48aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftctb"
Jul 02 12:38:54.403725 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.402966    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 02 12:38:54.403725 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.402989    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-host-var-lib-kubelet\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.403725 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403009    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/be2acf25-23d8-4ce7-bcb1-24aa464b48aa-device-dir\") pod \"aws-ebs-csi-driver-node-ftctb\" (UID: \"be2acf25-23d8-4ce7-bcb1-24aa464b48aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftctb"
Jul 02 12:38:54.403725 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403014    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/be2acf25-23d8-4ce7-bcb1-24aa464b48aa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ftctb\" (UID: \"be2acf25-23d8-4ce7-bcb1-24aa464b48aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftctb"
Jul 02 12:38:54.403725 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403038    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-multus-conf-dir\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.403725 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403149    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-host-run-netns\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.403725 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403194    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-host-cni-bin\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.403725 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403221    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3a25dc59-8e44-4f73-8275-342792a291b6-hosts-file\") pod \"node-resolver-cjwdw\" (UID: \"3a25dc59-8e44-4f73-8275-342792a291b6\") " pod="openshift-dns/node-resolver-cjwdw"
Jul 02 12:38:54.403725 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403289    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b1652685-279d-4a21-a047-736b5076b4e4-os-release\") pod \"multus-additional-cni-plugins-mtbbk\" (UID: \"b1652685-279d-4a21-a047-736b5076b4e4\") " pod="openshift-multus/multus-additional-cni-plugins-mtbbk"
Jul 02 12:38:54.403725 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403315    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-host-var-lib-cni-bin\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.403725 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403345    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-host-slash\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.403725 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403394    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-etc-kubernetes\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.403725 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403402    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3a25dc59-8e44-4f73-8275-342792a291b6-hosts-file\") pod \"node-resolver-cjwdw\" (UID: \"3a25dc59-8e44-4f73-8275-342792a291b6\") " pod="openshift-dns/node-resolver-cjwdw"
Jul 02 12:38:54.403725 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403407    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-host-cni-bin\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.404571 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403423    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2b74229e-eeba-4f5d-b05a-1d67c83d450c-etc-systemd\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.404571 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403439    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2b74229e-eeba-4f5d-b05a-1d67c83d450c-etc-tuned\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.404571 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403458    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-host-slash\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.404571 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403462    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9777c828-7acd-4d27-be6a-dbe5c3f6f18b-serviceca\") pod \"node-ca-cp22n\" (UID: \"9777c828-7acd-4d27-be6a-dbe5c3f6f18b\") " pod="openshift-image-registry/node-ca-cp22n"
Jul 02 12:38:54.404571 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403501    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-var-lib-openvswitch\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.404571 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403505    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-host-run-netns\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.404571 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403523    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/495e9fa8-b93f-42e0-95b5-813527df0993-env-overrides\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.404571 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403536    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/be2acf25-23d8-4ce7-bcb1-24aa464b48aa-sys-fs\") pod \"aws-ebs-csi-driver-node-ftctb\" (UID: \"be2acf25-23d8-4ce7-bcb1-24aa464b48aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftctb"
Jul 02 12:38:54.404571 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403567    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-var-lib-openvswitch\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.404571 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403609    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-cnibin\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.404571 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403625    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/495e9fa8-b93f-42e0-95b5-813527df0993-ovnkube-script-lib\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.404571 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403647    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-hostroot\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.404571 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403648    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/be2acf25-23d8-4ce7-bcb1-24aa464b48aa-sys-fs\") pod \"aws-ebs-csi-driver-node-ftctb\" (UID: \"be2acf25-23d8-4ce7-bcb1-24aa464b48aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftctb"
Jul 02 12:38:54.404571 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403681    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b74229e-eeba-4f5d-b05a-1d67c83d450c-etc-kubernetes\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.404571 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403708    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2718fba6-5e1c-4c48-a90c-fc26b0ccc385-original-pull-secret\") pod \"global-pull-secret-syncer-tsj7t\" (UID: \"2718fba6-5e1c-4c48-a90c-fc26b0ccc385\") " pod="kube-system/global-pull-secret-syncer-tsj7t"
Jul 02 12:38:54.404571 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403730    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/be2acf25-23d8-4ce7-bcb1-24aa464b48aa-socket-dir\") pod \"aws-ebs-csi-driver-node-ftctb\" (UID: \"be2acf25-23d8-4ce7-bcb1-24aa464b48aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftctb"
Jul 02 12:38:54.404571 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403786    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b1a6f623-3064-4500-be97-92644ef17328-iptables-alerter-script\") pod \"iptables-alerter-xplqk\" (UID: \"b1a6f623-3064-4500-be97-92644ef17328\") " pod="openshift-network-operator/iptables-alerter-xplqk"
Jul 02 12:38:54.405113 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403816    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2b74229e-eeba-4f5d-b05a-1d67c83d450c-etc-sysctl-conf\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.405113 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403843    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2718fba6-5e1c-4c48-a90c-fc26b0ccc385-kubelet-config\") pod \"global-pull-secret-syncer-tsj7t\" (UID: \"2718fba6-5e1c-4c48-a90c-fc26b0ccc385\") " pod="kube-system/global-pull-secret-syncer-tsj7t"
Jul 02 12:38:54.405113 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403870    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-node-log\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.405113 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403893    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1a6f623-3064-4500-be97-92644ef17328-host-slash\") pod \"iptables-alerter-xplqk\" (UID: \"b1a6f623-3064-4500-be97-92644ef17328\") " pod="openshift-network-operator/iptables-alerter-xplqk"
Jul 02 12:38:54.405113 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403918    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1652685-279d-4a21-a047-736b5076b4e4-cnibin\") pod \"multus-additional-cni-plugins-mtbbk\" (UID: \"b1652685-279d-4a21-a047-736b5076b4e4\") " pod="openshift-multus/multus-additional-cni-plugins-mtbbk"
Jul 02 12:38:54.405113 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403928    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-node-log\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.405113 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403945    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1652685-279d-4a21-a047-736b5076b4e4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mtbbk\" (UID: \"b1652685-279d-4a21-a047-736b5076b4e4\") " pod="openshift-multus/multus-additional-cni-plugins-mtbbk"
Jul 02 12:38:54.405113 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403951    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1a6f623-3064-4500-be97-92644ef17328-host-slash\") pod \"iptables-alerter-xplqk\" (UID: \"b1a6f623-3064-4500-be97-92644ef17328\") " pod="openshift-network-operator/iptables-alerter-xplqk"
Jul 02 12:38:54.405113 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403970    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx242\" (UniqueName: \"kubernetes.io/projected/b1652685-279d-4a21-a047-736b5076b4e4-kube-api-access-hx242\") pod \"multus-additional-cni-plugins-mtbbk\" (UID: \"b1652685-279d-4a21-a047-736b5076b4e4\") " pod="openshift-multus/multus-additional-cni-plugins-mtbbk"
Jul 02 12:38:54.405113 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.403995    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-system-cni-dir\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.405113 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.404019    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sm7v\" (UniqueName: \"kubernetes.io/projected/e2432aa2-93ec-46ce-afcd-e19f9dd03f8e-kube-api-access-5sm7v\") pod \"network-metrics-daemon-xsf2s\" (UID: \"e2432aa2-93ec-46ce-afcd-e19f9dd03f8e\") " pod="openshift-multus/network-metrics-daemon-xsf2s"
Jul 02 12:38:54.405113 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.404042    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/176a476f-a437-42c6-a32d-b6d23e4d2a9f-konnectivity-ca\") pod \"konnectivity-agent-9f8kj\" (UID: \"176a476f-a437-42c6-a32d-b6d23e4d2a9f\") " pod="kube-system/konnectivity-agent-9f8kj"
Jul 02 12:38:54.405113 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.404067    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-etc-openvswitch\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.405113 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.404090    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-run-openvswitch\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.405113 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.404113    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-run-ovn\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.405113 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.404118    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-etc-openvswitch\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.405113 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.404137    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcgjx\" (UniqueName: \"kubernetes.io/projected/b1a6f623-3064-4500-be97-92644ef17328-kube-api-access-kcgjx\") pod \"iptables-alerter-xplqk\" (UID: \"b1a6f623-3064-4500-be97-92644ef17328\") " pod="openshift-network-operator/iptables-alerter-xplqk"
Jul 02 12:38:54.405689 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.404161    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1652685-279d-4a21-a047-736b5076b4e4-cni-binary-copy\") pod \"multus-additional-cni-plugins-mtbbk\" (UID: \"b1652685-279d-4a21-a047-736b5076b4e4\") " pod="openshift-multus/multus-additional-cni-plugins-mtbbk"
Jul 02 12:38:54.405689 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.404165    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-run-openvswitch\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.405689 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.404200    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2432aa2-93ec-46ce-afcd-e19f9dd03f8e-metrics-certs\") pod \"network-metrics-daemon-xsf2s\" (UID: \"e2432aa2-93ec-46ce-afcd-e19f9dd03f8e\") " pod="openshift-multus/network-metrics-daemon-xsf2s"
Jul 02 12:38:54.405689 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.404224    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2b74229e-eeba-4f5d-b05a-1d67c83d450c-etc-sysconfig\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.405689 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.404199    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/495e9fa8-b93f-42e0-95b5-813527df0993-run-ovn\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.405689 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.404247    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2b74229e-eeba-4f5d-b05a-1d67c83d450c-run\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.405689 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.404394    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/be2acf25-23d8-4ce7-bcb1-24aa464b48aa-socket-dir\") pod \"aws-ebs-csi-driver-node-ftctb\" (UID: \"be2acf25-23d8-4ce7-bcb1-24aa464b48aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftctb"
Jul 02 12:38:54.405689 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.404458    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b1a6f623-3064-4500-be97-92644ef17328-iptables-alerter-script\") pod \"iptables-alerter-xplqk\" (UID: \"b1a6f623-3064-4500-be97-92644ef17328\") " pod="openshift-network-operator/iptables-alerter-xplqk"
Jul 02 12:38:54.406839 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.406818    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/495e9fa8-b93f-42e0-95b5-813527df0993-ovn-node-metrics-cert\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.409777 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.409756    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmqlb\" (UniqueName: \"kubernetes.io/projected/be2acf25-23d8-4ce7-bcb1-24aa464b48aa-kube-api-access-mmqlb\") pod \"aws-ebs-csi-driver-node-ftctb\" (UID: \"be2acf25-23d8-4ce7-bcb1-24aa464b48aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftctb"
Jul 02 12:38:54.410773 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.410745    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjn9x\" (UniqueName: \"kubernetes.io/projected/3a25dc59-8e44-4f73-8275-342792a291b6-kube-api-access-hjn9x\") pod \"node-resolver-cjwdw\" (UID: \"3a25dc59-8e44-4f73-8275-342792a291b6\") " pod="openshift-dns/node-resolver-cjwdw"
Jul 02 12:38:54.411151 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.411110    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dffv8\" (UniqueName: \"kubernetes.io/projected/495e9fa8-b93f-42e0-95b5-813527df0993-kube-api-access-dffv8\") pod \"ovnkube-node-95kwm\" (UID: \"495e9fa8-b93f-42e0-95b5-813527df0993\") " pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.411781 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.411764    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcgjx\" (UniqueName: \"kubernetes.io/projected/b1a6f623-3064-4500-be97-92644ef17328-kube-api-access-kcgjx\") pod \"iptables-alerter-xplqk\" (UID: \"b1a6f623-3064-4500-be97-92644ef17328\") " pod="openshift-network-operator/iptables-alerter-xplqk"
Jul 02 12:38:54.506854 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.505194    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7908d111-8ec6-4177-bd91-38966a57ac13-cni-binary-copy\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.506854 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.505247    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-host-run-k8s-cni-cncf-io\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.506854 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.505279    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b74229e-eeba-4f5d-b05a-1d67c83d450c-sys\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.506854 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.505311    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2b74229e-eeba-4f5d-b05a-1d67c83d450c-lib-modules\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.506854 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.505341    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxzvc\" (UniqueName: \"kubernetes.io/projected/2b74229e-eeba-4f5d-b05a-1d67c83d450c-kube-api-access-nxzvc\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.506854 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.505369    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-host-var-lib-kubelet\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.506854 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.505404    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-multus-conf-dir\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.506854 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.505437    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b1652685-279d-4a21-a047-736b5076b4e4-os-release\") pod \"multus-additional-cni-plugins-mtbbk\" (UID: \"b1652685-279d-4a21-a047-736b5076b4e4\") " pod="openshift-multus/multus-additional-cni-plugins-mtbbk"
Jul 02 12:38:54.506854 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.505546    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b1652685-279d-4a21-a047-736b5076b4e4-os-release\") pod \"multus-additional-cni-plugins-mtbbk\" (UID: \"b1652685-279d-4a21-a047-736b5076b4e4\") " pod="openshift-multus/multus-additional-cni-plugins-mtbbk"
Jul 02 12:38:54.506854 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.505543    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-host-var-lib-cni-bin\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.506854 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.505601    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-etc-kubernetes\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.506854 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.505630    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-host-var-lib-cni-bin\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.506854 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.505634    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2b74229e-eeba-4f5d-b05a-1d67c83d450c-etc-systemd\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.506854 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.505682    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2b74229e-eeba-4f5d-b05a-1d67c83d450c-etc-systemd\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.506854 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.505688    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2b74229e-eeba-4f5d-b05a-1d67c83d450c-etc-tuned\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.506854 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.505716    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9777c828-7acd-4d27-be6a-dbe5c3f6f18b-serviceca\") pod \"node-ca-cp22n\" (UID: \"9777c828-7acd-4d27-be6a-dbe5c3f6f18b\") " pod="openshift-image-registry/node-ca-cp22n"
Jul 02 12:38:54.506854 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.505752    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-cnibin\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.506854 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.505779    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-hostroot\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.507770 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.505805    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b74229e-eeba-4f5d-b05a-1d67c83d450c-etc-kubernetes\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.507770 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.505844    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2718fba6-5e1c-4c48-a90c-fc26b0ccc385-original-pull-secret\") pod \"global-pull-secret-syncer-tsj7t\" (UID: \"2718fba6-5e1c-4c48-a90c-fc26b0ccc385\") " pod="kube-system/global-pull-secret-syncer-tsj7t"
Jul 02 12:38:54.507770 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.505877    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2b74229e-eeba-4f5d-b05a-1d67c83d450c-etc-sysctl-conf\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.507770 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.505966    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-cnibin\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.507770 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.506130    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2b74229e-eeba-4f5d-b05a-1d67c83d450c-etc-sysctl-conf\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.507770 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.506431    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b74229e-eeba-4f5d-b05a-1d67c83d450c-sys\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.507770 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.506507    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-multus-conf-dir\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.507770 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.506545    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2b74229e-eeba-4f5d-b05a-1d67c83d450c-lib-modules\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.507770 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.506577    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-etc-kubernetes\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.507770 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.506588    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-host-var-lib-kubelet\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.507770 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.506646    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9777c828-7acd-4d27-be6a-dbe5c3f6f18b-serviceca\") pod \"node-ca-cp22n\" (UID: \"9777c828-7acd-4d27-be6a-dbe5c3f6f18b\") " pod="openshift-image-registry/node-ca-cp22n"
Jul 02 12:38:54.507770 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.506698    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2718fba6-5e1c-4c48-a90c-fc26b0ccc385-kubelet-config\") pod \"global-pull-secret-syncer-tsj7t\" (UID: \"2718fba6-5e1c-4c48-a90c-fc26b0ccc385\") " pod="kube-system/global-pull-secret-syncer-tsj7t"
Jul 02 12:38:54.507770 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.506736    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1652685-279d-4a21-a047-736b5076b4e4-cnibin\") pod \"multus-additional-cni-plugins-mtbbk\" (UID: \"b1652685-279d-4a21-a047-736b5076b4e4\") " pod="openshift-multus/multus-additional-cni-plugins-mtbbk"
Jul 02 12:38:54.507770 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.506766    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1652685-279d-4a21-a047-736b5076b4e4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mtbbk\" (UID: \"b1652685-279d-4a21-a047-736b5076b4e4\") " pod="openshift-multus/multus-additional-cni-plugins-mtbbk"
Jul 02 12:38:54.507770 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.506850    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2718fba6-5e1c-4c48-a90c-fc26b0ccc385-kubelet-config\") pod \"global-pull-secret-syncer-tsj7t\" (UID: \"2718fba6-5e1c-4c48-a90c-fc26b0ccc385\") " pod="kube-system/global-pull-secret-syncer-tsj7t"
Jul 02 12:38:54.507770 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.506885    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-hostroot\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.507770 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.506891    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-host-run-k8s-cni-cncf-io\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.507770 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.506898    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx242\" (UniqueName: \"kubernetes.io/projected/b1652685-279d-4a21-a047-736b5076b4e4-kube-api-access-hx242\") pod \"multus-additional-cni-plugins-mtbbk\" (UID: \"b1652685-279d-4a21-a047-736b5076b4e4\") " pod="openshift-multus/multus-additional-cni-plugins-mtbbk"
Jul 02 12:38:54.508579 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.506301    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7908d111-8ec6-4177-bd91-38966a57ac13-cni-binary-copy\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.508579 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.507037    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-system-cni-dir\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.508579 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.507084    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1652685-279d-4a21-a047-736b5076b4e4-cnibin\") pod \"multus-additional-cni-plugins-mtbbk\" (UID: \"b1652685-279d-4a21-a047-736b5076b4e4\") " pod="openshift-multus/multus-additional-cni-plugins-mtbbk"
Jul 02 12:38:54.508579 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.507102    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-system-cni-dir\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.508579 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.507140    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sm7v\" (UniqueName: \"kubernetes.io/projected/e2432aa2-93ec-46ce-afcd-e19f9dd03f8e-kube-api-access-5sm7v\") pod \"network-metrics-daemon-xsf2s\" (UID: \"e2432aa2-93ec-46ce-afcd-e19f9dd03f8e\") " pod="openshift-multus/network-metrics-daemon-xsf2s"
Jul 02 12:38:54.508579 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.507194    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/176a476f-a437-42c6-a32d-b6d23e4d2a9f-konnectivity-ca\") pod \"konnectivity-agent-9f8kj\" (UID: \"176a476f-a437-42c6-a32d-b6d23e4d2a9f\") " pod="kube-system/konnectivity-agent-9f8kj"
Jul 02 12:38:54.508579 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.507229    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1652685-279d-4a21-a047-736b5076b4e4-cni-binary-copy\") pod \"multus-additional-cni-plugins-mtbbk\" (UID: \"b1652685-279d-4a21-a047-736b5076b4e4\") " pod="openshift-multus/multus-additional-cni-plugins-mtbbk"
Jul 02 12:38:54.508579 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.507260    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2432aa2-93ec-46ce-afcd-e19f9dd03f8e-metrics-certs\") pod \"network-metrics-daemon-xsf2s\" (UID: \"e2432aa2-93ec-46ce-afcd-e19f9dd03f8e\") " pod="openshift-multus/network-metrics-daemon-xsf2s"
Jul 02 12:38:54.508579 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.507294    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2b74229e-eeba-4f5d-b05a-1d67c83d450c-etc-sysconfig\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.508579 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.507328    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2b74229e-eeba-4f5d-b05a-1d67c83d450c-run\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.508579 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.507361    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-host-var-lib-cni-multus\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.508579 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.507387    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9777c828-7acd-4d27-be6a-dbe5c3f6f18b-host\") pod \"node-ca-cp22n\" (UID: \"9777c828-7acd-4d27-be6a-dbe5c3f6f18b\") " pod="openshift-image-registry/node-ca-cp22n"
Jul 02 12:38:54.508579 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.507419    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2718fba6-5e1c-4c48-a90c-fc26b0ccc385-dbus\") pod \"global-pull-secret-syncer-tsj7t\" (UID: \"2718fba6-5e1c-4c48-a90c-fc26b0ccc385\") " pod="kube-system/global-pull-secret-syncer-tsj7t"
Jul 02 12:38:54.508579 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.507451    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-host-run-netns\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.508579 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.507457    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1652685-279d-4a21-a047-736b5076b4e4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mtbbk\" (UID: \"b1652685-279d-4a21-a047-736b5076b4e4\") " pod="openshift-multus/multus-additional-cni-plugins-mtbbk"
Jul 02 12:38:54.508579 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.507480    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/176a476f-a437-42c6-a32d-b6d23e4d2a9f-agent-certs\") pod \"konnectivity-agent-9f8kj\" (UID: \"176a476f-a437-42c6-a32d-b6d23e4d2a9f\") " pod="kube-system/konnectivity-agent-9f8kj"
Jul 02 12:38:54.508579 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.507533    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2b74229e-eeba-4f5d-b05a-1d67c83d450c-etc-sysconfig\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.508579 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.507620    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2b74229e-eeba-4f5d-b05a-1d67c83d450c-run\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.509360 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.507671    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-host-var-lib-cni-multus\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.509360 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.507721    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9777c828-7acd-4d27-be6a-dbe5c3f6f18b-host\") pod \"node-ca-cp22n\" (UID: \"9777c828-7acd-4d27-be6a-dbe5c3f6f18b\") " pod="openshift-image-registry/node-ca-cp22n"
Jul 02 12:38:54.509360 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.507868    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2718fba6-5e1c-4c48-a90c-fc26b0ccc385-dbus\") pod \"global-pull-secret-syncer-tsj7t\" (UID: \"2718fba6-5e1c-4c48-a90c-fc26b0ccc385\") " pod="kube-system/global-pull-secret-syncer-tsj7t"
Jul 02 12:38:54.509360 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.507932    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-host-run-netns\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.509360 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.508452    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/176a476f-a437-42c6-a32d-b6d23e4d2a9f-konnectivity-ca\") pod \"konnectivity-agent-9f8kj\" (UID: \"176a476f-a437-42c6-a32d-b6d23e4d2a9f\") " pod="kube-system/konnectivity-agent-9f8kj"
Jul 02 12:38:54.509360 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.508541    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b74229e-eeba-4f5d-b05a-1d67c83d450c-etc-kubernetes\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.509360 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:54.508641    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 12:38:54.509360 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:54.508724    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2718fba6-5e1c-4c48-a90c-fc26b0ccc385-original-pull-secret podName:2718fba6-5e1c-4c48-a90c-fc26b0ccc385 nodeName:}" failed. No retries permitted until 2026-07-02 12:38:55.008694018 +0000 UTC m=+3.141519857 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2718fba6-5e1c-4c48-a90c-fc26b0ccc385-original-pull-secret") pod "global-pull-secret-syncer-tsj7t" (UID: "2718fba6-5e1c-4c48-a90c-fc26b0ccc385") : object "kube-system"/"original-pull-secret" not registered
Jul 02 12:38:54.510235 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.510030    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2b74229e-eeba-4f5d-b05a-1d67c83d450c-etc-sysctl-d\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.510235 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.510089    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b74229e-eeba-4f5d-b05a-1d67c83d450c-var-lib-kubelet\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.510235 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:54.509303    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:38:54.510235 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.510120    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-multus-socket-dir-parent\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.510235 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.510129    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2b74229e-eeba-4f5d-b05a-1d67c83d450c-etc-tuned\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.510235 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.509844    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1652685-279d-4a21-a047-736b5076b4e4-cni-binary-copy\") pod \"multus-additional-cni-plugins-mtbbk\" (UID: \"b1652685-279d-4a21-a047-736b5076b4e4\") " pod="openshift-multus/multus-additional-cni-plugins-mtbbk"
Jul 02 12:38:54.510235 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.510161    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp5sc\" (UniqueName: \"kubernetes.io/projected/7908d111-8ec6-4177-bd91-38966a57ac13-kube-api-access-gp5sc\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.510235 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:54.510239    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2432aa2-93ec-46ce-afcd-e19f9dd03f8e-metrics-certs podName:e2432aa2-93ec-46ce-afcd-e19f9dd03f8e nodeName:}" failed. No retries permitted until 2026-07-02 12:38:55.010208809 +0000 UTC m=+3.143034651 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e2432aa2-93ec-46ce-afcd-e19f9dd03f8e-metrics-certs") pod "network-metrics-daemon-xsf2s" (UID: "e2432aa2-93ec-46ce-afcd-e19f9dd03f8e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:38:54.510725 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.510271    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b74229e-eeba-4f5d-b05a-1d67c83d450c-var-lib-kubelet\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.510725 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.510286    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b74229e-eeba-4f5d-b05a-1d67c83d450c-host\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.510725 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.510321    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1652685-279d-4a21-a047-736b5076b4e4-system-cni-dir\") pod \"multus-additional-cni-plugins-mtbbk\" (UID: \"b1652685-279d-4a21-a047-736b5076b4e4\") " pod="openshift-multus/multus-additional-cni-plugins-mtbbk"
Jul 02 12:38:54.510725 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.510406    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b74229e-eeba-4f5d-b05a-1d67c83d450c-host\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.510725 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.510423    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-multus-socket-dir-parent\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.510725 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.510453    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-multus-cni-dir\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.510725 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.510500    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1652685-279d-4a21-a047-736b5076b4e4-system-cni-dir\") pod \"multus-additional-cni-plugins-mtbbk\" (UID: \"b1652685-279d-4a21-a047-736b5076b4e4\") " pod="openshift-multus/multus-additional-cni-plugins-mtbbk"
Jul 02 12:38:54.510725 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.510498    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-host-run-multus-certs\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.510725 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.510544    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1652685-279d-4a21-a047-736b5076b4e4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mtbbk\" (UID: \"b1652685-279d-4a21-a047-736b5076b4e4\") " pod="openshift-multus/multus-additional-cni-plugins-mtbbk"
Jul 02 12:38:54.510725 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.510563    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2b74229e-eeba-4f5d-b05a-1d67c83d450c-etc-sysctl-d\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.510725 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.510537    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-host-run-multus-certs\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.510725 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.510576    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b74229e-eeba-4f5d-b05a-1d67c83d450c-tmp\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.510725 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.510619    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-multus-cni-dir\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.510725 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.510633    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1652685-279d-4a21-a047-736b5076b4e4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mtbbk\" (UID: \"b1652685-279d-4a21-a047-736b5076b4e4\") " pod="openshift-multus/multus-additional-cni-plugins-mtbbk"
Jul 02 12:38:54.510725 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.510676    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-os-release\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.510725 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.510706    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7908d111-8ec6-4177-bd91-38966a57ac13-multus-daemon-config\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.510725 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.510735    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpvs6\" (UniqueName: \"kubernetes.io/projected/cd215e6b-c112-412f-a031-30940e656f0d-kube-api-access-mpvs6\") pod \"network-check-target-4cw8v\" (UID: \"cd215e6b-c112-412f-a031-30940e656f0d\") " pod="openshift-network-diagnostics/network-check-target-4cw8v"
Jul 02 12:38:54.511466 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.510765    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2b74229e-eeba-4f5d-b05a-1d67c83d450c-etc-modprobe-d\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.511466 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.510796    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4qzp\" (UniqueName: \"kubernetes.io/projected/9777c828-7acd-4d27-be6a-dbe5c3f6f18b-kube-api-access-r4qzp\") pod \"node-ca-cp22n\" (UID: \"9777c828-7acd-4d27-be6a-dbe5c3f6f18b\") " pod="openshift-image-registry/node-ca-cp22n"
Jul 02 12:38:54.511466 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.510803    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7908d111-8ec6-4177-bd91-38966a57ac13-os-release\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.511466 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.510820    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1652685-279d-4a21-a047-736b5076b4e4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mtbbk\" (UID: \"b1652685-279d-4a21-a047-736b5076b4e4\") " pod="openshift-multus/multus-additional-cni-plugins-mtbbk"
Jul 02 12:38:54.511466 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.511292    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2b74229e-eeba-4f5d-b05a-1d67c83d450c-etc-modprobe-d\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.511466 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.511378    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/176a476f-a437-42c6-a32d-b6d23e4d2a9f-agent-certs\") pod \"konnectivity-agent-9f8kj\" (UID: \"176a476f-a437-42c6-a32d-b6d23e4d2a9f\") " pod="kube-system/konnectivity-agent-9f8kj"
Jul 02 12:38:54.511896 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.511843    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7908d111-8ec6-4177-bd91-38966a57ac13-multus-daemon-config\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.512078 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.511918    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1652685-279d-4a21-a047-736b5076b4e4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mtbbk\" (UID: \"b1652685-279d-4a21-a047-736b5076b4e4\") " pod="openshift-multus/multus-additional-cni-plugins-mtbbk"
Jul 02 12:38:54.513864 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.513842    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b74229e-eeba-4f5d-b05a-1d67c83d450c-tmp\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.514573 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.514515    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxzvc\" (UniqueName: \"kubernetes.io/projected/2b74229e-eeba-4f5d-b05a-1d67c83d450c-kube-api-access-nxzvc\") pod \"tuned-nj6tf\" (UID: \"2b74229e-eeba-4f5d-b05a-1d67c83d450c\") " pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.519452 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:54.519434    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 12:38:54.519569 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:54.519459    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 12:38:54.519569 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:54.519472    2578 projected.go:196] Error preparing data for projected volume kube-api-access-mpvs6 for pod openshift-network-diagnostics/network-check-target-4cw8v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:38:54.519569 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:54.519558    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd215e6b-c112-412f-a031-30940e656f0d-kube-api-access-mpvs6 podName:cd215e6b-c112-412f-a031-30940e656f0d nodeName:}" failed. No retries permitted until 2026-07-02 12:38:55.019541338 +0000 UTC m=+3.152367162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mpvs6" (UniqueName: "kubernetes.io/projected/cd215e6b-c112-412f-a031-30940e656f0d-kube-api-access-mpvs6") pod "network-check-target-4cw8v" (UID: "cd215e6b-c112-412f-a031-30940e656f0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:38:54.520316 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.520297    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx242\" (UniqueName: \"kubernetes.io/projected/b1652685-279d-4a21-a047-736b5076b4e4-kube-api-access-hx242\") pod \"multus-additional-cni-plugins-mtbbk\" (UID: \"b1652685-279d-4a21-a047-736b5076b4e4\") " pod="openshift-multus/multus-additional-cni-plugins-mtbbk"
Jul 02 12:38:54.520490 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.520469    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sm7v\" (UniqueName: \"kubernetes.io/projected/e2432aa2-93ec-46ce-afcd-e19f9dd03f8e-kube-api-access-5sm7v\") pod \"network-metrics-daemon-xsf2s\" (UID: \"e2432aa2-93ec-46ce-afcd-e19f9dd03f8e\") " pod="openshift-multus/network-metrics-daemon-xsf2s"
Jul 02 12:38:54.521148 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.521127    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp5sc\" (UniqueName: \"kubernetes.io/projected/7908d111-8ec6-4177-bd91-38966a57ac13-kube-api-access-gp5sc\") pod \"multus-kpfkc\" (UID: \"7908d111-8ec6-4177-bd91-38966a57ac13\") " pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.521613 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.521593    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4qzp\" (UniqueName: \"kubernetes.io/projected/9777c828-7acd-4d27-be6a-dbe5c3f6f18b-kube-api-access-r4qzp\") pod \"node-ca-cp22n\" (UID: \"9777c828-7acd-4d27-be6a-dbe5c3f6f18b\") " pod="openshift-image-registry/node-ca-cp22n"
Jul 02 12:38:54.596292 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.596257    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xplqk"
Jul 02 12:38:54.603943 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.603913    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:38:54.611923 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.611901    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftctb"
Jul 02 12:38:54.616788 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.616771    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cjwdw"
Jul 02 12:38:54.624415 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.624398    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cp22n"
Jul 02 12:38:54.631140 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.631113    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kpfkc"
Jul 02 12:38:54.640078 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.640056    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9f8kj"
Jul 02 12:38:54.645665 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.645648    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mtbbk"
Jul 02 12:38:54.651276 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.651254    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nj6tf"
Jul 02 12:38:54.686186 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:54.686145    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 12:38:55.015435 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:55.015395    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2432aa2-93ec-46ce-afcd-e19f9dd03f8e-metrics-certs\") pod \"network-metrics-daemon-xsf2s\" (UID: \"e2432aa2-93ec-46ce-afcd-e19f9dd03f8e\") " pod="openshift-multus/network-metrics-daemon-xsf2s"
Jul 02 12:38:55.015594 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:55.015476    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2718fba6-5e1c-4c48-a90c-fc26b0ccc385-original-pull-secret\") pod \"global-pull-secret-syncer-tsj7t\" (UID: \"2718fba6-5e1c-4c48-a90c-fc26b0ccc385\") " pod="kube-system/global-pull-secret-syncer-tsj7t"
Jul 02 12:38:55.015594 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:55.015551    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:38:55.015727 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:55.015597    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 12:38:55.015727 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:55.015622    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2432aa2-93ec-46ce-afcd-e19f9dd03f8e-metrics-certs podName:e2432aa2-93ec-46ce-afcd-e19f9dd03f8e nodeName:}" failed. No retries permitted until 2026-07-02 12:38:56.015603484 +0000 UTC m=+4.148429308 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e2432aa2-93ec-46ce-afcd-e19f9dd03f8e-metrics-certs") pod "network-metrics-daemon-xsf2s" (UID: "e2432aa2-93ec-46ce-afcd-e19f9dd03f8e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:38:55.015727 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:55.015646    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2718fba6-5e1c-4c48-a90c-fc26b0ccc385-original-pull-secret podName:2718fba6-5e1c-4c48-a90c-fc26b0ccc385 nodeName:}" failed. No retries permitted until 2026-07-02 12:38:56.015631361 +0000 UTC m=+4.148457187 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2718fba6-5e1c-4c48-a90c-fc26b0ccc385-original-pull-secret") pod "global-pull-secret-syncer-tsj7t" (UID: "2718fba6-5e1c-4c48-a90c-fc26b0ccc385") : object "kube-system"/"original-pull-secret" not registered
Jul 02 12:38:55.115907 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:55.115870    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpvs6\" (UniqueName: \"kubernetes.io/projected/cd215e6b-c112-412f-a031-30940e656f0d-kube-api-access-mpvs6\") pod \"network-check-target-4cw8v\" (UID: \"cd215e6b-c112-412f-a031-30940e656f0d\") " pod="openshift-network-diagnostics/network-check-target-4cw8v"
Jul 02 12:38:55.116076 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:55.116049    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 12:38:55.116076 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:55.116072    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 12:38:55.116220 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:55.116085    2578 projected.go:196] Error preparing data for projected volume kube-api-access-mpvs6 for pod openshift-network-diagnostics/network-check-target-4cw8v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:38:55.116220 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:55.116154    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd215e6b-c112-412f-a031-30940e656f0d-kube-api-access-mpvs6 podName:cd215e6b-c112-412f-a031-30940e656f0d nodeName:}" failed. No retries permitted until 2026-07-02 12:38:56.1161334 +0000 UTC m=+4.248959226 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mpvs6" (UniqueName: "kubernetes.io/projected/cd215e6b-c112-412f-a031-30940e656f0d-kube-api-access-mpvs6") pod "network-check-target-4cw8v" (UID: "cd215e6b-c112-412f-a031-30940e656f0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:38:55.208769 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:55.208732    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod176a476f_a437_42c6_a32d_b6d23e4d2a9f.slice/crio-cc08e02df6c025be737b7f1cbe54fbf49a7c521a1b27fb18f0cd44ac99abc957 WatchSource:0}: Error finding container cc08e02df6c025be737b7f1cbe54fbf49a7c521a1b27fb18f0cd44ac99abc957: Status 404 returned error can't find the container with id cc08e02df6c025be737b7f1cbe54fbf49a7c521a1b27fb18f0cd44ac99abc957
Jul 02 12:38:55.211164 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:55.210998    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a25dc59_8e44_4f73_8275_342792a291b6.slice/crio-f1191e085691579fc86c116bad289505b10d235f77a066f7e0fe94d66c23e45e WatchSource:0}: Error finding container f1191e085691579fc86c116bad289505b10d235f77a066f7e0fe94d66c23e45e: Status 404 returned error can't find the container with id f1191e085691579fc86c116bad289505b10d235f77a066f7e0fe94d66c23e45e
Jul 02 12:38:55.211782 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:55.211756    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9777c828_7acd_4d27_be6a_dbe5c3f6f18b.slice/crio-40efb2d06cc7e6de6046e32d907e6e7ea6262eda98c31d01491ba6692d72358d WatchSource:0}: Error finding container 40efb2d06cc7e6de6046e32d907e6e7ea6262eda98c31d01491ba6692d72358d: Status 404 returned error can't find the container with id 40efb2d06cc7e6de6046e32d907e6e7ea6262eda98c31d01491ba6692d72358d
Jul 02 12:38:55.213149 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:55.213127    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod495e9fa8_b93f_42e0_95b5_813527df0993.slice/crio-fa88a5d6aa10504524ce574993179e5229b049230da16d2d6268804162540468 WatchSource:0}: Error finding container fa88a5d6aa10504524ce574993179e5229b049230da16d2d6268804162540468: Status 404 returned error can't find the container with id fa88a5d6aa10504524ce574993179e5229b049230da16d2d6268804162540468
Jul 02 12:38:55.216349 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:38:55.215962    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b74229e_eeba_4f5d_b05a_1d67c83d450c.slice/crio-bc0a03dd43ce9c44d74366149f47e1b620efec01ea1e3da993b02704afeb0b63 WatchSource:0}: Error finding container bc0a03dd43ce9c44d74366149f47e1b620efec01ea1e3da993b02704afeb0b63: Status 404 returned error can't find the container with id bc0a03dd43ce9c44d74366149f47e1b620efec01ea1e3da993b02704afeb0b63
Jul 02 12:38:55.336585 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:55.336550    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 12:33:54 +0000 UTC" deadline="2027-12-07 02:22:02.7916039 +0000 UTC"
Jul 02 12:38:55.336585 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:55.336587    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12541h43m7.455020837s"
Jul 02 12:38:55.368307 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:55.368105    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftctb" event={"ID":"be2acf25-23d8-4ce7-bcb1-24aa464b48aa","Type":"ContainerStarted","Data":"1f6530905bcfdcea1ebfd64842a3bfee933122e01fac2609a8bda38c8039712b"}
Jul 02 12:38:55.369478 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:55.369451    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kpfkc" event={"ID":"7908d111-8ec6-4177-bd91-38966a57ac13","Type":"ContainerStarted","Data":"07b0ac558b0f051436fcacb314d4db3c1e4a12d4f2557b74bfe4f9401e361ba7"}
Jul 02 12:38:55.370542 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:55.370519    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mtbbk" event={"ID":"b1652685-279d-4a21-a047-736b5076b4e4","Type":"ContainerStarted","Data":"7d72750fc9c1add717533abc7abc14f81ed4e79c522d31771481944d714eef85"}
Jul 02 12:38:55.371659 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:55.371633    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-95kwm" event={"ID":"495e9fa8-b93f-42e0-95b5-813527df0993","Type":"ContainerStarted","Data":"fa88a5d6aa10504524ce574993179e5229b049230da16d2d6268804162540468"}
Jul 02 12:38:55.373295 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:55.373271    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-127.ec2.internal" event={"ID":"59308479622ecc528e66f3883871894d","Type":"ContainerStarted","Data":"5be811a28b79162a5e7d5e5ea881428aa8e30f45cd0c17daa89303b2f62b3635"}
Jul 02 12:38:55.374378 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:55.374352    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xplqk" event={"ID":"b1a6f623-3064-4500-be97-92644ef17328","Type":"ContainerStarted","Data":"bff167873085bbb512c7bec5dfe5f1ba664ab73a97655b154b07dd65cc850526"}
Jul 02 12:38:55.375294 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:55.375271    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nj6tf" event={"ID":"2b74229e-eeba-4f5d-b05a-1d67c83d450c","Type":"ContainerStarted","Data":"bc0a03dd43ce9c44d74366149f47e1b620efec01ea1e3da993b02704afeb0b63"}
Jul 02 12:38:55.376345 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:55.376321    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cp22n" event={"ID":"9777c828-7acd-4d27-be6a-dbe5c3f6f18b","Type":"ContainerStarted","Data":"40efb2d06cc7e6de6046e32d907e6e7ea6262eda98c31d01491ba6692d72358d"}
Jul 02 12:38:55.377433 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:55.377404    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cjwdw" event={"ID":"3a25dc59-8e44-4f73-8275-342792a291b6","Type":"ContainerStarted","Data":"f1191e085691579fc86c116bad289505b10d235f77a066f7e0fe94d66c23e45e"}
Jul 02 12:38:55.378390 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:55.378369    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9f8kj" event={"ID":"176a476f-a437-42c6-a32d-b6d23e4d2a9f","Type":"ContainerStarted","Data":"cc08e02df6c025be737b7f1cbe54fbf49a7c521a1b27fb18f0cd44ac99abc957"}
Jul 02 12:38:55.391302 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:55.391262    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-127.ec2.internal" podStartSLOduration=2.391248702 podStartE2EDuration="2.391248702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:38:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:38:55.391031486 +0000 UTC m=+3.523857345" watchObservedRunningTime="2026-07-02 12:38:55.391248702 +0000 UTC m=+3.524074548"
Jul 02 12:38:56.023916 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:56.023502    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2718fba6-5e1c-4c48-a90c-fc26b0ccc385-original-pull-secret\") pod \"global-pull-secret-syncer-tsj7t\" (UID: \"2718fba6-5e1c-4c48-a90c-fc26b0ccc385\") " pod="kube-system/global-pull-secret-syncer-tsj7t"
Jul 02 12:38:56.023916 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:56.023559    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2432aa2-93ec-46ce-afcd-e19f9dd03f8e-metrics-certs\") pod \"network-metrics-daemon-xsf2s\" (UID: \"e2432aa2-93ec-46ce-afcd-e19f9dd03f8e\") " pod="openshift-multus/network-metrics-daemon-xsf2s"
Jul 02 12:38:56.023916 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:56.023718    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:38:56.023916 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:56.023778    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2432aa2-93ec-46ce-afcd-e19f9dd03f8e-metrics-certs podName:e2432aa2-93ec-46ce-afcd-e19f9dd03f8e nodeName:}" failed. No retries permitted until 2026-07-02 12:38:58.023761027 +0000 UTC m=+6.156586856 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e2432aa2-93ec-46ce-afcd-e19f9dd03f8e-metrics-certs") pod "network-metrics-daemon-xsf2s" (UID: "e2432aa2-93ec-46ce-afcd-e19f9dd03f8e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:38:56.024288 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:56.024260    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 12:38:56.024341 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:56.024322    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2718fba6-5e1c-4c48-a90c-fc26b0ccc385-original-pull-secret podName:2718fba6-5e1c-4c48-a90c-fc26b0ccc385 nodeName:}" failed. No retries permitted until 2026-07-02 12:38:58.024298314 +0000 UTC m=+6.157124145 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2718fba6-5e1c-4c48-a90c-fc26b0ccc385-original-pull-secret") pod "global-pull-secret-syncer-tsj7t" (UID: "2718fba6-5e1c-4c48-a90c-fc26b0ccc385") : object "kube-system"/"original-pull-secret" not registered
Jul 02 12:38:56.124293 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:56.124254    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpvs6\" (UniqueName: \"kubernetes.io/projected/cd215e6b-c112-412f-a031-30940e656f0d-kube-api-access-mpvs6\") pod \"network-check-target-4cw8v\" (UID: \"cd215e6b-c112-412f-a031-30940e656f0d\") " pod="openshift-network-diagnostics/network-check-target-4cw8v"
Jul 02 12:38:56.124504 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:56.124465    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 12:38:56.124504 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:56.124484    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 12:38:56.124504 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:56.124497    2578 projected.go:196] Error preparing data for projected volume kube-api-access-mpvs6 for pod openshift-network-diagnostics/network-check-target-4cw8v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:38:56.124663 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:56.124561    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd215e6b-c112-412f-a031-30940e656f0d-kube-api-access-mpvs6 podName:cd215e6b-c112-412f-a031-30940e656f0d nodeName:}" failed. No retries permitted until 2026-07-02 12:38:58.124541751 +0000 UTC m=+6.257367580 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mpvs6" (UniqueName: "kubernetes.io/projected/cd215e6b-c112-412f-a031-30940e656f0d-kube-api-access-mpvs6") pod "network-check-target-4cw8v" (UID: "cd215e6b-c112-412f-a031-30940e656f0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:38:56.337791 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:56.337703    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 12:33:54 +0000 UTC" deadline="2027-12-11 06:57:30.616676792 +0000 UTC"
Jul 02 12:38:56.337791 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:56.337745    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12642h18m34.278935708s"
Jul 02 12:38:56.363336 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:56.363305    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsf2s"
Jul 02 12:38:56.363542 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:56.363479    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xsf2s" podUID="e2432aa2-93ec-46ce-afcd-e19f9dd03f8e"
Jul 02 12:38:56.363629 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:56.363580    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cw8v"
Jul 02 12:38:56.363690 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:56.363672    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cw8v" podUID="cd215e6b-c112-412f-a031-30940e656f0d"
Jul 02 12:38:56.364155 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:56.364108    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tsj7t"
Jul 02 12:38:56.364262 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:56.364242    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tsj7t" podUID="2718fba6-5e1c-4c48-a90c-fc26b0ccc385"
Jul 02 12:38:56.408216 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:56.408046    2578 generic.go:356] "Generic (PLEG): container finished" podID="4dcb23c7e9d915b5863a6ffa788e26c0" containerID="709bfc4c8df95fe8b5185c061f4d4e14d3d632aff01ff995e3b4d19498e8f724" exitCode=0
Jul 02 12:38:56.408383 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:56.408249    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-127.ec2.internal" event={"ID":"4dcb23c7e9d915b5863a6ffa788e26c0","Type":"ContainerDied","Data":"709bfc4c8df95fe8b5185c061f4d4e14d3d632aff01ff995e3b4d19498e8f724"}
Jul 02 12:38:57.429203 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:57.429146    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-127.ec2.internal" event={"ID":"4dcb23c7e9d915b5863a6ffa788e26c0","Type":"ContainerStarted","Data":"6401cce03b44ea0628bb9c6c325affc4c300802f313165e47989c419afcad272"}
Jul 02 12:38:58.040842 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:58.040788    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2718fba6-5e1c-4c48-a90c-fc26b0ccc385-original-pull-secret\") pod \"global-pull-secret-syncer-tsj7t\" (UID: \"2718fba6-5e1c-4c48-a90c-fc26b0ccc385\") " pod="kube-system/global-pull-secret-syncer-tsj7t"
Jul 02 12:38:58.041007 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:58.040864    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2432aa2-93ec-46ce-afcd-e19f9dd03f8e-metrics-certs\") pod \"network-metrics-daemon-xsf2s\" (UID: \"e2432aa2-93ec-46ce-afcd-e19f9dd03f8e\") " pod="openshift-multus/network-metrics-daemon-xsf2s"
Jul 02 12:38:58.041082 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:58.041017    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:38:58.041082 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:58.041079    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2432aa2-93ec-46ce-afcd-e19f9dd03f8e-metrics-certs podName:e2432aa2-93ec-46ce-afcd-e19f9dd03f8e nodeName:}" failed. No retries permitted until 2026-07-02 12:39:02.041061546 +0000 UTC m=+10.173887380 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e2432aa2-93ec-46ce-afcd-e19f9dd03f8e-metrics-certs") pod "network-metrics-daemon-xsf2s" (UID: "e2432aa2-93ec-46ce-afcd-e19f9dd03f8e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:38:58.041563 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:58.041543    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 12:38:58.041659 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:58.041601    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2718fba6-5e1c-4c48-a90c-fc26b0ccc385-original-pull-secret podName:2718fba6-5e1c-4c48-a90c-fc26b0ccc385 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:02.041585981 +0000 UTC m=+10.174411808 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2718fba6-5e1c-4c48-a90c-fc26b0ccc385-original-pull-secret") pod "global-pull-secret-syncer-tsj7t" (UID: "2718fba6-5e1c-4c48-a90c-fc26b0ccc385") : object "kube-system"/"original-pull-secret" not registered
Jul 02 12:38:58.141822 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:58.141785    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpvs6\" (UniqueName: \"kubernetes.io/projected/cd215e6b-c112-412f-a031-30940e656f0d-kube-api-access-mpvs6\") pod \"network-check-target-4cw8v\" (UID: \"cd215e6b-c112-412f-a031-30940e656f0d\") " pod="openshift-network-diagnostics/network-check-target-4cw8v"
Jul 02 12:38:58.142012 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:58.141995    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 12:38:58.142085 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:58.142020    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 12:38:58.142085 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:58.142032    2578 projected.go:196] Error preparing data for projected volume kube-api-access-mpvs6 for pod openshift-network-diagnostics/network-check-target-4cw8v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:38:58.142228 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:58.142089    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd215e6b-c112-412f-a031-30940e656f0d-kube-api-access-mpvs6 podName:cd215e6b-c112-412f-a031-30940e656f0d nodeName:}" failed. No retries permitted until 2026-07-02 12:39:02.142071301 +0000 UTC m=+10.274897131 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mpvs6" (UniqueName: "kubernetes.io/projected/cd215e6b-c112-412f-a031-30940e656f0d-kube-api-access-mpvs6") pod "network-check-target-4cw8v" (UID: "cd215e6b-c112-412f-a031-30940e656f0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:38:58.358246 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:58.357470    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsf2s"
Jul 02 12:38:58.358246 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:58.357503    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tsj7t"
Jul 02 12:38:58.358246 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:58.357645    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xsf2s" podUID="e2432aa2-93ec-46ce-afcd-e19f9dd03f8e"
Jul 02 12:38:58.358246 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:38:58.357697    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cw8v"
Jul 02 12:38:58.358246 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:58.357784    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cw8v" podUID="cd215e6b-c112-412f-a031-30940e656f0d"
Jul 02 12:38:58.358246 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:38:58.357863    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tsj7t" podUID="2718fba6-5e1c-4c48-a90c-fc26b0ccc385"
Jul 02 12:39:00.357939 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:00.357267    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsf2s"
Jul 02 12:39:00.357939 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:00.357397    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xsf2s" podUID="e2432aa2-93ec-46ce-afcd-e19f9dd03f8e"
Jul 02 12:39:00.357939 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:00.357750    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tsj7t"
Jul 02 12:39:00.357939 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:00.357835    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tsj7t" podUID="2718fba6-5e1c-4c48-a90c-fc26b0ccc385"
Jul 02 12:39:00.361208 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:00.357881    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cw8v"
Jul 02 12:39:00.361208 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:00.358959    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cw8v" podUID="cd215e6b-c112-412f-a031-30940e656f0d"
Jul 02 12:39:02.078719 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:02.078136    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2718fba6-5e1c-4c48-a90c-fc26b0ccc385-original-pull-secret\") pod \"global-pull-secret-syncer-tsj7t\" (UID: \"2718fba6-5e1c-4c48-a90c-fc26b0ccc385\") " pod="kube-system/global-pull-secret-syncer-tsj7t"
Jul 02 12:39:02.078719 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:02.078206    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2432aa2-93ec-46ce-afcd-e19f9dd03f8e-metrics-certs\") pod \"network-metrics-daemon-xsf2s\" (UID: \"e2432aa2-93ec-46ce-afcd-e19f9dd03f8e\") " pod="openshift-multus/network-metrics-daemon-xsf2s"
Jul 02 12:39:02.078719 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:02.078334    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:39:02.078719 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:02.078393    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2432aa2-93ec-46ce-afcd-e19f9dd03f8e-metrics-certs podName:e2432aa2-93ec-46ce-afcd-e19f9dd03f8e nodeName:}" failed. No retries permitted until 2026-07-02 12:39:10.078373523 +0000 UTC m=+18.211199370 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e2432aa2-93ec-46ce-afcd-e19f9dd03f8e-metrics-certs") pod "network-metrics-daemon-xsf2s" (UID: "e2432aa2-93ec-46ce-afcd-e19f9dd03f8e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:39:02.078719 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:02.078544    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 12:39:02.078719 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:02.078600    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2718fba6-5e1c-4c48-a90c-fc26b0ccc385-original-pull-secret podName:2718fba6-5e1c-4c48-a90c-fc26b0ccc385 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:10.078585608 +0000 UTC m=+18.211411435 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2718fba6-5e1c-4c48-a90c-fc26b0ccc385-original-pull-secret") pod "global-pull-secret-syncer-tsj7t" (UID: "2718fba6-5e1c-4c48-a90c-fc26b0ccc385") : object "kube-system"/"original-pull-secret" not registered
Jul 02 12:39:02.178827 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:02.178664    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpvs6\" (UniqueName: \"kubernetes.io/projected/cd215e6b-c112-412f-a031-30940e656f0d-kube-api-access-mpvs6\") pod \"network-check-target-4cw8v\" (UID: \"cd215e6b-c112-412f-a031-30940e656f0d\") " pod="openshift-network-diagnostics/network-check-target-4cw8v"
Jul 02 12:39:02.178827 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:02.178799    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 12:39:02.178827 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:02.178819    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 12:39:02.178827 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:02.178830    2578 projected.go:196] Error preparing data for projected volume kube-api-access-mpvs6 for pod openshift-network-diagnostics/network-check-target-4cw8v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:39:02.179136 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:02.178883    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd215e6b-c112-412f-a031-30940e656f0d-kube-api-access-mpvs6 podName:cd215e6b-c112-412f-a031-30940e656f0d nodeName:}" failed. No retries permitted until 2026-07-02 12:39:10.17886704 +0000 UTC m=+18.311692864 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mpvs6" (UniqueName: "kubernetes.io/projected/cd215e6b-c112-412f-a031-30940e656f0d-kube-api-access-mpvs6") pod "network-check-target-4cw8v" (UID: "cd215e6b-c112-412f-a031-30940e656f0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:39:02.358114 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:02.358017    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsf2s"
Jul 02 12:39:02.358276 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:02.358130    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tsj7t"
Jul 02 12:39:02.358276 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:02.358134    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xsf2s" podUID="e2432aa2-93ec-46ce-afcd-e19f9dd03f8e"
Jul 02 12:39:02.358375 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:02.358283    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cw8v"
Jul 02 12:39:02.358375 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:02.358276    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tsj7t" podUID="2718fba6-5e1c-4c48-a90c-fc26b0ccc385"
Jul 02 12:39:02.358482 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:02.358381    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cw8v" podUID="cd215e6b-c112-412f-a031-30940e656f0d"
Jul 02 12:39:04.356804 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:04.356621    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tsj7t"
Jul 02 12:39:04.356804 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:04.356627    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cw8v"
Jul 02 12:39:04.356804 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:04.356755    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tsj7t" podUID="2718fba6-5e1c-4c48-a90c-fc26b0ccc385"
Jul 02 12:39:04.356804 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:04.356779    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsf2s"
Jul 02 12:39:04.357424 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:04.356891    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cw8v" podUID="cd215e6b-c112-412f-a031-30940e656f0d"
Jul 02 12:39:04.357424 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:04.356981    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xsf2s" podUID="e2432aa2-93ec-46ce-afcd-e19f9dd03f8e"
Jul 02 12:39:06.357351 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:06.357317    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cw8v"
Jul 02 12:39:06.357351 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:06.357353    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsf2s"
Jul 02 12:39:06.357853 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:06.357317    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tsj7t"
Jul 02 12:39:06.357853 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:06.357440    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cw8v" podUID="cd215e6b-c112-412f-a031-30940e656f0d"
Jul 02 12:39:06.357853 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:06.357561    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tsj7t" podUID="2718fba6-5e1c-4c48-a90c-fc26b0ccc385"
Jul 02 12:39:06.357853 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:06.357682    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xsf2s" podUID="e2432aa2-93ec-46ce-afcd-e19f9dd03f8e"
Jul 02 12:39:08.360596 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:08.360564    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsf2s"
Jul 02 12:39:08.361041 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:08.360573    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tsj7t"
Jul 02 12:39:08.361041 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:08.360683    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xsf2s" podUID="e2432aa2-93ec-46ce-afcd-e19f9dd03f8e"
Jul 02 12:39:08.361041 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:08.360573    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cw8v"
Jul 02 12:39:08.361041 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:08.360741    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tsj7t" podUID="2718fba6-5e1c-4c48-a90c-fc26b0ccc385"
Jul 02 12:39:08.361041 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:08.360815    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cw8v" podUID="cd215e6b-c112-412f-a031-30940e656f0d"
Jul 02 12:39:10.139291 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:10.139250    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2718fba6-5e1c-4c48-a90c-fc26b0ccc385-original-pull-secret\") pod \"global-pull-secret-syncer-tsj7t\" (UID: \"2718fba6-5e1c-4c48-a90c-fc26b0ccc385\") " pod="kube-system/global-pull-secret-syncer-tsj7t"
Jul 02 12:39:10.139706 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:10.139306    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2432aa2-93ec-46ce-afcd-e19f9dd03f8e-metrics-certs\") pod \"network-metrics-daemon-xsf2s\" (UID: \"e2432aa2-93ec-46ce-afcd-e19f9dd03f8e\") " pod="openshift-multus/network-metrics-daemon-xsf2s"
Jul 02 12:39:10.139706 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:10.139410    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:39:10.139706 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:10.139413    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 12:39:10.139706 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:10.139473    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2432aa2-93ec-46ce-afcd-e19f9dd03f8e-metrics-certs podName:e2432aa2-93ec-46ce-afcd-e19f9dd03f8e nodeName:}" failed. No retries permitted until 2026-07-02 12:39:26.139458 +0000 UTC m=+34.272283824 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e2432aa2-93ec-46ce-afcd-e19f9dd03f8e-metrics-certs") pod "network-metrics-daemon-xsf2s" (UID: "e2432aa2-93ec-46ce-afcd-e19f9dd03f8e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:39:10.139706 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:10.139491    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2718fba6-5e1c-4c48-a90c-fc26b0ccc385-original-pull-secret podName:2718fba6-5e1c-4c48-a90c-fc26b0ccc385 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:26.139482445 +0000 UTC m=+34.272308270 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2718fba6-5e1c-4c48-a90c-fc26b0ccc385-original-pull-secret") pod "global-pull-secret-syncer-tsj7t" (UID: "2718fba6-5e1c-4c48-a90c-fc26b0ccc385") : object "kube-system"/"original-pull-secret" not registered
Jul 02 12:39:10.240196 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:10.240150    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpvs6\" (UniqueName: \"kubernetes.io/projected/cd215e6b-c112-412f-a031-30940e656f0d-kube-api-access-mpvs6\") pod \"network-check-target-4cw8v\" (UID: \"cd215e6b-c112-412f-a031-30940e656f0d\") " pod="openshift-network-diagnostics/network-check-target-4cw8v"
Jul 02 12:39:10.240360 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:10.240311    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 12:39:10.240360 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:10.240342    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 12:39:10.240360 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:10.240357    2578 projected.go:196] Error preparing data for projected volume kube-api-access-mpvs6 for pod openshift-network-diagnostics/network-check-target-4cw8v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:39:10.240527 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:10.240410    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd215e6b-c112-412f-a031-30940e656f0d-kube-api-access-mpvs6 podName:cd215e6b-c112-412f-a031-30940e656f0d nodeName:}" failed. No retries permitted until 2026-07-02 12:39:26.240397011 +0000 UTC m=+34.373222834 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mpvs6" (UniqueName: "kubernetes.io/projected/cd215e6b-c112-412f-a031-30940e656f0d-kube-api-access-mpvs6") pod "network-check-target-4cw8v" (UID: "cd215e6b-c112-412f-a031-30940e656f0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:39:10.356927 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:10.356892    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tsj7t"
Jul 02 12:39:10.356927 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:10.356928    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsf2s"
Jul 02 12:39:10.357133 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:10.357032    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tsj7t" podUID="2718fba6-5e1c-4c48-a90c-fc26b0ccc385"
Jul 02 12:39:10.357227 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:10.357116    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xsf2s" podUID="e2432aa2-93ec-46ce-afcd-e19f9dd03f8e"
Jul 02 12:39:10.357227 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:10.357193    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cw8v"
Jul 02 12:39:10.357300 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:10.357269    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cw8v" podUID="cd215e6b-c112-412f-a031-30940e656f0d"
Jul 02 12:39:12.359686 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:12.359637    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsf2s"
Jul 02 12:39:12.360103 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:12.359771    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cw8v"
Jul 02 12:39:12.360103 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:12.359801    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tsj7t"
Jul 02 12:39:12.360103 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:12.359808    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xsf2s" podUID="e2432aa2-93ec-46ce-afcd-e19f9dd03f8e"
Jul 02 12:39:12.360103 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:12.359868    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cw8v" podUID="cd215e6b-c112-412f-a031-30940e656f0d"
Jul 02 12:39:12.360103 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:12.359970    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tsj7t" podUID="2718fba6-5e1c-4c48-a90c-fc26b0ccc385"
Jul 02 12:39:13.486141 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:13.485857    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nj6tf" event={"ID":"2b74229e-eeba-4f5d-b05a-1d67c83d450c","Type":"ContainerStarted","Data":"d2d21394e19803164896459be12dbe15af165f330d0dffb70f494be94300e9eb"}
Jul 02 12:39:13.533571 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:13.533517    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-127.ec2.internal" podStartSLOduration=20.533497594 podStartE2EDuration="20.533497594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:38:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:38:57.446545641 +0000 UTC m=+5.579371489" watchObservedRunningTime="2026-07-02 12:39:13.533497594 +0000 UTC m=+21.666323441"
Jul 02 12:39:14.360056 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:14.359689    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cw8v"
Jul 02 12:39:14.360223 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:14.359700    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tsj7t"
Jul 02 12:39:14.360223 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:14.360080    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cw8v" podUID="cd215e6b-c112-412f-a031-30940e656f0d"
Jul 02 12:39:14.360223 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:14.359689    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsf2s"
Jul 02 12:39:14.360223 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:14.360150    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tsj7t" podUID="2718fba6-5e1c-4c48-a90c-fc26b0ccc385"
Jul 02 12:39:14.360350 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:14.360259    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xsf2s" podUID="e2432aa2-93ec-46ce-afcd-e19f9dd03f8e"
Jul 02 12:39:14.488849 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:14.488816    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cjwdw" event={"ID":"3a25dc59-8e44-4f73-8275-342792a291b6","Type":"ContainerStarted","Data":"87970d01a55540304acf7c4c75785f599424dec222a1380f3439e65418f03b06"}
Jul 02 12:39:14.490207 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:14.490161    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9f8kj" event={"ID":"176a476f-a437-42c6-a32d-b6d23e4d2a9f","Type":"ContainerStarted","Data":"9f4569cf35a48c14f473e631e080218e964a651d5fbab0f06483a4842b09f763"}
Jul 02 12:39:14.491511 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:14.491493    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftctb" event={"ID":"be2acf25-23d8-4ce7-bcb1-24aa464b48aa","Type":"ContainerStarted","Data":"1e221b5055dc8f7733f1b2125eb3d798ff4a2b46ce6357a97bcfad307cde6c3b"}
Jul 02 12:39:14.492696 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:14.492674    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kpfkc" event={"ID":"7908d111-8ec6-4177-bd91-38966a57ac13","Type":"ContainerStarted","Data":"58b2ab9c0ca5a3e5d566ce880b9f57b20d7ad785b130b6e3b8e784196559d0a0"}
Jul 02 12:39:14.494068 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:14.494043    2578 generic.go:356] "Generic (PLEG): container finished" podID="b1652685-279d-4a21-a047-736b5076b4e4" containerID="5a29aa2c6739d5a2bba743b0ab75f4a7ee512235b386449dbed8085078c8d85c" exitCode=0
Jul 02 12:39:14.494147 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:14.494133    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mtbbk" event={"ID":"b1652685-279d-4a21-a047-736b5076b4e4","Type":"ContainerDied","Data":"5a29aa2c6739d5a2bba743b0ab75f4a7ee512235b386449dbed8085078c8d85c"}
Jul 02 12:39:14.496639 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:14.496622    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95kwm_495e9fa8-b93f-42e0-95b5-813527df0993/ovn-acl-logging/0.log"
Jul 02 12:39:14.496924 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:14.496904    2578 generic.go:356] "Generic (PLEG): container finished" podID="495e9fa8-b93f-42e0-95b5-813527df0993" containerID="fa6890129113d08e0f598f1e922061acf54728f79608d2bcf400f07c71857a82" exitCode=1
Jul 02 12:39:14.497006 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:14.496976    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-95kwm" event={"ID":"495e9fa8-b93f-42e0-95b5-813527df0993","Type":"ContainerStarted","Data":"1398446c6f14ba6fbc57c35aa59c41d4fd938bc7dbb7e3fa3cadbf6facefef75"}
Jul 02 12:39:14.497063 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:14.497048    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-95kwm" event={"ID":"495e9fa8-b93f-42e0-95b5-813527df0993","Type":"ContainerStarted","Data":"31cf2dff574372590d44b315bc16cea930211bba402b013cbd14419f716e0156"}
Jul 02 12:39:14.497117 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:14.497100    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-95kwm" event={"ID":"495e9fa8-b93f-42e0-95b5-813527df0993","Type":"ContainerStarted","Data":"13e8c480138e0ba9a0ef72786bc2d35cb9fb91dc9b303b8f269af462ad1128f7"}
Jul 02 12:39:14.497187 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:14.497147    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-95kwm" event={"ID":"495e9fa8-b93f-42e0-95b5-813527df0993","Type":"ContainerStarted","Data":"d720d040e6502fc9d4bde0df6a547dd59b6310cadd9f8ecf1fbafbfc1f3e0768"}
Jul 02 12:39:14.497247 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:14.497213    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-95kwm" event={"ID":"495e9fa8-b93f-42e0-95b5-813527df0993","Type":"ContainerDied","Data":"fa6890129113d08e0f598f1e922061acf54728f79608d2bcf400f07c71857a82"}
Jul 02 12:39:14.497304 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:14.497264    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-95kwm" event={"ID":"495e9fa8-b93f-42e0-95b5-813527df0993","Type":"ContainerStarted","Data":"58a526dd826c3bcbc09adce3a708de4a08a2800cf074ff2e16a06e068f7383b9"}
Jul 02 12:39:14.498193 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:14.498163    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cp22n" event={"ID":"9777c828-7acd-4d27-be6a-dbe5c3f6f18b","Type":"ContainerStarted","Data":"3a9c5fc68f6d5d950e26209b5318453f2a173c6955bfc00effe74404f9ca95a2"}
Jul 02 12:39:14.516673 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:14.516634    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-cjwdw" podStartSLOduration=4.388262022 podStartE2EDuration="22.516621113s" totalImagesPullingTime="18.128359091s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:38:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:14.516543934 +0000 UTC m=+22.649369781" watchObservedRunningTime="2026-07-02 12:39:14.516621113 +0000 UTC m=+22.649446961"
Jul 02 12:39:14.517023 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:14.517001    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nj6tf" podStartSLOduration=4.422920691 podStartE2EDuration="22.516993857s" totalImagesPullingTime="18.094073166s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:38:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:13.534670795 +0000 UTC m=+21.667496661" watchObservedRunningTime="2026-07-02 12:39:14.516993857 +0000 UTC m=+22.649819702"
Jul 02 12:39:14.543211 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:14.543160    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cp22n" podStartSLOduration=4.413788734 podStartE2EDuration="22.543148828s" totalImagesPullingTime="18.129360094s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:38:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:14.542799088 +0000 UTC m=+22.675624934" watchObservedRunningTime="2026-07-02 12:39:14.543148828 +0000 UTC m=+22.675974673"
Jul 02 12:39:14.603611 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:14.603577    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-kpfkc" podStartSLOduration=4.475895571 podStartE2EDuration="22.603564546s" totalImagesPullingTime="18.127668975s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:38:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:14.579524721 +0000 UTC m=+22.712350566" watchObservedRunningTime="2026-07-02 12:39:14.603564546 +0000 UTC m=+22.736390392"
Jul 02 12:39:14.643107 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:14.643060    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9f8kj" podStartSLOduration=4.5415537 podStartE2EDuration="22.643042248s" totalImagesPullingTime="18.101488548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:38:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:14.60406668 +0000 UTC m=+22.736892520" watchObservedRunningTime="2026-07-02 12:39:14.643042248 +0000 UTC m=+22.775868095"
Jul 02 12:39:14.673021 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:14.672979    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:14.725587 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:14.725565    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 02 12:39:14.842369 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:14.842339    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:14.943189 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:14.943088    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:15.361091 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:15.360703    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-02T12:39:14.725582218Z","UUID":"e0afeefb-345f-4bf4-9b25-bca4353988b8","Handler":null,"Name":"","Endpoint":""}
Jul 02 12:39:15.362563 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:15.362535    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 02 12:39:15.362692 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:15.362586    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 02 12:39:15.503341 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:15.503302    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftctb" event={"ID":"be2acf25-23d8-4ce7-bcb1-24aa464b48aa","Type":"ContainerStarted","Data":"d66e33cd4d196e510d0f598cf4fd6c1aac331efcfb14102fd72933aa5c2b45bd"}
Jul 02 12:39:15.505008 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:15.504947    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xplqk" event={"ID":"b1a6f623-3064-4500-be97-92644ef17328","Type":"ContainerStarted","Data":"f061fc4e0e21ee2fc04684d757c99f33b429217dc1ef83f09a808631138c0d8a"}
Jul 02 12:39:15.521966 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:15.521919    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xplqk" podStartSLOduration=5.398571921 podStartE2EDuration="23.521903256s" totalImagesPullingTime="18.123331335s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:38:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:15.521776914 +0000 UTC m=+23.654602781" watchObservedRunningTime="2026-07-02 12:39:15.521903256 +0000 UTC m=+23.654729102"
Jul 02 12:39:15.644426 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:15.644319    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:15.698511 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:15.698483    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:16.356826 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:16.356793    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cw8v"
Jul 02 12:39:16.357001 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:16.356793    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsf2s"
Jul 02 12:39:16.357001 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:16.356925    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cw8v" podUID="cd215e6b-c112-412f-a031-30940e656f0d"
Jul 02 12:39:16.357001 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:16.356793    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tsj7t"
Jul 02 12:39:16.357189 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:16.357000    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xsf2s" podUID="e2432aa2-93ec-46ce-afcd-e19f9dd03f8e"
Jul 02 12:39:16.357189 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:16.357059    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tsj7t" podUID="2718fba6-5e1c-4c48-a90c-fc26b0ccc385"
Jul 02 12:39:16.510302 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:16.510264    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftctb" event={"ID":"be2acf25-23d8-4ce7-bcb1-24aa464b48aa","Type":"ContainerStarted","Data":"24f5d85613f9be0e6fe37434cbec6b7bc08d54acc3c2d64a26a1d2922de58ad1"}
Jul 02 12:39:16.513866 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:16.513836    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95kwm_495e9fa8-b93f-42e0-95b5-813527df0993/ovn-acl-logging/0.log"
Jul 02 12:39:16.514300 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:16.514271    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-95kwm" event={"ID":"495e9fa8-b93f-42e0-95b5-813527df0993","Type":"ContainerStarted","Data":"0c53c5eb0080fac97217d8f36d4a63e721528983bce652d3c5b288b36720b3fd"}
Jul 02 12:39:16.529763 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:16.529678    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftctb" podStartSLOduration=4.264955686 podStartE2EDuration="24.529657648s" totalImagesPullingTime="20.264701962s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:38:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:16.529535306 +0000 UTC m=+24.662361152" watchObservedRunningTime="2026-07-02 12:39:16.529657648 +0000 UTC m=+24.662483499"
Jul 02 12:39:16.645098 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:16.645059    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9f8kj"
Jul 02 12:39:16.645812 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:16.645794    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9f8kj"
Jul 02 12:39:16.994157 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:16.994105    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:17.517209 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:17.517163    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9f8kj"
Jul 02 12:39:17.517664 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:17.517648    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9f8kj"
Jul 02 12:39:17.635758 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:17.635717    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:18.222761 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:18.222726    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:18.338867 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:18.338821    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:18.356877 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:18.356839    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsf2s"
Jul 02 12:39:18.357044 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:18.356839    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tsj7t"
Jul 02 12:39:18.357044 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:18.356977    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xsf2s" podUID="e2432aa2-93ec-46ce-afcd-e19f9dd03f8e"
Jul 02 12:39:18.357044 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:18.356989    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cw8v"
Jul 02 12:39:18.357217 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:18.357106    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tsj7t" podUID="2718fba6-5e1c-4c48-a90c-fc26b0ccc385"
Jul 02 12:39:18.357217 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:18.357191    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cw8v" podUID="cd215e6b-c112-412f-a031-30940e656f0d"
Jul 02 12:39:19.524873 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:19.524656    2578 generic.go:356] "Generic (PLEG): container finished" podID="b1652685-279d-4a21-a047-736b5076b4e4" containerID="2833e204738c9cc59b71e5072a5fba82c001120ca31231cc43a3777785b71013" exitCode=0
Jul 02 12:39:19.525737 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:19.524738    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mtbbk" event={"ID":"b1652685-279d-4a21-a047-736b5076b4e4","Type":"ContainerDied","Data":"2833e204738c9cc59b71e5072a5fba82c001120ca31231cc43a3777785b71013"}
Jul 02 12:39:19.528697 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:19.528679    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95kwm_495e9fa8-b93f-42e0-95b5-813527df0993/ovn-acl-logging/0.log"
Jul 02 12:39:19.529065 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:19.529042    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-95kwm" event={"ID":"495e9fa8-b93f-42e0-95b5-813527df0993","Type":"ContainerStarted","Data":"dbaae2dedb5b72eb3aa6375c7acf9af843739424a74a75d6fdf3c65d17084fae"}
Jul 02 12:39:19.529560 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:19.529546    2578 scope.go:117] "RemoveContainer" containerID="fa6890129113d08e0f598f1e922061acf54728f79608d2bcf400f07c71857a82"
Jul 02 12:39:20.357089 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:20.356827    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tsj7t"
Jul 02 12:39:20.357252 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:20.356837    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cw8v"
Jul 02 12:39:20.357252 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:20.357224    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tsj7t" podUID="2718fba6-5e1c-4c48-a90c-fc26b0ccc385"
Jul 02 12:39:20.357328 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:20.356890    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsf2s"
Jul 02 12:39:20.357328 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:20.357293    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cw8v" podUID="cd215e6b-c112-412f-a031-30940e656f0d"
Jul 02 12:39:20.357390 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:20.357342    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xsf2s" podUID="e2432aa2-93ec-46ce-afcd-e19f9dd03f8e"
Jul 02 12:39:20.533952 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:20.533916    2578 generic.go:356] "Generic (PLEG): container finished" podID="b1652685-279d-4a21-a047-736b5076b4e4" containerID="8dda9e00531d5c9ecd769cdcd40c9428c0504f1ed4b1295f3c5768824966acda" exitCode=0
Jul 02 12:39:20.534399 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:20.534017    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mtbbk" event={"ID":"b1652685-279d-4a21-a047-736b5076b4e4","Type":"ContainerDied","Data":"8dda9e00531d5c9ecd769cdcd40c9428c0504f1ed4b1295f3c5768824966acda"}
Jul 02 12:39:20.538487 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:20.538462    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95kwm_495e9fa8-b93f-42e0-95b5-813527df0993/ovn-acl-logging/0.log"
Jul 02 12:39:20.538903 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:20.538874    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-95kwm" event={"ID":"495e9fa8-b93f-42e0-95b5-813527df0993","Type":"ContainerStarted","Data":"212f99f4227601a8adfdb4214782fcbf6466bb05a7430fc10f6779dcc93dc216"}
Jul 02 12:39:20.538996 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:20.538973    2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 02 12:39:20.539231 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:20.539213    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:39:20.539337 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:20.539282    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:39:20.555709 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:20.555680    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:39:20.555832 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:20.555797    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:39:20.603773 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:20.603722    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-95kwm" podStartSLOduration=10.428581391 podStartE2EDuration="28.603704849s" totalImagesPullingTime="18.175123458s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:38:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:20.601904051 +0000 UTC m=+28.734729915" watchObservedRunningTime="2026-07-02 12:39:20.603704849 +0000 UTC m=+28.736530727"
Jul 02 12:39:20.671827 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:20.671796    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:20.770478 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:20.770448    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:20.787912 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:20.787830    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tsj7t"]
Jul 02 12:39:20.788032 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:20.787923    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tsj7t"
Jul 02 12:39:20.788032 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:20.788005    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tsj7t" podUID="2718fba6-5e1c-4c48-a90c-fc26b0ccc385"
Jul 02 12:39:20.790406 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:20.790381    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xsf2s"]
Jul 02 12:39:20.790525 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:20.790482    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsf2s"
Jul 02 12:39:20.790601 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:20.790580    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xsf2s" podUID="e2432aa2-93ec-46ce-afcd-e19f9dd03f8e"
Jul 02 12:39:20.794096 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:20.794070    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4cw8v"]
Jul 02 12:39:20.794207 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:20.794146    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cw8v"
Jul 02 12:39:20.794249 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:20.794230    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cw8v" podUID="cd215e6b-c112-412f-a031-30940e656f0d"
Jul 02 12:39:21.540886 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:21.540860    2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 02 12:39:22.357376 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:22.357336    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cw8v"
Jul 02 12:39:22.357534 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:22.357429    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cw8v" podUID="cd215e6b-c112-412f-a031-30940e656f0d"
Jul 02 12:39:22.357534 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:22.357467    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tsj7t"
Jul 02 12:39:22.357655 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:22.357569    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsf2s"
Jul 02 12:39:22.357655 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:22.357586    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tsj7t" podUID="2718fba6-5e1c-4c48-a90c-fc26b0ccc385"
Jul 02 12:39:22.357655 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:22.357650    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xsf2s" podUID="e2432aa2-93ec-46ce-afcd-e19f9dd03f8e"
Jul 02 12:39:22.545520 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:22.545474    2578 generic.go:356] "Generic (PLEG): container finished" podID="b1652685-279d-4a21-a047-736b5076b4e4" containerID="9b4ba1a7819ca9a4a684952dc81a73b7180423f4d1bfde53614f0b9bc842c2f4" exitCode=0
Jul 02 12:39:22.545970 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:22.545559    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mtbbk" event={"ID":"b1652685-279d-4a21-a047-736b5076b4e4","Type":"ContainerDied","Data":"9b4ba1a7819ca9a4a684952dc81a73b7180423f4d1bfde53614f0b9bc842c2f4"}
Jul 02 12:39:22.545970 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:22.545637    2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 02 12:39:23.986522 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:23.986488    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:24.055070 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:24.055023    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:24.109678 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:24.109651    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:24.226792 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:24.226759    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:24.283982 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:24.283947    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:39:24.284265 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:24.284193    2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 02 12:39:24.295837 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:24.295792    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-95kwm" podUID="495e9fa8-b93f-42e0-95b5-813527df0993" containerName="ovnkube-controller" probeResult="failure" output=""
Jul 02 12:39:24.305878 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:24.305845    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-95kwm" podUID="495e9fa8-b93f-42e0-95b5-813527df0993" containerName="ovnkube-controller" probeResult="failure" output=""
Jul 02 12:39:24.357015 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:24.356989    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tsj7t"
Jul 02 12:39:24.357197 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:24.356997    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsf2s"
Jul 02 12:39:24.357197 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:24.357109    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cw8v"
Jul 02 12:39:24.357197 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:24.357117    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tsj7t" podUID="2718fba6-5e1c-4c48-a90c-fc26b0ccc385"
Jul 02 12:39:24.357365 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:24.357241    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xsf2s" podUID="e2432aa2-93ec-46ce-afcd-e19f9dd03f8e"
Jul 02 12:39:24.361038 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:24.357999    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cw8v" podUID="cd215e6b-c112-412f-a031-30940e656f0d"
Jul 02 12:39:24.465320 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:24.465290    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:24.561356 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:24.561267    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:24.619292 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:24.619249    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:25.022547 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:25.022366    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:25.122575 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:25.122533    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:25.248813 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:25.248781    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:25.349137 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:25.349055    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:25.669008 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:25.668934    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cjwdw_3a25dc59-8e44-4f73-8275-342792a291b6/dns-node-resolver/0.log"
Jul 02 12:39:26.134250 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.134223    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-127.ec2.internal" event="NodeReady"
Jul 02 12:39:26.134664 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.134376    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 02 12:39:26.160507 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.160472    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2718fba6-5e1c-4c48-a90c-fc26b0ccc385-original-pull-secret\") pod \"global-pull-secret-syncer-tsj7t\" (UID: \"2718fba6-5e1c-4c48-a90c-fc26b0ccc385\") " pod="kube-system/global-pull-secret-syncer-tsj7t"
Jul 02 12:39:26.160688 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.160521    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2432aa2-93ec-46ce-afcd-e19f9dd03f8e-metrics-certs\") pod \"network-metrics-daemon-xsf2s\" (UID: \"e2432aa2-93ec-46ce-afcd-e19f9dd03f8e\") " pod="openshift-multus/network-metrics-daemon-xsf2s"
Jul 02 12:39:26.160688 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:26.160638    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 12:39:26.160688 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:26.160647    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:39:26.160816 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:26.160709    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2432aa2-93ec-46ce-afcd-e19f9dd03f8e-metrics-certs podName:e2432aa2-93ec-46ce-afcd-e19f9dd03f8e nodeName:}" failed. No retries permitted until 2026-07-02 12:39:58.160691385 +0000 UTC m=+66.293517214 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e2432aa2-93ec-46ce-afcd-e19f9dd03f8e-metrics-certs") pod "network-metrics-daemon-xsf2s" (UID: "e2432aa2-93ec-46ce-afcd-e19f9dd03f8e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:39:26.160816 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:26.160728    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2718fba6-5e1c-4c48-a90c-fc26b0ccc385-original-pull-secret podName:2718fba6-5e1c-4c48-a90c-fc26b0ccc385 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:58.160719093 +0000 UTC m=+66.293544918 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2718fba6-5e1c-4c48-a90c-fc26b0ccc385-original-pull-secret") pod "global-pull-secret-syncer-tsj7t" (UID: "2718fba6-5e1c-4c48-a90c-fc26b0ccc385") : object "kube-system"/"original-pull-secret" not registered
Jul 02 12:39:26.206252 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.206220    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tg5m8"]
Jul 02 12:39:26.232897 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.232859    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4xx9d"]
Jul 02 12:39:26.233062 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.232992    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tg5m8"
Jul 02 12:39:26.235821 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.235795    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 02 12:39:26.235983 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.235799    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 02 12:39:26.235983 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.235934    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5qtp9\""
Jul 02 12:39:26.251244 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.251224    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tg5m8"]
Jul 02 12:39:26.251376 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.251257    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-764zc"]
Jul 02 12:39:26.251376 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.251362    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4xx9d"
Jul 02 12:39:26.254896 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.254880    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kzvtc\""
Jul 02 12:39:26.255095 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.255077    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 02 12:39:26.255190 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.255118    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 02 12:39:26.255190 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.255116    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 02 12:39:26.255300 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.255119    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 02 12:39:26.261106 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.261084    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpvs6\" (UniqueName: \"kubernetes.io/projected/cd215e6b-c112-412f-a031-30940e656f0d-kube-api-access-mpvs6\") pod \"network-check-target-4cw8v\" (UID: \"cd215e6b-c112-412f-a031-30940e656f0d\") " pod="openshift-network-diagnostics/network-check-target-4cw8v"
Jul 02 12:39:26.261289 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:26.261274    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 12:39:26.261357 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:26.261296    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 12:39:26.261357 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:26.261308    2578 projected.go:196] Error preparing data for projected volume kube-api-access-mpvs6 for pod openshift-network-diagnostics/network-check-target-4cw8v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:39:26.261505 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:26.261370    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd215e6b-c112-412f-a031-30940e656f0d-kube-api-access-mpvs6 podName:cd215e6b-c112-412f-a031-30940e656f0d nodeName:}" failed. No retries permitted until 2026-07-02 12:39:58.261352802 +0000 UTC m=+66.394178639 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mpvs6" (UniqueName: "kubernetes.io/projected/cd215e6b-c112-412f-a031-30940e656f0d-kube-api-access-mpvs6") pod "network-check-target-4cw8v" (UID: "cd215e6b-c112-412f-a031-30940e656f0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:39:26.269603 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.269584    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4xx9d"]
Jul 02 12:39:26.269715 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.269628    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-764zc"]
Jul 02 12:39:26.269715 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.269668    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-764zc"
Jul 02 12:39:26.273040 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.273015    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4g4pc\""
Jul 02 12:39:26.273234 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.273055    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 02 12:39:26.273402 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.273379    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 02 12:39:26.273402 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.273397    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 02 12:39:26.277940 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.277919    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:26.330307 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.330233    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:26.357255 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.357216    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsf2s"
Jul 02 12:39:26.357441 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.357224    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tsj7t"
Jul 02 12:39:26.357441 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.357370    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cw8v"
Jul 02 12:39:26.361916 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.361654    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/851415ab-7420-40c0-aa53-4e1d8899ee4c-config-volume\") pod \"dns-default-tg5m8\" (UID: \"851415ab-7420-40c0-aa53-4e1d8899ee4c\") " pod="openshift-dns/dns-default-tg5m8"
Jul 02 12:39:26.361916 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.361663    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 12:39:26.361916 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.361723    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bwrzl\""
Jul 02 12:39:26.361916 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.361764    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-crio-socket\") pod \"insights-runtime-extractor-4xx9d\" (UID: \"e3a44725-2f0a-44ea-9a50-4f13d28b0bd3\") " pod="openshift-insights/insights-runtime-extractor-4xx9d"
Jul 02 12:39:26.361916 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.361797    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2pw4\" (UniqueName: \"kubernetes.io/projected/9dcd5dc8-2736-4029-b386-bcecfe1bea3d-kube-api-access-l2pw4\") pod \"ingress-canary-764zc\" (UID: \"9dcd5dc8-2736-4029-b386-bcecfe1bea3d\") " pod="openshift-ingress-canary/ingress-canary-764zc"
Jul 02 12:39:26.361916 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.361806    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 12:39:26.361916 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.361832    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftzcb\" (UniqueName: \"kubernetes.io/projected/851415ab-7420-40c0-aa53-4e1d8899ee4c-kube-api-access-ftzcb\") pod \"dns-default-tg5m8\" (UID: \"851415ab-7420-40c0-aa53-4e1d8899ee4c\") " pod="openshift-dns/dns-default-tg5m8"
Jul 02 12:39:26.361916 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.361846    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 12:39:26.362431 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.361946    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-data-volume\") pod \"insights-runtime-extractor-4xx9d\" (UID: \"e3a44725-2f0a-44ea-9a50-4f13d28b0bd3\") " pod="openshift-insights/insights-runtime-extractor-4xx9d"
Jul 02 12:39:26.362431 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.362060    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4xx9d\" (UID: \"e3a44725-2f0a-44ea-9a50-4f13d28b0bd3\") " pod="openshift-insights/insights-runtime-extractor-4xx9d"
Jul 02 12:39:26.362431 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.362118    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9dcd5dc8-2736-4029-b386-bcecfe1bea3d-cert\") pod \"ingress-canary-764zc\" (UID: \"9dcd5dc8-2736-4029-b386-bcecfe1bea3d\") " pod="openshift-ingress-canary/ingress-canary-764zc"
Jul 02 12:39:26.362431 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.362171    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4xx9d\" (UID: \"e3a44725-2f0a-44ea-9a50-4f13d28b0bd3\") " pod="openshift-insights/insights-runtime-extractor-4xx9d"
Jul 02 12:39:26.362431 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.362195    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mgbjz\""
Jul 02 12:39:26.362431 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.362216    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/851415ab-7420-40c0-aa53-4e1d8899ee4c-tmp-dir\") pod \"dns-default-tg5m8\" (UID: \"851415ab-7420-40c0-aa53-4e1d8899ee4c\") " pod="openshift-dns/dns-default-tg5m8"
Jul 02 12:39:26.362431 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.362313    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/851415ab-7420-40c0-aa53-4e1d8899ee4c-metrics-tls\") pod \"dns-default-tg5m8\" (UID: \"851415ab-7420-40c0-aa53-4e1d8899ee4c\") " pod="openshift-dns/dns-default-tg5m8"
Jul 02 12:39:26.362431 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.362349    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx7v8\" (UniqueName: \"kubernetes.io/projected/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-kube-api-access-bx7v8\") pod \"insights-runtime-extractor-4xx9d\" (UID: \"e3a44725-2f0a-44ea-9a50-4f13d28b0bd3\") " pod="openshift-insights/insights-runtime-extractor-4xx9d"
Jul 02 12:39:26.362431 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.362408    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 12:39:26.463234 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.463195    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-crio-socket\") pod \"insights-runtime-extractor-4xx9d\" (UID: \"e3a44725-2f0a-44ea-9a50-4f13d28b0bd3\") " pod="openshift-insights/insights-runtime-extractor-4xx9d"
Jul 02 12:39:26.463407 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.463320    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2pw4\" (UniqueName: \"kubernetes.io/projected/9dcd5dc8-2736-4029-b386-bcecfe1bea3d-kube-api-access-l2pw4\") pod \"ingress-canary-764zc\" (UID: \"9dcd5dc8-2736-4029-b386-bcecfe1bea3d\") " pod="openshift-ingress-canary/ingress-canary-764zc"
Jul 02 12:39:26.463407 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.463347    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftzcb\" (UniqueName: \"kubernetes.io/projected/851415ab-7420-40c0-aa53-4e1d8899ee4c-kube-api-access-ftzcb\") pod \"dns-default-tg5m8\" (UID: \"851415ab-7420-40c0-aa53-4e1d8899ee4c\") " pod="openshift-dns/dns-default-tg5m8"
Jul 02 12:39:26.463407 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.463376    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-data-volume\") pod \"insights-runtime-extractor-4xx9d\" (UID: \"e3a44725-2f0a-44ea-9a50-4f13d28b0bd3\") " pod="openshift-insights/insights-runtime-extractor-4xx9d"
Jul 02 12:39:26.463567 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.463405    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4xx9d\" (UID: \"e3a44725-2f0a-44ea-9a50-4f13d28b0bd3\") " pod="openshift-insights/insights-runtime-extractor-4xx9d"
Jul 02 12:39:26.463567 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.463437    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9dcd5dc8-2736-4029-b386-bcecfe1bea3d-cert\") pod \"ingress-canary-764zc\" (UID: \"9dcd5dc8-2736-4029-b386-bcecfe1bea3d\") " pod="openshift-ingress-canary/ingress-canary-764zc"
Jul 02 12:39:26.463567 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.463458    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-crio-socket\") pod \"insights-runtime-extractor-4xx9d\" (UID: \"e3a44725-2f0a-44ea-9a50-4f13d28b0bd3\") " pod="openshift-insights/insights-runtime-extractor-4xx9d"
Jul 02 12:39:26.463567 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.463469    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4xx9d\" (UID: \"e3a44725-2f0a-44ea-9a50-4f13d28b0bd3\") " pod="openshift-insights/insights-runtime-extractor-4xx9d"
Jul 02 12:39:26.463567 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.463509    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/851415ab-7420-40c0-aa53-4e1d8899ee4c-tmp-dir\") pod \"dns-default-tg5m8\" (UID: \"851415ab-7420-40c0-aa53-4e1d8899ee4c\") " pod="openshift-dns/dns-default-tg5m8"
Jul 02 12:39:26.463793 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.463570    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/851415ab-7420-40c0-aa53-4e1d8899ee4c-metrics-tls\") pod \"dns-default-tg5m8\" (UID: \"851415ab-7420-40c0-aa53-4e1d8899ee4c\") " pod="openshift-dns/dns-default-tg5m8"
Jul 02 12:39:26.463793 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.463600    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx7v8\" (UniqueName: \"kubernetes.io/projected/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-kube-api-access-bx7v8\") pod \"insights-runtime-extractor-4xx9d\" (UID: \"e3a44725-2f0a-44ea-9a50-4f13d28b0bd3\") " pod="openshift-insights/insights-runtime-extractor-4xx9d"
Jul 02 12:39:26.463793 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.463668    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/851415ab-7420-40c0-aa53-4e1d8899ee4c-config-volume\") pod \"dns-default-tg5m8\" (UID: \"851415ab-7420-40c0-aa53-4e1d8899ee4c\") " pod="openshift-dns/dns-default-tg5m8"
Jul 02 12:39:26.463793 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:26.463752    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 12:39:26.463951 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:26.463819    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-kube-rbac-proxy-cm podName:e3a44725-2f0a-44ea-9a50-4f13d28b0bd3 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:26.963799674 +0000 UTC m=+35.096625511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4xx9d" (UID: "e3a44725-2f0a-44ea-9a50-4f13d28b0bd3") : configmap "kube-rbac-proxy" not found
Jul 02 12:39:26.463951 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.463856    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-data-volume\") pod \"insights-runtime-extractor-4xx9d\" (UID: \"e3a44725-2f0a-44ea-9a50-4f13d28b0bd3\") " pod="openshift-insights/insights-runtime-extractor-4xx9d"
Jul 02 12:39:26.464044 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:26.464000    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 12:39:26.464086 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:26.464064    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 12:39:26.464086 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:26.464083    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-insights-runtime-extractor-tls podName:e3a44725-2f0a-44ea-9a50-4f13d28b0bd3 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:26.964063022 +0000 UTC m=+35.096888877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4xx9d" (UID: "e3a44725-2f0a-44ea-9a50-4f13d28b0bd3") : secret "insights-runtime-extractor-tls" not found
Jul 02 12:39:26.464209 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.464099    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/851415ab-7420-40c0-aa53-4e1d8899ee4c-tmp-dir\") pod \"dns-default-tg5m8\" (UID: \"851415ab-7420-40c0-aa53-4e1d8899ee4c\") " pod="openshift-dns/dns-default-tg5m8"
Jul 02 12:39:26.464209 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:26.464111    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/851415ab-7420-40c0-aa53-4e1d8899ee4c-metrics-tls podName:851415ab-7420-40c0-aa53-4e1d8899ee4c nodeName:}" failed. No retries permitted until 2026-07-02 12:39:26.964100697 +0000 UTC m=+35.096926521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/851415ab-7420-40c0-aa53-4e1d8899ee4c-metrics-tls") pod "dns-default-tg5m8" (UID: "851415ab-7420-40c0-aa53-4e1d8899ee4c") : secret "dns-default-metrics-tls" not found
Jul 02 12:39:26.464294 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.464274    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/851415ab-7420-40c0-aa53-4e1d8899ee4c-config-volume\") pod \"dns-default-tg5m8\" (UID: \"851415ab-7420-40c0-aa53-4e1d8899ee4c\") " pod="openshift-dns/dns-default-tg5m8"
Jul 02 12:39:26.468267 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.468243    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9dcd5dc8-2736-4029-b386-bcecfe1bea3d-cert\") pod \"ingress-canary-764zc\" (UID: \"9dcd5dc8-2736-4029-b386-bcecfe1bea3d\") " pod="openshift-ingress-canary/ingress-canary-764zc"
Jul 02 12:39:26.474839 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.474818    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftzcb\" (UniqueName: \"kubernetes.io/projected/851415ab-7420-40c0-aa53-4e1d8899ee4c-kube-api-access-ftzcb\") pod \"dns-default-tg5m8\" (UID: \"851415ab-7420-40c0-aa53-4e1d8899ee4c\") " pod="openshift-dns/dns-default-tg5m8"
Jul 02 12:39:26.474959 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.474946    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2pw4\" (UniqueName: \"kubernetes.io/projected/9dcd5dc8-2736-4029-b386-bcecfe1bea3d-kube-api-access-l2pw4\") pod \"ingress-canary-764zc\" (UID: \"9dcd5dc8-2736-4029-b386-bcecfe1bea3d\") " pod="openshift-ingress-canary/ingress-canary-764zc"
Jul 02 12:39:26.475267 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.475246    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx7v8\" (UniqueName: \"kubernetes.io/projected/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-kube-api-access-bx7v8\") pod \"insights-runtime-extractor-4xx9d\" (UID: \"e3a44725-2f0a-44ea-9a50-4f13d28b0bd3\") " pod="openshift-insights/insights-runtime-extractor-4xx9d"
Jul 02 12:39:26.579608 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.579575    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-764zc"
Jul 02 12:39:26.669782 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.669757    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cp22n_9777c828-7acd-4d27-be6a-dbe5c3f6f18b/node-ca/0.log"
Jul 02 12:39:26.787787 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.787749    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-764zc"]
Jul 02 12:39:26.901464 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:39:26.901395    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dcd5dc8_2736_4029_b386_bcecfe1bea3d.slice/crio-bf42ec7e8a947ab074382e96523e472c87bc7720fac27949c7f5cd7eb1faffea WatchSource:0}: Error finding container bf42ec7e8a947ab074382e96523e472c87bc7720fac27949c7f5cd7eb1faffea: Status 404 returned error can't find the container with id bf42ec7e8a947ab074382e96523e472c87bc7720fac27949c7f5cd7eb1faffea
Jul 02 12:39:26.967966 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.967935    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4xx9d\" (UID: \"e3a44725-2f0a-44ea-9a50-4f13d28b0bd3\") " pod="openshift-insights/insights-runtime-extractor-4xx9d"
Jul 02 12:39:26.968109 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.967983    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/851415ab-7420-40c0-aa53-4e1d8899ee4c-metrics-tls\") pod \"dns-default-tg5m8\" (UID: \"851415ab-7420-40c0-aa53-4e1d8899ee4c\") " pod="openshift-dns/dns-default-tg5m8"
Jul 02 12:39:26.968109 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:26.968051    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4xx9d\" (UID: \"e3a44725-2f0a-44ea-9a50-4f13d28b0bd3\") " pod="openshift-insights/insights-runtime-extractor-4xx9d"
Jul 02 12:39:26.968109 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:26.968069    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 12:39:26.968285 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:26.968140    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 12:39:26.968285 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:26.968141    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 12:39:26.968285 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:26.968147    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-kube-rbac-proxy-cm podName:e3a44725-2f0a-44ea-9a50-4f13d28b0bd3 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:27.968126651 +0000 UTC m=+36.100952475 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4xx9d" (UID: "e3a44725-2f0a-44ea-9a50-4f13d28b0bd3") : configmap "kube-rbac-proxy" not found
Jul 02 12:39:26.968285 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:26.968208    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-insights-runtime-extractor-tls podName:e3a44725-2f0a-44ea-9a50-4f13d28b0bd3 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:27.968194731 +0000 UTC m=+36.101020555 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4xx9d" (UID: "e3a44725-2f0a-44ea-9a50-4f13d28b0bd3") : secret "insights-runtime-extractor-tls" not found
Jul 02 12:39:26.968285 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:26.968219    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/851415ab-7420-40c0-aa53-4e1d8899ee4c-metrics-tls podName:851415ab-7420-40c0-aa53-4e1d8899ee4c nodeName:}" failed. No retries permitted until 2026-07-02 12:39:27.968212569 +0000 UTC m=+36.101038392 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/851415ab-7420-40c0-aa53-4e1d8899ee4c-metrics-tls") pod "dns-default-tg5m8" (UID: "851415ab-7420-40c0-aa53-4e1d8899ee4c") : secret "dns-default-metrics-tls" not found
Jul 02 12:39:27.496361 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:27.496134    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-7bb4f4bcc7-hvxnc"]
Jul 02 12:39:27.514256 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:27.514229    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-7bb4f4bcc7-hvxnc"]
Jul 02 12:39:27.514416 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:27.514335    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-7bb4f4bcc7-hvxnc"
Jul 02 12:39:27.518120 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:27.518072    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 02 12:39:27.519169 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:27.519027    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 02 12:39:27.519169 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:27.519090    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 02 12:39:27.519169 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:27.519090    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-h2kmd\""
Jul 02 12:39:27.519408 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:27.519031    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 02 12:39:27.556756 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:27.556724    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:27.561320 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:27.561277    2578 generic.go:356] "Generic (PLEG): container finished" podID="b1652685-279d-4a21-a047-736b5076b4e4" containerID="7d18d0524d06e5b27f9909d82c8206541c242801e54f74442e4a7bcec8121a0f" exitCode=0
Jul 02 12:39:27.561458 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:27.561351    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mtbbk" event={"ID":"b1652685-279d-4a21-a047-736b5076b4e4","Type":"ContainerDied","Data":"7d18d0524d06e5b27f9909d82c8206541c242801e54f74442e4a7bcec8121a0f"}
Jul 02 12:39:27.562788 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:27.562761    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-764zc" event={"ID":"9dcd5dc8-2736-4029-b386-bcecfe1bea3d","Type":"ContainerStarted","Data":"bf42ec7e8a947ab074382e96523e472c87bc7720fac27949c7f5cd7eb1faffea"}
Jul 02 12:39:27.628037 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:27.628002    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:27.672552 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:27.672516    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3a887021-e49f-45a8-a45e-de17a9070633-signing-key\") pod \"service-ca-7bb4f4bcc7-hvxnc\" (UID: \"3a887021-e49f-45a8-a45e-de17a9070633\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-hvxnc"
Jul 02 12:39:27.672719 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:27.672671    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3a887021-e49f-45a8-a45e-de17a9070633-signing-cabundle\") pod \"service-ca-7bb4f4bcc7-hvxnc\" (UID: \"3a887021-e49f-45a8-a45e-de17a9070633\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-hvxnc"
Jul 02 12:39:27.672767 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:27.672739    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbzvr\" (UniqueName: \"kubernetes.io/projected/3a887021-e49f-45a8-a45e-de17a9070633-kube-api-access-lbzvr\") pod \"service-ca-7bb4f4bcc7-hvxnc\" (UID: \"3a887021-e49f-45a8-a45e-de17a9070633\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-hvxnc"
Jul 02 12:39:27.773479 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:27.773445    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbzvr\" (UniqueName: \"kubernetes.io/projected/3a887021-e49f-45a8-a45e-de17a9070633-kube-api-access-lbzvr\") pod \"service-ca-7bb4f4bcc7-hvxnc\" (UID: \"3a887021-e49f-45a8-a45e-de17a9070633\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-hvxnc"
Jul 02 12:39:27.773646 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:27.773538    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3a887021-e49f-45a8-a45e-de17a9070633-signing-key\") pod \"service-ca-7bb4f4bcc7-hvxnc\" (UID: \"3a887021-e49f-45a8-a45e-de17a9070633\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-hvxnc"
Jul 02 12:39:27.773749 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:27.773728    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3a887021-e49f-45a8-a45e-de17a9070633-signing-cabundle\") pod \"service-ca-7bb4f4bcc7-hvxnc\" (UID: \"3a887021-e49f-45a8-a45e-de17a9070633\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-hvxnc"
Jul 02 12:39:27.774478 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:27.774456    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3a887021-e49f-45a8-a45e-de17a9070633-signing-cabundle\") pod \"service-ca-7bb4f4bcc7-hvxnc\" (UID: \"3a887021-e49f-45a8-a45e-de17a9070633\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-hvxnc"
Jul 02 12:39:27.777057 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:27.777032    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3a887021-e49f-45a8-a45e-de17a9070633-signing-key\") pod \"service-ca-7bb4f4bcc7-hvxnc\" (UID: \"3a887021-e49f-45a8-a45e-de17a9070633\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-hvxnc"
Jul 02 12:39:27.783679 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:27.783653    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbzvr\" (UniqueName: \"kubernetes.io/projected/3a887021-e49f-45a8-a45e-de17a9070633-kube-api-access-lbzvr\") pod \"service-ca-7bb4f4bcc7-hvxnc\" (UID: \"3a887021-e49f-45a8-a45e-de17a9070633\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-hvxnc"
Jul 02 12:39:27.825929 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:27.825895    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-7bb4f4bcc7-hvxnc"
Jul 02 12:39:27.975800 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:27.975762    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4xx9d\" (UID: \"e3a44725-2f0a-44ea-9a50-4f13d28b0bd3\") " pod="openshift-insights/insights-runtime-extractor-4xx9d"
Jul 02 12:39:27.975987 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:27.975819    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4xx9d\" (UID: \"e3a44725-2f0a-44ea-9a50-4f13d28b0bd3\") " pod="openshift-insights/insights-runtime-extractor-4xx9d"
Jul 02 12:39:27.975987 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:27.975870    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/851415ab-7420-40c0-aa53-4e1d8899ee4c-metrics-tls\") pod \"dns-default-tg5m8\" (UID: \"851415ab-7420-40c0-aa53-4e1d8899ee4c\") " pod="openshift-dns/dns-default-tg5m8"
Jul 02 12:39:27.975987 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:27.975930    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 12:39:27.975987 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:27.975967    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 12:39:27.975987 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:27.975982    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 12:39:27.975987 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:27.975990    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-insights-runtime-extractor-tls podName:e3a44725-2f0a-44ea-9a50-4f13d28b0bd3 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:29.975975453 +0000 UTC m=+38.108801277 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4xx9d" (UID: "e3a44725-2f0a-44ea-9a50-4f13d28b0bd3") : secret "insights-runtime-extractor-tls" not found
Jul 02 12:39:27.976304 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:27.976029    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/851415ab-7420-40c0-aa53-4e1d8899ee4c-metrics-tls podName:851415ab-7420-40c0-aa53-4e1d8899ee4c nodeName:}" failed. No retries permitted until 2026-07-02 12:39:29.97601319 +0000 UTC m=+38.108839017 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/851415ab-7420-40c0-aa53-4e1d8899ee4c-metrics-tls") pod "dns-default-tg5m8" (UID: "851415ab-7420-40c0-aa53-4e1d8899ee4c") : secret "dns-default-metrics-tls" not found
Jul 02 12:39:27.976304 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:27.976045    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-kube-rbac-proxy-cm podName:e3a44725-2f0a-44ea-9a50-4f13d28b0bd3 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:29.976036698 +0000 UTC m=+38.108862525 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4xx9d" (UID: "e3a44725-2f0a-44ea-9a50-4f13d28b0bd3") : configmap "kube-rbac-proxy" not found
Jul 02 12:39:27.989543 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:27.989512    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-7bb4f4bcc7-hvxnc"]
Jul 02 12:39:27.993905 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:39:27.993873    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a887021_e49f_45a8_a45e_de17a9070633.slice/crio-4b65a0995825dac3293170b5038ab0b2e2703bfafa1d06b0553d1cfd00e4925a WatchSource:0}: Error finding container 4b65a0995825dac3293170b5038ab0b2e2703bfafa1d06b0553d1cfd00e4925a: Status 404 returned error can't find the container with id 4b65a0995825dac3293170b5038ab0b2e2703bfafa1d06b0553d1cfd00e4925a
Jul 02 12:39:28.571673 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:28.571637    2578 generic.go:356] "Generic (PLEG): container finished" podID="b1652685-279d-4a21-a047-736b5076b4e4" containerID="136858f6ea5b567717e51c2cb84562ff20c4cad05d872b21eaaca803627b5d35" exitCode=0
Jul 02 12:39:28.572286 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:28.571709    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mtbbk" event={"ID":"b1652685-279d-4a21-a047-736b5076b4e4","Type":"ContainerDied","Data":"136858f6ea5b567717e51c2cb84562ff20c4cad05d872b21eaaca803627b5d35"}
Jul 02 12:39:28.572998 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:28.572985    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-7bb4f4bcc7-hvxnc" event={"ID":"3a887021-e49f-45a8-a45e-de17a9070633","Type":"ContainerStarted","Data":"4b65a0995825dac3293170b5038ab0b2e2703bfafa1d06b0553d1cfd00e4925a"}
Jul 02 12:39:28.596522 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:28.596495    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:28.981283 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:28.981202    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:29.067056 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:29.067025    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:29.249454 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:29.249212    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:29.582852 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:29.582745    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mtbbk" event={"ID":"b1652685-279d-4a21-a047-736b5076b4e4","Type":"ContainerStarted","Data":"935f1569e956078115b14444f7044e2e99c39d7fb7a235b752c1fbccbcf76432"}
Jul 02 12:39:29.584489 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:29.584449    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-764zc" event={"ID":"9dcd5dc8-2736-4029-b386-bcecfe1bea3d","Type":"ContainerStarted","Data":"f12cd465c05b365d4beff38ff92c4dac4f3145a5b6abe37a751e1d3a6e990699"}
Jul 02 12:39:29.610402 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:29.610331    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mtbbk" podStartSLOduration=9.284340347 podStartE2EDuration="37.610309975s" totalImagesPullingTime="28.325969628s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:38:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:29.608524781 +0000 UTC m=+37.741350629" watchObservedRunningTime="2026-07-02 12:39:29.610309975 +0000 UTC m=+37.743135822"
Jul 02 12:39:29.628557 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:29.628503    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-764zc" podStartSLOduration=1.304891489 podStartE2EDuration="3.628487735s" totalImagesPullingTime="2.323596246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:39:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:29.627721454 +0000 UTC m=+37.760547321" watchObservedRunningTime="2026-07-02 12:39:29.628487735 +0000 UTC m=+37.761313580"
Jul 02 12:39:29.995277 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:29.995241    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/851415ab-7420-40c0-aa53-4e1d8899ee4c-metrics-tls\") pod \"dns-default-tg5m8\" (UID: \"851415ab-7420-40c0-aa53-4e1d8899ee4c\") " pod="openshift-dns/dns-default-tg5m8"
Jul 02 12:39:29.995458 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:29.995345    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4xx9d\" (UID: \"e3a44725-2f0a-44ea-9a50-4f13d28b0bd3\") " pod="openshift-insights/insights-runtime-extractor-4xx9d"
Jul 02 12:39:29.995458 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:29.995379    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4xx9d\" (UID: \"e3a44725-2f0a-44ea-9a50-4f13d28b0bd3\") " pod="openshift-insights/insights-runtime-extractor-4xx9d"
Jul 02 12:39:29.995458 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:29.995399    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 12:39:29.995609 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:29.995463    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/851415ab-7420-40c0-aa53-4e1d8899ee4c-metrics-tls podName:851415ab-7420-40c0-aa53-4e1d8899ee4c nodeName:}" failed. No retries permitted until 2026-07-02 12:39:33.995446279 +0000 UTC m=+42.128272107 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/851415ab-7420-40c0-aa53-4e1d8899ee4c-metrics-tls") pod "dns-default-tg5m8" (UID: "851415ab-7420-40c0-aa53-4e1d8899ee4c") : secret "dns-default-metrics-tls" not found
Jul 02 12:39:29.995609 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:29.995500    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 12:39:29.995609 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:29.995557    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-insights-runtime-extractor-tls podName:e3a44725-2f0a-44ea-9a50-4f13d28b0bd3 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:33.995542392 +0000 UTC m=+42.128368216 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4xx9d" (UID: "e3a44725-2f0a-44ea-9a50-4f13d28b0bd3") : secret "insights-runtime-extractor-tls" not found
Jul 02 12:39:29.995938 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:29.995918    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4xx9d\" (UID: \"e3a44725-2f0a-44ea-9a50-4f13d28b0bd3\") " pod="openshift-insights/insights-runtime-extractor-4xx9d"
Jul 02 12:39:31.591520 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:31.591422    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-7bb4f4bcc7-hvxnc" event={"ID":"3a887021-e49f-45a8-a45e-de17a9070633","Type":"ContainerStarted","Data":"52c491efda412311f74b8d116800eff635171386e06a53016b0cc30c0e9a999f"}
Jul 02 12:39:31.619827 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:31.619774    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-7bb4f4bcc7-hvxnc" podStartSLOduration=1.3061466 podStartE2EDuration="4.619758733s" totalImagesPullingTime="3.313612133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:39:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:31.61834289 +0000 UTC m=+39.751168739" watchObservedRunningTime="2026-07-02 12:39:31.619758733 +0000 UTC m=+39.752584578"
Jul 02 12:39:31.719068 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:31.719038    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:31.776853 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:31.776824    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:32.636660 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:32.636612    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:33.530670 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:33.530642    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:33.700938 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:33.700905    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:33.873734 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:33.873660    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:34.020311 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:34.020278    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/851415ab-7420-40c0-aa53-4e1d8899ee4c-metrics-tls\") pod \"dns-default-tg5m8\" (UID: \"851415ab-7420-40c0-aa53-4e1d8899ee4c\") " pod="openshift-dns/dns-default-tg5m8"
Jul 02 12:39:34.020511 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:34.020378    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4xx9d\" (UID: \"e3a44725-2f0a-44ea-9a50-4f13d28b0bd3\") " pod="openshift-insights/insights-runtime-extractor-4xx9d"
Jul 02 12:39:34.020511 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:34.020443    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 12:39:34.020511 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:34.020512    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/851415ab-7420-40c0-aa53-4e1d8899ee4c-metrics-tls podName:851415ab-7420-40c0-aa53-4e1d8899ee4c nodeName:}" failed. No retries permitted until 2026-07-02 12:39:42.020490722 +0000 UTC m=+50.153316561 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/851415ab-7420-40c0-aa53-4e1d8899ee4c-metrics-tls") pod "dns-default-tg5m8" (UID: "851415ab-7420-40c0-aa53-4e1d8899ee4c") : secret "dns-default-metrics-tls" not found
Jul 02 12:39:34.020673 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:34.020535    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 12:39:34.020673 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:39:34.020599    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-insights-runtime-extractor-tls podName:e3a44725-2f0a-44ea-9a50-4f13d28b0bd3 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:42.020582013 +0000 UTC m=+50.153407842 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4xx9d" (UID: "e3a44725-2f0a-44ea-9a50-4f13d28b0bd3") : secret "insights-runtime-extractor-tls" not found
Jul 02 12:39:34.053216 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:34.053168    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:34.149429 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:34.149351    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:34.246466 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:34.246421    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:34.367331 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:34.367295    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:34.826351 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:34.826325    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:35.230422 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:35.230340    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:36.023677 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:36.023648    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:36.221042 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:36.221005    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:36.615424 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:36.615394    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:36.820917 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:36.820887    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:37.214156 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:37.214126    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:37.316849 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:37.316818    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:37.421575 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:37.421537    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:37.825043 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:37.825008    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:37.922117 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:37.922081    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:38.453064 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:38.453032    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:38.633042 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:38.633015    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:38.828842 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:38.828809    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:39.202442 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:39.202357    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:39.785703 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:39.785672    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:40.012917 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:40.012888    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:40.220131 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:40.220050    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:40.597447 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:40.597414    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:40.731358 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:40.731318    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:40.802531 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:40.802497    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:41.277031 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:41.276989    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:41.379650 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:41.379562    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:41.440122 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:41.440089    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:41.796498 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:41.796468    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:42.081092 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:42.081001    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4xx9d\" (UID: \"e3a44725-2f0a-44ea-9a50-4f13d28b0bd3\") " pod="openshift-insights/insights-runtime-extractor-4xx9d"
Jul 02 12:39:42.081092 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:42.081066    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/851415ab-7420-40c0-aa53-4e1d8899ee4c-metrics-tls\") pod \"dns-default-tg5m8\" (UID: \"851415ab-7420-40c0-aa53-4e1d8899ee4c\") " pod="openshift-dns/dns-default-tg5m8"
Jul 02 12:39:42.084458 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:42.084428    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/851415ab-7420-40c0-aa53-4e1d8899ee4c-metrics-tls\") pod \"dns-default-tg5m8\" (UID: \"851415ab-7420-40c0-aa53-4e1d8899ee4c\") " pod="openshift-dns/dns-default-tg5m8"
Jul 02 12:39:42.084569 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:42.084465    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3a44725-2f0a-44ea-9a50-4f13d28b0bd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4xx9d\" (UID: \"e3a44725-2f0a-44ea-9a50-4f13d28b0bd3\") " pod="openshift-insights/insights-runtime-extractor-4xx9d"
Jul 02 12:39:42.145430 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:42.145390    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tg5m8"
Jul 02 12:39:42.160551 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:42.160522    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4xx9d"
Jul 02 12:39:42.206902 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:42.206877    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:42.322898 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:42.322868    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tg5m8"]
Jul 02 12:39:42.324718 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:39:42.324678    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod851415ab_7420_40c0_aa53_4e1d8899ee4c.slice/crio-e6808704567e35acebeff90cd216464d9e0d2a6c3a8f8ce4d58ced9607ef8863 WatchSource:0}: Error finding container e6808704567e35acebeff90cd216464d9e0d2a6c3a8f8ce4d58ced9607ef8863: Status 404 returned error can't find the container with id e6808704567e35acebeff90cd216464d9e0d2a6c3a8f8ce4d58ced9607ef8863
Jul 02 12:39:42.351260 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:42.351237    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4xx9d"]
Jul 02 12:39:42.353383 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:39:42.353355    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a44725_2f0a_44ea_9a50_4f13d28b0bd3.slice/crio-660161fd930c0ab51d17615b4e68f47811df1f0edb6b5aa60db5a16c90ca031a WatchSource:0}: Error finding container 660161fd930c0ab51d17615b4e68f47811df1f0edb6b5aa60db5a16c90ca031a: Status 404 returned error can't find the container with id 660161fd930c0ab51d17615b4e68f47811df1f0edb6b5aa60db5a16c90ca031a
Jul 02 12:39:42.397686 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:42.397659    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:42.620285 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:42.620185    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tg5m8" event={"ID":"851415ab-7420-40c0-aa53-4e1d8899ee4c","Type":"ContainerStarted","Data":"e6808704567e35acebeff90cd216464d9e0d2a6c3a8f8ce4d58ced9607ef8863"}
Jul 02 12:39:42.621324 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:42.621300    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4xx9d" event={"ID":"e3a44725-2f0a-44ea-9a50-4f13d28b0bd3","Type":"ContainerStarted","Data":"660161fd930c0ab51d17615b4e68f47811df1f0edb6b5aa60db5a16c90ca031a"}
Jul 02 12:39:42.797779 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:42.797745    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:43.601380 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:43.601338    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:43.887243 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:43.887156    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:44.009412 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:44.009372    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:44.226528 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:44.226449    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:46.088307 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:46.088269    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:46.185427 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:46.185347    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:47.637842 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:47.637802    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4xx9d" event={"ID":"e3a44725-2f0a-44ea-9a50-4f13d28b0bd3","Type":"ContainerStarted","Data":"98d5667c14e5e63cbee3f6839ef2c0b1948e7917371df3f0dd6f64f23cf33046"}
Jul 02 12:39:47.638291 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:47.637866    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4xx9d" event={"ID":"e3a44725-2f0a-44ea-9a50-4f13d28b0bd3","Type":"ContainerStarted","Data":"939f6cd2885bb7b88830d79ff0378632ef88c0839f5979f233582374cb88d1b0"}
Jul 02 12:39:47.639452 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:47.639429    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tg5m8" event={"ID":"851415ab-7420-40c0-aa53-4e1d8899ee4c","Type":"ContainerStarted","Data":"00ffc5c0f5a34db4b31367a9203e03fc9ca8d23b3a2b1024029e65897966be4f"}
Jul 02 12:39:47.639564 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:47.639470    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tg5m8" event={"ID":"851415ab-7420-40c0-aa53-4e1d8899ee4c","Type":"ContainerStarted","Data":"92701aeae1a580178151cb28855f2b021d1c88514b3b83a0ef789dcb51d13022"}
Jul 02 12:39:47.639564 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:47.639508    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tg5m8"
Jul 02 12:39:47.678603 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:47.678547    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tg5m8" podStartSLOduration=17.232994796 podStartE2EDuration="21.678532809s" totalImagesPullingTime="4.445538013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:39:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:47.677919349 +0000 UTC m=+55.810745206" watchObservedRunningTime="2026-07-02 12:39:47.678532809 +0000 UTC m=+55.811358671"
Jul 02 12:39:49.880691 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:49.880647    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6445d48586-vz2pq"]
Jul 02 12:39:49.892061 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:49.892027    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57564f8dc4-ckb4b"]
Jul 02 12:39:49.892276 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:49.892163    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6445d48586-vz2pq"
Jul 02 12:39:49.895214 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:49.895160    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57564f8dc4-ckb4b"
Jul 02 12:39:49.895964 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:49.895938    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 02 12:39:49.896080 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:49.895938    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 02 12:39:49.896080 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:49.896071    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 02 12:39:49.897706 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:49.897682    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 02 12:39:49.897814 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:49.897693    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 02 12:39:49.897814 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:49.897795    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 02 12:39:49.897939 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:49.897867    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 02 12:39:49.899217 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:49.899026    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-bzl9d\""
Jul 02 12:39:49.899217 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:49.899114    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 02 12:39:49.921788 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:49.921765    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57564f8dc4-ckb4b"]
Jul 02 12:39:49.926829 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:49.926800    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6445d48586-vz2pq"]
Jul 02 12:39:49.945067 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:49.945033    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/407ab2ed-b33f-4d2f-8e0e-72518d982621-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6445d48586-vz2pq\" (UID: \"407ab2ed-b33f-4d2f-8e0e-72518d982621\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6445d48586-vz2pq"
Jul 02 12:39:49.945248 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:49.945082    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg48m\" (UniqueName: \"kubernetes.io/projected/407ab2ed-b33f-4d2f-8e0e-72518d982621-kube-api-access-xg48m\") pod \"cluster-proxy-proxy-agent-6445d48586-vz2pq\" (UID: \"407ab2ed-b33f-4d2f-8e0e-72518d982621\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6445d48586-vz2pq"
Jul 02 12:39:49.945248 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:49.945154    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d05f7536-119d-4913-b014-c4b2b1d8bdc4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-57564f8dc4-ckb4b\" (UID: \"d05f7536-119d-4913-b014-c4b2b1d8bdc4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57564f8dc4-ckb4b"
Jul 02 12:39:49.945248 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:49.945225    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/407ab2ed-b33f-4d2f-8e0e-72518d982621-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6445d48586-vz2pq\" (UID: \"407ab2ed-b33f-4d2f-8e0e-72518d982621\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6445d48586-vz2pq"
Jul 02 12:39:49.945417 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:49.945281    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n66lp\" (UniqueName: \"kubernetes.io/projected/d05f7536-119d-4913-b014-c4b2b1d8bdc4-kube-api-access-n66lp\") pod \"managed-serviceaccount-addon-agent-57564f8dc4-ckb4b\" (UID: \"d05f7536-119d-4913-b014-c4b2b1d8bdc4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57564f8dc4-ckb4b"
Jul 02 12:39:49.945417 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:49.945305    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/407ab2ed-b33f-4d2f-8e0e-72518d982621-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6445d48586-vz2pq\" (UID: \"407ab2ed-b33f-4d2f-8e0e-72518d982621\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6445d48586-vz2pq"
Jul 02 12:39:49.945417 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:49.945374    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/407ab2ed-b33f-4d2f-8e0e-72518d982621-ca\") pod \"cluster-proxy-proxy-agent-6445d48586-vz2pq\" (UID: \"407ab2ed-b33f-4d2f-8e0e-72518d982621\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6445d48586-vz2pq"
Jul 02 12:39:49.945640 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:49.945416    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/407ab2ed-b33f-4d2f-8e0e-72518d982621-hub\") pod \"cluster-proxy-proxy-agent-6445d48586-vz2pq\" (UID: \"407ab2ed-b33f-4d2f-8e0e-72518d982621\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6445d48586-vz2pq"
Jul 02 12:39:50.046399 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:50.046365    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/407ab2ed-b33f-4d2f-8e0e-72518d982621-ca\") pod \"cluster-proxy-proxy-agent-6445d48586-vz2pq\" (UID: \"407ab2ed-b33f-4d2f-8e0e-72518d982621\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6445d48586-vz2pq"
Jul 02 12:39:50.046399 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:50.046400    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/407ab2ed-b33f-4d2f-8e0e-72518d982621-hub\") pod \"cluster-proxy-proxy-agent-6445d48586-vz2pq\" (UID: \"407ab2ed-b33f-4d2f-8e0e-72518d982621\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6445d48586-vz2pq"
Jul 02 12:39:50.046718 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:50.046448    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/407ab2ed-b33f-4d2f-8e0e-72518d982621-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6445d48586-vz2pq\" (UID: \"407ab2ed-b33f-4d2f-8e0e-72518d982621\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6445d48586-vz2pq"
Jul 02 12:39:50.046718 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:50.046619    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg48m\" (UniqueName: \"kubernetes.io/projected/407ab2ed-b33f-4d2f-8e0e-72518d982621-kube-api-access-xg48m\") pod \"cluster-proxy-proxy-agent-6445d48586-vz2pq\" (UID: \"407ab2ed-b33f-4d2f-8e0e-72518d982621\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6445d48586-vz2pq"
Jul 02 12:39:50.046718 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:50.046665    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d05f7536-119d-4913-b014-c4b2b1d8bdc4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-57564f8dc4-ckb4b\" (UID: \"d05f7536-119d-4913-b014-c4b2b1d8bdc4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57564f8dc4-ckb4b"
Jul 02 12:39:50.046718 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:50.046696    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/407ab2ed-b33f-4d2f-8e0e-72518d982621-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6445d48586-vz2pq\" (UID: \"407ab2ed-b33f-4d2f-8e0e-72518d982621\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6445d48586-vz2pq"
Jul 02 12:39:50.046926 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:50.046754    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n66lp\" (UniqueName: \"kubernetes.io/projected/d05f7536-119d-4913-b014-c4b2b1d8bdc4-kube-api-access-n66lp\") pod \"managed-serviceaccount-addon-agent-57564f8dc4-ckb4b\" (UID: \"d05f7536-119d-4913-b014-c4b2b1d8bdc4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57564f8dc4-ckb4b"
Jul 02 12:39:50.046926 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:50.046786    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/407ab2ed-b33f-4d2f-8e0e-72518d982621-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6445d48586-vz2pq\" (UID: \"407ab2ed-b33f-4d2f-8e0e-72518d982621\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6445d48586-vz2pq"
Jul 02 12:39:50.047509 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:50.047475    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/407ab2ed-b33f-4d2f-8e0e-72518d982621-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6445d48586-vz2pq\" (UID: \"407ab2ed-b33f-4d2f-8e0e-72518d982621\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6445d48586-vz2pq"
Jul 02 12:39:50.049418 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:50.049395    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/407ab2ed-b33f-4d2f-8e0e-72518d982621-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6445d48586-vz2pq\" (UID: \"407ab2ed-b33f-4d2f-8e0e-72518d982621\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6445d48586-vz2pq"
Jul 02 12:39:50.049523 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:50.049429    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/407ab2ed-b33f-4d2f-8e0e-72518d982621-ca\") pod \"cluster-proxy-proxy-agent-6445d48586-vz2pq\" (UID: \"407ab2ed-b33f-4d2f-8e0e-72518d982621\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6445d48586-vz2pq"
Jul 02 12:39:50.049523 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:50.049477    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d05f7536-119d-4913-b014-c4b2b1d8bdc4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-57564f8dc4-ckb4b\" (UID: \"d05f7536-119d-4913-b014-c4b2b1d8bdc4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57564f8dc4-ckb4b"
Jul 02 12:39:50.049523 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:50.049492    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/407ab2ed-b33f-4d2f-8e0e-72518d982621-hub\") pod \"cluster-proxy-proxy-agent-6445d48586-vz2pq\" (UID: \"407ab2ed-b33f-4d2f-8e0e-72518d982621\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6445d48586-vz2pq"
Jul 02 12:39:50.049737 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:50.049717    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/407ab2ed-b33f-4d2f-8e0e-72518d982621-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6445d48586-vz2pq\" (UID: \"407ab2ed-b33f-4d2f-8e0e-72518d982621\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6445d48586-vz2pq"
Jul 02 12:39:50.057592 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:50.057571    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg48m\" (UniqueName: \"kubernetes.io/projected/407ab2ed-b33f-4d2f-8e0e-72518d982621-kube-api-access-xg48m\") pod \"cluster-proxy-proxy-agent-6445d48586-vz2pq\" (UID: \"407ab2ed-b33f-4d2f-8e0e-72518d982621\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6445d48586-vz2pq"
Jul 02 12:39:50.061324 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:50.061302    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n66lp\" (UniqueName: \"kubernetes.io/projected/d05f7536-119d-4913-b014-c4b2b1d8bdc4-kube-api-access-n66lp\") pod \"managed-serviceaccount-addon-agent-57564f8dc4-ckb4b\" (UID: \"d05f7536-119d-4913-b014-c4b2b1d8bdc4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57564f8dc4-ckb4b"
Jul 02 12:39:50.102466 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:50.102446    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:50.200487 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:50.200414    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:50.210357 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:50.210329    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6445d48586-vz2pq"
Jul 02 12:39:50.216334 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:50.216314    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57564f8dc4-ckb4b"
Jul 02 12:39:50.303559 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:50.303456    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:50.347703 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:50.347640    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6445d48586-vz2pq"]
Jul 02 12:39:50.350048 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:39:50.350021    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod407ab2ed_b33f_4d2f_8e0e_72518d982621.slice/crio-d6564c9bb42589ab080676c8a6ede5b58f5ed095037115422a072dd9c8afff2d WatchSource:0}: Error finding container d6564c9bb42589ab080676c8a6ede5b58f5ed095037115422a072dd9c8afff2d: Status 404 returned error can't find the container with id d6564c9bb42589ab080676c8a6ede5b58f5ed095037115422a072dd9c8afff2d
Jul 02 12:39:50.360704 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:50.360685    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57564f8dc4-ckb4b"]
Jul 02 12:39:50.362900 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:39:50.362873    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd05f7536_119d_4913_b014_c4b2b1d8bdc4.slice/crio-72d334d97a4d3e9883c87b0e82eaa1813c27bdc8e13323e9a67f81c1c5294467 WatchSource:0}: Error finding container 72d334d97a4d3e9883c87b0e82eaa1813c27bdc8e13323e9a67f81c1c5294467: Status 404 returned error can't find the container with id 72d334d97a4d3e9883c87b0e82eaa1813c27bdc8e13323e9a67f81c1c5294467
Jul 02 12:39:50.369797 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:50.369777    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:50.653763 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:50.653724    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6445d48586-vz2pq" event={"ID":"407ab2ed-b33f-4d2f-8e0e-72518d982621","Type":"ContainerStarted","Data":"d6564c9bb42589ab080676c8a6ede5b58f5ed095037115422a072dd9c8afff2d"}
Jul 02 12:39:50.655655 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:50.655632    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4xx9d" event={"ID":"e3a44725-2f0a-44ea-9a50-4f13d28b0bd3","Type":"ContainerStarted","Data":"db55cb51d75bb20408868b48e949f41dc2d6a01cbd2652864e81a590fdc3071c"}
Jul 02 12:39:50.656695 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:50.656670    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57564f8dc4-ckb4b" event={"ID":"d05f7536-119d-4913-b014-c4b2b1d8bdc4","Type":"ContainerStarted","Data":"72d334d97a4d3e9883c87b0e82eaa1813c27bdc8e13323e9a67f81c1c5294467"}
Jul 02 12:39:50.680058 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:50.680017    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4xx9d" podStartSLOduration=17.070974067 podStartE2EDuration="24.680004199s" totalImagesPullingTime="7.609030132s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:39:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:50.678481361 +0000 UTC m=+58.811307207" watchObservedRunningTime="2026-07-02 12:39:50.680004199 +0000 UTC m=+58.812830045"
Jul 02 12:39:51.098934 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:51.098900    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:51.156804 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:51.156763    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:51.302393 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:51.302361    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:51.404428 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:51.404344    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:51.861723 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:51.861689    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:51.940808 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:51.940774    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:52.031772 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:52.028457    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:52.260634 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:52.260538    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:52.353123 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:52.353084    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:52.775219 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:52.775162    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:52.860790 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:52.860755    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:52.970641 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:52.970599    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:53.166950 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:53.166870    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:53.221913 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:53.221878    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:53.521671 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:53.521644    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:53.584026 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:53.583993    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:53.907762 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:53.907683    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:53.967140 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:53.967091    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:54.307483 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:54.307452    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-95kwm"
Jul 02 12:39:54.503677 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:54.503637    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:54.599379 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:54.599295    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:54.923671 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:54.923585    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:55.014877 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:55.014844    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:55.678642 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:55.678605    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6445d48586-vz2pq" event={"ID":"407ab2ed-b33f-4d2f-8e0e-72518d982621","Type":"ContainerStarted","Data":"ed1c774eb90aeb55c332920eb07eb824b813b8f1d5c00ac6d283bfbcea163850"}
Jul 02 12:39:55.679078 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:55.678657    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6445d48586-vz2pq" event={"ID":"407ab2ed-b33f-4d2f-8e0e-72518d982621","Type":"ContainerStarted","Data":"4bb4254be3d5c1e42f963de35799279a2bdc45419818a8e5ea8ecb30fea5fb5b"}
Jul 02 12:39:55.679078 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:55.678678    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6445d48586-vz2pq" event={"ID":"407ab2ed-b33f-4d2f-8e0e-72518d982621","Type":"ContainerStarted","Data":"2bdc0748f1d41beebc5801e58efb2dc463c15cc9df2bdd1605801487d32a9922"}
Jul 02 12:39:55.680073 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:55.680049    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57564f8dc4-ckb4b" event={"ID":"d05f7536-119d-4913-b014-c4b2b1d8bdc4","Type":"ContainerStarted","Data":"a1e98a15a4c30b7f7c927908a4727acbefbb0c8a5ad280e711dba51501aa4ad7"}
Jul 02 12:39:55.690627 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:55.690609    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:55.716243 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:55.716171    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6445d48586-vz2pq" podStartSLOduration=2.41616743 podStartE2EDuration="6.71615959s" totalImagesPullingTime="4.29999216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:39:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:55.715800014 +0000 UTC m=+63.848625861" watchObservedRunningTime="2026-07-02 12:39:55.71615959 +0000 UTC m=+63.848985414"
Jul 02 12:39:55.740784 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:55.740744    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57564f8dc4-ckb4b" podStartSLOduration=2.46362052 podStartE2EDuration="6.740730916s" totalImagesPullingTime="4.277110396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:39:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:55.740239512 +0000 UTC m=+63.873065357" watchObservedRunningTime="2026-07-02 12:39:55.740730916 +0000 UTC m=+63.873556761"
Jul 02 12:39:55.788911 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:55.788873    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:57.085476 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:57.085444    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:57.147368 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:57.147339    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:57.359537 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:57.359459    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:57.648327 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:57.648194    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tg5m8"
Jul 02 12:39:58.215304 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:58.215264    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2718fba6-5e1c-4c48-a90c-fc26b0ccc385-original-pull-secret\") pod \"global-pull-secret-syncer-tsj7t\" (UID: \"2718fba6-5e1c-4c48-a90c-fc26b0ccc385\") " pod="kube-system/global-pull-secret-syncer-tsj7t"
Jul 02 12:39:58.215771 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:58.215317    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2432aa2-93ec-46ce-afcd-e19f9dd03f8e-metrics-certs\") pod \"network-metrics-daemon-xsf2s\" (UID: \"e2432aa2-93ec-46ce-afcd-e19f9dd03f8e\") " pod="openshift-multus/network-metrics-daemon-xsf2s"
Jul 02 12:39:58.218657 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:58.218640    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 12:39:58.220006 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:58.219993    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 12:39:58.227734 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:58.227713    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2718fba6-5e1c-4c48-a90c-fc26b0ccc385-original-pull-secret\") pod \"global-pull-secret-syncer-tsj7t\" (UID: \"2718fba6-5e1c-4c48-a90c-fc26b0ccc385\") " pod="kube-system/global-pull-secret-syncer-tsj7t"
Jul 02 12:39:58.227849 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:58.227832    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2432aa2-93ec-46ce-afcd-e19f9dd03f8e-metrics-certs\") pod \"network-metrics-daemon-xsf2s\" (UID: \"e2432aa2-93ec-46ce-afcd-e19f9dd03f8e\") " pod="openshift-multus/network-metrics-daemon-xsf2s"
Jul 02 12:39:58.256752 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:58.256720    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:58.312191 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:58.312143    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:39:58.316251 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:58.316228    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpvs6\" (UniqueName: \"kubernetes.io/projected/cd215e6b-c112-412f-a031-30940e656f0d-kube-api-access-mpvs6\") pod \"network-check-target-4cw8v\" (UID: \"cd215e6b-c112-412f-a031-30940e656f0d\") " pod="openshift-network-diagnostics/network-check-target-4cw8v"
Jul 02 12:39:58.319303 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:58.319288    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 12:39:58.329563 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:58.329542    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 12:39:58.339419 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:58.339391    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpvs6\" (UniqueName: \"kubernetes.io/projected/cd215e6b-c112-412f-a031-30940e656f0d-kube-api-access-mpvs6\") pod \"network-check-target-4cw8v\" (UID: \"cd215e6b-c112-412f-a031-30940e656f0d\") " pod="openshift-network-diagnostics/network-check-target-4cw8v"
Jul 02 12:39:58.472582 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:58.472497    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tsj7t"
Jul 02 12:39:58.482428 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:58.482405    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bwrzl\""
Jul 02 12:39:58.488600 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:58.488581    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mgbjz\""
Jul 02 12:39:58.490655 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:58.490640    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsf2s"
Jul 02 12:39:58.496120 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:58.496107    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cw8v"
Jul 02 12:39:58.648111 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:58.648076    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tsj7t"]
Jul 02 12:39:58.651952 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:39:58.651910    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2718fba6_5e1c_4c48_a90c_fc26b0ccc385.slice/crio-a609a6c2619129b4abc14db23e2fc98208902a2623fd9bf2e7926813e19506d5 WatchSource:0}: Error finding container a609a6c2619129b4abc14db23e2fc98208902a2623fd9bf2e7926813e19506d5: Status 404 returned error can't find the container with id a609a6c2619129b4abc14db23e2fc98208902a2623fd9bf2e7926813e19506d5
Jul 02 12:39:58.681043 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:58.681011    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xsf2s"]
Jul 02 12:39:58.683786 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:39:58.683763    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2432aa2_93ec_46ce_afcd_e19f9dd03f8e.slice/crio-b4b3bac138a7d5e9fb00428f43da1bd618fb1fdc6566fb9b0ea771acd242ed55 WatchSource:0}: Error finding container b4b3bac138a7d5e9fb00428f43da1bd618fb1fdc6566fb9b0ea771acd242ed55: Status 404 returned error can't find the container with id b4b3bac138a7d5e9fb00428f43da1bd618fb1fdc6566fb9b0ea771acd242ed55
Jul 02 12:39:58.690916 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:58.690881    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xsf2s" event={"ID":"e2432aa2-93ec-46ce-afcd-e19f9dd03f8e","Type":"ContainerStarted","Data":"b4b3bac138a7d5e9fb00428f43da1bd618fb1fdc6566fb9b0ea771acd242ed55"}
Jul 02 12:39:58.691901 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:58.691879    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tsj7t" event={"ID":"2718fba6-5e1c-4c48-a90c-fc26b0ccc385","Type":"ContainerStarted","Data":"a609a6c2619129b4abc14db23e2fc98208902a2623fd9bf2e7926813e19506d5"}
Jul 02 12:39:58.700613 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:58.700590    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4cw8v"]
Jul 02 12:39:58.703625 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:39:58.703603    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd215e6b_c112_412f_a031_30940e656f0d.slice/crio-e53f02b69a4af5f2e2ee0c79eb510dc6c66fed98071c221e934a53c694a3ef7c WatchSource:0}: Error finding container e53f02b69a4af5f2e2ee0c79eb510dc6c66fed98071c221e934a53c694a3ef7c: Status 404 returned error can't find the container with id e53f02b69a4af5f2e2ee0c79eb510dc6c66fed98071c221e934a53c694a3ef7c
Jul 02 12:39:59.697859 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:59.697809    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4cw8v" event={"ID":"cd215e6b-c112-412f-a031-30940e656f0d","Type":"ContainerStarted","Data":"e53f02b69a4af5f2e2ee0c79eb510dc6c66fed98071c221e934a53c694a3ef7c"}
Jul 02 12:39:59.979915 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:39:59.979832    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:00.071530 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:00.071483    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:00.704473 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:00.704375    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xsf2s" event={"ID":"e2432aa2-93ec-46ce-afcd-e19f9dd03f8e","Type":"ContainerStarted","Data":"04b12925e1326c2e609ce3f2d934dd18246294fdf5055e12133ccdb55a487ea0"}
Jul 02 12:40:00.704473 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:00.704444    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xsf2s" event={"ID":"e2432aa2-93ec-46ce-afcd-e19f9dd03f8e","Type":"ContainerStarted","Data":"076581b2f23e951de9b611bdb93052ba2860c2f49843c6ebfd8d4da04c4d34cd"}
Jul 02 12:40:00.722146 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:00.722097    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xsf2s" podStartSLOduration=67.205620558 podStartE2EDuration="1m8.722080066s" totalImagesPullingTime="1.516459508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:38:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:40:00.720835255 +0000 UTC m=+68.853661125" watchObservedRunningTime="2026-07-02 12:40:00.722080066 +0000 UTC m=+68.854905911"
Jul 02 12:40:00.826952 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:00.826915    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:00.882798 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:00.882760    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:00.935457 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:00.935420    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:01.586921 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:01.586887    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:03.718996 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:03.718882    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tsj7t" event={"ID":"2718fba6-5e1c-4c48-a90c-fc26b0ccc385","Type":"ContainerStarted","Data":"cc6f0015e54a8dbfd8754f472d97ffd01ff11a31871f94e5fd51c14ab3e97fd8"}
Jul 02 12:40:03.720339 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:03.720316    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4cw8v" event={"ID":"cd215e6b-c112-412f-a031-30940e656f0d","Type":"ContainerStarted","Data":"a0be9233ddaa0605ea5fb559e38ec04ff3348d0f83468cbd5723697afd4132a0"}
Jul 02 12:40:03.720454 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:03.720387    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4cw8v"
Jul 02 12:40:03.736559 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:03.736508    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-tsj7t" podStartSLOduration=67.069266506 podStartE2EDuration="1m11.736493726s" totalImagesPullingTime="4.66722722s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:38:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:40:03.735816693 +0000 UTC m=+71.868642562" watchObservedRunningTime="2026-07-02 12:40:03.736493726 +0000 UTC m=+71.869319572"
Jul 02 12:40:03.753012 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:03.752968    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4cw8v" podStartSLOduration=67.138927894 podStartE2EDuration="1m11.752955628s" totalImagesPullingTime="4.614027734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:38:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:40:03.752253054 +0000 UTC m=+71.885078931" watchObservedRunningTime="2026-07-02 12:40:03.752955628 +0000 UTC m=+71.885781474"
Jul 02 12:40:04.235311 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.235275    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:04.337024 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.336996    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:04.442640 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.442602    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:04.509030 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.508990    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-p7k4z"]
Jul 02 12:40:04.511737 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.511719    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-p7k4z"
Jul 02 12:40:04.516413 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.516206    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-qswj9\""
Jul 02 12:40:04.516413 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.516229    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 02 12:40:04.516413 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.516236    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 02 12:40:04.516413 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.516297    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 02 12:40:04.516413 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.516231    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 02 12:40:04.517453 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.517430    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 02 12:40:04.527625 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.527594    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-p7k4z"]
Jul 02 12:40:04.561707 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.561674    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9dsgh"]
Jul 02 12:40:04.564374 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.564357    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9dsgh"
Jul 02 12:40:04.569639 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.569611    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d9e744f-62d0-46e6-a186-3b8d8e5efb94-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-p7k4z\" (UID: \"1d9e744f-62d0-46e6-a186-3b8d8e5efb94\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-p7k4z"
Jul 02 12:40:04.569777 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.569652    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d9e744f-62d0-46e6-a186-3b8d8e5efb94-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-p7k4z\" (UID: \"1d9e744f-62d0-46e6-a186-3b8d8e5efb94\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-p7k4z"
Jul 02 12:40:04.569777 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.569685    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dslp5\" (UniqueName: \"kubernetes.io/projected/1d9e744f-62d0-46e6-a186-3b8d8e5efb94-kube-api-access-dslp5\") pod \"openshift-state-metrics-57f99c8b49-p7k4z\" (UID: \"1d9e744f-62d0-46e6-a186-3b8d8e5efb94\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-p7k4z"
Jul 02 12:40:04.569944 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.569925    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d9e744f-62d0-46e6-a186-3b8d8e5efb94-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-p7k4z\" (UID: \"1d9e744f-62d0-46e6-a186-3b8d8e5efb94\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-p7k4z"
Jul 02 12:40:04.576900 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.576879    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 02 12:40:04.577003 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.576903    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 02 12:40:04.577206 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.577192    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 02 12:40:04.577206 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.577203    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jz7sf\""
Jul 02 12:40:04.670350 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.670314    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b45d14c-b4b5-4c87-bc99-ed18f33668ef-root\") pod \"node-exporter-9dsgh\" (UID: \"1b45d14c-b4b5-4c87-bc99-ed18f33668ef\") " pod="openshift-monitoring/node-exporter-9dsgh"
Jul 02 12:40:04.670503 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.670359    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9crt4\" (UniqueName: \"kubernetes.io/projected/1b45d14c-b4b5-4c87-bc99-ed18f33668ef-kube-api-access-9crt4\") pod \"node-exporter-9dsgh\" (UID: \"1b45d14c-b4b5-4c87-bc99-ed18f33668ef\") " pod="openshift-monitoring/node-exporter-9dsgh"
Jul 02 12:40:04.670503 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.670448    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d9e744f-62d0-46e6-a186-3b8d8e5efb94-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-p7k4z\" (UID: \"1d9e744f-62d0-46e6-a186-3b8d8e5efb94\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-p7k4z"
Jul 02 12:40:04.670503 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.670485    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dslp5\" (UniqueName: \"kubernetes.io/projected/1d9e744f-62d0-46e6-a186-3b8d8e5efb94-kube-api-access-dslp5\") pod \"openshift-state-metrics-57f99c8b49-p7k4z\" (UID: \"1d9e744f-62d0-46e6-a186-3b8d8e5efb94\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-p7k4z"
Jul 02 12:40:04.670630 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.670516    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b45d14c-b4b5-4c87-bc99-ed18f33668ef-node-exporter-tls\") pod \"node-exporter-9dsgh\" (UID: \"1b45d14c-b4b5-4c87-bc99-ed18f33668ef\") " pod="openshift-monitoring/node-exporter-9dsgh"
Jul 02 12:40:04.670630 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.670563    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b45d14c-b4b5-4c87-bc99-ed18f33668ef-sys\") pod \"node-exporter-9dsgh\" (UID: \"1b45d14c-b4b5-4c87-bc99-ed18f33668ef\") " pod="openshift-monitoring/node-exporter-9dsgh"
Jul 02 12:40:04.670630 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.670592    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b45d14c-b4b5-4c87-bc99-ed18f33668ef-node-exporter-wtmp\") pod \"node-exporter-9dsgh\" (UID: \"1b45d14c-b4b5-4c87-bc99-ed18f33668ef\") " pod="openshift-monitoring/node-exporter-9dsgh"
Jul 02 12:40:04.670781 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.670627    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b45d14c-b4b5-4c87-bc99-ed18f33668ef-node-exporter-textfile\") pod \"node-exporter-9dsgh\" (UID: \"1b45d14c-b4b5-4c87-bc99-ed18f33668ef\") " pod="openshift-monitoring/node-exporter-9dsgh"
Jul 02 12:40:04.670781 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.670688    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d9e744f-62d0-46e6-a186-3b8d8e5efb94-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-p7k4z\" (UID: \"1d9e744f-62d0-46e6-a186-3b8d8e5efb94\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-p7k4z"
Jul 02 12:40:04.670781 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.670731    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b45d14c-b4b5-4c87-bc99-ed18f33668ef-node-exporter-accelerators-collector-config\") pod \"node-exporter-9dsgh\" (UID: \"1b45d14c-b4b5-4c87-bc99-ed18f33668ef\") " pod="openshift-monitoring/node-exporter-9dsgh"
Jul 02 12:40:04.670781 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.670768    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b45d14c-b4b5-4c87-bc99-ed18f33668ef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9dsgh\" (UID: \"1b45d14c-b4b5-4c87-bc99-ed18f33668ef\") " pod="openshift-monitoring/node-exporter-9dsgh"
Jul 02 12:40:04.670958 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.670802    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d9e744f-62d0-46e6-a186-3b8d8e5efb94-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-p7k4z\" (UID: \"1d9e744f-62d0-46e6-a186-3b8d8e5efb94\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-p7k4z"
Jul 02 12:40:04.670958 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.670846    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b45d14c-b4b5-4c87-bc99-ed18f33668ef-metrics-client-ca\") pod \"node-exporter-9dsgh\" (UID: \"1b45d14c-b4b5-4c87-bc99-ed18f33668ef\") " pod="openshift-monitoring/node-exporter-9dsgh"
Jul 02 12:40:04.671498 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.671467    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d9e744f-62d0-46e6-a186-3b8d8e5efb94-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-p7k4z\" (UID: \"1d9e744f-62d0-46e6-a186-3b8d8e5efb94\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-p7k4z"
Jul 02 12:40:04.673017 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.672987    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d9e744f-62d0-46e6-a186-3b8d8e5efb94-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-p7k4z\" (UID: \"1d9e744f-62d0-46e6-a186-3b8d8e5efb94\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-p7k4z"
Jul 02 12:40:04.673250 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.673229    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d9e744f-62d0-46e6-a186-3b8d8e5efb94-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-p7k4z\" (UID: \"1d9e744f-62d0-46e6-a186-3b8d8e5efb94\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-p7k4z"
Jul 02 12:40:04.682045 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.682022    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dslp5\" (UniqueName: \"kubernetes.io/projected/1d9e744f-62d0-46e6-a186-3b8d8e5efb94-kube-api-access-dslp5\") pod \"openshift-state-metrics-57f99c8b49-p7k4z\" (UID: \"1d9e744f-62d0-46e6-a186-3b8d8e5efb94\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-p7k4z"
Jul 02 12:40:04.713006 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.712983    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:04.771763 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.771682    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b45d14c-b4b5-4c87-bc99-ed18f33668ef-root\") pod \"node-exporter-9dsgh\" (UID: \"1b45d14c-b4b5-4c87-bc99-ed18f33668ef\") " pod="openshift-monitoring/node-exporter-9dsgh"
Jul 02 12:40:04.771763 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.771727    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9crt4\" (UniqueName: \"kubernetes.io/projected/1b45d14c-b4b5-4c87-bc99-ed18f33668ef-kube-api-access-9crt4\") pod \"node-exporter-9dsgh\" (UID: \"1b45d14c-b4b5-4c87-bc99-ed18f33668ef\") " pod="openshift-monitoring/node-exporter-9dsgh"
Jul 02 12:40:04.771763 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.771761    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b45d14c-b4b5-4c87-bc99-ed18f33668ef-node-exporter-tls\") pod \"node-exporter-9dsgh\" (UID: \"1b45d14c-b4b5-4c87-bc99-ed18f33668ef\") " pod="openshift-monitoring/node-exporter-9dsgh"
Jul 02 12:40:04.772269 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.771791    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b45d14c-b4b5-4c87-bc99-ed18f33668ef-root\") pod \"node-exporter-9dsgh\" (UID: \"1b45d14c-b4b5-4c87-bc99-ed18f33668ef\") " pod="openshift-monitoring/node-exporter-9dsgh"
Jul 02 12:40:04.772269 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.771803    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b45d14c-b4b5-4c87-bc99-ed18f33668ef-sys\") pod \"node-exporter-9dsgh\" (UID: \"1b45d14c-b4b5-4c87-bc99-ed18f33668ef\") " pod="openshift-monitoring/node-exporter-9dsgh"
Jul 02 12:40:04.772269 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.771845    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b45d14c-b4b5-4c87-bc99-ed18f33668ef-sys\") pod \"node-exporter-9dsgh\" (UID: \"1b45d14c-b4b5-4c87-bc99-ed18f33668ef\") " pod="openshift-monitoring/node-exporter-9dsgh"
Jul 02 12:40:04.772269 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.771857    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b45d14c-b4b5-4c87-bc99-ed18f33668ef-node-exporter-wtmp\") pod \"node-exporter-9dsgh\" (UID: \"1b45d14c-b4b5-4c87-bc99-ed18f33668ef\") " pod="openshift-monitoring/node-exporter-9dsgh"
Jul 02 12:40:04.772269 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.771881    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b45d14c-b4b5-4c87-bc99-ed18f33668ef-node-exporter-textfile\") pod \"node-exporter-9dsgh\" (UID: \"1b45d14c-b4b5-4c87-bc99-ed18f33668ef\") " pod="openshift-monitoring/node-exporter-9dsgh"
Jul 02 12:40:04.772269 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.771909    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b45d14c-b4b5-4c87-bc99-ed18f33668ef-node-exporter-accelerators-collector-config\") pod \"node-exporter-9dsgh\" (UID: \"1b45d14c-b4b5-4c87-bc99-ed18f33668ef\") " pod="openshift-monitoring/node-exporter-9dsgh"
Jul 02 12:40:04.772269 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.771932    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b45d14c-b4b5-4c87-bc99-ed18f33668ef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9dsgh\" (UID: \"1b45d14c-b4b5-4c87-bc99-ed18f33668ef\") " pod="openshift-monitoring/node-exporter-9dsgh"
Jul 02 12:40:04.772269 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.771979    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b45d14c-b4b5-4c87-bc99-ed18f33668ef-metrics-client-ca\") pod \"node-exporter-9dsgh\" (UID: \"1b45d14c-b4b5-4c87-bc99-ed18f33668ef\") " pod="openshift-monitoring/node-exporter-9dsgh"
Jul 02 12:40:04.772269 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.772028    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b45d14c-b4b5-4c87-bc99-ed18f33668ef-node-exporter-wtmp\") pod \"node-exporter-9dsgh\" (UID: \"1b45d14c-b4b5-4c87-bc99-ed18f33668ef\") " pod="openshift-monitoring/node-exporter-9dsgh"
Jul 02 12:40:04.772676 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.772657    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b45d14c-b4b5-4c87-bc99-ed18f33668ef-node-exporter-textfile\") pod \"node-exporter-9dsgh\" (UID: \"1b45d14c-b4b5-4c87-bc99-ed18f33668ef\") " pod="openshift-monitoring/node-exporter-9dsgh"
Jul 02 12:40:04.772923 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.772900    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b45d14c-b4b5-4c87-bc99-ed18f33668ef-node-exporter-accelerators-collector-config\") pod \"node-exporter-9dsgh\" (UID: \"1b45d14c-b4b5-4c87-bc99-ed18f33668ef\") " pod="openshift-monitoring/node-exporter-9dsgh"
Jul 02 12:40:04.772994 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.772923    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b45d14c-b4b5-4c87-bc99-ed18f33668ef-metrics-client-ca\") pod \"node-exporter-9dsgh\" (UID: \"1b45d14c-b4b5-4c87-bc99-ed18f33668ef\") " pod="openshift-monitoring/node-exporter-9dsgh"
Jul 02 12:40:04.774325 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.774306    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b45d14c-b4b5-4c87-bc99-ed18f33668ef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9dsgh\" (UID: \"1b45d14c-b4b5-4c87-bc99-ed18f33668ef\") " pod="openshift-monitoring/node-exporter-9dsgh"
Jul 02 12:40:04.774426 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.774331    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b45d14c-b4b5-4c87-bc99-ed18f33668ef-node-exporter-tls\") pod \"node-exporter-9dsgh\" (UID: \"1b45d14c-b4b5-4c87-bc99-ed18f33668ef\") " pod="openshift-monitoring/node-exporter-9dsgh"
Jul 02 12:40:04.795293 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.795261    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9crt4\" (UniqueName: \"kubernetes.io/projected/1b45d14c-b4b5-4c87-bc99-ed18f33668ef-kube-api-access-9crt4\") pod \"node-exporter-9dsgh\" (UID: \"1b45d14c-b4b5-4c87-bc99-ed18f33668ef\") " pod="openshift-monitoring/node-exporter-9dsgh"
Jul 02 12:40:04.819002 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.818966    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:04.821942 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.821924    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-p7k4z"
Jul 02 12:40:04.873331 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.873306    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9dsgh"
Jul 02 12:40:04.883946 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:40:04.883918    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b45d14c_b4b5_4c87_bc99_ed18f33668ef.slice/crio-e4c453e23c58ad072f27354727d759b69ce17180b3115625277dd8f6f605e7a0 WatchSource:0}: Error finding container e4c453e23c58ad072f27354727d759b69ce17180b3115625277dd8f6f605e7a0: Status 404 returned error can't find the container with id e4c453e23c58ad072f27354727d759b69ce17180b3115625277dd8f6f605e7a0
Jul 02 12:40:04.916921 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.916868    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:04.971870 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:04.971844    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-p7k4z"]
Jul 02 12:40:04.973596 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:40:04.973570    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d9e744f_62d0_46e6_a186_3b8d8e5efb94.slice/crio-bd63727931833bfdc82aeef4bacac3904d3b91c36479bb0286cf5fbf9ebe694a WatchSource:0}: Error finding container bd63727931833bfdc82aeef4bacac3904d3b91c36479bb0286cf5fbf9ebe694a: Status 404 returned error can't find the container with id bd63727931833bfdc82aeef4bacac3904d3b91c36479bb0286cf5fbf9ebe694a
Jul 02 12:40:05.056737 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:05.056707    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:05.728726 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:05.728626    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9dsgh" event={"ID":"1b45d14c-b4b5-4c87-bc99-ed18f33668ef","Type":"ContainerStarted","Data":"e4c453e23c58ad072f27354727d759b69ce17180b3115625277dd8f6f605e7a0"}
Jul 02 12:40:05.730655 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:05.730617    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-p7k4z" event={"ID":"1d9e744f-62d0-46e6-a186-3b8d8e5efb94","Type":"ContainerStarted","Data":"d502ff795da04807f105279990c55bdf82c34fbfdf4bfdb69c76a9c3f5cf95e8"}
Jul 02 12:40:05.730771 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:05.730691    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-p7k4z" event={"ID":"1d9e744f-62d0-46e6-a186-3b8d8e5efb94","Type":"ContainerStarted","Data":"ac67adb47c3207807d83c6426bc46499ff88e9edd00f6363bed09c81ed733796"}
Jul 02 12:40:05.730771 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:05.730723    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-p7k4z" event={"ID":"1d9e744f-62d0-46e6-a186-3b8d8e5efb94","Type":"ContainerStarted","Data":"bd63727931833bfdc82aeef4bacac3904d3b91c36479bb0286cf5fbf9ebe694a"}
Jul 02 12:40:05.801817 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:05.801781    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:05.931955 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:05.931925    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:06.072061 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:06.072015    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:06.737472 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:06.737440    2578 generic.go:356] "Generic (PLEG): container finished" podID="1b45d14c-b4b5-4c87-bc99-ed18f33668ef" containerID="80487d208af6ba4e7293fbdb9a7095231d3f513660694322300e3d36b6b1ee87" exitCode=0
Jul 02 12:40:06.737635 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:06.737541    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9dsgh" event={"ID":"1b45d14c-b4b5-4c87-bc99-ed18f33668ef","Type":"ContainerDied","Data":"80487d208af6ba4e7293fbdb9a7095231d3f513660694322300e3d36b6b1ee87"}
Jul 02 12:40:06.742632 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:06.742590    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-p7k4z" event={"ID":"1d9e744f-62d0-46e6-a186-3b8d8e5efb94","Type":"ContainerStarted","Data":"af1c72f11149b552184cc1353a27676d9b47806b11e3551d101b6dd041795328"}
Jul 02 12:40:06.787314 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:06.787267    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-p7k4z" podStartSLOduration=1.211713325 podStartE2EDuration="2.787250869s" totalImagesPullingTime="1.575537544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:40:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:40:06.786369291 +0000 UTC m=+74.919195141" watchObservedRunningTime="2026-07-02 12:40:06.787250869 +0000 UTC m=+74.920076719"
Jul 02 12:40:07.582552 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:07.582514    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:07.748592 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:07.748558    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9dsgh" event={"ID":"1b45d14c-b4b5-4c87-bc99-ed18f33668ef","Type":"ContainerStarted","Data":"6797aef1777f52d80ac6f9411262b0d2dd25935e5021da97f5f39d12cd7e6c32"}
Jul 02 12:40:07.748776 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:07.748614    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9dsgh" event={"ID":"1b45d14c-b4b5-4c87-bc99-ed18f33668ef","Type":"ContainerStarted","Data":"c5e886bc880954dfec6d8c087760d1ea9e6a7c7a8c020eb1bdd68d82f22bfeb3"}
Jul 02 12:40:07.765255 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:07.765226    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-c4844dbfd-jjrvk"]
Jul 02 12:40:07.768871 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:07.768851    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-c4844dbfd-jjrvk"
Jul 02 12:40:07.771812 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:07.771793    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-e75p8uru4f56\""
Jul 02 12:40:07.771983 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:07.771965    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 02 12:40:07.772715 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:07.772697    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 02 12:40:07.773066 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:07.773053    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-df4rm\""
Jul 02 12:40:07.773298 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:07.773282    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 02 12:40:07.773354 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:07.773311    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 02 12:40:07.773403 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:07.773311    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 02 12:40:07.776532 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:07.776492    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9dsgh" podStartSLOduration=2.647666368 podStartE2EDuration="3.776465525s" totalImagesPullingTime="1.065739445s" totalInitContainerRuntime="63.059712ms" isStatefulPod=true podCreationTimestamp="2026-07-02 12:40:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:40:07.774855378 +0000 UTC m=+75.907681223" watchObservedRunningTime="2026-07-02 12:40:07.776465525 +0000 UTC m=+75.909291371"
Jul 02 12:40:07.791421 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:07.791401    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-c4844dbfd-jjrvk"]
Jul 02 12:40:07.897785 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:07.897683    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8-secret-grpc-tls\") pod \"thanos-querier-c4844dbfd-jjrvk\" (UID: \"67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8\") " pod="openshift-monitoring/thanos-querier-c4844dbfd-jjrvk"
Jul 02 12:40:07.897785 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:07.897721    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnzf5\" (UniqueName: \"kubernetes.io/projected/67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8-kube-api-access-bnzf5\") pod \"thanos-querier-c4844dbfd-jjrvk\" (UID: \"67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8\") " pod="openshift-monitoring/thanos-querier-c4844dbfd-jjrvk"
Jul 02 12:40:07.898014 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:07.897849    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c4844dbfd-jjrvk\" (UID: \"67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8\") " pod="openshift-monitoring/thanos-querier-c4844dbfd-jjrvk"
Jul 02 12:40:07.898014 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:07.897897    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8-secret-thanos-querier-tls\") pod \"thanos-querier-c4844dbfd-jjrvk\" (UID: \"67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8\") " pod="openshift-monitoring/thanos-querier-c4844dbfd-jjrvk"
Jul 02 12:40:07.898014 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:07.897915    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8-metrics-client-ca\") pod \"thanos-querier-c4844dbfd-jjrvk\" (UID: \"67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8\") " pod="openshift-monitoring/thanos-querier-c4844dbfd-jjrvk"
Jul 02 12:40:07.898014 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:07.897992    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c4844dbfd-jjrvk\" (UID: \"67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8\") " pod="openshift-monitoring/thanos-querier-c4844dbfd-jjrvk"
Jul 02 12:40:07.898245 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:07.898167    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c4844dbfd-jjrvk\" (UID: \"67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8\") " pod="openshift-monitoring/thanos-querier-c4844dbfd-jjrvk"
Jul 02 12:40:07.898245 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:07.898235    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c4844dbfd-jjrvk\" (UID: \"67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8\") " pod="openshift-monitoring/thanos-querier-c4844dbfd-jjrvk"
Jul 02 12:40:07.998929 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:07.998894    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c4844dbfd-jjrvk\" (UID: \"67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8\") " pod="openshift-monitoring/thanos-querier-c4844dbfd-jjrvk"
Jul 02 12:40:07.999112 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:07.998937    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c4844dbfd-jjrvk\" (UID: \"67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8\") " pod="openshift-monitoring/thanos-querier-c4844dbfd-jjrvk"
Jul 02 12:40:07.999112 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:07.998961    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8-secret-grpc-tls\") pod \"thanos-querier-c4844dbfd-jjrvk\" (UID: \"67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8\") " pod="openshift-monitoring/thanos-querier-c4844dbfd-jjrvk"
Jul 02 12:40:07.999112 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:07.998983    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnzf5\" (UniqueName: \"kubernetes.io/projected/67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8-kube-api-access-bnzf5\") pod \"thanos-querier-c4844dbfd-jjrvk\" (UID: \"67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8\") " pod="openshift-monitoring/thanos-querier-c4844dbfd-jjrvk"
Jul 02 12:40:07.999112 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:07.999011    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c4844dbfd-jjrvk\" (UID: \"67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8\") " pod="openshift-monitoring/thanos-querier-c4844dbfd-jjrvk"
Jul 02 12:40:07.999112 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:07.999044    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8-secret-thanos-querier-tls\") pod \"thanos-querier-c4844dbfd-jjrvk\" (UID: \"67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8\") " pod="openshift-monitoring/thanos-querier-c4844dbfd-jjrvk"
Jul 02 12:40:07.999112 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:07.999069    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8-metrics-client-ca\") pod \"thanos-querier-c4844dbfd-jjrvk\" (UID: \"67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8\") " pod="openshift-monitoring/thanos-querier-c4844dbfd-jjrvk"
Jul 02 12:40:07.999489 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:07.999117    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c4844dbfd-jjrvk\" (UID: \"67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8\") " pod="openshift-monitoring/thanos-querier-c4844dbfd-jjrvk"
Jul 02 12:40:07.999963 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:07.999937    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8-metrics-client-ca\") pod \"thanos-querier-c4844dbfd-jjrvk\" (UID: \"67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8\") " pod="openshift-monitoring/thanos-querier-c4844dbfd-jjrvk"
Jul 02 12:40:08.001749 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:08.001717    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c4844dbfd-jjrvk\" (UID: \"67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8\") " pod="openshift-monitoring/thanos-querier-c4844dbfd-jjrvk"
Jul 02 12:40:08.001864 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:08.001717    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8-secret-grpc-tls\") pod \"thanos-querier-c4844dbfd-jjrvk\" (UID: \"67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8\") " pod="openshift-monitoring/thanos-querier-c4844dbfd-jjrvk"
Jul 02 12:40:08.001972 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:08.001946    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c4844dbfd-jjrvk\" (UID: \"67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8\") " pod="openshift-monitoring/thanos-querier-c4844dbfd-jjrvk"
Jul 02 12:40:08.002019 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:08.001979    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c4844dbfd-jjrvk\" (UID: \"67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8\") " pod="openshift-monitoring/thanos-querier-c4844dbfd-jjrvk"
Jul 02 12:40:08.002058 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:08.002041    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c4844dbfd-jjrvk\" (UID: \"67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8\") " pod="openshift-monitoring/thanos-querier-c4844dbfd-jjrvk"
Jul 02 12:40:08.002337 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:08.002319    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8-secret-thanos-querier-tls\") pod \"thanos-querier-c4844dbfd-jjrvk\" (UID: \"67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8\") " pod="openshift-monitoring/thanos-querier-c4844dbfd-jjrvk"
Jul 02 12:40:08.017568 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:08.017543    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnzf5\" (UniqueName: \"kubernetes.io/projected/67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8-kube-api-access-bnzf5\") pod \"thanos-querier-c4844dbfd-jjrvk\" (UID: \"67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8\") " pod="openshift-monitoring/thanos-querier-c4844dbfd-jjrvk"
Jul 02 12:40:08.078256 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:08.078204    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-c4844dbfd-jjrvk"
Jul 02 12:40:08.081139 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:08.081117    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:08.181905 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:08.181874    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:08.226320 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:08.226299    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-c4844dbfd-jjrvk"]
Jul 02 12:40:08.227694 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:40:08.227667    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d53376_3f77_4cb4_aa14_f6ad8e8b7ef8.slice/crio-10c252c6aeea153a4fb8b79138e0dd86872cb9e8fe53a60cf16b137b5ee975df WatchSource:0}: Error finding container 10c252c6aeea153a4fb8b79138e0dd86872cb9e8fe53a60cf16b137b5ee975df: Status 404 returned error can't find the container with id 10c252c6aeea153a4fb8b79138e0dd86872cb9e8fe53a60cf16b137b5ee975df
Jul 02 12:40:08.309832 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:08.309810    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:08.398869 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:08.398842    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:08.753801 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:08.753765    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c4844dbfd-jjrvk" event={"ID":"67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8","Type":"ContainerStarted","Data":"10c252c6aeea153a4fb8b79138e0dd86872cb9e8fe53a60cf16b137b5ee975df"}
Jul 02 12:40:09.307398 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:09.307371    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:09.416642 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:09.416605    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:09.417164 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:09.417138    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-cd8656b8-sb6pw"]
Jul 02 12:40:09.420490 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:09.420470    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-cd8656b8-sb6pw"
Jul 02 12:40:09.430111 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:09.430088    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-vqc2q\""
Jul 02 12:40:09.430438 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:09.430417    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 02 12:40:09.444733 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:09.444714    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-cd8656b8-sb6pw"]
Jul 02 12:40:09.509317 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:09.509294    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0785238a-fca2-4d58-a34c-d1ec38d13133-monitoring-plugin-cert\") pod \"monitoring-plugin-cd8656b8-sb6pw\" (UID: \"0785238a-fca2-4d58-a34c-d1ec38d13133\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-sb6pw"
Jul 02 12:40:09.509440 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:09.509354    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlvlq\" (UniqueName: \"kubernetes.io/projected/0785238a-fca2-4d58-a34c-d1ec38d13133-kube-api-access-dlvlq\") pod \"monitoring-plugin-cd8656b8-sb6pw\" (UID: \"0785238a-fca2-4d58-a34c-d1ec38d13133\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-sb6pw"
Jul 02 12:40:09.610550 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:09.610490    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlvlq\" (UniqueName: \"kubernetes.io/projected/0785238a-fca2-4d58-a34c-d1ec38d13133-kube-api-access-dlvlq\") pod \"monitoring-plugin-cd8656b8-sb6pw\" (UID: \"0785238a-fca2-4d58-a34c-d1ec38d13133\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-sb6pw"
Jul 02 12:40:09.610550 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:09.610542    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0785238a-fca2-4d58-a34c-d1ec38d13133-monitoring-plugin-cert\") pod \"monitoring-plugin-cd8656b8-sb6pw\" (UID: \"0785238a-fca2-4d58-a34c-d1ec38d13133\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-sb6pw"
Jul 02 12:40:09.610693 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:40:09.610654    2578 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 02 12:40:09.610728 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:40:09.610713    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0785238a-fca2-4d58-a34c-d1ec38d13133-monitoring-plugin-cert podName:0785238a-fca2-4d58-a34c-d1ec38d13133 nodeName:}" failed. No retries permitted until 2026-07-02 12:40:10.110695864 +0000 UTC m=+78.243521688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/0785238a-fca2-4d58-a34c-d1ec38d13133-monitoring-plugin-cert") pod "monitoring-plugin-cd8656b8-sb6pw" (UID: "0785238a-fca2-4d58-a34c-d1ec38d13133") : secret "monitoring-plugin-cert" not found
Jul 02 12:40:09.621339 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:09.621316    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlvlq\" (UniqueName: \"kubernetes.io/projected/0785238a-fca2-4d58-a34c-d1ec38d13133-kube-api-access-dlvlq\") pod \"monitoring-plugin-cd8656b8-sb6pw\" (UID: \"0785238a-fca2-4d58-a34c-d1ec38d13133\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-sb6pw"
Jul 02 12:40:09.772190 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:09.772161    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:10.115048 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:10.115017    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0785238a-fca2-4d58-a34c-d1ec38d13133-monitoring-plugin-cert\") pod \"monitoring-plugin-cd8656b8-sb6pw\" (UID: \"0785238a-fca2-4d58-a34c-d1ec38d13133\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-sb6pw"
Jul 02 12:40:10.117827 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:10.117801    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0785238a-fca2-4d58-a34c-d1ec38d13133-monitoring-plugin-cert\") pod \"monitoring-plugin-cd8656b8-sb6pw\" (UID: \"0785238a-fca2-4d58-a34c-d1ec38d13133\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-sb6pw"
Jul 02 12:40:10.332444 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:10.332412    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-cd8656b8-sb6pw"
Jul 02 12:40:10.485916 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:10.485891    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-cd8656b8-sb6pw"]
Jul 02 12:40:10.490238 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:40:10.490218    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0785238a_fca2_4d58_a34c_d1ec38d13133.slice/crio-f92e849bc7a573e4b5539d7e88e6ad2debc2d373b790fe675077ef6080fb9bfb WatchSource:0}: Error finding container f92e849bc7a573e4b5539d7e88e6ad2debc2d373b790fe675077ef6080fb9bfb: Status 404 returned error can't find the container with id f92e849bc7a573e4b5539d7e88e6ad2debc2d373b790fe675077ef6080fb9bfb
Jul 02 12:40:10.763261 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:10.763224    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c4844dbfd-jjrvk" event={"ID":"67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8","Type":"ContainerStarted","Data":"d3c97f06119a3b19946200dc685899261bf79c848ba5a177f915b722295bc824"}
Jul 02 12:40:10.763423 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:10.763299    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c4844dbfd-jjrvk" event={"ID":"67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8","Type":"ContainerStarted","Data":"5381fc9356bd00629343c6c44d57e4da4e5d46fa637c86043358bf543eceffeb"}
Jul 02 12:40:10.763423 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:10.763333    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c4844dbfd-jjrvk" event={"ID":"67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8","Type":"ContainerStarted","Data":"f7920e789d4664ab4bbd2f3ec8a62f531b8782fb432d26e3472b0235bf038142"}
Jul 02 12:40:10.764328 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:10.764309    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-cd8656b8-sb6pw" event={"ID":"0785238a-fca2-4d58-a34c-d1ec38d13133","Type":"ContainerStarted","Data":"f92e849bc7a573e4b5539d7e88e6ad2debc2d373b790fe675077ef6080fb9bfb"}
Jul 02 12:40:11.105241 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:11.105141    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:11.279665 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:11.279631    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:11.380644 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:11.380620    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:11.771956 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:11.771918    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c4844dbfd-jjrvk" event={"ID":"67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8","Type":"ContainerStarted","Data":"6a14086efebd29a9f2fe12bf115ea0eac8be4df3a8f37d09091195fc48b94ee0"}
Jul 02 12:40:11.772144 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:11.772013    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c4844dbfd-jjrvk" event={"ID":"67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8","Type":"ContainerStarted","Data":"b4003337d09f7c2c0b29be990de0180d901b051a1d4333c63ee517d9b341d10b"}
Jul 02 12:40:11.772144 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:11.772064    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c4844dbfd-jjrvk" event={"ID":"67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8","Type":"ContainerStarted","Data":"96c2517e1e50fd59d91863b337ecb5e030cede23679e269085de76c32fd2ba64"}
Jul 02 12:40:11.772144 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:11.772120    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-c4844dbfd-jjrvk"
Jul 02 12:40:11.802255 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:11.802197    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-c4844dbfd-jjrvk" podStartSLOduration=1.93005625 podStartE2EDuration="4.802157707s" totalImagesPullingTime="2.872101457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:40:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:40:11.801064925 +0000 UTC m=+79.933890772" watchObservedRunningTime="2026-07-02 12:40:11.802157707 +0000 UTC m=+79.934983560"
Jul 02 12:40:12.260213 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:12.260161    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:12.318079 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:12.318045    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:12.777232 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:12.777195    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-cd8656b8-sb6pw" event={"ID":"0785238a-fca2-4d58-a34c-d1ec38d13133","Type":"ContainerStarted","Data":"97378f8861d9f246e8d594ffbe32f7db03a8a1d5f70c9d54d0bf8975a85ac317"}
Jul 02 12:40:12.777426 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:12.777410    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-cd8656b8-sb6pw"
Jul 02 12:40:12.781756 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:12.781735    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-cd8656b8-sb6pw"
Jul 02 12:40:12.795472 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:12.795432    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-cd8656b8-sb6pw" podStartSLOduration=2.443834634 podStartE2EDuration="3.795418996s" totalImagesPullingTime="1.351584362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:40:12.794941533 +0000 UTC m=+80.927767379" watchObservedRunningTime="2026-07-02 12:40:12.795418996 +0000 UTC m=+80.928244841"
Jul 02 12:40:13.284389 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:13.284362    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:13.415787 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:13.415758    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:14.943960 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:14.943915    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:15.047559 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:15.047531    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:17.184623 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:17.184590    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:17.298598 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:17.298570    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:17.784463 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:17.784233    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-c4844dbfd-jjrvk"
Jul 02 12:40:23.204295 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:23.204259    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:23.297311 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:23.297280    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:23.993397 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:23.993361    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:24.093321 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:24.093294    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:24.193704 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:24.193677    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:25.377367 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:25.377337    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:25.476740 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:25.476714    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:26.284208 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:26.284159    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:26.350559 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:26.350533    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:34.726266 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:34.726222    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4cw8v"
Jul 02 12:40:41.310801 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:41.310768    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:41.638698 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:41.638616    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:41.980020 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:41.979937    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:42.038382 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:42.038356    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:54.975300 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:54.975263    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:40:55.080698 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:40:55.080665    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:41:00.322016 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:41:00.321978    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:41:00.421605 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:41:00.421576    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:41:23.880160 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:41:23.880104    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:41:24.000649 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:41:24.000621    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:41:33.766202 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:41:33.766145    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:41:33.828464 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:41:33.828434    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:41:44.677783 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:41:44.677697    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:41:44.778240 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:41:44.778206    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:41:44.830623 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:41:44.830592    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:41:46.167391 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:41:46.167361    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:42:09.426221 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:42:09.426171    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:42:21.055821 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:42:21.055784    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:42:30.549972 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:42:30.549939    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:42:30.632510 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:42:30.632482    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:42:39.128088 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:42:39.128036    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:42:39.220596 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:42:39.220566    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:42:51.155789 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:42:51.155749    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:42:51.215660 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:42:51.215629    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:42:52.716853 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:42:52.716819    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:42:52.778696 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:42:52.778653    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:42:52.836053 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:42:52.836019    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:42:52.936356 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:42:52.936326    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:42:53.453916 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:42:53.453888    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:42:56.277119 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:42:56.277085    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:42:56.378109 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:42:56.378079    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:42:56.449114 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:42:56.449083    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:43:03.996563 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:43:03.996529    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:43:04.093534 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:43:04.093506    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:43:04.186656 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:43:04.186610    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:43:52.298617 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:43:52.298591    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95kwm_495e9fa8-b93f-42e0-95b5-813527df0993/ovn-acl-logging/0.log"
Jul 02 12:43:52.302009 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:43:52.301989    2578 kubelet.go:1648] "Image garbage collection succeeded"
Jul 02 12:43:52.305371 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:43:52.305353    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95kwm_495e9fa8-b93f-42e0-95b5-813527df0993/ovn-acl-logging/0.log"
Jul 02 12:45:23.014471 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:23.014436    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:45:24.739584 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:24.739554    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:45:45.929329 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:45.929298    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:45:46.024974 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:46.024945    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:45:53.076498 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:53.076457    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:45:53.383904 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:53.383767    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:45:53.563031 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:53.562974    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:45:53.849173 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:53.849111    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:45:54.094081 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:54.094041    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:45:54.247800 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:54.247770    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-nhs8p"]
Jul 02 12:45:54.250968 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:54.250949    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-nhs8p"
Jul 02 12:45:54.253990 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:54.253966    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-f7bxr\""
Jul 02 12:45:54.254084 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:54.253979    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 02 12:45:54.255399 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:54.255382    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 02 12:45:54.255472 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:54.255416    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 02 12:45:54.255472 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:54.255453    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 02 12:45:54.262532 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:54.262512    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-nhs8p"]
Jul 02 12:45:54.266458 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:54.266433    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29sdp\" (UniqueName: \"kubernetes.io/projected/18d1acdf-74e7-47b6-bd32-da4ddafb2c07-kube-api-access-29sdp\") pod \"keda-admission-5fbd5c4644-nhs8p\" (UID: \"18d1acdf-74e7-47b6-bd32-da4ddafb2c07\") " pod="openshift-keda/keda-admission-5fbd5c4644-nhs8p"
Jul 02 12:45:54.266566 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:54.266478    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/18d1acdf-74e7-47b6-bd32-da4ddafb2c07-certificates\") pod \"keda-admission-5fbd5c4644-nhs8p\" (UID: \"18d1acdf-74e7-47b6-bd32-da4ddafb2c07\") " pod="openshift-keda/keda-admission-5fbd5c4644-nhs8p"
Jul 02 12:45:54.267585 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:54.267563    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:45:54.365281 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:54.365250    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:45:54.367455 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:54.367438    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29sdp\" (UniqueName: \"kubernetes.io/projected/18d1acdf-74e7-47b6-bd32-da4ddafb2c07-kube-api-access-29sdp\") pod \"keda-admission-5fbd5c4644-nhs8p\" (UID: \"18d1acdf-74e7-47b6-bd32-da4ddafb2c07\") " pod="openshift-keda/keda-admission-5fbd5c4644-nhs8p"
Jul 02 12:45:54.367560 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:54.367467    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/18d1acdf-74e7-47b6-bd32-da4ddafb2c07-certificates\") pod \"keda-admission-5fbd5c4644-nhs8p\" (UID: \"18d1acdf-74e7-47b6-bd32-da4ddafb2c07\") " pod="openshift-keda/keda-admission-5fbd5c4644-nhs8p"
Jul 02 12:45:54.367616 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:45:54.367581    2578 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Jul 02 12:45:54.367616 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:45:54.367603    2578 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-5fbd5c4644-nhs8p: secret "keda-admission-webhooks-certs" not found
Jul 02 12:45:54.367717 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:45:54.367654    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18d1acdf-74e7-47b6-bd32-da4ddafb2c07-certificates podName:18d1acdf-74e7-47b6-bd32-da4ddafb2c07 nodeName:}" failed. No retries permitted until 2026-07-02 12:45:54.867636939 +0000 UTC m=+423.000462767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/18d1acdf-74e7-47b6-bd32-da4ddafb2c07-certificates") pod "keda-admission-5fbd5c4644-nhs8p" (UID: "18d1acdf-74e7-47b6-bd32-da4ddafb2c07") : secret "keda-admission-webhooks-certs" not found
Jul 02 12:45:54.380735 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:54.380708    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29sdp\" (UniqueName: \"kubernetes.io/projected/18d1acdf-74e7-47b6-bd32-da4ddafb2c07-kube-api-access-29sdp\") pod \"keda-admission-5fbd5c4644-nhs8p\" (UID: \"18d1acdf-74e7-47b6-bd32-da4ddafb2c07\") " pod="openshift-keda/keda-admission-5fbd5c4644-nhs8p"
Jul 02 12:45:54.588250 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:54.588148    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:45:54.788144 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:54.788115    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:45:54.842997 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:54.842932    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:45:54.871617 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:54.871593    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/18d1acdf-74e7-47b6-bd32-da4ddafb2c07-certificates\") pod \"keda-admission-5fbd5c4644-nhs8p\" (UID: \"18d1acdf-74e7-47b6-bd32-da4ddafb2c07\") " pod="openshift-keda/keda-admission-5fbd5c4644-nhs8p"
Jul 02 12:45:54.873849 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:54.873824    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/18d1acdf-74e7-47b6-bd32-da4ddafb2c07-certificates\") pod \"keda-admission-5fbd5c4644-nhs8p\" (UID: \"18d1acdf-74e7-47b6-bd32-da4ddafb2c07\") " pod="openshift-keda/keda-admission-5fbd5c4644-nhs8p"
Jul 02 12:45:54.917413 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:54.917394    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:45:55.016839 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:55.016813    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:45:55.161228 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:55.161124    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-nhs8p"
Jul 02 12:45:55.286056 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:55.286035    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-nhs8p"]
Jul 02 12:45:55.288234 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:45:55.288209    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18d1acdf_74e7_47b6_bd32_da4ddafb2c07.slice/crio-4132aa05f2829e26d19c7fa7ee3aa8a9993219ba601ec8be166a6e13411b5d47 WatchSource:0}: Error finding container 4132aa05f2829e26d19c7fa7ee3aa8a9993219ba601ec8be166a6e13411b5d47: Status 404 returned error can't find the container with id 4132aa05f2829e26d19c7fa7ee3aa8a9993219ba601ec8be166a6e13411b5d47
Jul 02 12:45:55.290122 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:55.290103    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 12:45:55.968602 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:55.968565    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-nhs8p" event={"ID":"18d1acdf-74e7-47b6-bd32-da4ddafb2c07","Type":"ContainerStarted","Data":"4132aa05f2829e26d19c7fa7ee3aa8a9993219ba601ec8be166a6e13411b5d47"}
Jul 02 12:45:58.982011 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:58.981973    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-nhs8p" event={"ID":"18d1acdf-74e7-47b6-bd32-da4ddafb2c07","Type":"ContainerStarted","Data":"582d0ffdc4673a3fb73dd85e119ea901916637dd670ed89a99a104a9926bb766"}
Jul 02 12:45:58.982427 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:58.982115    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-5fbd5c4644-nhs8p"
Jul 02 12:45:59.002020 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:45:59.001973    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-5fbd5c4644-nhs8p" podStartSLOduration=2.156918859 podStartE2EDuration="5.001958646s" totalImagesPullingTime="2.845039787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:45:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:45:58.999909136 +0000 UTC m=+427.132734981" watchObservedRunningTime="2026-07-02 12:45:59.001958646 +0000 UTC m=+427.134784491"
Jul 02 12:46:19.988304 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:46:19.988275    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-5fbd5c4644-nhs8p"
Jul 02 12:46:33.016189 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:46:33.016138    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:46:53.416041 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:46:53.416003    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:46:53.492667 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:46:53.492642    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:46:56.365736 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:46:56.365703    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:47:04.035761 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:04.035729    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:47:04.091455 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:04.091431    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:47:04.204456 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:04.204429    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:47:04.325809 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:04.325738    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-69b5d94596-2d254"]
Jul 02 12:47:04.328842 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:04.328825    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-69b5d94596-2d254"
Jul 02 12:47:04.333506 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:04.333486    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 02 12:47:04.333974 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:04.333956    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-lck5s\""
Jul 02 12:47:04.334397 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:04.334223    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 02 12:47:04.334397 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:04.334226    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 02 12:47:04.349090 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:04.349068    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:47:04.362172 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:04.362150    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-69b5d94596-2d254"]
Jul 02 12:47:04.370372 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:04.370353    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-lqvhj"]
Jul 02 12:47:04.373343 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:04.373322    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-lqvhj"
Jul 02 12:47:04.384116 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:04.383810    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 02 12:47:04.384786 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:04.384417    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-hx67w\""
Jul 02 12:47:04.386613 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:04.386596    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-lqvhj"]
Jul 02 12:47:04.442606 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:04.442585    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:47:04.448732 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:04.448714    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjdff\" (UniqueName: \"kubernetes.io/projected/f8297d26-2bc0-4f2a-9ba3-73a29907cc5f-kube-api-access-jjdff\") pod \"seaweedfs-64568bcd49-lqvhj\" (UID: \"f8297d26-2bc0-4f2a-9ba3-73a29907cc5f\") " pod="kserve/seaweedfs-64568bcd49-lqvhj"
Jul 02 12:47:04.448804 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:04.448749    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5be30942-6dd5-4842-89fd-d90997e930ec-cert\") pod \"kserve-controller-manager-69b5d94596-2d254\" (UID: \"5be30942-6dd5-4842-89fd-d90997e930ec\") " pod="kserve/kserve-controller-manager-69b5d94596-2d254"
Jul 02 12:47:04.448968 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:04.448941    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lc4m\" (UniqueName: \"kubernetes.io/projected/5be30942-6dd5-4842-89fd-d90997e930ec-kube-api-access-5lc4m\") pod \"kserve-controller-manager-69b5d94596-2d254\" (UID: \"5be30942-6dd5-4842-89fd-d90997e930ec\") " pod="kserve/kserve-controller-manager-69b5d94596-2d254"
Jul 02 12:47:04.449059 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:04.448990    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f8297d26-2bc0-4f2a-9ba3-73a29907cc5f-data\") pod \"seaweedfs-64568bcd49-lqvhj\" (UID: \"f8297d26-2bc0-4f2a-9ba3-73a29907cc5f\") " pod="kserve/seaweedfs-64568bcd49-lqvhj"
Jul 02 12:47:04.549749 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:04.549729    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjdff\" (UniqueName: \"kubernetes.io/projected/f8297d26-2bc0-4f2a-9ba3-73a29907cc5f-kube-api-access-jjdff\") pod \"seaweedfs-64568bcd49-lqvhj\" (UID: \"f8297d26-2bc0-4f2a-9ba3-73a29907cc5f\") " pod="kserve/seaweedfs-64568bcd49-lqvhj"
Jul 02 12:47:04.549860 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:04.549757    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5be30942-6dd5-4842-89fd-d90997e930ec-cert\") pod \"kserve-controller-manager-69b5d94596-2d254\" (UID: \"5be30942-6dd5-4842-89fd-d90997e930ec\") " pod="kserve/kserve-controller-manager-69b5d94596-2d254"
Jul 02 12:47:04.549860 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:04.549782    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lc4m\" (UniqueName: \"kubernetes.io/projected/5be30942-6dd5-4842-89fd-d90997e930ec-kube-api-access-5lc4m\") pod \"kserve-controller-manager-69b5d94596-2d254\" (UID: \"5be30942-6dd5-4842-89fd-d90997e930ec\") " pod="kserve/kserve-controller-manager-69b5d94596-2d254"
Jul 02 12:47:04.549860 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:04.549805    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f8297d26-2bc0-4f2a-9ba3-73a29907cc5f-data\") pod \"seaweedfs-64568bcd49-lqvhj\" (UID: \"f8297d26-2bc0-4f2a-9ba3-73a29907cc5f\") " pod="kserve/seaweedfs-64568bcd49-lqvhj"
Jul 02 12:47:04.550094 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:04.550080    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f8297d26-2bc0-4f2a-9ba3-73a29907cc5f-data\") pod \"seaweedfs-64568bcd49-lqvhj\" (UID: \"f8297d26-2bc0-4f2a-9ba3-73a29907cc5f\") " pod="kserve/seaweedfs-64568bcd49-lqvhj"
Jul 02 12:47:04.551605 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:04.551548    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:47:04.552276 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:04.552260    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5be30942-6dd5-4842-89fd-d90997e930ec-cert\") pod \"kserve-controller-manager-69b5d94596-2d254\" (UID: \"5be30942-6dd5-4842-89fd-d90997e930ec\") " pod="kserve/kserve-controller-manager-69b5d94596-2d254"
Jul 02 12:47:04.560757 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:04.560734    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjdff\" (UniqueName: \"kubernetes.io/projected/f8297d26-2bc0-4f2a-9ba3-73a29907cc5f-kube-api-access-jjdff\") pod \"seaweedfs-64568bcd49-lqvhj\" (UID: \"f8297d26-2bc0-4f2a-9ba3-73a29907cc5f\") " pod="kserve/seaweedfs-64568bcd49-lqvhj"
Jul 02 12:47:04.560836 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:04.560795    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lc4m\" (UniqueName: \"kubernetes.io/projected/5be30942-6dd5-4842-89fd-d90997e930ec-kube-api-access-5lc4m\") pod \"kserve-controller-manager-69b5d94596-2d254\" (UID: \"5be30942-6dd5-4842-89fd-d90997e930ec\") " pod="kserve/kserve-controller-manager-69b5d94596-2d254"
Jul 02 12:47:04.609488 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:04.609440    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:47:04.638613 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:04.638597    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-69b5d94596-2d254"
Jul 02 12:47:04.688671 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:04.688640    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-lqvhj"
Jul 02 12:47:04.768129 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:04.768103    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-69b5d94596-2d254"]
Jul 02 12:47:04.771267 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:47:04.771230    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5be30942_6dd5_4842_89fd_d90997e930ec.slice/crio-75f6276a8863e2c38f79de5ed6db6ab4b5deb347571dbeb3e14bb8bfb0136bf4 WatchSource:0}: Error finding container 75f6276a8863e2c38f79de5ed6db6ab4b5deb347571dbeb3e14bb8bfb0136bf4: Status 404 returned error can't find the container with id 75f6276a8863e2c38f79de5ed6db6ab4b5deb347571dbeb3e14bb8bfb0136bf4
Jul 02 12:47:04.819564 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:04.819537    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-lqvhj"]
Jul 02 12:47:04.821731 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:47:04.821708    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8297d26_2bc0_4f2a_9ba3_73a29907cc5f.slice/crio-7c782ef0522ae50129bb332504987760957c5103863d0e0544801593513650ca WatchSource:0}: Error finding container 7c782ef0522ae50129bb332504987760957c5103863d0e0544801593513650ca: Status 404 returned error can't find the container with id 7c782ef0522ae50129bb332504987760957c5103863d0e0544801593513650ca
Jul 02 12:47:05.157813 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:05.157776    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:47:05.211251 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:05.211212    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-69b5d94596-2d254" event={"ID":"5be30942-6dd5-4842-89fd-d90997e930ec","Type":"ContainerStarted","Data":"75f6276a8863e2c38f79de5ed6db6ab4b5deb347571dbeb3e14bb8bfb0136bf4"}
Jul 02 12:47:05.212458 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:05.212433    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-lqvhj" event={"ID":"f8297d26-2bc0-4f2a-9ba3-73a29907cc5f","Type":"ContainerStarted","Data":"7c782ef0522ae50129bb332504987760957c5103863d0e0544801593513650ca"}
Jul 02 12:47:05.255357 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:05.255335    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:47:05.311133 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:05.311057    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:47:09.230447 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:09.230374    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-69b5d94596-2d254" event={"ID":"5be30942-6dd5-4842-89fd-d90997e930ec","Type":"ContainerStarted","Data":"35d4ac001c9c3e2418d7aeb38651efd3ecfe6e40c2bf1a5120b70c0caab4f33e"}
Jul 02 12:47:09.230878 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:09.230488    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-69b5d94596-2d254"
Jul 02 12:47:09.232041 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:09.232019    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-lqvhj" event={"ID":"f8297d26-2bc0-4f2a-9ba3-73a29907cc5f","Type":"ContainerStarted","Data":"0dea80624242721f35a649058397f9f86e11212cb22bfc0e562ff8eb5cb2b984"}
Jul 02 12:47:09.232163 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:09.232060    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-lqvhj"
Jul 02 12:47:09.247420 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:09.247370    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-69b5d94596-2d254" podStartSLOduration=1.113475588 podStartE2EDuration="5.247357312s" totalImagesPullingTime="4.133881724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:47:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:47:09.246618221 +0000 UTC m=+497.379444083" watchObservedRunningTime="2026-07-02 12:47:09.247357312 +0000 UTC m=+497.380183159"
Jul 02 12:47:09.263393 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:09.263337    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-lqvhj" podStartSLOduration=1.129418157 podStartE2EDuration="5.263323436s" totalImagesPullingTime="4.133905279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:47:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:47:09.262082259 +0000 UTC m=+497.394908141" watchObservedRunningTime="2026-07-02 12:47:09.263323436 +0000 UTC m=+497.396149282"
Jul 02 12:47:15.237554 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:15.237517    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-lqvhj"
Jul 02 12:47:19.629902 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:19.629874    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:47:39.135392 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:39.135348    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-69b5d94596-2d254"]
Jul 02 12:47:39.135885 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:39.135580    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/kserve-controller-manager-69b5d94596-2d254" podUID="5be30942-6dd5-4842-89fd-d90997e930ec" containerName="manager" containerID="cri-o://35d4ac001c9c3e2418d7aeb38651efd3ecfe6e40c2bf1a5120b70c0caab4f33e" gracePeriod=10
Jul 02 12:47:39.140688 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:39.140658    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-69b5d94596-2d254"
Jul 02 12:47:39.160297 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:39.160275    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-69b5d94596-wdwkk"]
Jul 02 12:47:39.163544 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:39.163528    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-69b5d94596-wdwkk"
Jul 02 12:47:39.172310 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:39.172288    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-69b5d94596-wdwkk"]
Jul 02 12:47:39.210833 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:39.210809    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktcvg\" (UniqueName: \"kubernetes.io/projected/53050271-cef2-40e4-8f3b-d0b1bb9499ca-kube-api-access-ktcvg\") pod \"kserve-controller-manager-69b5d94596-wdwkk\" (UID: \"53050271-cef2-40e4-8f3b-d0b1bb9499ca\") " pod="kserve/kserve-controller-manager-69b5d94596-wdwkk"
Jul 02 12:47:39.210939 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:39.210847    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53050271-cef2-40e4-8f3b-d0b1bb9499ca-cert\") pod \"kserve-controller-manager-69b5d94596-wdwkk\" (UID: \"53050271-cef2-40e4-8f3b-d0b1bb9499ca\") " pod="kserve/kserve-controller-manager-69b5d94596-wdwkk"
Jul 02 12:47:39.311626 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:39.311599    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53050271-cef2-40e4-8f3b-d0b1bb9499ca-cert\") pod \"kserve-controller-manager-69b5d94596-wdwkk\" (UID: \"53050271-cef2-40e4-8f3b-d0b1bb9499ca\") " pod="kserve/kserve-controller-manager-69b5d94596-wdwkk"
Jul 02 12:47:39.311742 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:39.311678    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktcvg\" (UniqueName: \"kubernetes.io/projected/53050271-cef2-40e4-8f3b-d0b1bb9499ca-kube-api-access-ktcvg\") pod \"kserve-controller-manager-69b5d94596-wdwkk\" (UID: \"53050271-cef2-40e4-8f3b-d0b1bb9499ca\") " pod="kserve/kserve-controller-manager-69b5d94596-wdwkk"
Jul 02 12:47:39.314099 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:39.314058    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53050271-cef2-40e4-8f3b-d0b1bb9499ca-cert\") pod \"kserve-controller-manager-69b5d94596-wdwkk\" (UID: \"53050271-cef2-40e4-8f3b-d0b1bb9499ca\") " pod="kserve/kserve-controller-manager-69b5d94596-wdwkk"
Jul 02 12:47:39.320563 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:39.320537    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktcvg\" (UniqueName: \"kubernetes.io/projected/53050271-cef2-40e4-8f3b-d0b1bb9499ca-kube-api-access-ktcvg\") pod \"kserve-controller-manager-69b5d94596-wdwkk\" (UID: \"53050271-cef2-40e4-8f3b-d0b1bb9499ca\") " pod="kserve/kserve-controller-manager-69b5d94596-wdwkk"
Jul 02 12:47:39.341256 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:39.341231    2578 generic.go:356] "Generic (PLEG): container finished" podID="5be30942-6dd5-4842-89fd-d90997e930ec" containerID="35d4ac001c9c3e2418d7aeb38651efd3ecfe6e40c2bf1a5120b70c0caab4f33e" exitCode=0
Jul 02 12:47:39.341362 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:39.341295    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-69b5d94596-2d254" event={"ID":"5be30942-6dd5-4842-89fd-d90997e930ec","Type":"ContainerDied","Data":"35d4ac001c9c3e2418d7aeb38651efd3ecfe6e40c2bf1a5120b70c0caab4f33e"}
Jul 02 12:47:39.372940 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:39.372923    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-69b5d94596-2d254"
Jul 02 12:47:39.508313 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:39.508263    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-69b5d94596-wdwkk"
Jul 02 12:47:39.513358 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:39.513339    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5be30942-6dd5-4842-89fd-d90997e930ec-cert\") pod \"5be30942-6dd5-4842-89fd-d90997e930ec\" (UID: \"5be30942-6dd5-4842-89fd-d90997e930ec\") "
Jul 02 12:47:39.513434 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:39.513393    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lc4m\" (UniqueName: \"kubernetes.io/projected/5be30942-6dd5-4842-89fd-d90997e930ec-kube-api-access-5lc4m\") pod \"5be30942-6dd5-4842-89fd-d90997e930ec\" (UID: \"5be30942-6dd5-4842-89fd-d90997e930ec\") "
Jul 02 12:47:39.515352 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:39.515331    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5be30942-6dd5-4842-89fd-d90997e930ec-cert" (OuterVolumeSpecName: "cert") pod "5be30942-6dd5-4842-89fd-d90997e930ec" (UID: "5be30942-6dd5-4842-89fd-d90997e930ec"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:47:39.515454 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:39.515431    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be30942-6dd5-4842-89fd-d90997e930ec-kube-api-access-5lc4m" (OuterVolumeSpecName: "kube-api-access-5lc4m") pod "5be30942-6dd5-4842-89fd-d90997e930ec" (UID: "5be30942-6dd5-4842-89fd-d90997e930ec"). InnerVolumeSpecName "kube-api-access-5lc4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 12:47:39.614713 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:39.614686    2578 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5be30942-6dd5-4842-89fd-d90997e930ec-cert\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 12:47:39.614713 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:39.614712    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lc4m\" (UniqueName: \"kubernetes.io/projected/5be30942-6dd5-4842-89fd-d90997e930ec-kube-api-access-5lc4m\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 12:47:39.629233 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:39.629208    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-69b5d94596-wdwkk"]
Jul 02 12:47:39.632208 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:47:39.632166    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53050271_cef2_40e4_8f3b_d0b1bb9499ca.slice/crio-4de628375a0b4d69352b3ff646d7279aff8dc8d41d59ae2cb77f0542d6ef104a WatchSource:0}: Error finding container 4de628375a0b4d69352b3ff646d7279aff8dc8d41d59ae2cb77f0542d6ef104a: Status 404 returned error can't find the container with id 4de628375a0b4d69352b3ff646d7279aff8dc8d41d59ae2cb77f0542d6ef104a
Jul 02 12:47:40.346381 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:40.346307    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-69b5d94596-2d254"
Jul 02 12:47:40.346381 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:40.346320    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-69b5d94596-2d254" event={"ID":"5be30942-6dd5-4842-89fd-d90997e930ec","Type":"ContainerDied","Data":"75f6276a8863e2c38f79de5ed6db6ab4b5deb347571dbeb3e14bb8bfb0136bf4"}
Jul 02 12:47:40.346381 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:40.346373    2578 scope.go:117] "RemoveContainer" containerID="35d4ac001c9c3e2418d7aeb38651efd3ecfe6e40c2bf1a5120b70c0caab4f33e"
Jul 02 12:47:40.347964 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:40.347935    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-69b5d94596-wdwkk" event={"ID":"53050271-cef2-40e4-8f3b-d0b1bb9499ca","Type":"ContainerStarted","Data":"cb84a6c1a41a6565e096dce3683ef42d3bf4daba34363417062094dca84066eb"}
Jul 02 12:47:40.348064 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:40.347987    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-69b5d94596-wdwkk" event={"ID":"53050271-cef2-40e4-8f3b-d0b1bb9499ca","Type":"ContainerStarted","Data":"4de628375a0b4d69352b3ff646d7279aff8dc8d41d59ae2cb77f0542d6ef104a"}
Jul 02 12:47:40.348064 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:40.348017    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-69b5d94596-wdwkk"
Jul 02 12:47:40.369468 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:40.369432    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-69b5d94596-wdwkk" podStartSLOduration=0.955982623 podStartE2EDuration="1.369419708s" totalImagesPullingTime="413.437085ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:47:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:47:40.368259591 +0000 UTC m=+528.501085463" watchObservedRunningTime="2026-07-02 12:47:40.369419708 +0000 UTC m=+528.502245577"
Jul 02 12:47:40.383502 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:40.383485    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-69b5d94596-2d254"]
Jul 02 12:47:40.386539 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:40.386519    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-69b5d94596-2d254"]
Jul 02 12:47:42.361945 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:47:42.361857    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5be30942-6dd5-4842-89fd-d90997e930ec" path="/var/lib/kubelet/pods/5be30942-6dd5-4842-89fd-d90997e930ec/volumes"
Jul 02 12:48:11.357192 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:48:11.357150    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-69b5d94596-wdwkk"
Jul 02 12:48:12.251903 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:48:12.251865    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:48:12.401411 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:48:12.401376    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:48:12.570052 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:48:12.569968    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:48:12.672201 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:48:12.672148    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:48:13.062399 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:48:13.062371    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:48:13.133867 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:48:13.133835    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:48:52.327264 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:48:52.327240    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95kwm_495e9fa8-b93f-42e0-95b5-813527df0993/ovn-acl-logging/0.log"
Jul 02 12:48:52.331656 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:48:52.331637    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95kwm_495e9fa8-b93f-42e0-95b5-813527df0993/ovn-acl-logging/0.log"
Jul 02 12:50:01.660403 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:01.660369    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:50:01.747277 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:01.747244    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:50:12.511831 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:12.511794    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:50:12.701845 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:12.701809    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:50:12.705479 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:12.705454    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq"]
Jul 02 12:50:12.705779 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:12.705762    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5be30942-6dd5-4842-89fd-d90997e930ec" containerName="manager"
Jul 02 12:50:12.705837 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:12.705780    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5be30942-6dd5-4842-89fd-d90997e930ec" containerName="manager"
Jul 02 12:50:12.705837 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:12.705831    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5be30942-6dd5-4842-89fd-d90997e930ec" containerName="manager"
Jul 02 12:50:12.707610 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:12.707595    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq"
Jul 02 12:50:12.711194 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:12.711161    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-75b89-predictor-serving-cert\""
Jul 02 12:50:12.711454 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:12.711436    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-p74sx\""
Jul 02 12:50:12.711575 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:12.711440    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-75b89-kube-rbac-proxy-sar-config\""
Jul 02 12:50:12.712396 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:12.712379    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 12:50:12.712483 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:12.712405    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 12:50:12.719461 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:12.719443    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq"]
Jul 02 12:50:12.757761 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:12.757684    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5e22202-c96f-4bdd-81e2-8ea4bed2503e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq\" (UID: \"e5e22202-c96f-4bdd-81e2-8ea4bed2503e\") " pod="kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq"
Jul 02 12:50:12.757761 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:12.757757    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhcb6\" (UniqueName: \"kubernetes.io/projected/e5e22202-c96f-4bdd-81e2-8ea4bed2503e-kube-api-access-qhcb6\") pod \"success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq\" (UID: \"e5e22202-c96f-4bdd-81e2-8ea4bed2503e\") " pod="kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq"
Jul 02 12:50:12.757984 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:12.757844    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5e22202-c96f-4bdd-81e2-8ea4bed2503e-proxy-tls\") pod \"success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq\" (UID: \"e5e22202-c96f-4bdd-81e2-8ea4bed2503e\") " pod="kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq"
Jul 02 12:50:12.827052 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:12.826957    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:50:12.858932 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:12.858900    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5e22202-c96f-4bdd-81e2-8ea4bed2503e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq\" (UID: \"e5e22202-c96f-4bdd-81e2-8ea4bed2503e\") " pod="kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq"
Jul 02 12:50:12.859089 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:12.858946    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhcb6\" (UniqueName: \"kubernetes.io/projected/e5e22202-c96f-4bdd-81e2-8ea4bed2503e-kube-api-access-qhcb6\") pod \"success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq\" (UID: \"e5e22202-c96f-4bdd-81e2-8ea4bed2503e\") " pod="kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq"
Jul 02 12:50:12.859089 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:12.859012    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5e22202-c96f-4bdd-81e2-8ea4bed2503e-proxy-tls\") pod \"success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq\" (UID: \"e5e22202-c96f-4bdd-81e2-8ea4bed2503e\") " pod="kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq"
Jul 02 12:50:12.859250 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:50:12.859130    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-75b89-predictor-serving-cert: secret "success-200-isvc-75b89-predictor-serving-cert" not found
Jul 02 12:50:12.859250 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:50:12.859222    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5e22202-c96f-4bdd-81e2-8ea4bed2503e-proxy-tls podName:e5e22202-c96f-4bdd-81e2-8ea4bed2503e nodeName:}" failed. No retries permitted until 2026-07-02 12:50:13.359202248 +0000 UTC m=+681.492028091 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e5e22202-c96f-4bdd-81e2-8ea4bed2503e-proxy-tls") pod "success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq" (UID: "e5e22202-c96f-4bdd-81e2-8ea4bed2503e") : secret "success-200-isvc-75b89-predictor-serving-cert" not found
Jul 02 12:50:12.860100 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:12.860076    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5e22202-c96f-4bdd-81e2-8ea4bed2503e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq\" (UID: \"e5e22202-c96f-4bdd-81e2-8ea4bed2503e\") " pod="kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq"
Jul 02 12:50:12.869994 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:12.869970    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhcb6\" (UniqueName: \"kubernetes.io/projected/e5e22202-c96f-4bdd-81e2-8ea4bed2503e-kube-api-access-qhcb6\") pod \"success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq\" (UID: \"e5e22202-c96f-4bdd-81e2-8ea4bed2503e\") " pod="kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq"
Jul 02 12:50:12.904949 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:12.904924    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw"]
Jul 02 12:50:12.907415 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:12.907396    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw"
Jul 02 12:50:12.909100 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:12.909078    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:50:12.910246 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:12.910228    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-75b89-predictor-serving-cert\""
Jul 02 12:50:12.910333 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:12.910252    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-75b89-kube-rbac-proxy-sar-config\""
Jul 02 12:50:12.920688 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:12.920666    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw"]
Jul 02 12:50:12.960363 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:12.960333    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7812dc6f-e3a1-42af-ba09-068b9ceed016-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw\" (UID: \"7812dc6f-e3a1-42af-ba09-068b9ceed016\") " pod="kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw"
Jul 02 12:50:12.960471 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:12.960401    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58vjr\" (UniqueName: \"kubernetes.io/projected/7812dc6f-e3a1-42af-ba09-068b9ceed016-kube-api-access-58vjr\") pod \"error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw\" (UID: \"7812dc6f-e3a1-42af-ba09-068b9ceed016\") " pod="kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw"
Jul 02 12:50:12.960471 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:12.960456    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7812dc6f-e3a1-42af-ba09-068b9ceed016-proxy-tls\") pod \"error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw\" (UID: \"7812dc6f-e3a1-42af-ba09-068b9ceed016\") " pod="kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw"
Jul 02 12:50:12.997213 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:12.997166    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:50:13.061364 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:13.061313    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7812dc6f-e3a1-42af-ba09-068b9ceed016-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw\" (UID: \"7812dc6f-e3a1-42af-ba09-068b9ceed016\") " pod="kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw"
Jul 02 12:50:13.061498 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:13.061466    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58vjr\" (UniqueName: \"kubernetes.io/projected/7812dc6f-e3a1-42af-ba09-068b9ceed016-kube-api-access-58vjr\") pod \"error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw\" (UID: \"7812dc6f-e3a1-42af-ba09-068b9ceed016\") " pod="kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw"
Jul 02 12:50:13.061563 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:13.061509    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7812dc6f-e3a1-42af-ba09-068b9ceed016-proxy-tls\") pod \"error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw\" (UID: \"7812dc6f-e3a1-42af-ba09-068b9ceed016\") " pod="kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw"
Jul 02 12:50:13.061982 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:13.061961    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7812dc6f-e3a1-42af-ba09-068b9ceed016-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw\" (UID: \"7812dc6f-e3a1-42af-ba09-068b9ceed016\") " pod="kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw"
Jul 02 12:50:13.064099 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:13.064080    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7812dc6f-e3a1-42af-ba09-068b9ceed016-proxy-tls\") pod \"error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw\" (UID: \"7812dc6f-e3a1-42af-ba09-068b9ceed016\") " pod="kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw"
Jul 02 12:50:13.070849 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:13.070826    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58vjr\" (UniqueName: \"kubernetes.io/projected/7812dc6f-e3a1-42af-ba09-068b9ceed016-kube-api-access-58vjr\") pod \"error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw\" (UID: \"7812dc6f-e3a1-42af-ba09-068b9ceed016\") " pod="kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw"
Jul 02 12:50:13.103407 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:13.103344    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:50:13.197750 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:13.197719    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:50:13.218602 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:13.218568    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw"
Jul 02 12:50:13.346399 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:13.346375    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw"]
Jul 02 12:50:13.348738 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:50:13.348711    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7812dc6f_e3a1_42af_ba09_068b9ceed016.slice/crio-75d3feb3d34dd57b4a095779fa4dc322e8a86c9ecbe123657728cc0568b0a85c WatchSource:0}: Error finding container 75d3feb3d34dd57b4a095779fa4dc322e8a86c9ecbe123657728cc0568b0a85c: Status 404 returned error can't find the container with id 75d3feb3d34dd57b4a095779fa4dc322e8a86c9ecbe123657728cc0568b0a85c
Jul 02 12:50:13.363787 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:13.363764    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5e22202-c96f-4bdd-81e2-8ea4bed2503e-proxy-tls\") pod \"success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq\" (UID: \"e5e22202-c96f-4bdd-81e2-8ea4bed2503e\") " pod="kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq"
Jul 02 12:50:13.365916 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:13.365895    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5e22202-c96f-4bdd-81e2-8ea4bed2503e-proxy-tls\") pod \"success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq\" (UID: \"e5e22202-c96f-4bdd-81e2-8ea4bed2503e\") " pod="kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq"
Jul 02 12:50:13.618519 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:13.618447    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq"
Jul 02 12:50:13.745270 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:13.745242    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq"]
Jul 02 12:50:13.747871 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:50:13.747844    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e22202_c96f_4bdd_81e2_8ea4bed2503e.slice/crio-525692a0c2921a566f1d8657ff31a920036b35e7d9bfd7cd922b57b7114e8be8 WatchSource:0}: Error finding container 525692a0c2921a566f1d8657ff31a920036b35e7d9bfd7cd922b57b7114e8be8: Status 404 returned error can't find the container with id 525692a0c2921a566f1d8657ff31a920036b35e7d9bfd7cd922b57b7114e8be8
Jul 02 12:50:13.888274 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:13.888129    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw" event={"ID":"7812dc6f-e3a1-42af-ba09-068b9ceed016","Type":"ContainerStarted","Data":"75d3feb3d34dd57b4a095779fa4dc322e8a86c9ecbe123657728cc0568b0a85c"}
Jul 02 12:50:13.890092 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:13.890065    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq" event={"ID":"e5e22202-c96f-4bdd-81e2-8ea4bed2503e","Type":"ContainerStarted","Data":"525692a0c2921a566f1d8657ff31a920036b35e7d9bfd7cd922b57b7114e8be8"}
Jul 02 12:50:29.979250 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:29.979212    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw" event={"ID":"7812dc6f-e3a1-42af-ba09-068b9ceed016","Type":"ContainerStarted","Data":"585596caa545fa282e96a3a83cab723655a70cb072e4c1dceaebf8f4327fbbc6"}
Jul 02 12:50:29.981549 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:29.981524    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq" event={"ID":"e5e22202-c96f-4bdd-81e2-8ea4bed2503e","Type":"ContainerStarted","Data":"850579f6338eddbc6527f06d1e27d04a2e81d9f55a0a4e880b0b675b51ba8a82"}
Jul 02 12:50:31.992073 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:31.992040    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq" event={"ID":"e5e22202-c96f-4bdd-81e2-8ea4bed2503e","Type":"ContainerStarted","Data":"3a53bd50ab4efdcc6c1f2b4b36dc9013a074413ccc649a7910182ed2d94f3924"}
Jul 02 12:50:31.992487 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:31.992215    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq"
Jul 02 12:50:31.992487 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:31.992276    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq"
Jul 02 12:50:31.993741 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:31.993713    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq" podUID="e5e22202-c96f-4bdd-81e2-8ea4bed2503e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 02 12:50:31.993939 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:31.993910    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw" event={"ID":"7812dc6f-e3a1-42af-ba09-068b9ceed016","Type":"ContainerStarted","Data":"ece9d32b153443b4aad755fda6769bcd26355b0cf7cd5a61f7a590ffa8d730ab"}
Jul 02 12:50:31.994068 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:31.994051    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw"
Jul 02 12:50:31.994137 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:31.994092    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw"
Jul 02 12:50:31.994958 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:31.994936    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw" podUID="7812dc6f-e3a1-42af-ba09-068b9ceed016" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 02 12:50:32.010081 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:32.010043    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq" podStartSLOduration=1.9419340040000002 podStartE2EDuration="20.01003122s" totalImagesPullingTime="18.068097216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:50:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:50:32.008970118 +0000 UTC m=+700.141795992" watchObservedRunningTime="2026-07-02 12:50:32.01003122 +0000 UTC m=+700.142857065"
Jul 02 12:50:32.027019 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:32.026977    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw" podStartSLOduration=1.582167832 podStartE2EDuration="20.026964094s" totalImagesPullingTime="18.444796262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:50:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:50:32.026308982 +0000 UTC m=+700.159134828" watchObservedRunningTime="2026-07-02 12:50:32.026964094 +0000 UTC m=+700.159789942"
Jul 02 12:50:32.997489 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:32.997447    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw" podUID="7812dc6f-e3a1-42af-ba09-068b9ceed016" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 02 12:50:32.997856 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:32.997569    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq" podUID="e5e22202-c96f-4bdd-81e2-8ea4bed2503e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 02 12:50:38.001551 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:38.001521    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw"
Jul 02 12:50:38.001973 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:38.001955    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq"
Jul 02 12:50:38.002057 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:38.002028    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw" podUID="7812dc6f-e3a1-42af-ba09-068b9ceed016" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 02 12:50:38.002363 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:38.002338    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq" podUID="e5e22202-c96f-4bdd-81e2-8ea4bed2503e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 02 12:50:48.002699 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:48.002620    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq" podUID="e5e22202-c96f-4bdd-81e2-8ea4bed2503e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 02 12:50:48.002699 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:48.002632    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw" podUID="7812dc6f-e3a1-42af-ba09-068b9ceed016" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 02 12:50:58.002459 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:58.002408    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw" podUID="7812dc6f-e3a1-42af-ba09-068b9ceed016" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 02 12:50:58.002459 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:50:58.002433    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq" podUID="e5e22202-c96f-4bdd-81e2-8ea4bed2503e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 02 12:51:07.126677 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:07.126647    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:51:08.002642 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:08.002589    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw" podUID="7812dc6f-e3a1-42af-ba09-068b9ceed016" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 02 12:51:08.002859 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:08.002681    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq" podUID="e5e22202-c96f-4bdd-81e2-8ea4bed2503e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 02 12:51:18.002361 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:18.002336    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw"
Jul 02 12:51:18.003115 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:18.003097    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq"
Jul 02 12:51:30.687694 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:30.687658    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:51:32.507662 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:32.507622    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:51:32.744778 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:32.744747    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:51:42.649398 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:42.649366    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:51:42.798637 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:42.798592    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:51:42.804973 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:42.804948    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq"]
Jul 02 12:51:42.805245 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:42.805219    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq" podUID="e5e22202-c96f-4bdd-81e2-8ea4bed2503e" containerName="kserve-container" containerID="cri-o://850579f6338eddbc6527f06d1e27d04a2e81d9f55a0a4e880b0b675b51ba8a82" gracePeriod=30
Jul 02 12:51:42.805579 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:42.805305    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq" podUID="e5e22202-c96f-4bdd-81e2-8ea4bed2503e" containerName="kube-rbac-proxy" containerID="cri-o://3a53bd50ab4efdcc6c1f2b4b36dc9013a074413ccc649a7910182ed2d94f3924" gracePeriod=30
Jul 02 12:51:42.816026 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:42.816002    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv"]
Jul 02 12:51:42.819423 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:42.819406    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv"
Jul 02 12:51:42.821966 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:42.821947    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-04e3b-predictor-serving-cert\""
Jul 02 12:51:42.822067 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:42.821980    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-04e3b-kube-rbac-proxy-sar-config\""
Jul 02 12:51:42.829988 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:42.829966    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv"]
Jul 02 12:51:42.906885 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:42.906798    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:51:42.907588 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:42.907562    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92"]
Jul 02 12:51:42.910894 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:42.910871    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92"
Jul 02 12:51:42.913707 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:42.913673    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-04e3b-predictor-serving-cert\""
Jul 02 12:51:42.913820 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:42.913767    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-04e3b-kube-rbac-proxy-sar-config\""
Jul 02 12:51:42.922648 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:42.922626    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92"]
Jul 02 12:51:42.963406 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:42.963379    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw"]
Jul 02 12:51:42.963633 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:42.963603    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw" podUID="7812dc6f-e3a1-42af-ba09-068b9ceed016" containerName="kserve-container" containerID="cri-o://585596caa545fa282e96a3a83cab723655a70cb072e4c1dceaebf8f4327fbbc6" gracePeriod=30
Jul 02 12:51:42.963774 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:42.963635    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw" podUID="7812dc6f-e3a1-42af-ba09-068b9ceed016" containerName="kube-rbac-proxy" containerID="cri-o://ece9d32b153443b4aad755fda6769bcd26355b0cf7cd5a61f7a590ffa8d730ab" gracePeriod=30
Jul 02 12:51:42.974859 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:42.974834    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3-proxy-tls\") pod \"success-200-isvc-04e3b-predictor-7448945865-gfvcv\" (UID: \"7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3\") " pod="kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv"
Jul 02 12:51:42.974978 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:42.974873    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf9f6\" (UniqueName: \"kubernetes.io/projected/f41086ca-3185-4e23-9d33-bb1539172af1-kube-api-access-sf9f6\") pod \"error-404-isvc-04e3b-predictor-5756f97b8-66z92\" (UID: \"f41086ca-3185-4e23-9d33-bb1539172af1\") " pod="kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92"
Jul 02 12:51:42.974978 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:42.974905    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f41086ca-3185-4e23-9d33-bb1539172af1-proxy-tls\") pod \"error-404-isvc-04e3b-predictor-5756f97b8-66z92\" (UID: \"f41086ca-3185-4e23-9d33-bb1539172af1\") " pod="kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92"
Jul 02 12:51:42.974978 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:42.974963    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f41086ca-3185-4e23-9d33-bb1539172af1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-04e3b-predictor-5756f97b8-66z92\" (UID: \"f41086ca-3185-4e23-9d33-bb1539172af1\") " pod="kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92"
Jul 02 12:51:42.975115 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:42.975025    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5f8j\" (UniqueName: \"kubernetes.io/projected/7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3-kube-api-access-m5f8j\") pod \"success-200-isvc-04e3b-predictor-7448945865-gfvcv\" (UID: \"7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3\") " pod="kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv"
Jul 02 12:51:42.975115 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:42.975069    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-04e3b-predictor-7448945865-gfvcv\" (UID: \"7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3\") " pod="kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv"
Jul 02 12:51:42.998344 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:42.998316    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw" podUID="7812dc6f-e3a1-42af-ba09-068b9ceed016" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Jul 02 12:51:42.998344 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:42.998331    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq" podUID="e5e22202-c96f-4bdd-81e2-8ea4bed2503e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Jul 02 12:51:43.027188 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:43.027160    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:51:43.075999 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:43.075970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3-proxy-tls\") pod \"success-200-isvc-04e3b-predictor-7448945865-gfvcv\" (UID: \"7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3\") " pod="kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv"
Jul 02 12:51:43.076141 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:43.076018    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf9f6\" (UniqueName: \"kubernetes.io/projected/f41086ca-3185-4e23-9d33-bb1539172af1-kube-api-access-sf9f6\") pod \"error-404-isvc-04e3b-predictor-5756f97b8-66z92\" (UID: \"f41086ca-3185-4e23-9d33-bb1539172af1\") " pod="kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92"
Jul 02 12:51:43.076141 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:43.076056    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f41086ca-3185-4e23-9d33-bb1539172af1-proxy-tls\") pod \"error-404-isvc-04e3b-predictor-5756f97b8-66z92\" (UID: \"f41086ca-3185-4e23-9d33-bb1539172af1\") " pod="kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92"
Jul 02 12:51:43.076272 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:43.076234    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f41086ca-3185-4e23-9d33-bb1539172af1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-04e3b-predictor-5756f97b8-66z92\" (UID: \"f41086ca-3185-4e23-9d33-bb1539172af1\") " pod="kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92"
Jul 02 12:51:43.076333 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:43.076300    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5f8j\" (UniqueName: \"kubernetes.io/projected/7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3-kube-api-access-m5f8j\") pod \"success-200-isvc-04e3b-predictor-7448945865-gfvcv\" (UID: \"7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3\") " pod="kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv"
Jul 02 12:51:43.076391 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:43.076334    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-04e3b-predictor-7448945865-gfvcv\" (UID: \"7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3\") " pod="kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv"
Jul 02 12:51:43.077080 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:43.077052    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-04e3b-predictor-7448945865-gfvcv\" (UID: \"7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3\") " pod="kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv"
Jul 02 12:51:43.077210 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:43.077052    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f41086ca-3185-4e23-9d33-bb1539172af1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-04e3b-predictor-5756f97b8-66z92\" (UID: \"f41086ca-3185-4e23-9d33-bb1539172af1\") " pod="kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92"
Jul 02 12:51:43.078475 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:43.078449    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f41086ca-3185-4e23-9d33-bb1539172af1-proxy-tls\") pod \"error-404-isvc-04e3b-predictor-5756f97b8-66z92\" (UID: \"f41086ca-3185-4e23-9d33-bb1539172af1\") " pod="kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92"
Jul 02 12:51:43.078573 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:43.078526    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3-proxy-tls\") pod \"success-200-isvc-04e3b-predictor-7448945865-gfvcv\" (UID: \"7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3\") " pod="kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv"
Jul 02 12:51:43.085792 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:43.085766    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf9f6\" (UniqueName: \"kubernetes.io/projected/f41086ca-3185-4e23-9d33-bb1539172af1-kube-api-access-sf9f6\") pod \"error-404-isvc-04e3b-predictor-5756f97b8-66z92\" (UID: \"f41086ca-3185-4e23-9d33-bb1539172af1\") " pod="kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92"
Jul 02 12:51:43.086335 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:43.086317    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5f8j\" (UniqueName: \"kubernetes.io/projected/7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3-kube-api-access-m5f8j\") pod \"success-200-isvc-04e3b-predictor-7448945865-gfvcv\" (UID: \"7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3\") " pod="kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv"
Jul 02 12:51:43.127187 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:43.127157    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:51:43.130061 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:43.130042    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv"
Jul 02 12:51:43.225050 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:43.224835    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92"
Jul 02 12:51:43.256605 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:43.256578    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv"]
Jul 02 12:51:43.258437 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:51:43.258405    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ea27268_cfef_4c78_a7e0_7b6a57bc0ee3.slice/crio-ec42ede6fbc9b16b775212768d17c1fe39e94e172b1c1169c78d22bb9f962d17 WatchSource:0}: Error finding container ec42ede6fbc9b16b775212768d17c1fe39e94e172b1c1169c78d22bb9f962d17: Status 404 returned error can't find the container with id ec42ede6fbc9b16b775212768d17c1fe39e94e172b1c1169c78d22bb9f962d17
Jul 02 12:51:43.267982 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:43.267958    2578 generic.go:356] "Generic (PLEG): container finished" podID="7812dc6f-e3a1-42af-ba09-068b9ceed016" containerID="ece9d32b153443b4aad755fda6769bcd26355b0cf7cd5a61f7a590ffa8d730ab" exitCode=2
Jul 02 12:51:43.268057 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:43.268010    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw" event={"ID":"7812dc6f-e3a1-42af-ba09-068b9ceed016","Type":"ContainerDied","Data":"ece9d32b153443b4aad755fda6769bcd26355b0cf7cd5a61f7a590ffa8d730ab"}
Jul 02 12:51:43.269303 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:43.269280    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv" event={"ID":"7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3","Type":"ContainerStarted","Data":"ec42ede6fbc9b16b775212768d17c1fe39e94e172b1c1169c78d22bb9f962d17"}
Jul 02 12:51:43.271192 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:43.271154    2578 generic.go:356] "Generic (PLEG): container finished" podID="e5e22202-c96f-4bdd-81e2-8ea4bed2503e" containerID="3a53bd50ab4efdcc6c1f2b4b36dc9013a074413ccc649a7910182ed2d94f3924" exitCode=2
Jul 02 12:51:43.271288 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:43.271216    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq" event={"ID":"e5e22202-c96f-4bdd-81e2-8ea4bed2503e","Type":"ContainerDied","Data":"3a53bd50ab4efdcc6c1f2b4b36dc9013a074413ccc649a7910182ed2d94f3924"}
Jul 02 12:51:43.362883 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:43.362861    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92"]
Jul 02 12:51:43.365648 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:51:43.365625    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf41086ca_3185_4e23_9d33_bb1539172af1.slice/crio-f90387409c0fa589fb6ad41d49862b96807c3b2ee1776e194b6ae8a5f14e4e97 WatchSource:0}: Error finding container f90387409c0fa589fb6ad41d49862b96807c3b2ee1776e194b6ae8a5f14e4e97: Status 404 returned error can't find the container with id f90387409c0fa589fb6ad41d49862b96807c3b2ee1776e194b6ae8a5f14e4e97
Jul 02 12:51:44.276465 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:44.276433    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv" event={"ID":"7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3","Type":"ContainerStarted","Data":"cee9d1aadf3aa6ebcdf341b90015c24bba9f3eb45ca2c31fa43dd30e5a92d609"}
Jul 02 12:51:44.276888 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:44.276485    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv" event={"ID":"7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3","Type":"ContainerStarted","Data":"638559a25388dc950fa3e2f393c54ea7b286a029c3db4ceae5b4de053b264aa6"}
Jul 02 12:51:44.276888 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:44.276515    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv"
Jul 02 12:51:44.276888 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:44.276670    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv"
Jul 02 12:51:44.277628 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:44.277601    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv" podUID="7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 02 12:51:44.278340 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:44.278320    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92" event={"ID":"f41086ca-3185-4e23-9d33-bb1539172af1","Type":"ContainerStarted","Data":"4ed2108c51ac72ca86683168bac33b647fbdc97f04fce19833f01b6860479b41"}
Jul 02 12:51:44.278454 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:44.278360    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92" event={"ID":"f41086ca-3185-4e23-9d33-bb1539172af1","Type":"ContainerStarted","Data":"76b29931d5942c67d0692559f3687b9c1ca89f75cae7edb9e98d0f63931bfde4"}
Jul 02 12:51:44.278454 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:44.278380    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92" event={"ID":"f41086ca-3185-4e23-9d33-bb1539172af1","Type":"ContainerStarted","Data":"f90387409c0fa589fb6ad41d49862b96807c3b2ee1776e194b6ae8a5f14e4e97"}
Jul 02 12:51:44.278454 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:44.278401    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92"
Jul 02 12:51:44.278573 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:44.278459    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92"
Jul 02 12:51:44.279372 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:44.279353    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92" podUID="f41086ca-3185-4e23-9d33-bb1539172af1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 02 12:51:44.297570 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:44.297534    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv" podStartSLOduration=2.297521372 podStartE2EDuration="2.297521372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:51:44.296017863 +0000 UTC m=+772.428843719" watchObservedRunningTime="2026-07-02 12:51:44.297521372 +0000 UTC m=+772.430347218"
Jul 02 12:51:44.315167 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:44.315130    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92" podStartSLOduration=2.3151191349999998 podStartE2EDuration="2.315119135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:51:44.31416735 +0000 UTC m=+772.446993195" watchObservedRunningTime="2026-07-02 12:51:44.315119135 +0000 UTC m=+772.447944981"
Jul 02 12:51:45.282157 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:45.282113    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv" podUID="7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 02 12:51:45.282628 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:45.282220    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92" podUID="f41086ca-3185-4e23-9d33-bb1539172af1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 02 12:51:45.963282 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:45.963252    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq"
Jul 02 12:51:46.007618 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.007591    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw"
Jul 02 12:51:46.097935 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.097905    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7812dc6f-e3a1-42af-ba09-068b9ceed016-kube-rbac-proxy-sar-config\") pod \"7812dc6f-e3a1-42af-ba09-068b9ceed016\" (UID: \"7812dc6f-e3a1-42af-ba09-068b9ceed016\") "
Jul 02 12:51:46.097935 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.097935    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7812dc6f-e3a1-42af-ba09-068b9ceed016-proxy-tls\") pod \"7812dc6f-e3a1-42af-ba09-068b9ceed016\" (UID: \"7812dc6f-e3a1-42af-ba09-068b9ceed016\") "
Jul 02 12:51:46.098134 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.097957    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5e22202-c96f-4bdd-81e2-8ea4bed2503e-proxy-tls\") pod \"e5e22202-c96f-4bdd-81e2-8ea4bed2503e\" (UID: \"e5e22202-c96f-4bdd-81e2-8ea4bed2503e\") "
Jul 02 12:51:46.098134 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.097972    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58vjr\" (UniqueName: \"kubernetes.io/projected/7812dc6f-e3a1-42af-ba09-068b9ceed016-kube-api-access-58vjr\") pod \"7812dc6f-e3a1-42af-ba09-068b9ceed016\" (UID: \"7812dc6f-e3a1-42af-ba09-068b9ceed016\") "
Jul 02 12:51:46.098134 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.098023    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhcb6\" (UniqueName: \"kubernetes.io/projected/e5e22202-c96f-4bdd-81e2-8ea4bed2503e-kube-api-access-qhcb6\") pod \"e5e22202-c96f-4bdd-81e2-8ea4bed2503e\" (UID: \"e5e22202-c96f-4bdd-81e2-8ea4bed2503e\") "
Jul 02 12:51:46.098134 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.098091    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5e22202-c96f-4bdd-81e2-8ea4bed2503e-kube-rbac-proxy-sar-config\") pod \"e5e22202-c96f-4bdd-81e2-8ea4bed2503e\" (UID: \"e5e22202-c96f-4bdd-81e2-8ea4bed2503e\") "
Jul 02 12:51:46.098427 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.098363    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7812dc6f-e3a1-42af-ba09-068b9ceed016-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7812dc6f-e3a1-42af-ba09-068b9ceed016" (UID: "7812dc6f-e3a1-42af-ba09-068b9ceed016"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:51:46.098523 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.098475    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5e22202-c96f-4bdd-81e2-8ea4bed2503e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e5e22202-c96f-4bdd-81e2-8ea4bed2503e" (UID: "e5e22202-c96f-4bdd-81e2-8ea4bed2503e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:51:46.100821 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.100796    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5e22202-c96f-4bdd-81e2-8ea4bed2503e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e5e22202-c96f-4bdd-81e2-8ea4bed2503e" (UID: "e5e22202-c96f-4bdd-81e2-8ea4bed2503e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:51:46.100908 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.100832    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7812dc6f-e3a1-42af-ba09-068b9ceed016-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7812dc6f-e3a1-42af-ba09-068b9ceed016" (UID: "7812dc6f-e3a1-42af-ba09-068b9ceed016"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:51:46.100908 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.100897    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5e22202-c96f-4bdd-81e2-8ea4bed2503e-kube-api-access-qhcb6" (OuterVolumeSpecName: "kube-api-access-qhcb6") pod "e5e22202-c96f-4bdd-81e2-8ea4bed2503e" (UID: "e5e22202-c96f-4bdd-81e2-8ea4bed2503e"). InnerVolumeSpecName "kube-api-access-qhcb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 12:51:46.100989 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.100920    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7812dc6f-e3a1-42af-ba09-068b9ceed016-kube-api-access-58vjr" (OuterVolumeSpecName: "kube-api-access-58vjr") pod "7812dc6f-e3a1-42af-ba09-068b9ceed016" (UID: "7812dc6f-e3a1-42af-ba09-068b9ceed016"). InnerVolumeSpecName "kube-api-access-58vjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 12:51:46.199448 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.199421    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5e22202-c96f-4bdd-81e2-8ea4bed2503e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 12:51:46.199448 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.199445    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7812dc6f-e3a1-42af-ba09-068b9ceed016-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 12:51:46.199587 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.199454    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7812dc6f-e3a1-42af-ba09-068b9ceed016-proxy-tls\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 12:51:46.199587 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.199463    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5e22202-c96f-4bdd-81e2-8ea4bed2503e-proxy-tls\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 12:51:46.199587 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.199471    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58vjr\" (UniqueName: \"kubernetes.io/projected/7812dc6f-e3a1-42af-ba09-068b9ceed016-kube-api-access-58vjr\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 12:51:46.199587 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.199480    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhcb6\" (UniqueName: \"kubernetes.io/projected/e5e22202-c96f-4bdd-81e2-8ea4bed2503e-kube-api-access-qhcb6\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 12:51:46.287576 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.287493    2578 generic.go:356] "Generic (PLEG): container finished" podID="e5e22202-c96f-4bdd-81e2-8ea4bed2503e" containerID="850579f6338eddbc6527f06d1e27d04a2e81d9f55a0a4e880b0b675b51ba8a82" exitCode=0
Jul 02 12:51:46.287576 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.287553    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq"
Jul 02 12:51:46.288061 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.287576    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq" event={"ID":"e5e22202-c96f-4bdd-81e2-8ea4bed2503e","Type":"ContainerDied","Data":"850579f6338eddbc6527f06d1e27d04a2e81d9f55a0a4e880b0b675b51ba8a82"}
Jul 02 12:51:46.288061 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.287645    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq" event={"ID":"e5e22202-c96f-4bdd-81e2-8ea4bed2503e","Type":"ContainerDied","Data":"525692a0c2921a566f1d8657ff31a920036b35e7d9bfd7cd922b57b7114e8be8"}
Jul 02 12:51:46.288061 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.287676    2578 scope.go:117] "RemoveContainer" containerID="3a53bd50ab4efdcc6c1f2b4b36dc9013a074413ccc649a7910182ed2d94f3924"
Jul 02 12:51:46.289077 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.289052    2578 generic.go:356] "Generic (PLEG): container finished" podID="7812dc6f-e3a1-42af-ba09-068b9ceed016" containerID="585596caa545fa282e96a3a83cab723655a70cb072e4c1dceaebf8f4327fbbc6" exitCode=0
Jul 02 12:51:46.289229 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.289084    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw" event={"ID":"7812dc6f-e3a1-42af-ba09-068b9ceed016","Type":"ContainerDied","Data":"585596caa545fa282e96a3a83cab723655a70cb072e4c1dceaebf8f4327fbbc6"}
Jul 02 12:51:46.289229 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.289101    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw"
Jul 02 12:51:46.289229 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.289135    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw" event={"ID":"7812dc6f-e3a1-42af-ba09-068b9ceed016","Type":"ContainerDied","Data":"75d3feb3d34dd57b4a095779fa4dc322e8a86c9ecbe123657728cc0568b0a85c"}
Jul 02 12:51:46.296282 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.296266    2578 scope.go:117] "RemoveContainer" containerID="850579f6338eddbc6527f06d1e27d04a2e81d9f55a0a4e880b0b675b51ba8a82"
Jul 02 12:51:46.303698 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.303672    2578 scope.go:117] "RemoveContainer" containerID="3a53bd50ab4efdcc6c1f2b4b36dc9013a074413ccc649a7910182ed2d94f3924"
Jul 02 12:51:46.303929 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:51:46.303911    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a53bd50ab4efdcc6c1f2b4b36dc9013a074413ccc649a7910182ed2d94f3924\": container with ID starting with 3a53bd50ab4efdcc6c1f2b4b36dc9013a074413ccc649a7910182ed2d94f3924 not found: ID does not exist" containerID="3a53bd50ab4efdcc6c1f2b4b36dc9013a074413ccc649a7910182ed2d94f3924"
Jul 02 12:51:46.303977 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.303937    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a53bd50ab4efdcc6c1f2b4b36dc9013a074413ccc649a7910182ed2d94f3924"} err="failed to get container status \"3a53bd50ab4efdcc6c1f2b4b36dc9013a074413ccc649a7910182ed2d94f3924\": rpc error: code = NotFound desc = could not find container \"3a53bd50ab4efdcc6c1f2b4b36dc9013a074413ccc649a7910182ed2d94f3924\": container with ID starting with 3a53bd50ab4efdcc6c1f2b4b36dc9013a074413ccc649a7910182ed2d94f3924 not found: ID does not exist"
Jul 02 12:51:46.303977 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.303968    2578 scope.go:117] "RemoveContainer" containerID="850579f6338eddbc6527f06d1e27d04a2e81d9f55a0a4e880b0b675b51ba8a82"
Jul 02 12:51:46.304230 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:51:46.304193    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"850579f6338eddbc6527f06d1e27d04a2e81d9f55a0a4e880b0b675b51ba8a82\": container with ID starting with 850579f6338eddbc6527f06d1e27d04a2e81d9f55a0a4e880b0b675b51ba8a82 not found: ID does not exist" containerID="850579f6338eddbc6527f06d1e27d04a2e81d9f55a0a4e880b0b675b51ba8a82"
Jul 02 12:51:46.304269 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.304238    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"850579f6338eddbc6527f06d1e27d04a2e81d9f55a0a4e880b0b675b51ba8a82"} err="failed to get container status \"850579f6338eddbc6527f06d1e27d04a2e81d9f55a0a4e880b0b675b51ba8a82\": rpc error: code = NotFound desc = could not find container \"850579f6338eddbc6527f06d1e27d04a2e81d9f55a0a4e880b0b675b51ba8a82\": container with ID starting with 850579f6338eddbc6527f06d1e27d04a2e81d9f55a0a4e880b0b675b51ba8a82 not found: ID does not exist"
Jul 02 12:51:46.304269 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.304252    2578 scope.go:117] "RemoveContainer" containerID="ece9d32b153443b4aad755fda6769bcd26355b0cf7cd5a61f7a590ffa8d730ab"
Jul 02 12:51:46.311037 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.311022    2578 scope.go:117] "RemoveContainer" containerID="585596caa545fa282e96a3a83cab723655a70cb072e4c1dceaebf8f4327fbbc6"
Jul 02 12:51:46.314281 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.313223    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw"]
Jul 02 12:51:46.317911 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.317885    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-75b89-predictor-7f5dc87c4-z2qpw"]
Jul 02 12:51:46.320256 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.320239    2578 scope.go:117] "RemoveContainer" containerID="ece9d32b153443b4aad755fda6769bcd26355b0cf7cd5a61f7a590ffa8d730ab"
Jul 02 12:51:46.320541 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:51:46.320523    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ece9d32b153443b4aad755fda6769bcd26355b0cf7cd5a61f7a590ffa8d730ab\": container with ID starting with ece9d32b153443b4aad755fda6769bcd26355b0cf7cd5a61f7a590ffa8d730ab not found: ID does not exist" containerID="ece9d32b153443b4aad755fda6769bcd26355b0cf7cd5a61f7a590ffa8d730ab"
Jul 02 12:51:46.320593 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.320546    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ece9d32b153443b4aad755fda6769bcd26355b0cf7cd5a61f7a590ffa8d730ab"} err="failed to get container status \"ece9d32b153443b4aad755fda6769bcd26355b0cf7cd5a61f7a590ffa8d730ab\": rpc error: code = NotFound desc = could not find container \"ece9d32b153443b4aad755fda6769bcd26355b0cf7cd5a61f7a590ffa8d730ab\": container with ID starting with ece9d32b153443b4aad755fda6769bcd26355b0cf7cd5a61f7a590ffa8d730ab not found: ID does not exist"
Jul 02 12:51:46.320593 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.320562    2578 scope.go:117] "RemoveContainer" containerID="585596caa545fa282e96a3a83cab723655a70cb072e4c1dceaebf8f4327fbbc6"
Jul 02 12:51:46.320784 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:51:46.320769    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"585596caa545fa282e96a3a83cab723655a70cb072e4c1dceaebf8f4327fbbc6\": container with ID starting with 585596caa545fa282e96a3a83cab723655a70cb072e4c1dceaebf8f4327fbbc6 not found: ID does not exist" containerID="585596caa545fa282e96a3a83cab723655a70cb072e4c1dceaebf8f4327fbbc6"
Jul 02 12:51:46.320824 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.320786    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"585596caa545fa282e96a3a83cab723655a70cb072e4c1dceaebf8f4327fbbc6"} err="failed to get container status \"585596caa545fa282e96a3a83cab723655a70cb072e4c1dceaebf8f4327fbbc6\": rpc error: code = NotFound desc = could not find container \"585596caa545fa282e96a3a83cab723655a70cb072e4c1dceaebf8f4327fbbc6\": container with ID starting with 585596caa545fa282e96a3a83cab723655a70cb072e4c1dceaebf8f4327fbbc6 not found: ID does not exist"
Jul 02 12:51:46.329128 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.329108    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq"]
Jul 02 12:51:46.336588 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.336569    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-75b89-predictor-864c8fdb5f-k6tdq"]
Jul 02 12:51:46.362285 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.362264    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7812dc6f-e3a1-42af-ba09-068b9ceed016" path="/var/lib/kubelet/pods/7812dc6f-e3a1-42af-ba09-068b9ceed016/volumes"
Jul 02 12:51:46.362675 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:46.362662    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5e22202-c96f-4bdd-81e2-8ea4bed2503e" path="/var/lib/kubelet/pods/e5e22202-c96f-4bdd-81e2-8ea4bed2503e/volumes"
Jul 02 12:51:50.286357 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:50.286330    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92"
Jul 02 12:51:50.286744 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:50.286624    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv"
Jul 02 12:51:50.286862 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:50.286836    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92" podUID="f41086ca-3185-4e23-9d33-bb1539172af1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 02 12:51:50.287310 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:51:50.287288    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv" podUID="7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 02 12:52:00.287666 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:00.287623    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv" podUID="7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 02 12:52:00.288124 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:00.287623    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92" podUID="f41086ca-3185-4e23-9d33-bb1539172af1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 02 12:52:10.287201 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:10.287143    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92" podUID="f41086ca-3185-4e23-9d33-bb1539172af1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 02 12:52:10.287594 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:10.287320    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv" podUID="7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 02 12:52:12.487466 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:12.487371    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:52:13.080620 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:13.080592    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:52:20.287481 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:20.287441    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92" podUID="f41086ca-3185-4e23-9d33-bb1539172af1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 02 12:52:20.287481 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:20.287442    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv" podUID="7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 02 12:52:22.593003 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.592964    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:52:22.716447 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.716414    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:52:22.728861 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.728810    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm"]
Jul 02 12:52:22.729136 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.729124    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7812dc6f-e3a1-42af-ba09-068b9ceed016" containerName="kserve-container"
Jul 02 12:52:22.729195 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.729138    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7812dc6f-e3a1-42af-ba09-068b9ceed016" containerName="kserve-container"
Jul 02 12:52:22.729195 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.729149    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5e22202-c96f-4bdd-81e2-8ea4bed2503e" containerName="kserve-container"
Jul 02 12:52:22.729195 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.729154    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5e22202-c96f-4bdd-81e2-8ea4bed2503e" containerName="kserve-container"
Jul 02 12:52:22.729195 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.729170    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7812dc6f-e3a1-42af-ba09-068b9ceed016" containerName="kube-rbac-proxy"
Jul 02 12:52:22.729195 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.729191    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7812dc6f-e3a1-42af-ba09-068b9ceed016" containerName="kube-rbac-proxy"
Jul 02 12:52:22.729340 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.729201    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5e22202-c96f-4bdd-81e2-8ea4bed2503e" containerName="kube-rbac-proxy"
Jul 02 12:52:22.729340 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.729208    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5e22202-c96f-4bdd-81e2-8ea4bed2503e" containerName="kube-rbac-proxy"
Jul 02 12:52:22.729340 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.729284    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5e22202-c96f-4bdd-81e2-8ea4bed2503e" containerName="kserve-container"
Jul 02 12:52:22.729340 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.729294    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7812dc6f-e3a1-42af-ba09-068b9ceed016" containerName="kserve-container"
Jul 02 12:52:22.729340 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.729303    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7812dc6f-e3a1-42af-ba09-068b9ceed016" containerName="kube-rbac-proxy"
Jul 02 12:52:22.729340 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.729316    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5e22202-c96f-4bdd-81e2-8ea4bed2503e" containerName="kube-rbac-proxy"
Jul 02 12:52:22.732705 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.732676    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm"
Jul 02 12:52:22.736331 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.735538    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8bebb-predictor-serving-cert\""
Jul 02 12:52:22.736866 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.736599    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8bebb-kube-rbac-proxy-sar-config\""
Jul 02 12:52:22.744135 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.743670    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm"]
Jul 02 12:52:22.775601 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.775579    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/689be393-9525-45ff-93b3-ba50eeaa6dde-proxy-tls\") pod \"success-200-isvc-8bebb-predictor-674985687f-rxfmm\" (UID: \"689be393-9525-45ff-93b3-ba50eeaa6dde\") " pod="kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm"
Jul 02 12:52:22.775727 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.775615    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/689be393-9525-45ff-93b3-ba50eeaa6dde-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8bebb-predictor-674985687f-rxfmm\" (UID: \"689be393-9525-45ff-93b3-ba50eeaa6dde\") " pod="kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm"
Jul 02 12:52:22.775727 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.775660    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8ckw\" (UniqueName: \"kubernetes.io/projected/689be393-9525-45ff-93b3-ba50eeaa6dde-kube-api-access-f8ckw\") pod \"success-200-isvc-8bebb-predictor-674985687f-rxfmm\" (UID: \"689be393-9525-45ff-93b3-ba50eeaa6dde\") " pod="kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm"
Jul 02 12:52:22.802171 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.802145    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:52:22.852868 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.852786    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm"]
Jul 02 12:52:22.856053 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.856037    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm"
Jul 02 12:52:22.859084 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.859054    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8bebb-predictor-serving-cert\""
Jul 02 12:52:22.859348 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.859333    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8bebb-kube-rbac-proxy-sar-config\""
Jul 02 12:52:22.865900 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.865878    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm"]
Jul 02 12:52:22.868947 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.868921    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:52:22.876251 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.876229    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95ade19c-8fea-4214-8d6a-7c5264e03c0e-proxy-tls\") pod \"error-404-isvc-8bebb-predictor-5796bd8b66-hvggm\" (UID: \"95ade19c-8fea-4214-8d6a-7c5264e03c0e\") " pod="kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm"
Jul 02 12:52:22.876370 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.876259    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/689be393-9525-45ff-93b3-ba50eeaa6dde-proxy-tls\") pod \"success-200-isvc-8bebb-predictor-674985687f-rxfmm\" (UID: \"689be393-9525-45ff-93b3-ba50eeaa6dde\") " pod="kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm"
Jul 02 12:52:22.876370 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.876278    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95ade19c-8fea-4214-8d6a-7c5264e03c0e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8bebb-predictor-5796bd8b66-hvggm\" (UID: \"95ade19c-8fea-4214-8d6a-7c5264e03c0e\") " pod="kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm"
Jul 02 12:52:22.876370 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.876306    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/689be393-9525-45ff-93b3-ba50eeaa6dde-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8bebb-predictor-674985687f-rxfmm\" (UID: \"689be393-9525-45ff-93b3-ba50eeaa6dde\") " pod="kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm"
Jul 02 12:52:22.876522 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.876383    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcjb9\" (UniqueName: \"kubernetes.io/projected/95ade19c-8fea-4214-8d6a-7c5264e03c0e-kube-api-access-kcjb9\") pod \"error-404-isvc-8bebb-predictor-5796bd8b66-hvggm\" (UID: \"95ade19c-8fea-4214-8d6a-7c5264e03c0e\") " pod="kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm"
Jul 02 12:52:22.876522 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.876492    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8ckw\" (UniqueName: \"kubernetes.io/projected/689be393-9525-45ff-93b3-ba50eeaa6dde-kube-api-access-f8ckw\") pod \"success-200-isvc-8bebb-predictor-674985687f-rxfmm\" (UID: \"689be393-9525-45ff-93b3-ba50eeaa6dde\") " pod="kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm"
Jul 02 12:52:22.877024 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.877007    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/689be393-9525-45ff-93b3-ba50eeaa6dde-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8bebb-predictor-674985687f-rxfmm\" (UID: \"689be393-9525-45ff-93b3-ba50eeaa6dde\") " pod="kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm"
Jul 02 12:52:22.878745 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.878728    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/689be393-9525-45ff-93b3-ba50eeaa6dde-proxy-tls\") pod \"success-200-isvc-8bebb-predictor-674985687f-rxfmm\" (UID: \"689be393-9525-45ff-93b3-ba50eeaa6dde\") " pod="kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm"
Jul 02 12:52:22.888704 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.888680    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8ckw\" (UniqueName: \"kubernetes.io/projected/689be393-9525-45ff-93b3-ba50eeaa6dde-kube-api-access-f8ckw\") pod \"success-200-isvc-8bebb-predictor-674985687f-rxfmm\" (UID: \"689be393-9525-45ff-93b3-ba50eeaa6dde\") " pod="kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm"
Jul 02 12:52:22.941062 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.941021    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:52:22.977199 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.977151    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcjb9\" (UniqueName: \"kubernetes.io/projected/95ade19c-8fea-4214-8d6a-7c5264e03c0e-kube-api-access-kcjb9\") pod \"error-404-isvc-8bebb-predictor-5796bd8b66-hvggm\" (UID: \"95ade19c-8fea-4214-8d6a-7c5264e03c0e\") " pod="kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm"
Jul 02 12:52:22.977334 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.977235    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95ade19c-8fea-4214-8d6a-7c5264e03c0e-proxy-tls\") pod \"error-404-isvc-8bebb-predictor-5796bd8b66-hvggm\" (UID: \"95ade19c-8fea-4214-8d6a-7c5264e03c0e\") " pod="kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm"
Jul 02 12:52:22.977334 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.977285    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95ade19c-8fea-4214-8d6a-7c5264e03c0e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8bebb-predictor-5796bd8b66-hvggm\" (UID: \"95ade19c-8fea-4214-8d6a-7c5264e03c0e\") " pod="kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm"
Jul 02 12:52:22.977439 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:52:22.977372    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-serving-cert: secret "error-404-isvc-8bebb-predictor-serving-cert" not found
Jul 02 12:52:22.977487 ip-10-0-137-127 kubenswrapper[2578]: E0702 12:52:22.977475    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95ade19c-8fea-4214-8d6a-7c5264e03c0e-proxy-tls podName:95ade19c-8fea-4214-8d6a-7c5264e03c0e nodeName:}" failed. No retries permitted until 2026-07-02 12:52:23.477442057 +0000 UTC m=+811.610267881 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/95ade19c-8fea-4214-8d6a-7c5264e03c0e-proxy-tls") pod "error-404-isvc-8bebb-predictor-5796bd8b66-hvggm" (UID: "95ade19c-8fea-4214-8d6a-7c5264e03c0e") : secret "error-404-isvc-8bebb-predictor-serving-cert" not found
Jul 02 12:52:22.977851 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.977834    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95ade19c-8fea-4214-8d6a-7c5264e03c0e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8bebb-predictor-5796bd8b66-hvggm\" (UID: \"95ade19c-8fea-4214-8d6a-7c5264e03c0e\") " pod="kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm"
Jul 02 12:52:22.989740 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:22.989716    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcjb9\" (UniqueName: \"kubernetes.io/projected/95ade19c-8fea-4214-8d6a-7c5264e03c0e-kube-api-access-kcjb9\") pod \"error-404-isvc-8bebb-predictor-5796bd8b66-hvggm\" (UID: \"95ade19c-8fea-4214-8d6a-7c5264e03c0e\") " pod="kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm"
Jul 02 12:52:23.049237 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:23.049165    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm"
Jul 02 12:52:23.074569 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:23.074535    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:52:23.186630 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:23.186605    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm"]
Jul 02 12:52:23.188309 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:52:23.188284    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod689be393_9525_45ff_93b3_ba50eeaa6dde.slice/crio-4d300dc7c025a2c98cde17c0b979902529714b48eadea00ae21067ccd9a3bebc WatchSource:0}: Error finding container 4d300dc7c025a2c98cde17c0b979902529714b48eadea00ae21067ccd9a3bebc: Status 404 returned error can't find the container with id 4d300dc7c025a2c98cde17c0b979902529714b48eadea00ae21067ccd9a3bebc
Jul 02 12:52:23.427479 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:23.427395    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm" event={"ID":"689be393-9525-45ff-93b3-ba50eeaa6dde","Type":"ContainerStarted","Data":"b6545602a0fabb0ab686261da092016abcbcb3afa8b9db107cb46dc72afeb7dd"}
Jul 02 12:52:23.427630 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:23.427502    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm"
Jul 02 12:52:23.427630 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:23.427540    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm" event={"ID":"689be393-9525-45ff-93b3-ba50eeaa6dde","Type":"ContainerStarted","Data":"65cdbc7f4daee456dcc6008a75f3a7523a1807a7e0615927df79b3a02e066eb7"}
Jul 02 12:52:23.427630 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:23.427570    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm" event={"ID":"689be393-9525-45ff-93b3-ba50eeaa6dde","Type":"ContainerStarted","Data":"4d300dc7c025a2c98cde17c0b979902529714b48eadea00ae21067ccd9a3bebc"}
Jul 02 12:52:23.427630 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:23.427598    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm"
Jul 02 12:52:23.429202 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:23.429129    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm" podUID="689be393-9525-45ff-93b3-ba50eeaa6dde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 02 12:52:23.447070 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:23.447020    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm" podStartSLOduration=1.447002919 podStartE2EDuration="1.447002919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:52:23.446704049 +0000 UTC m=+811.579529895" watchObservedRunningTime="2026-07-02 12:52:23.447002919 +0000 UTC m=+811.579828768"
Jul 02 12:52:23.484405 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:23.484379    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95ade19c-8fea-4214-8d6a-7c5264e03c0e-proxy-tls\") pod \"error-404-isvc-8bebb-predictor-5796bd8b66-hvggm\" (UID: \"95ade19c-8fea-4214-8d6a-7c5264e03c0e\") " pod="kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm"
Jul 02 12:52:23.486614 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:23.486587    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95ade19c-8fea-4214-8d6a-7c5264e03c0e-proxy-tls\") pod \"error-404-isvc-8bebb-predictor-5796bd8b66-hvggm\" (UID: \"95ade19c-8fea-4214-8d6a-7c5264e03c0e\") " pod="kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm"
Jul 02 12:52:23.766615 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:23.766580    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm"
Jul 02 12:52:23.912578 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:23.912541    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm"]
Jul 02 12:52:23.913277 ip-10-0-137-127 kubenswrapper[2578]: W0702 12:52:23.913240    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95ade19c_8fea_4214_8d6a_7c5264e03c0e.slice/crio-7d2d080bee57d61c54d9bbf3f0367639f2d0746c5ad976ca00fd3e40219f0333 WatchSource:0}: Error finding container 7d2d080bee57d61c54d9bbf3f0367639f2d0746c5ad976ca00fd3e40219f0333: Status 404 returned error can't find the container with id 7d2d080bee57d61c54d9bbf3f0367639f2d0746c5ad976ca00fd3e40219f0333
Jul 02 12:52:24.436305 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:24.436199    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm" event={"ID":"95ade19c-8fea-4214-8d6a-7c5264e03c0e","Type":"ContainerStarted","Data":"a1307c8f79945de84e5625bb23e6972497f55df0d9b40676dcf838c310bb13ab"}
Jul 02 12:52:24.436473 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:24.436313    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm" event={"ID":"95ade19c-8fea-4214-8d6a-7c5264e03c0e","Type":"ContainerStarted","Data":"5da8ca99ea3959b7e6e8a07750aa6ad861092610da5cb8f15ca11723cef0ecc5"}
Jul 02 12:52:24.436473 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:24.436348    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm" event={"ID":"95ade19c-8fea-4214-8d6a-7c5264e03c0e","Type":"ContainerStarted","Data":"7d2d080bee57d61c54d9bbf3f0367639f2d0746c5ad976ca00fd3e40219f0333"}
Jul 02 12:52:24.436823 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:24.436790    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm" podUID="689be393-9525-45ff-93b3-ba50eeaa6dde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 02 12:52:24.456025 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:24.455978    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm" podStartSLOduration=2.455962334 podStartE2EDuration="2.455962334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:52:24.454519288 +0000 UTC m=+812.587345153" watchObservedRunningTime="2026-07-02 12:52:24.455962334 +0000 UTC m=+812.588788257"
Jul 02 12:52:25.440997 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:25.440971    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm"
Jul 02 12:52:25.441373 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:25.441033    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm"
Jul 02 12:52:25.442069 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:25.442043    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm" podUID="95ade19c-8fea-4214-8d6a-7c5264e03c0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 02 12:52:26.445270 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:26.445234    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm" podUID="95ade19c-8fea-4214-8d6a-7c5264e03c0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 02 12:52:29.440971 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:29.440944    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm"
Jul 02 12:52:29.441390 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:29.441351    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm" podUID="689be393-9525-45ff-93b3-ba50eeaa6dde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 02 12:52:30.287385 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:30.287352    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92"
Jul 02 12:52:30.287815 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:30.287795    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv"
Jul 02 12:52:31.450446 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:31.450417    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm"
Jul 02 12:52:31.450916 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:31.450890    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm" podUID="95ade19c-8fea-4214-8d6a-7c5264e03c0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 02 12:52:39.442109 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:39.442074    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm" podUID="689be393-9525-45ff-93b3-ba50eeaa6dde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 02 12:52:41.451862 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:41.451819    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm" podUID="95ade19c-8fea-4214-8d6a-7c5264e03c0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 02 12:52:42.879366 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:42.879324    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:52:43.029680 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:43.029638    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:52:49.441654 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:49.441615    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm" podUID="689be393-9525-45ff-93b3-ba50eeaa6dde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 02 12:52:51.451459 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:51.451422    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm" podUID="95ade19c-8fea-4214-8d6a-7c5264e03c0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 02 12:52:59.441595 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:52:59.441549    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm" podUID="689be393-9525-45ff-93b3-ba50eeaa6dde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 02 12:53:01.451446 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:53:01.451404    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm" podUID="95ade19c-8fea-4214-8d6a-7c5264e03c0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 02 12:53:09.442340 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:53:09.442310    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm"
Jul 02 12:53:11.452126 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:53:11.452088    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm"
Jul 02 12:53:22.788970 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:53:22.788926    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:53:23.008358 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:53:23.008323    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:53:52.351799 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:53:52.351718    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95kwm_495e9fa8-b93f-42e0-95b5-813527df0993/ovn-acl-logging/0.log"
Jul 02 12:53:52.354606 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:53:52.354580    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95kwm_495e9fa8-b93f-42e0-95b5-813527df0993/ovn-acl-logging/0.log"
Jul 02 12:54:05.458459 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:54:05.458420    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:54:05.533458 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:54:05.533409    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:55:01.559459 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:55:01.559429    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:55:44.414737 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:55:44.414707    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:58:52.382520 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:58:52.382491    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95kwm_495e9fa8-b93f-42e0-95b5-813527df0993/ovn-acl-logging/0.log"
Jul 02 12:58:52.383576 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:58:52.383556    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95kwm_495e9fa8-b93f-42e0-95b5-813527df0993/ovn-acl-logging/0.log"
Jul 02 12:59:03.594114 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:59:03.594081    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 12:59:10.028192 ip-10-0-137-127 kubenswrapper[2578]: I0702 12:59:10.028131    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:00:57.513595 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:57.513558    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:00:57.698377 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:57.698339    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:00:57.771862 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:57.771779    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z"]
Jul 02 13:00:57.775236 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:57.775213    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z"
Jul 02 13:00:57.778405 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:57.778379    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv"]
Jul 02 13:00:57.778653 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:57.778599    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv" podUID="7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3" containerName="kserve-container" containerID="cri-o://638559a25388dc950fa3e2f393c54ea7b286a029c3db4ceae5b4de053b264aa6" gracePeriod=30
Jul 02 13:00:57.778759 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:57.778658    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv" podUID="7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3" containerName="kube-rbac-proxy" containerID="cri-o://cee9d1aadf3aa6ebcdf341b90015c24bba9f3eb45ca2c31fa43dd30e5a92d609" gracePeriod=30
Jul 02 13:00:57.779590 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:57.779381    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c0346-kube-rbac-proxy-sar-config\""
Jul 02 13:00:57.779590 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:57.779415    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c0346-predictor-serving-cert\""
Jul 02 13:00:57.788121 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:57.788095    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z"]
Jul 02 13:00:57.797705 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:57.797682    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:00:57.879164 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:57.879127    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:00:57.879395 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:57.879375    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk"]
Jul 02 13:00:57.882783 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:57.882764    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk"
Jul 02 13:00:57.886003 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:57.885977    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c0346-predictor-serving-cert\""
Jul 02 13:00:57.886003 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:57.885993    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c0346-kube-rbac-proxy-sar-config\""
Jul 02 13:00:57.892868 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:57.892844    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b200e0e-cf4f-450b-8b90-57be599a90d3-proxy-tls\") pod \"success-200-isvc-c0346-predictor-848ccd5d77-kzr6z\" (UID: \"1b200e0e-cf4f-450b-8b90-57be599a90d3\") " pod="kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z"
Jul 02 13:00:57.892996 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:57.892905    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b200e0e-cf4f-450b-8b90-57be599a90d3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c0346-predictor-848ccd5d77-kzr6z\" (UID: \"1b200e0e-cf4f-450b-8b90-57be599a90d3\") " pod="kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z"
Jul 02 13:00:57.892996 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:57.892950    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht62b\" (UniqueName: \"kubernetes.io/projected/1b200e0e-cf4f-450b-8b90-57be599a90d3-kube-api-access-ht62b\") pod \"success-200-isvc-c0346-predictor-848ccd5d77-kzr6z\" (UID: \"1b200e0e-cf4f-450b-8b90-57be599a90d3\") " pod="kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z"
Jul 02 13:00:57.900948 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:57.900919    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk"]
Jul 02 13:00:57.928786 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:57.928748    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92"]
Jul 02 13:00:57.929048 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:57.929022    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92" podUID="f41086ca-3185-4e23-9d33-bb1539172af1" containerName="kserve-container" containerID="cri-o://76b29931d5942c67d0692559f3687b9c1ca89f75cae7edb9e98d0f63931bfde4" gracePeriod=30
Jul 02 13:00:57.929136 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:57.929100    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92" podUID="f41086ca-3185-4e23-9d33-bb1539172af1" containerName="kube-rbac-proxy" containerID="cri-o://4ed2108c51ac72ca86683168bac33b647fbdc97f04fce19833f01b6860479b41" gracePeriod=30
Jul 02 13:00:57.944391 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:57.944365    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:00:57.994203 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:57.994142    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b200e0e-cf4f-450b-8b90-57be599a90d3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c0346-predictor-848ccd5d77-kzr6z\" (UID: \"1b200e0e-cf4f-450b-8b90-57be599a90d3\") " pod="kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z"
Jul 02 13:00:57.994203 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:57.994197    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht62b\" (UniqueName: \"kubernetes.io/projected/1b200e0e-cf4f-450b-8b90-57be599a90d3-kube-api-access-ht62b\") pod \"success-200-isvc-c0346-predictor-848ccd5d77-kzr6z\" (UID: \"1b200e0e-cf4f-450b-8b90-57be599a90d3\") " pod="kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z"
Jul 02 13:00:57.994448 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:57.994270    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fea819a-f3de-4ac8-92fe-acc5a8227857-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk\" (UID: \"5fea819a-f3de-4ac8-92fe-acc5a8227857\") " pod="kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk"
Jul 02 13:00:57.994448 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:57.994304    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fea819a-f3de-4ac8-92fe-acc5a8227857-proxy-tls\") pod \"error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk\" (UID: \"5fea819a-f3de-4ac8-92fe-acc5a8227857\") " pod="kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk"
Jul 02 13:00:57.994448 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:57.994335    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b200e0e-cf4f-450b-8b90-57be599a90d3-proxy-tls\") pod \"success-200-isvc-c0346-predictor-848ccd5d77-kzr6z\" (UID: \"1b200e0e-cf4f-450b-8b90-57be599a90d3\") " pod="kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z"
Jul 02 13:00:57.994448 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:57.994356    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2kkk\" (UniqueName: \"kubernetes.io/projected/5fea819a-f3de-4ac8-92fe-acc5a8227857-kube-api-access-f2kkk\") pod \"error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk\" (UID: \"5fea819a-f3de-4ac8-92fe-acc5a8227857\") " pod="kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk"
Jul 02 13:00:57.994881 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:57.994859    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b200e0e-cf4f-450b-8b90-57be599a90d3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c0346-predictor-848ccd5d77-kzr6z\" (UID: \"1b200e0e-cf4f-450b-8b90-57be599a90d3\") " pod="kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z"
Jul 02 13:00:57.996941 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:57.996916    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b200e0e-cf4f-450b-8b90-57be599a90d3-proxy-tls\") pod \"success-200-isvc-c0346-predictor-848ccd5d77-kzr6z\" (UID: \"1b200e0e-cf4f-450b-8b90-57be599a90d3\") " pod="kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z"
Jul 02 13:00:58.002941 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:58.002918    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht62b\" (UniqueName: \"kubernetes.io/projected/1b200e0e-cf4f-450b-8b90-57be599a90d3-kube-api-access-ht62b\") pod \"success-200-isvc-c0346-predictor-848ccd5d77-kzr6z\" (UID: \"1b200e0e-cf4f-450b-8b90-57be599a90d3\") " pod="kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z"
Jul 02 13:00:58.086940 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:58.086905    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z"
Jul 02 13:00:58.094924 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:58.094901    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2kkk\" (UniqueName: \"kubernetes.io/projected/5fea819a-f3de-4ac8-92fe-acc5a8227857-kube-api-access-f2kkk\") pod \"error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk\" (UID: \"5fea819a-f3de-4ac8-92fe-acc5a8227857\") " pod="kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk"
Jul 02 13:00:58.095025 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:58.094999    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fea819a-f3de-4ac8-92fe-acc5a8227857-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk\" (UID: \"5fea819a-f3de-4ac8-92fe-acc5a8227857\") " pod="kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk"
Jul 02 13:00:58.095086 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:58.095034    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fea819a-f3de-4ac8-92fe-acc5a8227857-proxy-tls\") pod \"error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk\" (UID: \"5fea819a-f3de-4ac8-92fe-acc5a8227857\") " pod="kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk"
Jul 02 13:00:58.095645 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:58.095618    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fea819a-f3de-4ac8-92fe-acc5a8227857-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk\" (UID: \"5fea819a-f3de-4ac8-92fe-acc5a8227857\") " pod="kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk"
Jul 02 13:00:58.097413 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:58.097389    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fea819a-f3de-4ac8-92fe-acc5a8227857-proxy-tls\") pod \"error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk\" (UID: \"5fea819a-f3de-4ac8-92fe-acc5a8227857\") " pod="kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk"
Jul 02 13:00:58.104506 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:58.104482    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2kkk\" (UniqueName: \"kubernetes.io/projected/5fea819a-f3de-4ac8-92fe-acc5a8227857-kube-api-access-f2kkk\") pod \"error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk\" (UID: \"5fea819a-f3de-4ac8-92fe-acc5a8227857\") " pod="kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk"
Jul 02 13:00:58.193660 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:58.193621    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk"
Jul 02 13:00:58.225483 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:58.225461    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z"]
Jul 02 13:00:58.228171 ip-10-0-137-127 kubenswrapper[2578]: W0702 13:00:58.228144    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b200e0e_cf4f_450b_8b90_57be599a90d3.slice/crio-c4fcbb3b034165d501036c3a9acf987f24fd75326de30d5d3b844dfbb25c9641 WatchSource:0}: Error finding container c4fcbb3b034165d501036c3a9acf987f24fd75326de30d5d3b844dfbb25c9641: Status 404 returned error can't find the container with id c4fcbb3b034165d501036c3a9acf987f24fd75326de30d5d3b844dfbb25c9641
Jul 02 13:00:58.340076 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:58.340050    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk"]
Jul 02 13:00:58.342717 ip-10-0-137-127 kubenswrapper[2578]: W0702 13:00:58.342687    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fea819a_f3de_4ac8_92fe_acc5a8227857.slice/crio-e30bea835fc35ad13f82ce46ab2dfa409c18e4c2c35c0993feb268e453a8c4af WatchSource:0}: Error finding container e30bea835fc35ad13f82ce46ab2dfa409c18e4c2c35c0993feb268e453a8c4af: Status 404 returned error can't find the container with id e30bea835fc35ad13f82ce46ab2dfa409c18e4c2c35c0993feb268e453a8c4af
Jul 02 13:00:58.393515 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:58.393489    2578 generic.go:356] "Generic (PLEG): container finished" podID="f41086ca-3185-4e23-9d33-bb1539172af1" containerID="4ed2108c51ac72ca86683168bac33b647fbdc97f04fce19833f01b6860479b41" exitCode=2
Jul 02 13:00:58.393618 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:58.393557    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92" event={"ID":"f41086ca-3185-4e23-9d33-bb1539172af1","Type":"ContainerDied","Data":"4ed2108c51ac72ca86683168bac33b647fbdc97f04fce19833f01b6860479b41"}
Jul 02 13:00:58.395465 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:58.395434    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z" event={"ID":"1b200e0e-cf4f-450b-8b90-57be599a90d3","Type":"ContainerStarted","Data":"3b6b43d48a6e8a771e90cac5e5487aa2f197dbf04605d7e43ea7c57d847ac28c"}
Jul 02 13:00:58.395579 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:58.395483    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z" event={"ID":"1b200e0e-cf4f-450b-8b90-57be599a90d3","Type":"ContainerStarted","Data":"f84bea15a5082c0f343031a6f91c3f6850c701043d724ac8565a92e9ad381734"}
Jul 02 13:00:58.395579 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:58.395505    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z" event={"ID":"1b200e0e-cf4f-450b-8b90-57be599a90d3","Type":"ContainerStarted","Data":"c4fcbb3b034165d501036c3a9acf987f24fd75326de30d5d3b844dfbb25c9641"}
Jul 02 13:00:58.395741 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:58.395654    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z"
Jul 02 13:00:58.395741 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:58.395704    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z"
Jul 02 13:00:58.397013 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:58.396973    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z" podUID="1b200e0e-cf4f-450b-8b90-57be599a90d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 02 13:00:58.397117 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:58.397086    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk" event={"ID":"5fea819a-f3de-4ac8-92fe-acc5a8227857","Type":"ContainerStarted","Data":"e30bea835fc35ad13f82ce46ab2dfa409c18e4c2c35c0993feb268e453a8c4af"}
Jul 02 13:00:58.398847 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:58.398827    2578 generic.go:356] "Generic (PLEG): container finished" podID="7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3" containerID="cee9d1aadf3aa6ebcdf341b90015c24bba9f3eb45ca2c31fa43dd30e5a92d609" exitCode=2
Jul 02 13:00:58.398939 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:58.398854    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv" event={"ID":"7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3","Type":"ContainerDied","Data":"cee9d1aadf3aa6ebcdf341b90015c24bba9f3eb45ca2c31fa43dd30e5a92d609"}
Jul 02 13:00:58.415093 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:58.415052    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z" podStartSLOduration=1.415039414 podStartE2EDuration="1.415039414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:00:58.413651291 +0000 UTC m=+1326.546477161" watchObservedRunningTime="2026-07-02 13:00:58.415039414 +0000 UTC m=+1326.547865262"
Jul 02 13:00:59.404415 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:59.404373    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk" event={"ID":"5fea819a-f3de-4ac8-92fe-acc5a8227857","Type":"ContainerStarted","Data":"413abeeae022edfa7d846a232a7e8ab9e98c4f321fd1419f5a451cf6b703d057"}
Jul 02 13:00:59.404978 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:59.404430    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk" event={"ID":"5fea819a-f3de-4ac8-92fe-acc5a8227857","Type":"ContainerStarted","Data":"68318ae2aaedcc2518c7324441adc1d43549c3db7f6eaa9c467abd5c60fd40a2"}
Jul 02 13:00:59.404978 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:59.404637    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk"
Jul 02 13:00:59.404978 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:59.404687    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk"
Jul 02 13:00:59.404978 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:59.404782    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z" podUID="1b200e0e-cf4f-450b-8b90-57be599a90d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 02 13:00:59.405770 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:59.405741    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk" podUID="5fea819a-f3de-4ac8-92fe-acc5a8227857" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 02 13:00:59.423415 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:00:59.423367    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk" podStartSLOduration=2.423352717 podStartE2EDuration="2.423352717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:00:59.421341486 +0000 UTC m=+1327.554167333" watchObservedRunningTime="2026-07-02 13:00:59.423352717 +0000 UTC m=+1327.556178561"
Jul 02 13:01:00.282425 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:00.282381    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv" podUID="7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 02 13:01:00.282621 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:00.282381    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92" podUID="f41086ca-3185-4e23-9d33-bb1539172af1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 02 13:01:00.287014 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:00.286985    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92" podUID="f41086ca-3185-4e23-9d33-bb1539172af1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 02 13:01:00.287286 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:00.287260    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv" podUID="7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 02 13:01:00.409035 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:00.408992    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk" podUID="5fea819a-f3de-4ac8-92fe-acc5a8227857" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 02 13:01:02.141547 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.141523    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv"
Jul 02 13:01:02.189222 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.189138    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92"
Jul 02 13:01:02.230209 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.230153    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sf9f6\" (UniqueName: \"kubernetes.io/projected/f41086ca-3185-4e23-9d33-bb1539172af1-kube-api-access-sf9f6\") pod \"f41086ca-3185-4e23-9d33-bb1539172af1\" (UID: \"f41086ca-3185-4e23-9d33-bb1539172af1\") "
Jul 02 13:01:02.230209 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.230214    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f41086ca-3185-4e23-9d33-bb1539172af1-kube-rbac-proxy-sar-config\") pod \"f41086ca-3185-4e23-9d33-bb1539172af1\" (UID: \"f41086ca-3185-4e23-9d33-bb1539172af1\") "
Jul 02 13:01:02.230466 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.230238    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3-proxy-tls\") pod \"7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3\" (UID: \"7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3\") "
Jul 02 13:01:02.230562 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.230540    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f41086ca-3185-4e23-9d33-bb1539172af1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f41086ca-3185-4e23-9d33-bb1539172af1" (UID: "f41086ca-3185-4e23-9d33-bb1539172af1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:01:02.232372 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.232337    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f41086ca-3185-4e23-9d33-bb1539172af1-kube-api-access-sf9f6" (OuterVolumeSpecName: "kube-api-access-sf9f6") pod "f41086ca-3185-4e23-9d33-bb1539172af1" (UID: "f41086ca-3185-4e23-9d33-bb1539172af1"). InnerVolumeSpecName "kube-api-access-sf9f6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:01:02.232774 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.232747    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3" (UID: "7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:01:02.331238 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.331195    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5f8j\" (UniqueName: \"kubernetes.io/projected/7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3-kube-api-access-m5f8j\") pod \"7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3\" (UID: \"7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3\") "
Jul 02 13:01:02.331424 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.331255    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f41086ca-3185-4e23-9d33-bb1539172af1-proxy-tls\") pod \"f41086ca-3185-4e23-9d33-bb1539172af1\" (UID: \"f41086ca-3185-4e23-9d33-bb1539172af1\") "
Jul 02 13:01:02.331424 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.331319    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3-kube-rbac-proxy-sar-config\") pod \"7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3\" (UID: \"7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3\") "
Jul 02 13:01:02.331572 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.331547    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf9f6\" (UniqueName: \"kubernetes.io/projected/f41086ca-3185-4e23-9d33-bb1539172af1-kube-api-access-sf9f6\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:01:02.331572 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.331571    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f41086ca-3185-4e23-9d33-bb1539172af1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:01:02.331768 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.331587    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3-proxy-tls\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:01:02.331768 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.331739    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3" (UID: "7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:01:02.333322 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.333299    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3-kube-api-access-m5f8j" (OuterVolumeSpecName: "kube-api-access-m5f8j") pod "7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3" (UID: "7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3"). InnerVolumeSpecName "kube-api-access-m5f8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:01:02.333704 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.333681    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f41086ca-3185-4e23-9d33-bb1539172af1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f41086ca-3185-4e23-9d33-bb1539172af1" (UID: "f41086ca-3185-4e23-9d33-bb1539172af1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:01:02.419257 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.419221    2578 generic.go:356] "Generic (PLEG): container finished" podID="7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3" containerID="638559a25388dc950fa3e2f393c54ea7b286a029c3db4ceae5b4de053b264aa6" exitCode=0
Jul 02 13:01:02.419422 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.419273    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv"
Jul 02 13:01:02.419422 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.419301    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv" event={"ID":"7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3","Type":"ContainerDied","Data":"638559a25388dc950fa3e2f393c54ea7b286a029c3db4ceae5b4de053b264aa6"}
Jul 02 13:01:02.419422 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.419375    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv" event={"ID":"7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3","Type":"ContainerDied","Data":"ec42ede6fbc9b16b775212768d17c1fe39e94e172b1c1169c78d22bb9f962d17"}
Jul 02 13:01:02.419422 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.419416    2578 scope.go:117] "RemoveContainer" containerID="cee9d1aadf3aa6ebcdf341b90015c24bba9f3eb45ca2c31fa43dd30e5a92d609"
Jul 02 13:01:02.420931 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.420905    2578 generic.go:356] "Generic (PLEG): container finished" podID="f41086ca-3185-4e23-9d33-bb1539172af1" containerID="76b29931d5942c67d0692559f3687b9c1ca89f75cae7edb9e98d0f63931bfde4" exitCode=0
Jul 02 13:01:02.421072 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.420930    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92" event={"ID":"f41086ca-3185-4e23-9d33-bb1539172af1","Type":"ContainerDied","Data":"76b29931d5942c67d0692559f3687b9c1ca89f75cae7edb9e98d0f63931bfde4"}
Jul 02 13:01:02.421072 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.420970    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92"
Jul 02 13:01:02.421072 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.420980    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92" event={"ID":"f41086ca-3185-4e23-9d33-bb1539172af1","Type":"ContainerDied","Data":"f90387409c0fa589fb6ad41d49862b96807c3b2ee1776e194b6ae8a5f14e4e97"}
Jul 02 13:01:02.427915 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.427890    2578 scope.go:117] "RemoveContainer" containerID="638559a25388dc950fa3e2f393c54ea7b286a029c3db4ceae5b4de053b264aa6"
Jul 02 13:01:02.431982 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.431949    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:01:02.431982 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.431979    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5f8j\" (UniqueName: \"kubernetes.io/projected/7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3-kube-api-access-m5f8j\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:01:02.432122 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.431997    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f41086ca-3185-4e23-9d33-bb1539172af1-proxy-tls\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:01:02.436381 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.436361    2578 scope.go:117] "RemoveContainer" containerID="cee9d1aadf3aa6ebcdf341b90015c24bba9f3eb45ca2c31fa43dd30e5a92d609"
Jul 02 13:01:02.436682 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:01:02.436662    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cee9d1aadf3aa6ebcdf341b90015c24bba9f3eb45ca2c31fa43dd30e5a92d609\": container with ID starting with cee9d1aadf3aa6ebcdf341b90015c24bba9f3eb45ca2c31fa43dd30e5a92d609 not found: ID does not exist" containerID="cee9d1aadf3aa6ebcdf341b90015c24bba9f3eb45ca2c31fa43dd30e5a92d609"
Jul 02 13:01:02.436756 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.436696    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cee9d1aadf3aa6ebcdf341b90015c24bba9f3eb45ca2c31fa43dd30e5a92d609"} err="failed to get container status \"cee9d1aadf3aa6ebcdf341b90015c24bba9f3eb45ca2c31fa43dd30e5a92d609\": rpc error: code = NotFound desc = could not find container \"cee9d1aadf3aa6ebcdf341b90015c24bba9f3eb45ca2c31fa43dd30e5a92d609\": container with ID starting with cee9d1aadf3aa6ebcdf341b90015c24bba9f3eb45ca2c31fa43dd30e5a92d609 not found: ID does not exist"
Jul 02 13:01:02.436756 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.436722    2578 scope.go:117] "RemoveContainer" containerID="638559a25388dc950fa3e2f393c54ea7b286a029c3db4ceae5b4de053b264aa6"
Jul 02 13:01:02.436960 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:01:02.436942    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"638559a25388dc950fa3e2f393c54ea7b286a029c3db4ceae5b4de053b264aa6\": container with ID starting with 638559a25388dc950fa3e2f393c54ea7b286a029c3db4ceae5b4de053b264aa6 not found: ID does not exist" containerID="638559a25388dc950fa3e2f393c54ea7b286a029c3db4ceae5b4de053b264aa6"
Jul 02 13:01:02.437049 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.436969    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"638559a25388dc950fa3e2f393c54ea7b286a029c3db4ceae5b4de053b264aa6"} err="failed to get container status \"638559a25388dc950fa3e2f393c54ea7b286a029c3db4ceae5b4de053b264aa6\": rpc error: code = NotFound desc = could not find container \"638559a25388dc950fa3e2f393c54ea7b286a029c3db4ceae5b4de053b264aa6\": container with ID starting with 638559a25388dc950fa3e2f393c54ea7b286a029c3db4ceae5b4de053b264aa6 not found: ID does not exist"
Jul 02 13:01:02.437049 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.436991    2578 scope.go:117] "RemoveContainer" containerID="4ed2108c51ac72ca86683168bac33b647fbdc97f04fce19833f01b6860479b41"
Jul 02 13:01:02.440476 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.440419    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92"]
Jul 02 13:01:02.445273 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.445245    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-04e3b-predictor-5756f97b8-66z92"]
Jul 02 13:01:02.447321 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.447296    2578 scope.go:117] "RemoveContainer" containerID="76b29931d5942c67d0692559f3687b9c1ca89f75cae7edb9e98d0f63931bfde4"
Jul 02 13:01:02.454646 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.454620    2578 scope.go:117] "RemoveContainer" containerID="4ed2108c51ac72ca86683168bac33b647fbdc97f04fce19833f01b6860479b41"
Jul 02 13:01:02.454897 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:01:02.454878    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ed2108c51ac72ca86683168bac33b647fbdc97f04fce19833f01b6860479b41\": container with ID starting with 4ed2108c51ac72ca86683168bac33b647fbdc97f04fce19833f01b6860479b41 not found: ID does not exist" containerID="4ed2108c51ac72ca86683168bac33b647fbdc97f04fce19833f01b6860479b41"
Jul 02 13:01:02.454942 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.454905    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ed2108c51ac72ca86683168bac33b647fbdc97f04fce19833f01b6860479b41"} err="failed to get container status \"4ed2108c51ac72ca86683168bac33b647fbdc97f04fce19833f01b6860479b41\": rpc error: code = NotFound desc = could not find container \"4ed2108c51ac72ca86683168bac33b647fbdc97f04fce19833f01b6860479b41\": container with ID starting with 4ed2108c51ac72ca86683168bac33b647fbdc97f04fce19833f01b6860479b41 not found: ID does not exist"
Jul 02 13:01:02.454942 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.454926    2578 scope.go:117] "RemoveContainer" containerID="76b29931d5942c67d0692559f3687b9c1ca89f75cae7edb9e98d0f63931bfde4"
Jul 02 13:01:02.455154 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:01:02.455138    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76b29931d5942c67d0692559f3687b9c1ca89f75cae7edb9e98d0f63931bfde4\": container with ID starting with 76b29931d5942c67d0692559f3687b9c1ca89f75cae7edb9e98d0f63931bfde4 not found: ID does not exist" containerID="76b29931d5942c67d0692559f3687b9c1ca89f75cae7edb9e98d0f63931bfde4"
Jul 02 13:01:02.455224 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.455160    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76b29931d5942c67d0692559f3687b9c1ca89f75cae7edb9e98d0f63931bfde4"} err="failed to get container status \"76b29931d5942c67d0692559f3687b9c1ca89f75cae7edb9e98d0f63931bfde4\": rpc error: code = NotFound desc = could not find container \"76b29931d5942c67d0692559f3687b9c1ca89f75cae7edb9e98d0f63931bfde4\": container with ID starting with 76b29931d5942c67d0692559f3687b9c1ca89f75cae7edb9e98d0f63931bfde4 not found: ID does not exist"
Jul 02 13:01:02.459891 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.459869    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv"]
Jul 02 13:01:02.463934 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:02.463913    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-04e3b-predictor-7448945865-gfvcv"]
Jul 02 13:01:04.362108 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:04.362064    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3" path="/var/lib/kubelet/pods/7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3/volumes"
Jul 02 13:01:04.362704 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:04.362684    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f41086ca-3185-4e23-9d33-bb1539172af1" path="/var/lib/kubelet/pods/f41086ca-3185-4e23-9d33-bb1539172af1/volumes"
Jul 02 13:01:04.409295 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:04.409266    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z"
Jul 02 13:01:04.409900 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:04.409872    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z" podUID="1b200e0e-cf4f-450b-8b90-57be599a90d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 02 13:01:05.413817 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:05.413787    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk"
Jul 02 13:01:05.414283 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:05.414258    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk" podUID="5fea819a-f3de-4ac8-92fe-acc5a8227857" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 02 13:01:14.409975 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:14.409888    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z" podUID="1b200e0e-cf4f-450b-8b90-57be599a90d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 02 13:01:15.415218 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:15.415161    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk" podUID="5fea819a-f3de-4ac8-92fe-acc5a8227857" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 02 13:01:24.410470 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:24.410431    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z" podUID="1b200e0e-cf4f-450b-8b90-57be599a90d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 02 13:01:25.415324 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:25.415273    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk" podUID="5fea819a-f3de-4ac8-92fe-acc5a8227857" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 02 13:01:34.410221 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:34.410160    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z" podUID="1b200e0e-cf4f-450b-8b90-57be599a90d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 02 13:01:35.415103 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:35.415062    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk" podUID="5fea819a-f3de-4ac8-92fe-acc5a8227857" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 02 13:01:44.410511 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:44.410476    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z"
Jul 02 13:01:45.415372 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:45.415331    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk"
Jul 02 13:01:47.693355 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.693320    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm"]
Jul 02 13:01:47.693750 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.693583    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm" podUID="689be393-9525-45ff-93b3-ba50eeaa6dde" containerName="kserve-container" containerID="cri-o://65cdbc7f4daee456dcc6008a75f3a7523a1807a7e0615927df79b3a02e066eb7" gracePeriod=30
Jul 02 13:01:47.693750 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.693622    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm" podUID="689be393-9525-45ff-93b3-ba50eeaa6dde" containerName="kube-rbac-proxy" containerID="cri-o://b6545602a0fabb0ab686261da092016abcbcb3afa8b9db107cb46dc72afeb7dd" gracePeriod=30
Jul 02 13:01:47.711436 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.711408    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd"]
Jul 02 13:01:47.711807 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.711790    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3" containerName="kserve-container"
Jul 02 13:01:47.711889 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.711809    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3" containerName="kserve-container"
Jul 02 13:01:47.711889 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.711839    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f41086ca-3185-4e23-9d33-bb1539172af1" containerName="kserve-container"
Jul 02 13:01:47.711889 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.711848    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f41086ca-3185-4e23-9d33-bb1539172af1" containerName="kserve-container"
Jul 02 13:01:47.711889 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.711861    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3" containerName="kube-rbac-proxy"
Jul 02 13:01:47.711889 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.711869    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3" containerName="kube-rbac-proxy"
Jul 02 13:01:47.711889 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.711880    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f41086ca-3185-4e23-9d33-bb1539172af1" containerName="kube-rbac-proxy"
Jul 02 13:01:47.711889 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.711888    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f41086ca-3185-4e23-9d33-bb1539172af1" containerName="kube-rbac-proxy"
Jul 02 13:01:47.712247 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.711984    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3" containerName="kube-rbac-proxy"
Jul 02 13:01:47.712247 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.711996    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f41086ca-3185-4e23-9d33-bb1539172af1" containerName="kserve-container"
Jul 02 13:01:47.712247 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.712009    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ea27268-cfef-4c78-a7e0-7b6a57bc0ee3" containerName="kserve-container"
Jul 02 13:01:47.712247 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.712020    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f41086ca-3185-4e23-9d33-bb1539172af1" containerName="kube-rbac-proxy"
Jul 02 13:01:47.715284 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.715265    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd"
Jul 02 13:01:47.718220 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.718200    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7a990-predictor-serving-cert\""
Jul 02 13:01:47.718220 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.718215    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7a990-kube-rbac-proxy-sar-config\""
Jul 02 13:01:47.725259 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.725238    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd"]
Jul 02 13:01:47.744098 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.744065    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4d57fc5-4f72-40e4-bbbc-fcff17489250-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7a990-predictor-7b76c758b9-pbfdd\" (UID: \"c4d57fc5-4f72-40e4-bbbc-fcff17489250\") " pod="kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd"
Jul 02 13:01:47.744098 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.744100    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxdbb\" (UniqueName: \"kubernetes.io/projected/c4d57fc5-4f72-40e4-bbbc-fcff17489250-kube-api-access-cxdbb\") pod \"success-200-isvc-7a990-predictor-7b76c758b9-pbfdd\" (UID: \"c4d57fc5-4f72-40e4-bbbc-fcff17489250\") " pod="kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd"
Jul 02 13:01:47.744299 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.744134    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4d57fc5-4f72-40e4-bbbc-fcff17489250-proxy-tls\") pod \"success-200-isvc-7a990-predictor-7b76c758b9-pbfdd\" (UID: \"c4d57fc5-4f72-40e4-bbbc-fcff17489250\") " pod="kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd"
Jul 02 13:01:47.823611 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.823578    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k"]
Jul 02 13:01:47.827386 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.827362    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k"
Jul 02 13:01:47.830230 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.830207    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7a990-predictor-serving-cert\""
Jul 02 13:01:47.830317 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.830307    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7a990-kube-rbac-proxy-sar-config\""
Jul 02 13:01:47.839501 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.839481    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k"]
Jul 02 13:01:47.844892 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.844870    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4d57fc5-4f72-40e4-bbbc-fcff17489250-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7a990-predictor-7b76c758b9-pbfdd\" (UID: \"c4d57fc5-4f72-40e4-bbbc-fcff17489250\") " pod="kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd"
Jul 02 13:01:47.845026 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.844903    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxdbb\" (UniqueName: \"kubernetes.io/projected/c4d57fc5-4f72-40e4-bbbc-fcff17489250-kube-api-access-cxdbb\") pod \"success-200-isvc-7a990-predictor-7b76c758b9-pbfdd\" (UID: \"c4d57fc5-4f72-40e4-bbbc-fcff17489250\") " pod="kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd"
Jul 02 13:01:47.845026 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.844947    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1413e900-a9c6-4606-bcfe-b60c3813e400-proxy-tls\") pod \"error-404-isvc-7a990-predictor-9f9c44d9b-9v57k\" (UID: \"1413e900-a9c6-4606-bcfe-b60c3813e400\") " pod="kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k"
Jul 02 13:01:47.845026 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.844979    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4gsf\" (UniqueName: \"kubernetes.io/projected/1413e900-a9c6-4606-bcfe-b60c3813e400-kube-api-access-k4gsf\") pod \"error-404-isvc-7a990-predictor-9f9c44d9b-9v57k\" (UID: \"1413e900-a9c6-4606-bcfe-b60c3813e400\") " pod="kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k"
Jul 02 13:01:47.845248 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.845031    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4d57fc5-4f72-40e4-bbbc-fcff17489250-proxy-tls\") pod \"success-200-isvc-7a990-predictor-7b76c758b9-pbfdd\" (UID: \"c4d57fc5-4f72-40e4-bbbc-fcff17489250\") " pod="kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd"
Jul 02 13:01:47.845248 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.845062    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1413e900-a9c6-4606-bcfe-b60c3813e400-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7a990-predictor-9f9c44d9b-9v57k\" (UID: \"1413e900-a9c6-4606-bcfe-b60c3813e400\") " pod="kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k"
Jul 02 13:01:47.845592 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.845574    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4d57fc5-4f72-40e4-bbbc-fcff17489250-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7a990-predictor-7b76c758b9-pbfdd\" (UID: \"c4d57fc5-4f72-40e4-bbbc-fcff17489250\") " pod="kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd"
Jul 02 13:01:47.848039 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.847948    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4d57fc5-4f72-40e4-bbbc-fcff17489250-proxy-tls\") pod \"success-200-isvc-7a990-predictor-7b76c758b9-pbfdd\" (UID: \"c4d57fc5-4f72-40e4-bbbc-fcff17489250\") " pod="kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd"
Jul 02 13:01:47.865596 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.865570    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxdbb\" (UniqueName: \"kubernetes.io/projected/c4d57fc5-4f72-40e4-bbbc-fcff17489250-kube-api-access-cxdbb\") pod \"success-200-isvc-7a990-predictor-7b76c758b9-pbfdd\" (UID: \"c4d57fc5-4f72-40e4-bbbc-fcff17489250\") " pod="kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd"
Jul 02 13:01:47.913406 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.913375    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm"]
Jul 02 13:01:47.913641 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.913595    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm" podUID="95ade19c-8fea-4214-8d6a-7c5264e03c0e" containerName="kserve-container" containerID="cri-o://5da8ca99ea3959b7e6e8a07750aa6ad861092610da5cb8f15ca11723cef0ecc5" gracePeriod=30
Jul 02 13:01:47.913731 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.913630    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm" podUID="95ade19c-8fea-4214-8d6a-7c5264e03c0e" containerName="kube-rbac-proxy" containerID="cri-o://a1307c8f79945de84e5625bb23e6972497f55df0d9b40676dcf838c310bb13ab" gracePeriod=30
Jul 02 13:01:47.946210 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.946096    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1413e900-a9c6-4606-bcfe-b60c3813e400-proxy-tls\") pod \"error-404-isvc-7a990-predictor-9f9c44d9b-9v57k\" (UID: \"1413e900-a9c6-4606-bcfe-b60c3813e400\") " pod="kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k"
Jul 02 13:01:47.946210 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.946149    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4gsf\" (UniqueName: \"kubernetes.io/projected/1413e900-a9c6-4606-bcfe-b60c3813e400-kube-api-access-k4gsf\") pod \"error-404-isvc-7a990-predictor-9f9c44d9b-9v57k\" (UID: \"1413e900-a9c6-4606-bcfe-b60c3813e400\") " pod="kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k"
Jul 02 13:01:47.946428 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:01:47.946351    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-7a990-predictor-serving-cert: secret "error-404-isvc-7a990-predictor-serving-cert" not found
Jul 02 13:01:47.946428 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:01:47.946415    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1413e900-a9c6-4606-bcfe-b60c3813e400-proxy-tls podName:1413e900-a9c6-4606-bcfe-b60c3813e400 nodeName:}" failed. No retries permitted until 2026-07-02 13:01:48.446393313 +0000 UTC m=+1376.579219141 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1413e900-a9c6-4606-bcfe-b60c3813e400-proxy-tls") pod "error-404-isvc-7a990-predictor-9f9c44d9b-9v57k" (UID: "1413e900-a9c6-4606-bcfe-b60c3813e400") : secret "error-404-isvc-7a990-predictor-serving-cert" not found
Jul 02 13:01:47.946538 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.946479    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1413e900-a9c6-4606-bcfe-b60c3813e400-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7a990-predictor-9f9c44d9b-9v57k\" (UID: \"1413e900-a9c6-4606-bcfe-b60c3813e400\") " pod="kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k"
Jul 02 13:01:47.947296 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.947272    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1413e900-a9c6-4606-bcfe-b60c3813e400-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7a990-predictor-9f9c44d9b-9v57k\" (UID: \"1413e900-a9c6-4606-bcfe-b60c3813e400\") " pod="kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k"
Jul 02 13:01:47.958122 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:47.958089    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4gsf\" (UniqueName: \"kubernetes.io/projected/1413e900-a9c6-4606-bcfe-b60c3813e400-kube-api-access-k4gsf\") pod \"error-404-isvc-7a990-predictor-9f9c44d9b-9v57k\" (UID: \"1413e900-a9c6-4606-bcfe-b60c3813e400\") " pod="kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k"
Jul 02 13:01:48.026913 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:48.026877    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd"
Jul 02 13:01:48.081903 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:48.081872    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:01:48.183454 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:48.183431    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd"]
Jul 02 13:01:48.186065 ip-10-0-137-127 kubenswrapper[2578]: W0702 13:01:48.186013    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4d57fc5_4f72_40e4_bbbc_fcff17489250.slice/crio-74e51770466c5c952b4c2161b4c245170467b0b9aeb4a2e248fa92ff05840f20 WatchSource:0}: Error finding container 74e51770466c5c952b4c2161b4c245170467b0b9aeb4a2e248fa92ff05840f20: Status 404 returned error can't find the container with id 74e51770466c5c952b4c2161b4c245170467b0b9aeb4a2e248fa92ff05840f20
Jul 02 13:01:48.220552 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:48.220522    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:01:48.309503 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:48.309471    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:01:48.450578 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:48.450482    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1413e900-a9c6-4606-bcfe-b60c3813e400-proxy-tls\") pod \"error-404-isvc-7a990-predictor-9f9c44d9b-9v57k\" (UID: \"1413e900-a9c6-4606-bcfe-b60c3813e400\") " pod="kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k"
Jul 02 13:01:48.452841 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:48.452816    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1413e900-a9c6-4606-bcfe-b60c3813e400-proxy-tls\") pod \"error-404-isvc-7a990-predictor-9f9c44d9b-9v57k\" (UID: \"1413e900-a9c6-4606-bcfe-b60c3813e400\") " pod="kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k"
Jul 02 13:01:48.617648 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:48.617616    2578 generic.go:356] "Generic (PLEG): container finished" podID="689be393-9525-45ff-93b3-ba50eeaa6dde" containerID="b6545602a0fabb0ab686261da092016abcbcb3afa8b9db107cb46dc72afeb7dd" exitCode=2
Jul 02 13:01:48.617833 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:48.617688    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm" event={"ID":"689be393-9525-45ff-93b3-ba50eeaa6dde","Type":"ContainerDied","Data":"b6545602a0fabb0ab686261da092016abcbcb3afa8b9db107cb46dc72afeb7dd"}
Jul 02 13:01:48.624725 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:48.624676    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd" event={"ID":"c4d57fc5-4f72-40e4-bbbc-fcff17489250","Type":"ContainerStarted","Data":"b2634cdd74703280219d697804f8cffa7f21321a30ad9f73fa395c322599f260"}
Jul 02 13:01:48.624851 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:48.624795    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd"
Jul 02 13:01:48.624851 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:48.624830    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd" event={"ID":"c4d57fc5-4f72-40e4-bbbc-fcff17489250","Type":"ContainerStarted","Data":"5404f478cc1281a56b11699a42d4b2279a869d60eed353e86043c78be697ad89"}
Jul 02 13:01:48.624992 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:48.624857    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd" event={"ID":"c4d57fc5-4f72-40e4-bbbc-fcff17489250","Type":"ContainerStarted","Data":"74e51770466c5c952b4c2161b4c245170467b0b9aeb4a2e248fa92ff05840f20"}
Jul 02 13:01:48.626336 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:48.626303    2578 generic.go:356] "Generic (PLEG): container finished" podID="95ade19c-8fea-4214-8d6a-7c5264e03c0e" containerID="a1307c8f79945de84e5625bb23e6972497f55df0d9b40676dcf838c310bb13ab" exitCode=2
Jul 02 13:01:48.626439 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:48.626363    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm" event={"ID":"95ade19c-8fea-4214-8d6a-7c5264e03c0e","Type":"ContainerDied","Data":"a1307c8f79945de84e5625bb23e6972497f55df0d9b40676dcf838c310bb13ab"}
Jul 02 13:01:48.666581 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:48.666532    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd" podStartSLOduration=1.666518494 podStartE2EDuration="1.666518494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:01:48.665121239 +0000 UTC m=+1376.797947084" watchObservedRunningTime="2026-07-02 13:01:48.666518494 +0000 UTC m=+1376.799344339"
Jul 02 13:01:48.738345 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:48.738271    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k"
Jul 02 13:01:48.883985 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:48.883949    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k"]
Jul 02 13:01:48.888601 ip-10-0-137-127 kubenswrapper[2578]: W0702 13:01:48.888569    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1413e900_a9c6_4606_bcfe_b60c3813e400.slice/crio-3d5569a67f2e1b3fa1757fadfbce9f83d308fbc525f0df17a6c361ab36a333af WatchSource:0}: Error finding container 3d5569a67f2e1b3fa1757fadfbce9f83d308fbc525f0df17a6c361ab36a333af: Status 404 returned error can't find the container with id 3d5569a67f2e1b3fa1757fadfbce9f83d308fbc525f0df17a6c361ab36a333af
Jul 02 13:01:48.901943 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:48.901907    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:01:49.437339 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:49.437224    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm" podUID="689be393-9525-45ff-93b3-ba50eeaa6dde" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 02 13:01:49.441594 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:49.441567    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm" podUID="689be393-9525-45ff-93b3-ba50eeaa6dde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 02 13:01:49.633550 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:49.633483    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k" event={"ID":"1413e900-a9c6-4606-bcfe-b60c3813e400","Type":"ContainerStarted","Data":"c457f9641362ad1a4c45fcc35c5714ad0f7cbe831d300d25f4ce666a51e2fa96"}
Jul 02 13:01:49.633733 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:49.633563    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k" event={"ID":"1413e900-a9c6-4606-bcfe-b60c3813e400","Type":"ContainerStarted","Data":"06550560ab336914c6d1e0b199916a3abb2d38d5edaf062ec872796927a0ef35"}
Jul 02 13:01:49.633733 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:49.633597    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k" event={"ID":"1413e900-a9c6-4606-bcfe-b60c3813e400","Type":"ContainerStarted","Data":"3d5569a67f2e1b3fa1757fadfbce9f83d308fbc525f0df17a6c361ab36a333af"}
Jul 02 13:01:49.634145 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:49.634118    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd"
Jul 02 13:01:49.635874 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:49.635830    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd" podUID="c4d57fc5-4f72-40e4-bbbc-fcff17489250" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 02 13:01:49.652890 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:49.652844    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k" podStartSLOduration=2.652829375 podStartE2EDuration="2.652829375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:01:49.652077707 +0000 UTC m=+1377.784903556" watchObservedRunningTime="2026-07-02 13:01:49.652829375 +0000 UTC m=+1377.785655224"
Jul 02 13:01:50.639412 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:50.639378    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k"
Jul 02 13:01:50.639864 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:50.639499    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd" podUID="c4d57fc5-4f72-40e4-bbbc-fcff17489250" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 02 13:01:50.639864 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:50.639626    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k"
Jul 02 13:01:50.640681 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:50.640657    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k" podUID="1413e900-a9c6-4606-bcfe-b60c3813e400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 02 13:01:51.140793 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.140765    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm"
Jul 02 13:01:51.172954 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.172926    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/689be393-9525-45ff-93b3-ba50eeaa6dde-proxy-tls\") pod \"689be393-9525-45ff-93b3-ba50eeaa6dde\" (UID: \"689be393-9525-45ff-93b3-ba50eeaa6dde\") "
Jul 02 13:01:51.173083 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.172985    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8ckw\" (UniqueName: \"kubernetes.io/projected/689be393-9525-45ff-93b3-ba50eeaa6dde-kube-api-access-f8ckw\") pod \"689be393-9525-45ff-93b3-ba50eeaa6dde\" (UID: \"689be393-9525-45ff-93b3-ba50eeaa6dde\") "
Jul 02 13:01:51.173158 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.173083    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/689be393-9525-45ff-93b3-ba50eeaa6dde-kube-rbac-proxy-sar-config\") pod \"689be393-9525-45ff-93b3-ba50eeaa6dde\" (UID: \"689be393-9525-45ff-93b3-ba50eeaa6dde\") "
Jul 02 13:01:51.173389 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.173362    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/689be393-9525-45ff-93b3-ba50eeaa6dde-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "689be393-9525-45ff-93b3-ba50eeaa6dde" (UID: "689be393-9525-45ff-93b3-ba50eeaa6dde"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:01:51.175016 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.174997    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/689be393-9525-45ff-93b3-ba50eeaa6dde-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "689be393-9525-45ff-93b3-ba50eeaa6dde" (UID: "689be393-9525-45ff-93b3-ba50eeaa6dde"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:01:51.175126 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.175039    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/689be393-9525-45ff-93b3-ba50eeaa6dde-kube-api-access-f8ckw" (OuterVolumeSpecName: "kube-api-access-f8ckw") pod "689be393-9525-45ff-93b3-ba50eeaa6dde" (UID: "689be393-9525-45ff-93b3-ba50eeaa6dde"). InnerVolumeSpecName "kube-api-access-f8ckw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:01:51.274621 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.274578    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/689be393-9525-45ff-93b3-ba50eeaa6dde-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:01:51.274786 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.274626    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/689be393-9525-45ff-93b3-ba50eeaa6dde-proxy-tls\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:01:51.274786 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.274645    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8ckw\" (UniqueName: \"kubernetes.io/projected/689be393-9525-45ff-93b3-ba50eeaa6dde-kube-api-access-f8ckw\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:01:51.357921 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.357896    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm"
Jul 02 13:01:51.375697 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.375675    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcjb9\" (UniqueName: \"kubernetes.io/projected/95ade19c-8fea-4214-8d6a-7c5264e03c0e-kube-api-access-kcjb9\") pod \"95ade19c-8fea-4214-8d6a-7c5264e03c0e\" (UID: \"95ade19c-8fea-4214-8d6a-7c5264e03c0e\") "
Jul 02 13:01:51.375825 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.375730    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95ade19c-8fea-4214-8d6a-7c5264e03c0e-kube-rbac-proxy-sar-config\") pod \"95ade19c-8fea-4214-8d6a-7c5264e03c0e\" (UID: \"95ade19c-8fea-4214-8d6a-7c5264e03c0e\") "
Jul 02 13:01:51.375888 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.375843    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95ade19c-8fea-4214-8d6a-7c5264e03c0e-proxy-tls\") pod \"95ade19c-8fea-4214-8d6a-7c5264e03c0e\" (UID: \"95ade19c-8fea-4214-8d6a-7c5264e03c0e\") "
Jul 02 13:01:51.376074 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.376052    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95ade19c-8fea-4214-8d6a-7c5264e03c0e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "95ade19c-8fea-4214-8d6a-7c5264e03c0e" (UID: "95ade19c-8fea-4214-8d6a-7c5264e03c0e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:01:51.376174 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.376157    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95ade19c-8fea-4214-8d6a-7c5264e03c0e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:01:51.377742 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.377723    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95ade19c-8fea-4214-8d6a-7c5264e03c0e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "95ade19c-8fea-4214-8d6a-7c5264e03c0e" (UID: "95ade19c-8fea-4214-8d6a-7c5264e03c0e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:01:51.377812 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.377776    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95ade19c-8fea-4214-8d6a-7c5264e03c0e-kube-api-access-kcjb9" (OuterVolumeSpecName: "kube-api-access-kcjb9") pod "95ade19c-8fea-4214-8d6a-7c5264e03c0e" (UID: "95ade19c-8fea-4214-8d6a-7c5264e03c0e"). InnerVolumeSpecName "kube-api-access-kcjb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:01:51.477447 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.477357    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcjb9\" (UniqueName: \"kubernetes.io/projected/95ade19c-8fea-4214-8d6a-7c5264e03c0e-kube-api-access-kcjb9\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:01:51.477447 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.477387    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95ade19c-8fea-4214-8d6a-7c5264e03c0e-proxy-tls\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:01:51.644315 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.644278    2578 generic.go:356] "Generic (PLEG): container finished" podID="95ade19c-8fea-4214-8d6a-7c5264e03c0e" containerID="5da8ca99ea3959b7e6e8a07750aa6ad861092610da5cb8f15ca11723cef0ecc5" exitCode=0
Jul 02 13:01:51.644770 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.644316    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm" event={"ID":"95ade19c-8fea-4214-8d6a-7c5264e03c0e","Type":"ContainerDied","Data":"5da8ca99ea3959b7e6e8a07750aa6ad861092610da5cb8f15ca11723cef0ecc5"}
Jul 02 13:01:51.644770 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.644339    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm"
Jul 02 13:01:51.644770 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.644380    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm" event={"ID":"95ade19c-8fea-4214-8d6a-7c5264e03c0e","Type":"ContainerDied","Data":"7d2d080bee57d61c54d9bbf3f0367639f2d0746c5ad976ca00fd3e40219f0333"}
Jul 02 13:01:51.644770 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.644406    2578 scope.go:117] "RemoveContainer" containerID="a1307c8f79945de84e5625bb23e6972497f55df0d9b40676dcf838c310bb13ab"
Jul 02 13:01:51.646003 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.645977    2578 generic.go:356] "Generic (PLEG): container finished" podID="689be393-9525-45ff-93b3-ba50eeaa6dde" containerID="65cdbc7f4daee456dcc6008a75f3a7523a1807a7e0615927df79b3a02e066eb7" exitCode=0
Jul 02 13:01:51.646133 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.646049    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm" event={"ID":"689be393-9525-45ff-93b3-ba50eeaa6dde","Type":"ContainerDied","Data":"65cdbc7f4daee456dcc6008a75f3a7523a1807a7e0615927df79b3a02e066eb7"}
Jul 02 13:01:51.646133 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.646061    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm"
Jul 02 13:01:51.646133 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.646109    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm" event={"ID":"689be393-9525-45ff-93b3-ba50eeaa6dde","Type":"ContainerDied","Data":"4d300dc7c025a2c98cde17c0b979902529714b48eadea00ae21067ccd9a3bebc"}
Jul 02 13:01:51.646371 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.646346    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k" podUID="1413e900-a9c6-4606-bcfe-b60c3813e400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 02 13:01:51.653461 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.653446    2578 scope.go:117] "RemoveContainer" containerID="5da8ca99ea3959b7e6e8a07750aa6ad861092610da5cb8f15ca11723cef0ecc5"
Jul 02 13:01:51.660680 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.660658    2578 scope.go:117] "RemoveContainer" containerID="a1307c8f79945de84e5625bb23e6972497f55df0d9b40676dcf838c310bb13ab"
Jul 02 13:01:51.660924 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:01:51.660907    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1307c8f79945de84e5625bb23e6972497f55df0d9b40676dcf838c310bb13ab\": container with ID starting with a1307c8f79945de84e5625bb23e6972497f55df0d9b40676dcf838c310bb13ab not found: ID does not exist" containerID="a1307c8f79945de84e5625bb23e6972497f55df0d9b40676dcf838c310bb13ab"
Jul 02 13:01:51.660972 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.660931    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1307c8f79945de84e5625bb23e6972497f55df0d9b40676dcf838c310bb13ab"} err="failed to get container status \"a1307c8f79945de84e5625bb23e6972497f55df0d9b40676dcf838c310bb13ab\": rpc error: code = NotFound desc = could not find container \"a1307c8f79945de84e5625bb23e6972497f55df0d9b40676dcf838c310bb13ab\": container with ID starting with a1307c8f79945de84e5625bb23e6972497f55df0d9b40676dcf838c310bb13ab not found: ID does not exist"
Jul 02 13:01:51.660972 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.660947    2578 scope.go:117] "RemoveContainer" containerID="5da8ca99ea3959b7e6e8a07750aa6ad861092610da5cb8f15ca11723cef0ecc5"
Jul 02 13:01:51.661172 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:01:51.661156    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5da8ca99ea3959b7e6e8a07750aa6ad861092610da5cb8f15ca11723cef0ecc5\": container with ID starting with 5da8ca99ea3959b7e6e8a07750aa6ad861092610da5cb8f15ca11723cef0ecc5 not found: ID does not exist" containerID="5da8ca99ea3959b7e6e8a07750aa6ad861092610da5cb8f15ca11723cef0ecc5"
Jul 02 13:01:51.661653 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.661199    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5da8ca99ea3959b7e6e8a07750aa6ad861092610da5cb8f15ca11723cef0ecc5"} err="failed to get container status \"5da8ca99ea3959b7e6e8a07750aa6ad861092610da5cb8f15ca11723cef0ecc5\": rpc error: code = NotFound desc = could not find container \"5da8ca99ea3959b7e6e8a07750aa6ad861092610da5cb8f15ca11723cef0ecc5\": container with ID starting with 5da8ca99ea3959b7e6e8a07750aa6ad861092610da5cb8f15ca11723cef0ecc5 not found: ID does not exist"
Jul 02 13:01:51.661653 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.661217    2578 scope.go:117] "RemoveContainer" containerID="b6545602a0fabb0ab686261da092016abcbcb3afa8b9db107cb46dc72afeb7dd"
Jul 02 13:01:51.668533 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.668517    2578 scope.go:117] "RemoveContainer" containerID="65cdbc7f4daee456dcc6008a75f3a7523a1807a7e0615927df79b3a02e066eb7"
Jul 02 13:01:51.675339 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.675305    2578 scope.go:117] "RemoveContainer" containerID="b6545602a0fabb0ab686261da092016abcbcb3afa8b9db107cb46dc72afeb7dd"
Jul 02 13:01:51.675611 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:01:51.675585    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6545602a0fabb0ab686261da092016abcbcb3afa8b9db107cb46dc72afeb7dd\": container with ID starting with b6545602a0fabb0ab686261da092016abcbcb3afa8b9db107cb46dc72afeb7dd not found: ID does not exist" containerID="b6545602a0fabb0ab686261da092016abcbcb3afa8b9db107cb46dc72afeb7dd"
Jul 02 13:01:51.675670 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.675608    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6545602a0fabb0ab686261da092016abcbcb3afa8b9db107cb46dc72afeb7dd"} err="failed to get container status \"b6545602a0fabb0ab686261da092016abcbcb3afa8b9db107cb46dc72afeb7dd\": rpc error: code = NotFound desc = could not find container \"b6545602a0fabb0ab686261da092016abcbcb3afa8b9db107cb46dc72afeb7dd\": container with ID starting with b6545602a0fabb0ab686261da092016abcbcb3afa8b9db107cb46dc72afeb7dd not found: ID does not exist"
Jul 02 13:01:51.675670 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.675624    2578 scope.go:117] "RemoveContainer" containerID="65cdbc7f4daee456dcc6008a75f3a7523a1807a7e0615927df79b3a02e066eb7"
Jul 02 13:01:51.675892 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:01:51.675874    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65cdbc7f4daee456dcc6008a75f3a7523a1807a7e0615927df79b3a02e066eb7\": container with ID starting with 65cdbc7f4daee456dcc6008a75f3a7523a1807a7e0615927df79b3a02e066eb7 not found: ID does not exist" containerID="65cdbc7f4daee456dcc6008a75f3a7523a1807a7e0615927df79b3a02e066eb7"
Jul 02 13:01:51.675942 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.675897    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65cdbc7f4daee456dcc6008a75f3a7523a1807a7e0615927df79b3a02e066eb7"} err="failed to get container status \"65cdbc7f4daee456dcc6008a75f3a7523a1807a7e0615927df79b3a02e066eb7\": rpc error: code = NotFound desc = could not find container \"65cdbc7f4daee456dcc6008a75f3a7523a1807a7e0615927df79b3a02e066eb7\": container with ID starting with 65cdbc7f4daee456dcc6008a75f3a7523a1807a7e0615927df79b3a02e066eb7 not found: ID does not exist"
Jul 02 13:01:51.677903 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.677882    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm"]
Jul 02 13:01:51.684972 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.684953    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8bebb-predictor-674985687f-rxfmm"]
Jul 02 13:01:51.707992 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.707970    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm"]
Jul 02 13:01:51.711585 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:51.711566    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8bebb-predictor-5796bd8b66-hvggm"]
Jul 02 13:01:52.367283 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:52.363284    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="689be393-9525-45ff-93b3-ba50eeaa6dde" path="/var/lib/kubelet/pods/689be393-9525-45ff-93b3-ba50eeaa6dde/volumes"
Jul 02 13:01:52.367283 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:52.363884    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95ade19c-8fea-4214-8d6a-7c5264e03c0e" path="/var/lib/kubelet/pods/95ade19c-8fea-4214-8d6a-7c5264e03c0e/volumes"
Jul 02 13:01:55.644376 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:55.644346    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd"
Jul 02 13:01:55.644877 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:55.644846    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd" podUID="c4d57fc5-4f72-40e4-bbbc-fcff17489250" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 02 13:01:56.651004 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:56.650976    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k"
Jul 02 13:01:56.651589 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:56.651561    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k" podUID="1413e900-a9c6-4606-bcfe-b60c3813e400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 02 13:01:57.735924 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:57.735880    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:01:57.961379 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:01:57.961343    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:02:05.644822 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:05.644778    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd" podUID="c4d57fc5-4f72-40e4-bbbc-fcff17489250" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 02 13:02:06.651809 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:06.651772    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k" podUID="1413e900-a9c6-4606-bcfe-b60c3813e400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 02 13:02:07.811673 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:07.811639    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:02:07.921002 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:07.920971    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk"]
Jul 02 13:02:07.921235 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:07.921208    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk" podUID="5fea819a-f3de-4ac8-92fe-acc5a8227857" containerName="kserve-container" containerID="cri-o://68318ae2aaedcc2518c7324441adc1d43549c3db7f6eaa9c467abd5c60fd40a2" gracePeriod=30
Jul 02 13:02:07.921357 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:07.921243    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk" podUID="5fea819a-f3de-4ac8-92fe-acc5a8227857" containerName="kube-rbac-proxy" containerID="cri-o://413abeeae022edfa7d846a232a7e8ab9e98c4f321fd1419f5a451cf6b703d057" gracePeriod=30
Jul 02 13:02:07.982741 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:07.982713    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:02:08.048914 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.048883    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw"]
Jul 02 13:02:08.049233 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.049219    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95ade19c-8fea-4214-8d6a-7c5264e03c0e" containerName="kube-rbac-proxy"
Jul 02 13:02:08.049314 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.049234    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="95ade19c-8fea-4214-8d6a-7c5264e03c0e" containerName="kube-rbac-proxy"
Jul 02 13:02:08.049314 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.049250    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95ade19c-8fea-4214-8d6a-7c5264e03c0e" containerName="kserve-container"
Jul 02 13:02:08.049314 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.049255    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="95ade19c-8fea-4214-8d6a-7c5264e03c0e" containerName="kserve-container"
Jul 02 13:02:08.049314 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.049263    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="689be393-9525-45ff-93b3-ba50eeaa6dde" containerName="kube-rbac-proxy"
Jul 02 13:02:08.049314 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.049268    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="689be393-9525-45ff-93b3-ba50eeaa6dde" containerName="kube-rbac-proxy"
Jul 02 13:02:08.049314 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.049279    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="689be393-9525-45ff-93b3-ba50eeaa6dde" containerName="kserve-container"
Jul 02 13:02:08.049314 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.049285    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="689be393-9525-45ff-93b3-ba50eeaa6dde" containerName="kserve-container"
Jul 02 13:02:08.049691 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.049334    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="95ade19c-8fea-4214-8d6a-7c5264e03c0e" containerName="kserve-container"
Jul 02 13:02:08.049691 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.049343    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="689be393-9525-45ff-93b3-ba50eeaa6dde" containerName="kube-rbac-proxy"
Jul 02 13:02:08.049691 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.049350    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="689be393-9525-45ff-93b3-ba50eeaa6dde" containerName="kserve-container"
Jul 02 13:02:08.049691 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.049356    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="95ade19c-8fea-4214-8d6a-7c5264e03c0e" containerName="kube-rbac-proxy"
Jul 02 13:02:08.052232 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.052215    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw"
Jul 02 13:02:08.058655 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.058622    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b2e72-kube-rbac-proxy-sar-config\""
Jul 02 13:02:08.073621 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.073550    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b2e72-predictor-serving-cert\""
Jul 02 13:02:08.082304 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.082278    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw"]
Jul 02 13:02:08.111325 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.111296    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f5xj\" (UniqueName: \"kubernetes.io/projected/2a63b743-a196-48a1-a6cb-dd7dcb38efad-kube-api-access-6f5xj\") pod \"success-200-isvc-b2e72-predictor-6cbff7944f-jcljw\" (UID: \"2a63b743-a196-48a1-a6cb-dd7dcb38efad\") " pod="kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw"
Jul 02 13:02:08.111516 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.111339    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a63b743-a196-48a1-a6cb-dd7dcb38efad-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b2e72-predictor-6cbff7944f-jcljw\" (UID: \"2a63b743-a196-48a1-a6cb-dd7dcb38efad\") " pod="kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw"
Jul 02 13:02:08.111516 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.111455    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a63b743-a196-48a1-a6cb-dd7dcb38efad-proxy-tls\") pod \"success-200-isvc-b2e72-predictor-6cbff7944f-jcljw\" (UID: \"2a63b743-a196-48a1-a6cb-dd7dcb38efad\") " pod="kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw"
Jul 02 13:02:08.121061 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.121026    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:02:08.194333 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.194304    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z"]
Jul 02 13:02:08.194588 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.194559    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z" podUID="1b200e0e-cf4f-450b-8b90-57be599a90d3" containerName="kserve-container" containerID="cri-o://f84bea15a5082c0f343031a6f91c3f6850c701043d724ac8565a92e9ad381734" gracePeriod=30
Jul 02 13:02:08.194751 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.194592    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z" podUID="1b200e0e-cf4f-450b-8b90-57be599a90d3" containerName="kube-rbac-proxy" containerID="cri-o://3b6b43d48a6e8a771e90cac5e5487aa2f197dbf04605d7e43ea7c57d847ac28c" gracePeriod=30
Jul 02 13:02:08.211969 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.211943    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f5xj\" (UniqueName: \"kubernetes.io/projected/2a63b743-a196-48a1-a6cb-dd7dcb38efad-kube-api-access-6f5xj\") pod \"success-200-isvc-b2e72-predictor-6cbff7944f-jcljw\" (UID: \"2a63b743-a196-48a1-a6cb-dd7dcb38efad\") " pod="kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw"
Jul 02 13:02:08.212067 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.211987    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a63b743-a196-48a1-a6cb-dd7dcb38efad-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b2e72-predictor-6cbff7944f-jcljw\" (UID: \"2a63b743-a196-48a1-a6cb-dd7dcb38efad\") " pod="kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw"
Jul 02 13:02:08.212067 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.212052    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a63b743-a196-48a1-a6cb-dd7dcb38efad-proxy-tls\") pod \"success-200-isvc-b2e72-predictor-6cbff7944f-jcljw\" (UID: \"2a63b743-a196-48a1-a6cb-dd7dcb38efad\") " pod="kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw"
Jul 02 13:02:08.212205 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:02:08.212170    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-serving-cert: secret "success-200-isvc-b2e72-predictor-serving-cert" not found
Jul 02 13:02:08.212273 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:02:08.212251    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a63b743-a196-48a1-a6cb-dd7dcb38efad-proxy-tls podName:2a63b743-a196-48a1-a6cb-dd7dcb38efad nodeName:}" failed. No retries permitted until 2026-07-02 13:02:08.712230137 +0000 UTC m=+1396.845055975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2a63b743-a196-48a1-a6cb-dd7dcb38efad-proxy-tls") pod "success-200-isvc-b2e72-predictor-6cbff7944f-jcljw" (UID: "2a63b743-a196-48a1-a6cb-dd7dcb38efad") : secret "success-200-isvc-b2e72-predictor-serving-cert" not found
Jul 02 13:02:08.212781 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.212760    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a63b743-a196-48a1-a6cb-dd7dcb38efad-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b2e72-predictor-6cbff7944f-jcljw\" (UID: \"2a63b743-a196-48a1-a6cb-dd7dcb38efad\") " pod="kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw"
Jul 02 13:02:08.220020 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.219999    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978"]
Jul 02 13:02:08.222254 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.222228    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:02:08.223469 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.223452    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978"
Jul 02 13:02:08.224778 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.224757    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f5xj\" (UniqueName: \"kubernetes.io/projected/2a63b743-a196-48a1-a6cb-dd7dcb38efad-kube-api-access-6f5xj\") pod \"success-200-isvc-b2e72-predictor-6cbff7944f-jcljw\" (UID: \"2a63b743-a196-48a1-a6cb-dd7dcb38efad\") " pod="kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw"
Jul 02 13:02:08.226666 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.226650    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b2e72-predictor-serving-cert\""
Jul 02 13:02:08.226767 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.226752    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b2e72-kube-rbac-proxy-sar-config\""
Jul 02 13:02:08.235380 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.235358    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978"]
Jul 02 13:02:08.288382 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.288350    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:02:08.313234 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.313203    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn9nd\" (UniqueName: \"kubernetes.io/projected/4ff8e35a-c9be-462d-abce-3ae66c142bd1-kube-api-access-jn9nd\") pod \"error-404-isvc-b2e72-predictor-59d9b5c96d-xl978\" (UID: \"4ff8e35a-c9be-462d-abce-3ae66c142bd1\") " pod="kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978"
Jul 02 13:02:08.313428 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.313253    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ff8e35a-c9be-462d-abce-3ae66c142bd1-proxy-tls\") pod \"error-404-isvc-b2e72-predictor-59d9b5c96d-xl978\" (UID: \"4ff8e35a-c9be-462d-abce-3ae66c142bd1\") " pod="kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978"
Jul 02 13:02:08.313428 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.313403    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ff8e35a-c9be-462d-abce-3ae66c142bd1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b2e72-predictor-59d9b5c96d-xl978\" (UID: \"4ff8e35a-c9be-462d-abce-3ae66c142bd1\") " pod="kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978"
Jul 02 13:02:08.414345 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.414261    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn9nd\" (UniqueName: \"kubernetes.io/projected/4ff8e35a-c9be-462d-abce-3ae66c142bd1-kube-api-access-jn9nd\") pod \"error-404-isvc-b2e72-predictor-59d9b5c96d-xl978\" (UID: \"4ff8e35a-c9be-462d-abce-3ae66c142bd1\") " pod="kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978"
Jul 02 13:02:08.414345 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.414313    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ff8e35a-c9be-462d-abce-3ae66c142bd1-proxy-tls\") pod \"error-404-isvc-b2e72-predictor-59d9b5c96d-xl978\" (UID: \"4ff8e35a-c9be-462d-abce-3ae66c142bd1\") " pod="kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978"
Jul 02 13:02:08.414579 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:02:08.414412    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-serving-cert: secret "error-404-isvc-b2e72-predictor-serving-cert" not found
Jul 02 13:02:08.414579 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:02:08.414494    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ff8e35a-c9be-462d-abce-3ae66c142bd1-proxy-tls podName:4ff8e35a-c9be-462d-abce-3ae66c142bd1 nodeName:}" failed. No retries permitted until 2026-07-02 13:02:08.914474216 +0000 UTC m=+1397.047300057 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4ff8e35a-c9be-462d-abce-3ae66c142bd1-proxy-tls") pod "error-404-isvc-b2e72-predictor-59d9b5c96d-xl978" (UID: "4ff8e35a-c9be-462d-abce-3ae66c142bd1") : secret "error-404-isvc-b2e72-predictor-serving-cert" not found
Jul 02 13:02:08.414579 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.414422    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ff8e35a-c9be-462d-abce-3ae66c142bd1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b2e72-predictor-59d9b5c96d-xl978\" (UID: \"4ff8e35a-c9be-462d-abce-3ae66c142bd1\") " pod="kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978"
Jul 02 13:02:08.415127 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.415094    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ff8e35a-c9be-462d-abce-3ae66c142bd1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b2e72-predictor-59d9b5c96d-xl978\" (UID: \"4ff8e35a-c9be-462d-abce-3ae66c142bd1\") " pod="kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978"
Jul 02 13:02:08.426574 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.426545    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn9nd\" (UniqueName: \"kubernetes.io/projected/4ff8e35a-c9be-462d-abce-3ae66c142bd1-kube-api-access-jn9nd\") pod \"error-404-isvc-b2e72-predictor-59d9b5c96d-xl978\" (UID: \"4ff8e35a-c9be-462d-abce-3ae66c142bd1\") " pod="kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978"
Jul 02 13:02:08.637806 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.637742    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:02:08.713050 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.712966    2578 generic.go:356] "Generic (PLEG): container finished" podID="1b200e0e-cf4f-450b-8b90-57be599a90d3" containerID="3b6b43d48a6e8a771e90cac5e5487aa2f197dbf04605d7e43ea7c57d847ac28c" exitCode=2
Jul 02 13:02:08.713236 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.713040    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z" event={"ID":"1b200e0e-cf4f-450b-8b90-57be599a90d3","Type":"ContainerDied","Data":"3b6b43d48a6e8a771e90cac5e5487aa2f197dbf04605d7e43ea7c57d847ac28c"}
Jul 02 13:02:08.714747 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.714725    2578 generic.go:356] "Generic (PLEG): container finished" podID="5fea819a-f3de-4ac8-92fe-acc5a8227857" containerID="413abeeae022edfa7d846a232a7e8ab9e98c4f321fd1419f5a451cf6b703d057" exitCode=2
Jul 02 13:02:08.714846 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.714781    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk" event={"ID":"5fea819a-f3de-4ac8-92fe-acc5a8227857","Type":"ContainerDied","Data":"413abeeae022edfa7d846a232a7e8ab9e98c4f321fd1419f5a451cf6b703d057"}
Jul 02 13:02:08.717136 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.717119    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a63b743-a196-48a1-a6cb-dd7dcb38efad-proxy-tls\") pod \"success-200-isvc-b2e72-predictor-6cbff7944f-jcljw\" (UID: \"2a63b743-a196-48a1-a6cb-dd7dcb38efad\") " pod="kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw"
Jul 02 13:02:08.719460 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.719439    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a63b743-a196-48a1-a6cb-dd7dcb38efad-proxy-tls\") pod \"success-200-isvc-b2e72-predictor-6cbff7944f-jcljw\" (UID: \"2a63b743-a196-48a1-a6cb-dd7dcb38efad\") " pod="kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw"
Jul 02 13:02:08.738960 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.738929    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:02:08.919473 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.919423    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ff8e35a-c9be-462d-abce-3ae66c142bd1-proxy-tls\") pod \"error-404-isvc-b2e72-predictor-59d9b5c96d-xl978\" (UID: \"4ff8e35a-c9be-462d-abce-3ae66c142bd1\") " pod="kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978"
Jul 02 13:02:08.921824 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.921804    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ff8e35a-c9be-462d-abce-3ae66c142bd1-proxy-tls\") pod \"error-404-isvc-b2e72-predictor-59d9b5c96d-xl978\" (UID: \"4ff8e35a-c9be-462d-abce-3ae66c142bd1\") " pod="kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978"
Jul 02 13:02:08.961864 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:08.961831    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw"
Jul 02 13:02:09.094227 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:09.094065    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw"]
Jul 02 13:02:09.096326 ip-10-0-137-127 kubenswrapper[2578]: W0702 13:02:09.096301    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a63b743_a196_48a1_a6cb_dd7dcb38efad.slice/crio-590e5b93f2c5080d73f510c3af93113210cf9d89b743ece21aa3bde2f5c8c045 WatchSource:0}: Error finding container 590e5b93f2c5080d73f510c3af93113210cf9d89b743ece21aa3bde2f5c8c045: Status 404 returned error can't find the container with id 590e5b93f2c5080d73f510c3af93113210cf9d89b743ece21aa3bde2f5c8c045
Jul 02 13:02:09.143805 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:09.143782    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978"
Jul 02 13:02:09.287525 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:09.287463    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978"]
Jul 02 13:02:09.289479 ip-10-0-137-127 kubenswrapper[2578]: W0702 13:02:09.289452    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ff8e35a_c9be_462d_abce_3ae66c142bd1.slice/crio-04bc83ac08459c0f9f8bd6adda54fd3db375fb31ab40ed6e1f8c1a7e168dc736 WatchSource:0}: Error finding container 04bc83ac08459c0f9f8bd6adda54fd3db375fb31ab40ed6e1f8c1a7e168dc736: Status 404 returned error can't find the container with id 04bc83ac08459c0f9f8bd6adda54fd3db375fb31ab40ed6e1f8c1a7e168dc736
Jul 02 13:02:09.405265 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:09.405233    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z" podUID="1b200e0e-cf4f-450b-8b90-57be599a90d3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 02 13:02:09.720794 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:09.720752    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw" event={"ID":"2a63b743-a196-48a1-a6cb-dd7dcb38efad","Type":"ContainerStarted","Data":"587ee091b770d62f4fb9433ec0f4b5c2d5b2593a72e497779e6020a95a364e96"}
Jul 02 13:02:09.720997 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:09.720833    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw"
Jul 02 13:02:09.720997 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:09.720868    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw" event={"ID":"2a63b743-a196-48a1-a6cb-dd7dcb38efad","Type":"ContainerStarted","Data":"0ca0e2398e7cd29a9ac74943bf7deb0a6451aea1f11f27ca2dd92fbbc6a25269"}
Jul 02 13:02:09.720997 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:09.720899    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw"
Jul 02 13:02:09.720997 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:09.720925    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw" event={"ID":"2a63b743-a196-48a1-a6cb-dd7dcb38efad","Type":"ContainerStarted","Data":"590e5b93f2c5080d73f510c3af93113210cf9d89b743ece21aa3bde2f5c8c045"}
Jul 02 13:02:09.722620 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:09.722580    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw" podUID="2a63b743-a196-48a1-a6cb-dd7dcb38efad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 02 13:02:09.722745 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:09.722722    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978" event={"ID":"4ff8e35a-c9be-462d-abce-3ae66c142bd1","Type":"ContainerStarted","Data":"06dfce3b7f6c54630136fad5cebff508cfbdae8eafdb077d3802aa186021930c"}
Jul 02 13:02:09.722829 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:09.722780    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978" event={"ID":"4ff8e35a-c9be-462d-abce-3ae66c142bd1","Type":"ContainerStarted","Data":"e4e825e0e6ab8920715a6d80792f3abff44de4cdd2b33967d816206fda979b28"}
Jul 02 13:02:09.722829 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:09.722820    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978"
Jul 02 13:02:09.722935 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:09.722852    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978" event={"ID":"4ff8e35a-c9be-462d-abce-3ae66c142bd1","Type":"ContainerStarted","Data":"04bc83ac08459c0f9f8bd6adda54fd3db375fb31ab40ed6e1f8c1a7e168dc736"}
Jul 02 13:02:09.743243 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:09.743170    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw" podStartSLOduration=2.743154809 podStartE2EDuration="2.743154809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:02:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:02:09.74284345 +0000 UTC m=+1397.875669346" watchObservedRunningTime="2026-07-02 13:02:09.743154809 +0000 UTC m=+1397.875980654"
Jul 02 13:02:09.760974 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:09.760931    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978" podStartSLOduration=1.760919936 podStartE2EDuration="1.760919936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:02:09.760049941 +0000 UTC m=+1397.892875788" watchObservedRunningTime="2026-07-02 13:02:09.760919936 +0000 UTC m=+1397.893745836"
Jul 02 13:02:10.409372 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:10.409329    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk" podUID="5fea819a-f3de-4ac8-92fe-acc5a8227857" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 02 13:02:10.727135 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:10.727030    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw" podUID="2a63b743-a196-48a1-a6cb-dd7dcb38efad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 02 13:02:10.727135 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:10.727119    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978"
Jul 02 13:02:10.728583 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:10.728542    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978" podUID="4ff8e35a-c9be-462d-abce-3ae66c142bd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 02 13:02:11.616796 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:11.616770    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk"
Jul 02 13:02:11.738912 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:11.738883    2578 generic.go:356] "Generic (PLEG): container finished" podID="1b200e0e-cf4f-450b-8b90-57be599a90d3" containerID="f84bea15a5082c0f343031a6f91c3f6850c701043d724ac8565a92e9ad381734" exitCode=0
Jul 02 13:02:11.739046 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:11.738957    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z" event={"ID":"1b200e0e-cf4f-450b-8b90-57be599a90d3","Type":"ContainerDied","Data":"f84bea15a5082c0f343031a6f91c3f6850c701043d724ac8565a92e9ad381734"}
Jul 02 13:02:11.739046 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:11.739034    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z" event={"ID":"1b200e0e-cf4f-450b-8b90-57be599a90d3","Type":"ContainerDied","Data":"c4fcbb3b034165d501036c3a9acf987f24fd75326de30d5d3b844dfbb25c9641"}
Jul 02 13:02:11.739203 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:11.739067    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4fcbb3b034165d501036c3a9acf987f24fd75326de30d5d3b844dfbb25c9641"
Jul 02 13:02:11.740520 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:11.740497    2578 generic.go:356] "Generic (PLEG): container finished" podID="5fea819a-f3de-4ac8-92fe-acc5a8227857" containerID="68318ae2aaedcc2518c7324441adc1d43549c3db7f6eaa9c467abd5c60fd40a2" exitCode=0
Jul 02 13:02:11.740622 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:11.740524    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk" event={"ID":"5fea819a-f3de-4ac8-92fe-acc5a8227857","Type":"ContainerDied","Data":"68318ae2aaedcc2518c7324441adc1d43549c3db7f6eaa9c467abd5c60fd40a2"}
Jul 02 13:02:11.740622 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:11.740556    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk"
Jul 02 13:02:11.740622 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:11.740572    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk" event={"ID":"5fea819a-f3de-4ac8-92fe-acc5a8227857","Type":"ContainerDied","Data":"e30bea835fc35ad13f82ce46ab2dfa409c18e4c2c35c0993feb268e453a8c4af"}
Jul 02 13:02:11.740622 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:11.740601    2578 scope.go:117] "RemoveContainer" containerID="413abeeae022edfa7d846a232a7e8ab9e98c4f321fd1419f5a451cf6b703d057"
Jul 02 13:02:11.740989 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:11.740955    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978" podUID="4ff8e35a-c9be-462d-abce-3ae66c142bd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 02 13:02:11.741936 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:11.741921    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z"
Jul 02 13:02:11.746657 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:11.746641    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2kkk\" (UniqueName: \"kubernetes.io/projected/5fea819a-f3de-4ac8-92fe-acc5a8227857-kube-api-access-f2kkk\") pod \"5fea819a-f3de-4ac8-92fe-acc5a8227857\" (UID: \"5fea819a-f3de-4ac8-92fe-acc5a8227857\") "
Jul 02 13:02:11.746751 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:11.746676    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fea819a-f3de-4ac8-92fe-acc5a8227857-proxy-tls\") pod \"5fea819a-f3de-4ac8-92fe-acc5a8227857\" (UID: \"5fea819a-f3de-4ac8-92fe-acc5a8227857\") "
Jul 02 13:02:11.746751 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:11.746718    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fea819a-f3de-4ac8-92fe-acc5a8227857-kube-rbac-proxy-sar-config\") pod \"5fea819a-f3de-4ac8-92fe-acc5a8227857\" (UID: \"5fea819a-f3de-4ac8-92fe-acc5a8227857\") "
Jul 02 13:02:11.747110 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:11.747087    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fea819a-f3de-4ac8-92fe-acc5a8227857-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5fea819a-f3de-4ac8-92fe-acc5a8227857" (UID: "5fea819a-f3de-4ac8-92fe-acc5a8227857"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:02:11.747329 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:11.747312    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fea819a-f3de-4ac8-92fe-acc5a8227857-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:02:11.748437 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:11.748416    2578 scope.go:117] "RemoveContainer" containerID="68318ae2aaedcc2518c7324441adc1d43549c3db7f6eaa9c467abd5c60fd40a2"
Jul 02 13:02:11.748733 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:11.748708    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fea819a-f3de-4ac8-92fe-acc5a8227857-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5fea819a-f3de-4ac8-92fe-acc5a8227857" (UID: "5fea819a-f3de-4ac8-92fe-acc5a8227857"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:02:11.748868 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:11.748726    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fea819a-f3de-4ac8-92fe-acc5a8227857-kube-api-access-f2kkk" (OuterVolumeSpecName: "kube-api-access-f2kkk") pod "5fea819a-f3de-4ac8-92fe-acc5a8227857" (UID: "5fea819a-f3de-4ac8-92fe-acc5a8227857"). InnerVolumeSpecName "kube-api-access-f2kkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:02:11.755593 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:11.755574    2578 scope.go:117] "RemoveContainer" containerID="413abeeae022edfa7d846a232a7e8ab9e98c4f321fd1419f5a451cf6b703d057"
Jul 02 13:02:11.755856 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:02:11.755839    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"413abeeae022edfa7d846a232a7e8ab9e98c4f321fd1419f5a451cf6b703d057\": container with ID starting with 413abeeae022edfa7d846a232a7e8ab9e98c4f321fd1419f5a451cf6b703d057 not found: ID does not exist" containerID="413abeeae022edfa7d846a232a7e8ab9e98c4f321fd1419f5a451cf6b703d057"
Jul 02 13:02:11.755934 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:11.755869    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"413abeeae022edfa7d846a232a7e8ab9e98c4f321fd1419f5a451cf6b703d057"} err="failed to get container status \"413abeeae022edfa7d846a232a7e8ab9e98c4f321fd1419f5a451cf6b703d057\": rpc error: code = NotFound desc = could not find container \"413abeeae022edfa7d846a232a7e8ab9e98c4f321fd1419f5a451cf6b703d057\": container with ID starting with 413abeeae022edfa7d846a232a7e8ab9e98c4f321fd1419f5a451cf6b703d057 not found: ID does not exist"
Jul 02 13:02:11.755934 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:11.755891    2578 scope.go:117] "RemoveContainer" containerID="68318ae2aaedcc2518c7324441adc1d43549c3db7f6eaa9c467abd5c60fd40a2"
Jul 02 13:02:11.756190 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:02:11.756157    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68318ae2aaedcc2518c7324441adc1d43549c3db7f6eaa9c467abd5c60fd40a2\": container with ID starting with 68318ae2aaedcc2518c7324441adc1d43549c3db7f6eaa9c467abd5c60fd40a2 not found: ID does not exist" containerID="68318ae2aaedcc2518c7324441adc1d43549c3db7f6eaa9c467abd5c60fd40a2"
Jul 02 13:02:11.756285 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:11.756202    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68318ae2aaedcc2518c7324441adc1d43549c3db7f6eaa9c467abd5c60fd40a2"} err="failed to get container status \"68318ae2aaedcc2518c7324441adc1d43549c3db7f6eaa9c467abd5c60fd40a2\": rpc error: code = NotFound desc = could not find container \"68318ae2aaedcc2518c7324441adc1d43549c3db7f6eaa9c467abd5c60fd40a2\": container with ID starting with 68318ae2aaedcc2518c7324441adc1d43549c3db7f6eaa9c467abd5c60fd40a2 not found: ID does not exist"
Jul 02 13:02:11.847597 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:11.847561    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b200e0e-cf4f-450b-8b90-57be599a90d3-proxy-tls\") pod \"1b200e0e-cf4f-450b-8b90-57be599a90d3\" (UID: \"1b200e0e-cf4f-450b-8b90-57be599a90d3\") "
Jul 02 13:02:11.847787 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:11.847611    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ht62b\" (UniqueName: \"kubernetes.io/projected/1b200e0e-cf4f-450b-8b90-57be599a90d3-kube-api-access-ht62b\") pod \"1b200e0e-cf4f-450b-8b90-57be599a90d3\" (UID: \"1b200e0e-cf4f-450b-8b90-57be599a90d3\") "
Jul 02 13:02:11.847787 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:11.847693    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b200e0e-cf4f-450b-8b90-57be599a90d3-kube-rbac-proxy-sar-config\") pod \"1b200e0e-cf4f-450b-8b90-57be599a90d3\" (UID: \"1b200e0e-cf4f-450b-8b90-57be599a90d3\") "
Jul 02 13:02:11.848225 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:11.847933    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2kkk\" (UniqueName: \"kubernetes.io/projected/5fea819a-f3de-4ac8-92fe-acc5a8227857-kube-api-access-f2kkk\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:02:11.848225 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:11.847953    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fea819a-f3de-4ac8-92fe-acc5a8227857-proxy-tls\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:02:11.848225 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:11.848132    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b200e0e-cf4f-450b-8b90-57be599a90d3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1b200e0e-cf4f-450b-8b90-57be599a90d3" (UID: "1b200e0e-cf4f-450b-8b90-57be599a90d3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:02:11.849790 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:11.849732    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b200e0e-cf4f-450b-8b90-57be599a90d3-kube-api-access-ht62b" (OuterVolumeSpecName: "kube-api-access-ht62b") pod "1b200e0e-cf4f-450b-8b90-57be599a90d3" (UID: "1b200e0e-cf4f-450b-8b90-57be599a90d3"). InnerVolumeSpecName "kube-api-access-ht62b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:02:11.849865 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:11.849788    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b200e0e-cf4f-450b-8b90-57be599a90d3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1b200e0e-cf4f-450b-8b90-57be599a90d3" (UID: "1b200e0e-cf4f-450b-8b90-57be599a90d3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:02:11.948545 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:11.948512    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b200e0e-cf4f-450b-8b90-57be599a90d3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:02:11.948545 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:11.948538    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b200e0e-cf4f-450b-8b90-57be599a90d3-proxy-tls\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:02:11.948545 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:11.948550    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht62b\" (UniqueName: \"kubernetes.io/projected/1b200e0e-cf4f-450b-8b90-57be599a90d3-kube-api-access-ht62b\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:02:12.065073 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:12.065042    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk"]
Jul 02 13:02:12.067462 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:12.067433    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c0346-predictor-6fd846fb8d-bd5rk"]
Jul 02 13:02:12.362440 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:12.362409    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fea819a-f3de-4ac8-92fe-acc5a8227857" path="/var/lib/kubelet/pods/5fea819a-f3de-4ac8-92fe-acc5a8227857/volumes"
Jul 02 13:02:12.745101 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:12.745080    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z"
Jul 02 13:02:12.764923 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:12.764894    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z"]
Jul 02 13:02:12.769743 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:12.769716    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c0346-predictor-848ccd5d77-kzr6z"]
Jul 02 13:02:14.362326 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:14.362297    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b200e0e-cf4f-450b-8b90-57be599a90d3" path="/var/lib/kubelet/pods/1b200e0e-cf4f-450b-8b90-57be599a90d3/volumes"
Jul 02 13:02:15.645089 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:15.645055    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd" podUID="c4d57fc5-4f72-40e4-bbbc-fcff17489250" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 02 13:02:15.732488 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:15.732454    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw"
Jul 02 13:02:15.733080 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:15.733051    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw" podUID="2a63b743-a196-48a1-a6cb-dd7dcb38efad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 02 13:02:16.651877 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:16.651837    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k" podUID="1413e900-a9c6-4606-bcfe-b60c3813e400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 02 13:02:16.746827 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:16.746789    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978"
Jul 02 13:02:16.747245 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:16.747217    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978" podUID="4ff8e35a-c9be-462d-abce-3ae66c142bd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 02 13:02:25.646222 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:25.645993    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd" podUID="c4d57fc5-4f72-40e4-bbbc-fcff17489250" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 02 13:02:25.733962 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:25.733910    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw" podUID="2a63b743-a196-48a1-a6cb-dd7dcb38efad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 02 13:02:26.651729 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:26.651689    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k" podUID="1413e900-a9c6-4606-bcfe-b60c3813e400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 02 13:02:26.748050 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:26.748013    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978" podUID="4ff8e35a-c9be-462d-abce-3ae66c142bd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 02 13:02:35.644902 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:35.644860    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd" podUID="c4d57fc5-4f72-40e4-bbbc-fcff17489250" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 02 13:02:35.734110 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:35.734064    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw" podUID="2a63b743-a196-48a1-a6cb-dd7dcb38efad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 02 13:02:36.651936 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:36.651899    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k" podUID="1413e900-a9c6-4606-bcfe-b60c3813e400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 02 13:02:36.747428 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:36.747389    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978" podUID="4ff8e35a-c9be-462d-abce-3ae66c142bd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 02 13:02:45.646451 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:45.646373    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd"
Jul 02 13:02:45.733211 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:45.733163    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw" podUID="2a63b743-a196-48a1-a6cb-dd7dcb38efad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 02 13:02:46.652361 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:46.652328    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k"
Jul 02 13:02:46.747391 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:46.747358    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978" podUID="4ff8e35a-c9be-462d-abce-3ae66c142bd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 02 13:02:55.732995 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:55.732952    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw" podUID="2a63b743-a196-48a1-a6cb-dd7dcb38efad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 02 13:02:56.747796 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:56.747765    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978"
Jul 02 13:02:57.732655 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:57.732623    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:02:57.998196 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:02:57.998142    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:03:05.734328 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:05.734299    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw"
Jul 02 13:03:07.813578 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:07.813540    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:03:07.894035 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:07.894006    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k"]
Jul 02 13:03:07.894350 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:07.894318    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k" podUID="1413e900-a9c6-4606-bcfe-b60c3813e400" containerName="kserve-container" containerID="cri-o://06550560ab336914c6d1e0b199916a3abb2d38d5edaf062ec872796927a0ef35" gracePeriod=30
Jul 02 13:03:07.894491 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:07.894366    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k" podUID="1413e900-a9c6-4606-bcfe-b60c3813e400" containerName="kube-rbac-proxy" containerID="cri-o://c457f9641362ad1a4c45fcc35c5714ad0f7cbe831d300d25f4ce666a51e2fa96" gracePeriod=30
Jul 02 13:03:07.948265 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:07.948236    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:03:07.948875 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:07.948854    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5"]
Jul 02 13:03:07.949234 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:07.949220    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b200e0e-cf4f-450b-8b90-57be599a90d3" containerName="kube-rbac-proxy"
Jul 02 13:03:07.949286 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:07.949236    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b200e0e-cf4f-450b-8b90-57be599a90d3" containerName="kube-rbac-proxy"
Jul 02 13:03:07.949286 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:07.949249    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fea819a-f3de-4ac8-92fe-acc5a8227857" containerName="kserve-container"
Jul 02 13:03:07.949286 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:07.949257    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fea819a-f3de-4ac8-92fe-acc5a8227857" containerName="kserve-container"
Jul 02 13:03:07.949286 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:07.949269    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b200e0e-cf4f-450b-8b90-57be599a90d3" containerName="kserve-container"
Jul 02 13:03:07.949286 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:07.949278    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b200e0e-cf4f-450b-8b90-57be599a90d3" containerName="kserve-container"
Jul 02 13:03:07.949286 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:07.949286    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fea819a-f3de-4ac8-92fe-acc5a8227857" containerName="kube-rbac-proxy"
Jul 02 13:03:07.949500 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:07.949294    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fea819a-f3de-4ac8-92fe-acc5a8227857" containerName="kube-rbac-proxy"
Jul 02 13:03:07.949500 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:07.949379    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fea819a-f3de-4ac8-92fe-acc5a8227857" containerName="kserve-container"
Jul 02 13:03:07.949500 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:07.949394    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b200e0e-cf4f-450b-8b90-57be599a90d3" containerName="kserve-container"
Jul 02 13:03:07.949500 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:07.949405    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fea819a-f3de-4ac8-92fe-acc5a8227857" containerName="kube-rbac-proxy"
Jul 02 13:03:07.949500 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:07.949414    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b200e0e-cf4f-450b-8b90-57be599a90d3" containerName="kube-rbac-proxy"
Jul 02 13:03:07.953694 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:07.953676    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5"
Jul 02 13:03:07.957508 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:07.957483    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5b6c7-predictor-serving-cert\""
Jul 02 13:03:07.958147 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:07.958132    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5b6c7-kube-rbac-proxy-sar-config\""
Jul 02 13:03:07.964252 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:07.964232    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5"]
Jul 02 13:03:07.973342 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:07.973315    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59c38447-8a34-40df-9cbb-71e57dbb7e5a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5\" (UID: \"59c38447-8a34-40df-9cbb-71e57dbb7e5a\") " pod="kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5"
Jul 02 13:03:07.973425 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:07.973383    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59c38447-8a34-40df-9cbb-71e57dbb7e5a-proxy-tls\") pod \"success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5\" (UID: \"59c38447-8a34-40df-9cbb-71e57dbb7e5a\") " pod="kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5"
Jul 02 13:03:07.973466 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:07.973426    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85k2m\" (UniqueName: \"kubernetes.io/projected/59c38447-8a34-40df-9cbb-71e57dbb7e5a-kube-api-access-85k2m\") pod \"success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5\" (UID: \"59c38447-8a34-40df-9cbb-71e57dbb7e5a\") " pod="kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5"
Jul 02 13:03:08.026934 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.026847    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:03:08.034852 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.034818    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd"]
Jul 02 13:03:08.035061 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.035025    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd" podUID="c4d57fc5-4f72-40e4-bbbc-fcff17489250" containerName="kserve-container" containerID="cri-o://5404f478cc1281a56b11699a42d4b2279a869d60eed353e86043c78be697ad89" gracePeriod=30
Jul 02 13:03:08.035198 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.035080    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd" podUID="c4d57fc5-4f72-40e4-bbbc-fcff17489250" containerName="kube-rbac-proxy" containerID="cri-o://b2634cdd74703280219d697804f8cffa7f21321a30ad9f73fa395c322599f260" gracePeriod=30
Jul 02 13:03:08.051815 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.051790    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8"]
Jul 02 13:03:08.055054 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.055036    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8"
Jul 02 13:03:08.057987 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.057970    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-5b6c7-predictor-serving-cert\""
Jul 02 13:03:08.058076 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.057974    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-5b6c7-kube-rbac-proxy-sar-config\""
Jul 02 13:03:08.069415 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.069361    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8"]
Jul 02 13:03:08.074150 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.074125    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59c38447-8a34-40df-9cbb-71e57dbb7e5a-proxy-tls\") pod \"success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5\" (UID: \"59c38447-8a34-40df-9cbb-71e57dbb7e5a\") " pod="kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5"
Jul 02 13:03:08.074265 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.074173    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85k2m\" (UniqueName: \"kubernetes.io/projected/59c38447-8a34-40df-9cbb-71e57dbb7e5a-kube-api-access-85k2m\") pod \"success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5\" (UID: \"59c38447-8a34-40df-9cbb-71e57dbb7e5a\") " pod="kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5"
Jul 02 13:03:08.074265 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.074259    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36d5af22-ebaa-484e-a59f-6f404f343543-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8\" (UID: \"36d5af22-ebaa-484e-a59f-6f404f343543\") " pod="kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8"
Jul 02 13:03:08.074380 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.074296    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59c38447-8a34-40df-9cbb-71e57dbb7e5a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5\" (UID: \"59c38447-8a34-40df-9cbb-71e57dbb7e5a\") " pod="kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5"
Jul 02 13:03:08.074380 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:03:08.074307    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-serving-cert: secret "success-200-isvc-5b6c7-predictor-serving-cert" not found
Jul 02 13:03:08.074380 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.074326    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkg4n\" (UniqueName: \"kubernetes.io/projected/36d5af22-ebaa-484e-a59f-6f404f343543-kube-api-access-lkg4n\") pod \"error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8\" (UID: \"36d5af22-ebaa-484e-a59f-6f404f343543\") " pod="kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8"
Jul 02 13:03:08.074380 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.074355    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36d5af22-ebaa-484e-a59f-6f404f343543-proxy-tls\") pod \"error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8\" (UID: \"36d5af22-ebaa-484e-a59f-6f404f343543\") " pod="kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8"
Jul 02 13:03:08.074380 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:03:08.074375    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59c38447-8a34-40df-9cbb-71e57dbb7e5a-proxy-tls podName:59c38447-8a34-40df-9cbb-71e57dbb7e5a nodeName:}" failed. No retries permitted until 2026-07-02 13:03:08.574355702 +0000 UTC m=+1456.707181540 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/59c38447-8a34-40df-9cbb-71e57dbb7e5a-proxy-tls") pod "success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5" (UID: "59c38447-8a34-40df-9cbb-71e57dbb7e5a") : secret "success-200-isvc-5b6c7-predictor-serving-cert" not found
Jul 02 13:03:08.075027 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.075004    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59c38447-8a34-40df-9cbb-71e57dbb7e5a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5\" (UID: \"59c38447-8a34-40df-9cbb-71e57dbb7e5a\") " pod="kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5"
Jul 02 13:03:08.085266 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.085244    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85k2m\" (UniqueName: \"kubernetes.io/projected/59c38447-8a34-40df-9cbb-71e57dbb7e5a-kube-api-access-85k2m\") pod \"success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5\" (UID: \"59c38447-8a34-40df-9cbb-71e57dbb7e5a\") " pod="kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5"
Jul 02 13:03:08.121559 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.121536    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:03:08.174854 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.174827    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36d5af22-ebaa-484e-a59f-6f404f343543-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8\" (UID: \"36d5af22-ebaa-484e-a59f-6f404f343543\") " pod="kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8"
Jul 02 13:03:08.174942 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.174868    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkg4n\" (UniqueName: \"kubernetes.io/projected/36d5af22-ebaa-484e-a59f-6f404f343543-kube-api-access-lkg4n\") pod \"error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8\" (UID: \"36d5af22-ebaa-484e-a59f-6f404f343543\") " pod="kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8"
Jul 02 13:03:08.174942 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.174891    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36d5af22-ebaa-484e-a59f-6f404f343543-proxy-tls\") pod \"error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8\" (UID: \"36d5af22-ebaa-484e-a59f-6f404f343543\") " pod="kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8"
Jul 02 13:03:08.175461 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.175443    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36d5af22-ebaa-484e-a59f-6f404f343543-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8\" (UID: \"36d5af22-ebaa-484e-a59f-6f404f343543\") " pod="kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8"
Jul 02 13:03:08.177088 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.177067    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36d5af22-ebaa-484e-a59f-6f404f343543-proxy-tls\") pod \"error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8\" (UID: \"36d5af22-ebaa-484e-a59f-6f404f343543\") " pod="kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8"
Jul 02 13:03:08.185227 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.185198    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkg4n\" (UniqueName: \"kubernetes.io/projected/36d5af22-ebaa-484e-a59f-6f404f343543-kube-api-access-lkg4n\") pod \"error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8\" (UID: \"36d5af22-ebaa-484e-a59f-6f404f343543\") " pod="kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8"
Jul 02 13:03:08.221691 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.221666    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:03:08.365859 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.365789    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8"
Jul 02 13:03:08.496314 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.496291    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8"]
Jul 02 13:03:08.498763 ip-10-0-137-127 kubenswrapper[2578]: W0702 13:03:08.498738    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36d5af22_ebaa_484e_a59f_6f404f343543.slice/crio-683f42cb1e50b2777c215ce12ab833159984b3af21ecba665ab66e54073a61db WatchSource:0}: Error finding container 683f42cb1e50b2777c215ce12ab833159984b3af21ecba665ab66e54073a61db: Status 404 returned error can't find the container with id 683f42cb1e50b2777c215ce12ab833159984b3af21ecba665ab66e54073a61db
Jul 02 13:03:08.578527 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.578503    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59c38447-8a34-40df-9cbb-71e57dbb7e5a-proxy-tls\") pod \"success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5\" (UID: \"59c38447-8a34-40df-9cbb-71e57dbb7e5a\") " pod="kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5"
Jul 02 13:03:08.581230 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.581211    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59c38447-8a34-40df-9cbb-71e57dbb7e5a-proxy-tls\") pod \"success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5\" (UID: \"59c38447-8a34-40df-9cbb-71e57dbb7e5a\") " pod="kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5"
Jul 02 13:03:08.864328 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.864296    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5"
Jul 02 13:03:08.976159 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.976131    2578 generic.go:356] "Generic (PLEG): container finished" podID="1413e900-a9c6-4606-bcfe-b60c3813e400" containerID="c457f9641362ad1a4c45fcc35c5714ad0f7cbe831d300d25f4ce666a51e2fa96" exitCode=2
Jul 02 13:03:08.976325 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.976218    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k" event={"ID":"1413e900-a9c6-4606-bcfe-b60c3813e400","Type":"ContainerDied","Data":"c457f9641362ad1a4c45fcc35c5714ad0f7cbe831d300d25f4ce666a51e2fa96"}
Jul 02 13:03:08.977856 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.977829    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8" event={"ID":"36d5af22-ebaa-484e-a59f-6f404f343543","Type":"ContainerStarted","Data":"227f4e4854d676893b6d42e9f959e41cdbde59a5761806cbf71830507e043add"}
Jul 02 13:03:08.978000 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.977982    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8"
Jul 02 13:03:08.978073 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.978035    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8"
Jul 02 13:03:08.978073 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.978063    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8" event={"ID":"36d5af22-ebaa-484e-a59f-6f404f343543","Type":"ContainerStarted","Data":"06ad1cf27d14560e3cd34078d8b539287124638e68d6a47c5e3b524de3f5519b"}
Jul 02 13:03:08.978157 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.978092    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8" event={"ID":"36d5af22-ebaa-484e-a59f-6f404f343543","Type":"ContainerStarted","Data":"683f42cb1e50b2777c215ce12ab833159984b3af21ecba665ab66e54073a61db"}
Jul 02 13:03:08.978964 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.978928    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8" podUID="36d5af22-ebaa-484e-a59f-6f404f343543" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 02 13:03:08.979914 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.979892    2578 generic.go:356] "Generic (PLEG): container finished" podID="c4d57fc5-4f72-40e4-bbbc-fcff17489250" containerID="b2634cdd74703280219d697804f8cffa7f21321a30ad9f73fa395c322599f260" exitCode=2
Jul 02 13:03:08.980011 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.979927    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd" event={"ID":"c4d57fc5-4f72-40e4-bbbc-fcff17489250","Type":"ContainerDied","Data":"b2634cdd74703280219d697804f8cffa7f21321a30ad9f73fa395c322599f260"}
Jul 02 13:03:08.994374 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.993988    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5"]
Jul 02 13:03:08.996577 ip-10-0-137-127 kubenswrapper[2578]: W0702 13:03:08.996556    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59c38447_8a34_40df_9cbb_71e57dbb7e5a.slice/crio-fb1c1c64740aed8e205f141edbb2f91c1b769f0fd0f9d07dce418896965416e3 WatchSource:0}: Error finding container fb1c1c64740aed8e205f141edbb2f91c1b769f0fd0f9d07dce418896965416e3: Status 404 returned error can't find the container with id fb1c1c64740aed8e205f141edbb2f91c1b769f0fd0f9d07dce418896965416e3
Jul 02 13:03:08.999515 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:08.999473    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8" podStartSLOduration=0.999458504 podStartE2EDuration="999.458504ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:03:08.997444157 +0000 UTC m=+1457.130270003" watchObservedRunningTime="2026-07-02 13:03:08.999458504 +0000 UTC m=+1457.132284350"
Jul 02 13:03:09.985622 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:09.985585    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5" event={"ID":"59c38447-8a34-40df-9cbb-71e57dbb7e5a","Type":"ContainerStarted","Data":"e51d97fb85bac058792fce166cde13f60133ad7c47212398bb8db5072e729f2f"}
Jul 02 13:03:09.985994 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:09.985654    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5" event={"ID":"59c38447-8a34-40df-9cbb-71e57dbb7e5a","Type":"ContainerStarted","Data":"fd38296b53d41ae63554264b0b19fa0d5294d59d1008a47a91d1f5f4f2511dd7"}
Jul 02 13:03:09.985994 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:09.985677    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5" event={"ID":"59c38447-8a34-40df-9cbb-71e57dbb7e5a","Type":"ContainerStarted","Data":"fb1c1c64740aed8e205f141edbb2f91c1b769f0fd0f9d07dce418896965416e3"}
Jul 02 13:03:09.986069 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:09.986003    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8" podUID="36d5af22-ebaa-484e-a59f-6f404f343543" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 02 13:03:10.005407 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:10.005367    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5" podStartSLOduration=3.005351872 podStartE2EDuration="3.005351872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:03:10.004944774 +0000 UTC m=+1458.137770646" watchObservedRunningTime="2026-07-02 13:03:10.005351872 +0000 UTC m=+1458.138177717"
Jul 02 13:03:10.640372 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:10.640334    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd" podUID="c4d57fc5-4f72-40e4-bbbc-fcff17489250" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 02 13:03:10.990254 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:10.990230    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5"
Jul 02 13:03:10.990584 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:10.990301    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5"
Jul 02 13:03:10.991443 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:10.991415    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5" podUID="59c38447-8a34-40df-9cbb-71e57dbb7e5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 02 13:03:11.134917 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:11.134893    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k"
Jul 02 13:03:11.197501 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:11.197434    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1413e900-a9c6-4606-bcfe-b60c3813e400-proxy-tls\") pod \"1413e900-a9c6-4606-bcfe-b60c3813e400\" (UID: \"1413e900-a9c6-4606-bcfe-b60c3813e400\") "
Jul 02 13:03:11.197501 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:11.197480    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1413e900-a9c6-4606-bcfe-b60c3813e400-kube-rbac-proxy-sar-config\") pod \"1413e900-a9c6-4606-bcfe-b60c3813e400\" (UID: \"1413e900-a9c6-4606-bcfe-b60c3813e400\") "
Jul 02 13:03:11.197705 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:11.197525    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4gsf\" (UniqueName: \"kubernetes.io/projected/1413e900-a9c6-4606-bcfe-b60c3813e400-kube-api-access-k4gsf\") pod \"1413e900-a9c6-4606-bcfe-b60c3813e400\" (UID: \"1413e900-a9c6-4606-bcfe-b60c3813e400\") "
Jul 02 13:03:11.197854 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:11.197825    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1413e900-a9c6-4606-bcfe-b60c3813e400-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1413e900-a9c6-4606-bcfe-b60c3813e400" (UID: "1413e900-a9c6-4606-bcfe-b60c3813e400"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:03:11.199451 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:11.199429    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1413e900-a9c6-4606-bcfe-b60c3813e400-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1413e900-a9c6-4606-bcfe-b60c3813e400" (UID: "1413e900-a9c6-4606-bcfe-b60c3813e400"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:03:11.199538 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:11.199496    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1413e900-a9c6-4606-bcfe-b60c3813e400-kube-api-access-k4gsf" (OuterVolumeSpecName: "kube-api-access-k4gsf") pod "1413e900-a9c6-4606-bcfe-b60c3813e400" (UID: "1413e900-a9c6-4606-bcfe-b60c3813e400"). InnerVolumeSpecName "kube-api-access-k4gsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:03:11.272867 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:11.272845    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd"
Jul 02 13:03:11.298886 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:11.298858    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxdbb\" (UniqueName: \"kubernetes.io/projected/c4d57fc5-4f72-40e4-bbbc-fcff17489250-kube-api-access-cxdbb\") pod \"c4d57fc5-4f72-40e4-bbbc-fcff17489250\" (UID: \"c4d57fc5-4f72-40e4-bbbc-fcff17489250\") "
Jul 02 13:03:11.299028 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:11.298937    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4d57fc5-4f72-40e4-bbbc-fcff17489250-proxy-tls\") pod \"c4d57fc5-4f72-40e4-bbbc-fcff17489250\" (UID: \"c4d57fc5-4f72-40e4-bbbc-fcff17489250\") "
Jul 02 13:03:11.299028 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:11.298968    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4d57fc5-4f72-40e4-bbbc-fcff17489250-kube-rbac-proxy-sar-config\") pod \"c4d57fc5-4f72-40e4-bbbc-fcff17489250\" (UID: \"c4d57fc5-4f72-40e4-bbbc-fcff17489250\") "
Jul 02 13:03:11.299231 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:11.299213    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1413e900-a9c6-4606-bcfe-b60c3813e400-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:03:11.299307 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:11.299235    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4gsf\" (UniqueName: \"kubernetes.io/projected/1413e900-a9c6-4606-bcfe-b60c3813e400-kube-api-access-k4gsf\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:03:11.299307 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:11.299251    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1413e900-a9c6-4606-bcfe-b60c3813e400-proxy-tls\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:03:11.299410 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:11.299357    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4d57fc5-4f72-40e4-bbbc-fcff17489250-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c4d57fc5-4f72-40e4-bbbc-fcff17489250" (UID: "c4d57fc5-4f72-40e4-bbbc-fcff17489250"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:03:11.300877 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:11.300857    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4d57fc5-4f72-40e4-bbbc-fcff17489250-kube-api-access-cxdbb" (OuterVolumeSpecName: "kube-api-access-cxdbb") pod "c4d57fc5-4f72-40e4-bbbc-fcff17489250" (UID: "c4d57fc5-4f72-40e4-bbbc-fcff17489250"). InnerVolumeSpecName "kube-api-access-cxdbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:03:11.300934 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:11.300894    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4d57fc5-4f72-40e4-bbbc-fcff17489250-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c4d57fc5-4f72-40e4-bbbc-fcff17489250" (UID: "c4d57fc5-4f72-40e4-bbbc-fcff17489250"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:03:11.400560 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:11.400532    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxdbb\" (UniqueName: \"kubernetes.io/projected/c4d57fc5-4f72-40e4-bbbc-fcff17489250-kube-api-access-cxdbb\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:03:11.400560 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:11.400557    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4d57fc5-4f72-40e4-bbbc-fcff17489250-proxy-tls\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:03:11.400560 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:11.400566    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4d57fc5-4f72-40e4-bbbc-fcff17489250-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:03:11.995686 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:11.995654    2578 generic.go:356] "Generic (PLEG): container finished" podID="1413e900-a9c6-4606-bcfe-b60c3813e400" containerID="06550560ab336914c6d1e0b199916a3abb2d38d5edaf062ec872796927a0ef35" exitCode=0
Jul 02 13:03:11.996121 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:11.995699    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k"
Jul 02 13:03:11.996121 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:11.995750    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k" event={"ID":"1413e900-a9c6-4606-bcfe-b60c3813e400","Type":"ContainerDied","Data":"06550560ab336914c6d1e0b199916a3abb2d38d5edaf062ec872796927a0ef35"}
Jul 02 13:03:11.996121 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:11.995804    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k" event={"ID":"1413e900-a9c6-4606-bcfe-b60c3813e400","Type":"ContainerDied","Data":"3d5569a67f2e1b3fa1757fadfbce9f83d308fbc525f0df17a6c361ab36a333af"}
Jul 02 13:03:11.996121 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:11.995831    2578 scope.go:117] "RemoveContainer" containerID="c457f9641362ad1a4c45fcc35c5714ad0f7cbe831d300d25f4ce666a51e2fa96"
Jul 02 13:03:11.997381 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:11.997356    2578 generic.go:356] "Generic (PLEG): container finished" podID="c4d57fc5-4f72-40e4-bbbc-fcff17489250" containerID="5404f478cc1281a56b11699a42d4b2279a869d60eed353e86043c78be697ad89" exitCode=0
Jul 02 13:03:11.997497 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:11.997398    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd"
Jul 02 13:03:11.997497 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:11.997436    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd" event={"ID":"c4d57fc5-4f72-40e4-bbbc-fcff17489250","Type":"ContainerDied","Data":"5404f478cc1281a56b11699a42d4b2279a869d60eed353e86043c78be697ad89"}
Jul 02 13:03:11.997611 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:11.997500    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd" event={"ID":"c4d57fc5-4f72-40e4-bbbc-fcff17489250","Type":"ContainerDied","Data":"74e51770466c5c952b4c2161b4c245170467b0b9aeb4a2e248fa92ff05840f20"}
Jul 02 13:03:11.997843 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:11.997794    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5" podUID="59c38447-8a34-40df-9cbb-71e57dbb7e5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 02 13:03:12.005426 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:12.005404    2578 scope.go:117] "RemoveContainer" containerID="06550560ab336914c6d1e0b199916a3abb2d38d5edaf062ec872796927a0ef35"
Jul 02 13:03:12.012634 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:12.012617    2578 scope.go:117] "RemoveContainer" containerID="c457f9641362ad1a4c45fcc35c5714ad0f7cbe831d300d25f4ce666a51e2fa96"
Jul 02 13:03:12.012839 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:03:12.012822    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c457f9641362ad1a4c45fcc35c5714ad0f7cbe831d300d25f4ce666a51e2fa96\": container with ID starting with c457f9641362ad1a4c45fcc35c5714ad0f7cbe831d300d25f4ce666a51e2fa96 not found: ID does not exist" containerID="c457f9641362ad1a4c45fcc35c5714ad0f7cbe831d300d25f4ce666a51e2fa96"
Jul 02 13:03:12.012890 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:12.012850    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c457f9641362ad1a4c45fcc35c5714ad0f7cbe831d300d25f4ce666a51e2fa96"} err="failed to get container status \"c457f9641362ad1a4c45fcc35c5714ad0f7cbe831d300d25f4ce666a51e2fa96\": rpc error: code = NotFound desc = could not find container \"c457f9641362ad1a4c45fcc35c5714ad0f7cbe831d300d25f4ce666a51e2fa96\": container with ID starting with c457f9641362ad1a4c45fcc35c5714ad0f7cbe831d300d25f4ce666a51e2fa96 not found: ID does not exist"
Jul 02 13:03:12.012890 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:12.012872    2578 scope.go:117] "RemoveContainer" containerID="06550560ab336914c6d1e0b199916a3abb2d38d5edaf062ec872796927a0ef35"
Jul 02 13:03:12.013127 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:03:12.013108    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06550560ab336914c6d1e0b199916a3abb2d38d5edaf062ec872796927a0ef35\": container with ID starting with 06550560ab336914c6d1e0b199916a3abb2d38d5edaf062ec872796927a0ef35 not found: ID does not exist" containerID="06550560ab336914c6d1e0b199916a3abb2d38d5edaf062ec872796927a0ef35"
Jul 02 13:03:12.013204 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:12.013131    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06550560ab336914c6d1e0b199916a3abb2d38d5edaf062ec872796927a0ef35"} err="failed to get container status \"06550560ab336914c6d1e0b199916a3abb2d38d5edaf062ec872796927a0ef35\": rpc error: code = NotFound desc = could not find container \"06550560ab336914c6d1e0b199916a3abb2d38d5edaf062ec872796927a0ef35\": container with ID starting with 06550560ab336914c6d1e0b199916a3abb2d38d5edaf062ec872796927a0ef35 not found: ID does not exist"
Jul 02 13:03:12.013204 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:12.013145    2578 scope.go:117] "RemoveContainer" containerID="b2634cdd74703280219d697804f8cffa7f21321a30ad9f73fa395c322599f260"
Jul 02 13:03:12.019650 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:12.019630    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k"]
Jul 02 13:03:12.020243 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:12.020228    2578 scope.go:117] "RemoveContainer" containerID="5404f478cc1281a56b11699a42d4b2279a869d60eed353e86043c78be697ad89"
Jul 02 13:03:12.027607 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:12.027592    2578 scope.go:117] "RemoveContainer" containerID="b2634cdd74703280219d697804f8cffa7f21321a30ad9f73fa395c322599f260"
Jul 02 13:03:12.027914 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:03:12.027891    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2634cdd74703280219d697804f8cffa7f21321a30ad9f73fa395c322599f260\": container with ID starting with b2634cdd74703280219d697804f8cffa7f21321a30ad9f73fa395c322599f260 not found: ID does not exist" containerID="b2634cdd74703280219d697804f8cffa7f21321a30ad9f73fa395c322599f260"
Jul 02 13:03:12.027984 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:12.027923    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2634cdd74703280219d697804f8cffa7f21321a30ad9f73fa395c322599f260"} err="failed to get container status \"b2634cdd74703280219d697804f8cffa7f21321a30ad9f73fa395c322599f260\": rpc error: code = NotFound desc = could not find container \"b2634cdd74703280219d697804f8cffa7f21321a30ad9f73fa395c322599f260\": container with ID starting with b2634cdd74703280219d697804f8cffa7f21321a30ad9f73fa395c322599f260 not found: ID does not exist"
Jul 02 13:03:12.027984 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:12.027945    2578 scope.go:117] "RemoveContainer" containerID="5404f478cc1281a56b11699a42d4b2279a869d60eed353e86043c78be697ad89"
Jul 02 13:03:12.028114 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:12.028099    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7a990-predictor-9f9c44d9b-9v57k"]
Jul 02 13:03:12.028241 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:03:12.028216    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5404f478cc1281a56b11699a42d4b2279a869d60eed353e86043c78be697ad89\": container with ID starting with 5404f478cc1281a56b11699a42d4b2279a869d60eed353e86043c78be697ad89 not found: ID does not exist" containerID="5404f478cc1281a56b11699a42d4b2279a869d60eed353e86043c78be697ad89"
Jul 02 13:03:12.028343 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:12.028248    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5404f478cc1281a56b11699a42d4b2279a869d60eed353e86043c78be697ad89"} err="failed to get container status \"5404f478cc1281a56b11699a42d4b2279a869d60eed353e86043c78be697ad89\": rpc error: code = NotFound desc = could not find container \"5404f478cc1281a56b11699a42d4b2279a869d60eed353e86043c78be697ad89\": container with ID starting with 5404f478cc1281a56b11699a42d4b2279a869d60eed353e86043c78be697ad89 not found: ID does not exist"
Jul 02 13:03:12.040497 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:12.040480    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd"]
Jul 02 13:03:12.044257 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:12.044238    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7a990-predictor-7b76c758b9-pbfdd"]
Jul 02 13:03:12.362275 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:12.362246    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1413e900-a9c6-4606-bcfe-b60c3813e400" path="/var/lib/kubelet/pods/1413e900-a9c6-4606-bcfe-b60c3813e400/volumes"
Jul 02 13:03:12.362667 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:12.362653    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4d57fc5-4f72-40e4-bbbc-fcff17489250" path="/var/lib/kubelet/pods/c4d57fc5-4f72-40e4-bbbc-fcff17489250/volumes"
Jul 02 13:03:14.990857 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:14.990832    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8"
Jul 02 13:03:14.991307 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:14.991278    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8" podUID="36d5af22-ebaa-484e-a59f-6f404f343543" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 02 13:03:17.002259 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:17.002228    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5"
Jul 02 13:03:17.002705 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:17.002670    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5" podUID="59c38447-8a34-40df-9cbb-71e57dbb7e5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 02 13:03:18.039997 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:18.039968    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:03:18.326662 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:18.326575    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:03:24.991838 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:24.991791    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8" podUID="36d5af22-ebaa-484e-a59f-6f404f343543" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 02 13:03:27.002752 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:27.002709    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5" podUID="59c38447-8a34-40df-9cbb-71e57dbb7e5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 02 13:03:34.991423 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:34.991378    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8" podUID="36d5af22-ebaa-484e-a59f-6f404f343543" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 02 13:03:37.003239 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:37.003198    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5" podUID="59c38447-8a34-40df-9cbb-71e57dbb7e5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 02 13:03:44.991517 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:44.991472    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8" podUID="36d5af22-ebaa-484e-a59f-6f404f343543" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 02 13:03:47.003520 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:47.003478    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5" podUID="59c38447-8a34-40df-9cbb-71e57dbb7e5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 02 13:03:52.407479 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:52.407452    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95kwm_495e9fa8-b93f-42e0-95b5-813527df0993/ovn-acl-logging/0.log"
Jul 02 13:03:52.408209 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:52.408191    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95kwm_495e9fa8-b93f-42e0-95b5-813527df0993/ovn-acl-logging/0.log"
Jul 02 13:03:54.992430 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:54.992400    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8"
Jul 02 13:03:57.004238 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:03:57.004199    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5"
Jul 02 13:04:09.403750 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:04:09.403714    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:04:11.792964 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:04:11.792929    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:06:47.523194 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:06:47.523155    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:06:47.635389 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:06:47.635356    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:07:11.308360 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:07:11.308326    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:07:52.488102 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:07:52.488073    2578 scope.go:117] "RemoveContainer" containerID="3b6b43d48a6e8a771e90cac5e5487aa2f197dbf04605d7e43ea7c57d847ac28c"
Jul 02 13:07:52.496188 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:07:52.496158    2578 scope.go:117] "RemoveContainer" containerID="f84bea15a5082c0f343031a6f91c3f6850c701043d724ac8565a92e9ad381734"
Jul 02 13:08:52.431872 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:08:52.431844    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95kwm_495e9fa8-b93f-42e0-95b5-813527df0993/ovn-acl-logging/0.log"
Jul 02 13:08:52.432862 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:08:52.432836    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95kwm_495e9fa8-b93f-42e0-95b5-813527df0993/ovn-acl-logging/0.log"
Jul 02 13:11:32.849777 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:32.849722    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:11:33.010919 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.010885    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx"]
Jul 02 13:11:33.011414 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.011399    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1413e900-a9c6-4606-bcfe-b60c3813e400" containerName="kserve-container"
Jul 02 13:11:33.011481 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.011417    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1413e900-a9c6-4606-bcfe-b60c3813e400" containerName="kserve-container"
Jul 02 13:11:33.011481 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.011428    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1413e900-a9c6-4606-bcfe-b60c3813e400" containerName="kube-rbac-proxy"
Jul 02 13:11:33.011481 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.011437    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1413e900-a9c6-4606-bcfe-b60c3813e400" containerName="kube-rbac-proxy"
Jul 02 13:11:33.011481 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.011453    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4d57fc5-4f72-40e4-bbbc-fcff17489250" containerName="kube-rbac-proxy"
Jul 02 13:11:33.011481 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.011463    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4d57fc5-4f72-40e4-bbbc-fcff17489250" containerName="kube-rbac-proxy"
Jul 02 13:11:33.011656 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.011484    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4d57fc5-4f72-40e4-bbbc-fcff17489250" containerName="kserve-container"
Jul 02 13:11:33.011656 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.011492    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4d57fc5-4f72-40e4-bbbc-fcff17489250" containerName="kserve-container"
Jul 02 13:11:33.011656 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.011559    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1413e900-a9c6-4606-bcfe-b60c3813e400" containerName="kserve-container"
Jul 02 13:11:33.011656 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.011572    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4d57fc5-4f72-40e4-bbbc-fcff17489250" containerName="kserve-container"
Jul 02 13:11:33.011656 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.011584    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1413e900-a9c6-4606-bcfe-b60c3813e400" containerName="kube-rbac-proxy"
Jul 02 13:11:33.011656 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.011593    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4d57fc5-4f72-40e4-bbbc-fcff17489250" containerName="kube-rbac-proxy"
Jul 02 13:11:33.014817 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.014797    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx"
Jul 02 13:11:33.019191 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.019152    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-63f60-kube-rbac-proxy-sar-config\""
Jul 02 13:11:33.019310 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.019157    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-63f60-predictor-serving-cert\""
Jul 02 13:11:33.029095 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.029069    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx"]
Jul 02 13:11:33.086518 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.086479    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw"]
Jul 02 13:11:33.086794 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.086767    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw" podUID="2a63b743-a196-48a1-a6cb-dd7dcb38efad" containerName="kserve-container" containerID="cri-o://0ca0e2398e7cd29a9ac74943bf7deb0a6451aea1f11f27ca2dd92fbbc6a25269" gracePeriod=30
Jul 02 13:11:33.086908 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.086793    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw" podUID="2a63b743-a196-48a1-a6cb-dd7dcb38efad" containerName="kube-rbac-proxy" containerID="cri-o://587ee091b770d62f4fb9433ec0f4b5c2d5b2593a72e497779e6020a95a364e96" gracePeriod=30
Jul 02 13:11:33.096027 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.096003    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/233babba-5bbc-4832-9ba6-06063f0edd75-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-63f60-predictor-fc76d56f5-rrcpx\" (UID: \"233babba-5bbc-4832-9ba6-06063f0edd75\") " pod="kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx"
Jul 02 13:11:33.096133 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.096083    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/233babba-5bbc-4832-9ba6-06063f0edd75-proxy-tls\") pod \"success-200-isvc-63f60-predictor-fc76d56f5-rrcpx\" (UID: \"233babba-5bbc-4832-9ba6-06063f0edd75\") " pod="kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx"
Jul 02 13:11:33.096133 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.096124    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v29k5\" (UniqueName: \"kubernetes.io/projected/233babba-5bbc-4832-9ba6-06063f0edd75-kube-api-access-v29k5\") pod \"success-200-isvc-63f60-predictor-fc76d56f5-rrcpx\" (UID: \"233babba-5bbc-4832-9ba6-06063f0edd75\") " pod="kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx"
Jul 02 13:11:33.100111 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.100048    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf"]
Jul 02 13:11:33.103350 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.103333    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf"
Jul 02 13:11:33.107995 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.107977    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-63f60-predictor-serving-cert\""
Jul 02 13:11:33.108094 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.107977    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-63f60-kube-rbac-proxy-sar-config\""
Jul 02 13:11:33.122979 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.122954    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf"]
Jul 02 13:11:33.164584 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.164554    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978"]
Jul 02 13:11:33.164864 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.164834    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978" podUID="4ff8e35a-c9be-462d-abce-3ae66c142bd1" containerName="kserve-container" containerID="cri-o://e4e825e0e6ab8920715a6d80792f3abff44de4cdd2b33967d816206fda979b28" gracePeriod=30
Jul 02 13:11:33.165017 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.164902    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978" podUID="4ff8e35a-c9be-462d-abce-3ae66c142bd1" containerName="kube-rbac-proxy" containerID="cri-o://06dfce3b7f6c54630136fad5cebff508cfbdae8eafdb077d3802aa186021930c" gracePeriod=30
Jul 02 13:11:33.197520 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.197487    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v29k5\" (UniqueName: \"kubernetes.io/projected/233babba-5bbc-4832-9ba6-06063f0edd75-kube-api-access-v29k5\") pod \"success-200-isvc-63f60-predictor-fc76d56f5-rrcpx\" (UID: \"233babba-5bbc-4832-9ba6-06063f0edd75\") " pod="kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx"
Jul 02 13:11:33.197685 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.197546    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/233babba-5bbc-4832-9ba6-06063f0edd75-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-63f60-predictor-fc76d56f5-rrcpx\" (UID: \"233babba-5bbc-4832-9ba6-06063f0edd75\") " pod="kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx"
Jul 02 13:11:33.197685 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.197585    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72b2c\" (UniqueName: \"kubernetes.io/projected/b6d34e5f-d9f1-4a6e-889f-90ea3304f93d-kube-api-access-72b2c\") pod \"error-404-isvc-63f60-predictor-58dc685fdb-6f6zf\" (UID: \"b6d34e5f-d9f1-4a6e-889f-90ea3304f93d\") " pod="kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf"
Jul 02 13:11:33.197795 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.197715    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6d34e5f-d9f1-4a6e-889f-90ea3304f93d-proxy-tls\") pod \"error-404-isvc-63f60-predictor-58dc685fdb-6f6zf\" (UID: \"b6d34e5f-d9f1-4a6e-889f-90ea3304f93d\") " pod="kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf"
Jul 02 13:11:33.197795 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.197766    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/233babba-5bbc-4832-9ba6-06063f0edd75-proxy-tls\") pod \"success-200-isvc-63f60-predictor-fc76d56f5-rrcpx\" (UID: \"233babba-5bbc-4832-9ba6-06063f0edd75\") " pod="kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx"
Jul 02 13:11:33.197891 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.197795    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6d34e5f-d9f1-4a6e-889f-90ea3304f93d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-63f60-predictor-58dc685fdb-6f6zf\" (UID: \"b6d34e5f-d9f1-4a6e-889f-90ea3304f93d\") " pod="kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf"
Jul 02 13:11:33.197943 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:11:33.197875    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-63f60-predictor-serving-cert: secret "success-200-isvc-63f60-predictor-serving-cert" not found
Jul 02 13:11:33.198008 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:11:33.197994    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/233babba-5bbc-4832-9ba6-06063f0edd75-proxy-tls podName:233babba-5bbc-4832-9ba6-06063f0edd75 nodeName:}" failed. No retries permitted until 2026-07-02 13:11:33.697972378 +0000 UTC m=+1961.830798209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/233babba-5bbc-4832-9ba6-06063f0edd75-proxy-tls") pod "success-200-isvc-63f60-predictor-fc76d56f5-rrcpx" (UID: "233babba-5bbc-4832-9ba6-06063f0edd75") : secret "success-200-isvc-63f60-predictor-serving-cert" not found
Jul 02 13:11:33.198828 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.198802    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/233babba-5bbc-4832-9ba6-06063f0edd75-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-63f60-predictor-fc76d56f5-rrcpx\" (UID: \"233babba-5bbc-4832-9ba6-06063f0edd75\") " pod="kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx"
Jul 02 13:11:33.207358 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.207329    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v29k5\" (UniqueName: \"kubernetes.io/projected/233babba-5bbc-4832-9ba6-06063f0edd75-kube-api-access-v29k5\") pod \"success-200-isvc-63f60-predictor-fc76d56f5-rrcpx\" (UID: \"233babba-5bbc-4832-9ba6-06063f0edd75\") " pod="kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx"
Jul 02 13:11:33.298948 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.298907    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72b2c\" (UniqueName: \"kubernetes.io/projected/b6d34e5f-d9f1-4a6e-889f-90ea3304f93d-kube-api-access-72b2c\") pod \"error-404-isvc-63f60-predictor-58dc685fdb-6f6zf\" (UID: \"b6d34e5f-d9f1-4a6e-889f-90ea3304f93d\") " pod="kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf"
Jul 02 13:11:33.299103 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.298975    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6d34e5f-d9f1-4a6e-889f-90ea3304f93d-proxy-tls\") pod \"error-404-isvc-63f60-predictor-58dc685fdb-6f6zf\" (UID: \"b6d34e5f-d9f1-4a6e-889f-90ea3304f93d\") " pod="kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf"
Jul 02 13:11:33.299103 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.299025    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6d34e5f-d9f1-4a6e-889f-90ea3304f93d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-63f60-predictor-58dc685fdb-6f6zf\" (UID: \"b6d34e5f-d9f1-4a6e-889f-90ea3304f93d\") " pod="kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf"
Jul 02 13:11:33.299241 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:11:33.299120    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-63f60-predictor-serving-cert: secret "error-404-isvc-63f60-predictor-serving-cert" not found
Jul 02 13:11:33.299241 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:11:33.299209    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6d34e5f-d9f1-4a6e-889f-90ea3304f93d-proxy-tls podName:b6d34e5f-d9f1-4a6e-889f-90ea3304f93d nodeName:}" failed. No retries permitted until 2026-07-02 13:11:33.799168734 +0000 UTC m=+1961.931994558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b6d34e5f-d9f1-4a6e-889f-90ea3304f93d-proxy-tls") pod "error-404-isvc-63f60-predictor-58dc685fdb-6f6zf" (UID: "b6d34e5f-d9f1-4a6e-889f-90ea3304f93d") : secret "error-404-isvc-63f60-predictor-serving-cert" not found
Jul 02 13:11:33.299811 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.299790    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6d34e5f-d9f1-4a6e-889f-90ea3304f93d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-63f60-predictor-58dc685fdb-6f6zf\" (UID: \"b6d34e5f-d9f1-4a6e-889f-90ea3304f93d\") " pod="kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf"
Jul 02 13:11:33.310192 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.310153    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72b2c\" (UniqueName: \"kubernetes.io/projected/b6d34e5f-d9f1-4a6e-889f-90ea3304f93d-kube-api-access-72b2c\") pod \"error-404-isvc-63f60-predictor-58dc685fdb-6f6zf\" (UID: \"b6d34e5f-d9f1-4a6e-889f-90ea3304f93d\") " pod="kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf"
Jul 02 13:11:33.703255 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.703218    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/233babba-5bbc-4832-9ba6-06063f0edd75-proxy-tls\") pod \"success-200-isvc-63f60-predictor-fc76d56f5-rrcpx\" (UID: \"233babba-5bbc-4832-9ba6-06063f0edd75\") " pod="kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx"
Jul 02 13:11:33.705734 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.705701    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/233babba-5bbc-4832-9ba6-06063f0edd75-proxy-tls\") pod \"success-200-isvc-63f60-predictor-fc76d56f5-rrcpx\" (UID: \"233babba-5bbc-4832-9ba6-06063f0edd75\") " pod="kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx"
Jul 02 13:11:33.804611 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.804561    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6d34e5f-d9f1-4a6e-889f-90ea3304f93d-proxy-tls\") pod \"error-404-isvc-63f60-predictor-58dc685fdb-6f6zf\" (UID: \"b6d34e5f-d9f1-4a6e-889f-90ea3304f93d\") " pod="kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf"
Jul 02 13:11:33.807012 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.806986    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6d34e5f-d9f1-4a6e-889f-90ea3304f93d-proxy-tls\") pod \"error-404-isvc-63f60-predictor-58dc685fdb-6f6zf\" (UID: \"b6d34e5f-d9f1-4a6e-889f-90ea3304f93d\") " pod="kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf"
Jul 02 13:11:33.928121 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.928083    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx"
Jul 02 13:11:33.939394 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.939368    2578 generic.go:356] "Generic (PLEG): container finished" podID="4ff8e35a-c9be-462d-abce-3ae66c142bd1" containerID="06dfce3b7f6c54630136fad5cebff508cfbdae8eafdb077d3802aa186021930c" exitCode=2
Jul 02 13:11:33.939497 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.939395    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978" event={"ID":"4ff8e35a-c9be-462d-abce-3ae66c142bd1","Type":"ContainerDied","Data":"06dfce3b7f6c54630136fad5cebff508cfbdae8eafdb077d3802aa186021930c"}
Jul 02 13:11:33.941311 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.941287    2578 generic.go:356] "Generic (PLEG): container finished" podID="2a63b743-a196-48a1-a6cb-dd7dcb38efad" containerID="587ee091b770d62f4fb9433ec0f4b5c2d5b2593a72e497779e6020a95a364e96" exitCode=2
Jul 02 13:11:33.941403 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:33.941319    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw" event={"ID":"2a63b743-a196-48a1-a6cb-dd7dcb38efad","Type":"ContainerDied","Data":"587ee091b770d62f4fb9433ec0f4b5c2d5b2593a72e497779e6020a95a364e96"}
Jul 02 13:11:34.016481 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:34.016452    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf"
Jul 02 13:11:34.126090 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:34.126012    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx"]
Jul 02 13:11:34.128985 ip-10-0-137-127 kubenswrapper[2578]: W0702 13:11:34.128954    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod233babba_5bbc_4832_9ba6_06063f0edd75.slice/crio-70bc1c5b8a6743d2f8427e3bafb8f5a802be9de56521a500e403e23cea28c5a7 WatchSource:0}: Error finding container 70bc1c5b8a6743d2f8427e3bafb8f5a802be9de56521a500e403e23cea28c5a7: Status 404 returned error can't find the container with id 70bc1c5b8a6743d2f8427e3bafb8f5a802be9de56521a500e403e23cea28c5a7
Jul 02 13:11:34.152983 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:34.152954    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf"]
Jul 02 13:11:34.155274 ip-10-0-137-127 kubenswrapper[2578]: W0702 13:11:34.155250    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d34e5f_d9f1_4a6e_889f_90ea3304f93d.slice/crio-0d8c0379607e2ed4ed0ecd3d0fe9cc865cde33442d34f8db97c7f03d0f88dbe1 WatchSource:0}: Error finding container 0d8c0379607e2ed4ed0ecd3d0fe9cc865cde33442d34f8db97c7f03d0f88dbe1: Status 404 returned error can't find the container with id 0d8c0379607e2ed4ed0ecd3d0fe9cc865cde33442d34f8db97c7f03d0f88dbe1
Jul 02 13:11:34.920932 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:34.920896    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:11:34.946805 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:34.946772    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx" event={"ID":"233babba-5bbc-4832-9ba6-06063f0edd75","Type":"ContainerStarted","Data":"c22cdc945f713b76224eebfcb3a98d707fb8061de9532a40c4c03c974be0af82"}
Jul 02 13:11:34.947253 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:34.946954    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx"
Jul 02 13:11:34.947253 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:34.947010    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx" event={"ID":"233babba-5bbc-4832-9ba6-06063f0edd75","Type":"ContainerStarted","Data":"275f67115a00f95a220b8efc40f309492f686d0a49bd23adb6eb3ed3e13ba7f9"}
Jul 02 13:11:34.947253 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:34.947045    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx"
Jul 02 13:11:34.947253 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:34.947069    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx" event={"ID":"233babba-5bbc-4832-9ba6-06063f0edd75","Type":"ContainerStarted","Data":"70bc1c5b8a6743d2f8427e3bafb8f5a802be9de56521a500e403e23cea28c5a7"}
Jul 02 13:11:34.948496 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:34.948465    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx" podUID="233babba-5bbc-4832-9ba6-06063f0edd75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 02 13:11:34.948872 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:34.948852    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf" event={"ID":"b6d34e5f-d9f1-4a6e-889f-90ea3304f93d","Type":"ContainerStarted","Data":"621170f28ceed5d957ca1b5d2b04094b1580c58a2e680f85f0272c5070ae0b85"}
Jul 02 13:11:34.948935 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:34.948896    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf" event={"ID":"b6d34e5f-d9f1-4a6e-889f-90ea3304f93d","Type":"ContainerStarted","Data":"122560edf39f71df2b6481767f0f25fc52b7545c7581d0fe302eeb633a9cf218"}
Jul 02 13:11:34.948935 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:34.948921    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf" event={"ID":"b6d34e5f-d9f1-4a6e-889f-90ea3304f93d","Type":"ContainerStarted","Data":"0d8c0379607e2ed4ed0ecd3d0fe9cc865cde33442d34f8db97c7f03d0f88dbe1"}
Jul 02 13:11:34.949015 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:34.948942    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf"
Jul 02 13:11:34.949015 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:34.948991    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf"
Jul 02 13:11:34.949902 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:34.949880    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf" podUID="b6d34e5f-d9f1-4a6e-889f-90ea3304f93d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 02 13:11:34.967118 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:34.967076    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx" podStartSLOduration=2.96706267 podStartE2EDuration="2.96706267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:11:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:11:34.965388248 +0000 UTC m=+1963.098214094" watchObservedRunningTime="2026-07-02 13:11:34.96706267 +0000 UTC m=+1963.099888516"
Jul 02 13:11:34.986973 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:34.986931    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf" podStartSLOduration=1.986918947 podStartE2EDuration="1.986918947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:11:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:11:34.986199445 +0000 UTC m=+1963.119025310" watchObservedRunningTime="2026-07-02 13:11:34.986918947 +0000 UTC m=+1963.119744792"
Jul 02 13:11:35.727932 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:35.727895    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw" podUID="2a63b743-a196-48a1-a6cb-dd7dcb38efad" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 02 13:11:35.733457 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:35.733418    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw" podUID="2a63b743-a196-48a1-a6cb-dd7dcb38efad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 02 13:11:35.953122 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:35.953081    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf" podUID="b6d34e5f-d9f1-4a6e-889f-90ea3304f93d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 02 13:11:35.953601 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:35.953166    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx" podUID="233babba-5bbc-4832-9ba6-06063f0edd75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 02 13:11:36.646670 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.646641    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw"
Jul 02 13:11:36.716783 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.716761    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978"
Jul 02 13:11:36.730895 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.730866    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6f5xj\" (UniqueName: \"kubernetes.io/projected/2a63b743-a196-48a1-a6cb-dd7dcb38efad-kube-api-access-6f5xj\") pod \"2a63b743-a196-48a1-a6cb-dd7dcb38efad\" (UID: \"2a63b743-a196-48a1-a6cb-dd7dcb38efad\") "
Jul 02 13:11:36.731048 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.730908    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a63b743-a196-48a1-a6cb-dd7dcb38efad-kube-rbac-proxy-sar-config\") pod \"2a63b743-a196-48a1-a6cb-dd7dcb38efad\" (UID: \"2a63b743-a196-48a1-a6cb-dd7dcb38efad\") "
Jul 02 13:11:36.731048 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.730980    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a63b743-a196-48a1-a6cb-dd7dcb38efad-proxy-tls\") pod \"2a63b743-a196-48a1-a6cb-dd7dcb38efad\" (UID: \"2a63b743-a196-48a1-a6cb-dd7dcb38efad\") "
Jul 02 13:11:36.731350 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.731316    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a63b743-a196-48a1-a6cb-dd7dcb38efad-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2a63b743-a196-48a1-a6cb-dd7dcb38efad" (UID: "2a63b743-a196-48a1-a6cb-dd7dcb38efad"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:11:36.733079 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.733052    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a63b743-a196-48a1-a6cb-dd7dcb38efad-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2a63b743-a196-48a1-a6cb-dd7dcb38efad" (UID: "2a63b743-a196-48a1-a6cb-dd7dcb38efad"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:11:36.733169 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.733076    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a63b743-a196-48a1-a6cb-dd7dcb38efad-kube-api-access-6f5xj" (OuterVolumeSpecName: "kube-api-access-6f5xj") pod "2a63b743-a196-48a1-a6cb-dd7dcb38efad" (UID: "2a63b743-a196-48a1-a6cb-dd7dcb38efad"). InnerVolumeSpecName "kube-api-access-6f5xj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:11:36.832106 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.832066    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ff8e35a-c9be-462d-abce-3ae66c142bd1-proxy-tls\") pod \"4ff8e35a-c9be-462d-abce-3ae66c142bd1\" (UID: \"4ff8e35a-c9be-462d-abce-3ae66c142bd1\") "
Jul 02 13:11:36.832318 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.832129    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ff8e35a-c9be-462d-abce-3ae66c142bd1-kube-rbac-proxy-sar-config\") pod \"4ff8e35a-c9be-462d-abce-3ae66c142bd1\" (UID: \"4ff8e35a-c9be-462d-abce-3ae66c142bd1\") "
Jul 02 13:11:36.832318 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.832156    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jn9nd\" (UniqueName: \"kubernetes.io/projected/4ff8e35a-c9be-462d-abce-3ae66c142bd1-kube-api-access-jn9nd\") pod \"4ff8e35a-c9be-462d-abce-3ae66c142bd1\" (UID: \"4ff8e35a-c9be-462d-abce-3ae66c142bd1\") "
Jul 02 13:11:36.832440 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.832359    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a63b743-a196-48a1-a6cb-dd7dcb38efad-proxy-tls\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:11:36.832440 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.832377    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f5xj\" (UniqueName: \"kubernetes.io/projected/2a63b743-a196-48a1-a6cb-dd7dcb38efad-kube-api-access-6f5xj\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:11:36.832440 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.832392    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a63b743-a196-48a1-a6cb-dd7dcb38efad-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:11:36.832655 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.832618    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ff8e35a-c9be-462d-abce-3ae66c142bd1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4ff8e35a-c9be-462d-abce-3ae66c142bd1" (UID: "4ff8e35a-c9be-462d-abce-3ae66c142bd1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:11:36.834236 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.834206    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ff8e35a-c9be-462d-abce-3ae66c142bd1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4ff8e35a-c9be-462d-abce-3ae66c142bd1" (UID: "4ff8e35a-c9be-462d-abce-3ae66c142bd1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:11:36.834361 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.834317    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ff8e35a-c9be-462d-abce-3ae66c142bd1-kube-api-access-jn9nd" (OuterVolumeSpecName: "kube-api-access-jn9nd") pod "4ff8e35a-c9be-462d-abce-3ae66c142bd1" (UID: "4ff8e35a-c9be-462d-abce-3ae66c142bd1"). InnerVolumeSpecName "kube-api-access-jn9nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:11:36.933316 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.933278    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ff8e35a-c9be-462d-abce-3ae66c142bd1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:11:36.933316 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.933304    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn9nd\" (UniqueName: \"kubernetes.io/projected/4ff8e35a-c9be-462d-abce-3ae66c142bd1-kube-api-access-jn9nd\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:11:36.933316 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.933314    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ff8e35a-c9be-462d-abce-3ae66c142bd1-proxy-tls\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:11:36.958722 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.958630    2578 generic.go:356] "Generic (PLEG): container finished" podID="2a63b743-a196-48a1-a6cb-dd7dcb38efad" containerID="0ca0e2398e7cd29a9ac74943bf7deb0a6451aea1f11f27ca2dd92fbbc6a25269" exitCode=0
Jul 02 13:11:36.958722 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.958675    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw"
Jul 02 13:11:36.959264 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.958746    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw" event={"ID":"2a63b743-a196-48a1-a6cb-dd7dcb38efad","Type":"ContainerDied","Data":"0ca0e2398e7cd29a9ac74943bf7deb0a6451aea1f11f27ca2dd92fbbc6a25269"}
Jul 02 13:11:36.959264 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.958812    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw" event={"ID":"2a63b743-a196-48a1-a6cb-dd7dcb38efad","Type":"ContainerDied","Data":"590e5b93f2c5080d73f510c3af93113210cf9d89b743ece21aa3bde2f5c8c045"}
Jul 02 13:11:36.959264 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.958852    2578 scope.go:117] "RemoveContainer" containerID="587ee091b770d62f4fb9433ec0f4b5c2d5b2593a72e497779e6020a95a364e96"
Jul 02 13:11:36.960577 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.960552    2578 generic.go:356] "Generic (PLEG): container finished" podID="4ff8e35a-c9be-462d-abce-3ae66c142bd1" containerID="e4e825e0e6ab8920715a6d80792f3abff44de4cdd2b33967d816206fda979b28" exitCode=0
Jul 02 13:11:36.960711 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.960590    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978"
Jul 02 13:11:36.960711 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.960634    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978" event={"ID":"4ff8e35a-c9be-462d-abce-3ae66c142bd1","Type":"ContainerDied","Data":"e4e825e0e6ab8920715a6d80792f3abff44de4cdd2b33967d816206fda979b28"}
Jul 02 13:11:36.960711 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.960698    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978" event={"ID":"4ff8e35a-c9be-462d-abce-3ae66c142bd1","Type":"ContainerDied","Data":"04bc83ac08459c0f9f8bd6adda54fd3db375fb31ab40ed6e1f8c1a7e168dc736"}
Jul 02 13:11:36.968350 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.968331    2578 scope.go:117] "RemoveContainer" containerID="0ca0e2398e7cd29a9ac74943bf7deb0a6451aea1f11f27ca2dd92fbbc6a25269"
Jul 02 13:11:36.976636 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.976621    2578 scope.go:117] "RemoveContainer" containerID="587ee091b770d62f4fb9433ec0f4b5c2d5b2593a72e497779e6020a95a364e96"
Jul 02 13:11:36.976915 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:11:36.976897    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"587ee091b770d62f4fb9433ec0f4b5c2d5b2593a72e497779e6020a95a364e96\": container with ID starting with 587ee091b770d62f4fb9433ec0f4b5c2d5b2593a72e497779e6020a95a364e96 not found: ID does not exist" containerID="587ee091b770d62f4fb9433ec0f4b5c2d5b2593a72e497779e6020a95a364e96"
Jul 02 13:11:36.976980 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.976922    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"587ee091b770d62f4fb9433ec0f4b5c2d5b2593a72e497779e6020a95a364e96"} err="failed to get container status \"587ee091b770d62f4fb9433ec0f4b5c2d5b2593a72e497779e6020a95a364e96\": rpc error: code = NotFound desc = could not find container \"587ee091b770d62f4fb9433ec0f4b5c2d5b2593a72e497779e6020a95a364e96\": container with ID starting with 587ee091b770d62f4fb9433ec0f4b5c2d5b2593a72e497779e6020a95a364e96 not found: ID does not exist"
Jul 02 13:11:36.976980 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.976938    2578 scope.go:117] "RemoveContainer" containerID="0ca0e2398e7cd29a9ac74943bf7deb0a6451aea1f11f27ca2dd92fbbc6a25269"
Jul 02 13:11:36.977163 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:11:36.977147    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ca0e2398e7cd29a9ac74943bf7deb0a6451aea1f11f27ca2dd92fbbc6a25269\": container with ID starting with 0ca0e2398e7cd29a9ac74943bf7deb0a6451aea1f11f27ca2dd92fbbc6a25269 not found: ID does not exist" containerID="0ca0e2398e7cd29a9ac74943bf7deb0a6451aea1f11f27ca2dd92fbbc6a25269"
Jul 02 13:11:36.977237 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.977166    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ca0e2398e7cd29a9ac74943bf7deb0a6451aea1f11f27ca2dd92fbbc6a25269"} err="failed to get container status \"0ca0e2398e7cd29a9ac74943bf7deb0a6451aea1f11f27ca2dd92fbbc6a25269\": rpc error: code = NotFound desc = could not find container \"0ca0e2398e7cd29a9ac74943bf7deb0a6451aea1f11f27ca2dd92fbbc6a25269\": container with ID starting with 0ca0e2398e7cd29a9ac74943bf7deb0a6451aea1f11f27ca2dd92fbbc6a25269 not found: ID does not exist"
Jul 02 13:11:36.977237 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.977199    2578 scope.go:117] "RemoveContainer" containerID="06dfce3b7f6c54630136fad5cebff508cfbdae8eafdb077d3802aa186021930c"
Jul 02 13:11:36.984277 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.984260    2578 scope.go:117] "RemoveContainer" containerID="e4e825e0e6ab8920715a6d80792f3abff44de4cdd2b33967d816206fda979b28"
Jul 02 13:11:36.985794 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.985773    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978"]
Jul 02 13:11:36.991853 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.991837    2578 scope.go:117] "RemoveContainer" containerID="06dfce3b7f6c54630136fad5cebff508cfbdae8eafdb077d3802aa186021930c"
Jul 02 13:11:36.992111 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:11:36.992095    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06dfce3b7f6c54630136fad5cebff508cfbdae8eafdb077d3802aa186021930c\": container with ID starting with 06dfce3b7f6c54630136fad5cebff508cfbdae8eafdb077d3802aa186021930c not found: ID does not exist" containerID="06dfce3b7f6c54630136fad5cebff508cfbdae8eafdb077d3802aa186021930c"
Jul 02 13:11:36.992272 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.992131    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06dfce3b7f6c54630136fad5cebff508cfbdae8eafdb077d3802aa186021930c"} err="failed to get container status \"06dfce3b7f6c54630136fad5cebff508cfbdae8eafdb077d3802aa186021930c\": rpc error: code = NotFound desc = could not find container \"06dfce3b7f6c54630136fad5cebff508cfbdae8eafdb077d3802aa186021930c\": container with ID starting with 06dfce3b7f6c54630136fad5cebff508cfbdae8eafdb077d3802aa186021930c not found: ID does not exist"
Jul 02 13:11:36.992272 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.992155    2578 scope.go:117] "RemoveContainer" containerID="e4e825e0e6ab8920715a6d80792f3abff44de4cdd2b33967d816206fda979b28"
Jul 02 13:11:36.992404 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:11:36.992385    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4e825e0e6ab8920715a6d80792f3abff44de4cdd2b33967d816206fda979b28\": container with ID starting with e4e825e0e6ab8920715a6d80792f3abff44de4cdd2b33967d816206fda979b28 not found: ID does not exist" containerID="e4e825e0e6ab8920715a6d80792f3abff44de4cdd2b33967d816206fda979b28"
Jul 02 13:11:36.992442 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.992410    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4e825e0e6ab8920715a6d80792f3abff44de4cdd2b33967d816206fda979b28"} err="failed to get container status \"e4e825e0e6ab8920715a6d80792f3abff44de4cdd2b33967d816206fda979b28\": rpc error: code = NotFound desc = could not find container \"e4e825e0e6ab8920715a6d80792f3abff44de4cdd2b33967d816206fda979b28\": container with ID starting with e4e825e0e6ab8920715a6d80792f3abff44de4cdd2b33967d816206fda979b28 not found: ID does not exist"
Jul 02 13:11:36.992559 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:36.992544    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b2e72-predictor-59d9b5c96d-xl978"]
Jul 02 13:11:37.005701 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:37.005681    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw"]
Jul 02 13:11:37.012362 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:37.012343    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b2e72-predictor-6cbff7944f-jcljw"]
Jul 02 13:11:38.362249 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:38.362218    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a63b743-a196-48a1-a6cb-dd7dcb38efad" path="/var/lib/kubelet/pods/2a63b743-a196-48a1-a6cb-dd7dcb38efad/volumes"
Jul 02 13:11:38.362659 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:38.362644    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ff8e35a-c9be-462d-abce-3ae66c142bd1" path="/var/lib/kubelet/pods/4ff8e35a-c9be-462d-abce-3ae66c142bd1/volumes"
Jul 02 13:11:40.958357 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:40.958329    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf"
Jul 02 13:11:40.958886 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:40.958827    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf" podUID="b6d34e5f-d9f1-4a6e-889f-90ea3304f93d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 02 13:11:40.959095 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:40.959079    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx"
Jul 02 13:11:40.959543 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:40.959519    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx" podUID="233babba-5bbc-4832-9ba6-06063f0edd75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 02 13:11:50.959759 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:50.959714    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf" podUID="b6d34e5f-d9f1-4a6e-889f-90ea3304f93d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 02 13:11:50.960137 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:11:50.959714    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx" podUID="233babba-5bbc-4832-9ba6-06063f0edd75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 02 13:12:00.958783 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:00.958739    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf" podUID="b6d34e5f-d9f1-4a6e-889f-90ea3304f93d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 02 13:12:00.959958 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:00.959934    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx" podUID="233babba-5bbc-4832-9ba6-06063f0edd75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 02 13:12:10.959038 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:10.958996    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf" podUID="b6d34e5f-d9f1-4a6e-889f-90ea3304f93d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 02 13:12:10.959485 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:10.959458    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx" podUID="233babba-5bbc-4832-9ba6-06063f0edd75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 02 13:12:20.959484 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:20.959450    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf"
Jul 02 13:12:20.960231 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:20.960211    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx"
Jul 02 13:12:22.750215 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:22.750157    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:12:22.908897 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:22.908857    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:12:22.920463 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:22.920430    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5"]
Jul 02 13:12:22.920697 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:22.920671    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5" podUID="59c38447-8a34-40df-9cbb-71e57dbb7e5a" containerName="kserve-container" containerID="cri-o://fd38296b53d41ae63554264b0b19fa0d5294d59d1008a47a91d1f5f4f2511dd7" gracePeriod=30
Jul 02 13:12:22.920773 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:22.920708    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5" podUID="59c38447-8a34-40df-9cbb-71e57dbb7e5a" containerName="kube-rbac-proxy" containerID="cri-o://e51d97fb85bac058792fce166cde13f60133ad7c47212398bb8db5072e729f2f" gracePeriod=30
Jul 02 13:12:22.939433 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:22.939405    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z"]
Jul 02 13:12:22.939800 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:22.939783    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ff8e35a-c9be-462d-abce-3ae66c142bd1" containerName="kserve-container"
Jul 02 13:12:22.939880 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:22.939803    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ff8e35a-c9be-462d-abce-3ae66c142bd1" containerName="kserve-container"
Jul 02 13:12:22.939880 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:22.939813    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ff8e35a-c9be-462d-abce-3ae66c142bd1" containerName="kube-rbac-proxy"
Jul 02 13:12:22.939880 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:22.939822    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ff8e35a-c9be-462d-abce-3ae66c142bd1" containerName="kube-rbac-proxy"
Jul 02 13:12:22.939880 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:22.939835    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a63b743-a196-48a1-a6cb-dd7dcb38efad" containerName="kserve-container"
Jul 02 13:12:22.939880 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:22.939844    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a63b743-a196-48a1-a6cb-dd7dcb38efad" containerName="kserve-container"
Jul 02 13:12:22.939880 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:22.939860    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a63b743-a196-48a1-a6cb-dd7dcb38efad" containerName="kube-rbac-proxy"
Jul 02 13:12:22.939880 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:22.939868    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a63b743-a196-48a1-a6cb-dd7dcb38efad" containerName="kube-rbac-proxy"
Jul 02 13:12:22.940246 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:22.939959    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ff8e35a-c9be-462d-abce-3ae66c142bd1" containerName="kserve-container"
Jul 02 13:12:22.940246 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:22.939973    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a63b743-a196-48a1-a6cb-dd7dcb38efad" containerName="kserve-container"
Jul 02 13:12:22.940246 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:22.939984    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ff8e35a-c9be-462d-abce-3ae66c142bd1" containerName="kube-rbac-proxy"
Jul 02 13:12:22.940246 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:22.939994    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a63b743-a196-48a1-a6cb-dd7dcb38efad" containerName="kube-rbac-proxy"
Jul 02 13:12:22.943452 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:22.943431    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z"
Jul 02 13:12:22.946322 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:22.946300    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f7674-predictor-serving-cert\""
Jul 02 13:12:22.946411 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:22.946307    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f7674-kube-rbac-proxy-sar-config\""
Jul 02 13:12:22.955195 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:22.955146    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z"]
Jul 02 13:12:23.011593 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:23.011505    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11d14f5a-762b-4186-abe5-0d694152b0d3-proxy-tls\") pod \"success-200-isvc-f7674-predictor-84c5db9945-fr67z\" (UID: \"11d14f5a-762b-4186-abe5-0d694152b0d3\") " pod="kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z"
Jul 02 13:12:23.011593 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:23.011542    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11d14f5a-762b-4186-abe5-0d694152b0d3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f7674-predictor-84c5db9945-fr67z\" (UID: \"11d14f5a-762b-4186-abe5-0d694152b0d3\") " pod="kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z"
Jul 02 13:12:23.011593 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:23.011561    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxjzl\" (UniqueName: \"kubernetes.io/projected/11d14f5a-762b-4186-abe5-0d694152b0d3-kube-api-access-mxjzl\") pod \"success-200-isvc-f7674-predictor-84c5db9945-fr67z\" (UID: \"11d14f5a-762b-4186-abe5-0d694152b0d3\") " pod="kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z"
Jul 02 13:12:23.013239 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:23.013209    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:12:23.059912 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:23.059874    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj"]
Jul 02 13:12:23.063259 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:23.063236    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj"
Jul 02 13:12:23.070356 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:23.070325    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f7674-predictor-serving-cert\""
Jul 02 13:12:23.070499 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:23.070445    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f7674-kube-rbac-proxy-sar-config\""
Jul 02 13:12:23.083538 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:23.083509    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj"]
Jul 02 13:12:23.112686 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:23.112653    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11d14f5a-762b-4186-abe5-0d694152b0d3-proxy-tls\") pod \"success-200-isvc-f7674-predictor-84c5db9945-fr67z\" (UID: \"11d14f5a-762b-4186-abe5-0d694152b0d3\") " pod="kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z"
Jul 02 13:12:23.112860 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:23.112691    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11d14f5a-762b-4186-abe5-0d694152b0d3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f7674-predictor-84c5db9945-fr67z\" (UID: \"11d14f5a-762b-4186-abe5-0d694152b0d3\") " pod="kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z"
Jul 02 13:12:23.112860 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:23.112718    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxjzl\" (UniqueName: \"kubernetes.io/projected/11d14f5a-762b-4186-abe5-0d694152b0d3-kube-api-access-mxjzl\") pod \"success-200-isvc-f7674-predictor-84c5db9945-fr67z\" (UID: \"11d14f5a-762b-4186-abe5-0d694152b0d3\") " pod="kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z"
Jul 02 13:12:23.112860 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:23.112764    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7d3331c-2c7d-4880-ab5a-aa63b2107a19-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f7674-predictor-79cc69864-jh7vj\" (UID: \"d7d3331c-2c7d-4880-ab5a-aa63b2107a19\") " pod="kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj"
Jul 02 13:12:23.112860 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:23.112798    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7d3331c-2c7d-4880-ab5a-aa63b2107a19-proxy-tls\") pod \"error-404-isvc-f7674-predictor-79cc69864-jh7vj\" (UID: \"d7d3331c-2c7d-4880-ab5a-aa63b2107a19\") " pod="kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj"
Jul 02 13:12:23.112860 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:23.112823    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvvss\" (UniqueName: \"kubernetes.io/projected/d7d3331c-2c7d-4880-ab5a-aa63b2107a19-kube-api-access-pvvss\") pod \"error-404-isvc-f7674-predictor-79cc69864-jh7vj\" (UID: \"d7d3331c-2c7d-4880-ab5a-aa63b2107a19\") " pod="kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj"
Jul 02 13:12:23.113570 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:23.113545    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11d14f5a-762b-4186-abe5-0d694152b0d3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f7674-predictor-84c5db9945-fr67z\" (UID: \"11d14f5a-762b-4186-abe5-0d694152b0d3\") " pod="kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z"
Jul 02 13:12:23.113974 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:23.113948    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:12:23.115724 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:23.115699    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11d14f5a-762b-4186-abe5-0d694152b0d3-proxy-tls\") pod \"success-200-isvc-f7674-predictor-84c5db9945-fr67z\" (UID: \"11d14f5a-762b-4186-abe5-0d694152b0d3\") " pod="kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z"
Jul 02 13:12:23.123572 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:23.123548    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxjzl\" (UniqueName: \"kubernetes.io/projected/11d14f5a-762b-4186-abe5-0d694152b0d3-kube-api-access-mxjzl\") pod \"success-200-isvc-f7674-predictor-84c5db9945-fr67z\" (UID: \"11d14f5a-762b-4186-abe5-0d694152b0d3\") " pod="kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z"
Jul 02 13:12:23.150816 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:23.150786    2578 generic.go:356] "Generic (PLEG): container finished" podID="59c38447-8a34-40df-9cbb-71e57dbb7e5a" containerID="e51d97fb85bac058792fce166cde13f60133ad7c47212398bb8db5072e729f2f" exitCode=2
Jul 02 13:12:23.150983 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:23.150868    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5" event={"ID":"59c38447-8a34-40df-9cbb-71e57dbb7e5a","Type":"ContainerDied","Data":"e51d97fb85bac058792fce166cde13f60133ad7c47212398bb8db5072e729f2f"}
Jul 02 13:12:23.213943 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:23.213904    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7d3331c-2c7d-4880-ab5a-aa63b2107a19-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f7674-predictor-79cc69864-jh7vj\" (UID: \"d7d3331c-2c7d-4880-ab5a-aa63b2107a19\") " pod="kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj"
Jul 02 13:12:23.213943 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:23.213953    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7d3331c-2c7d-4880-ab5a-aa63b2107a19-proxy-tls\") pod \"error-404-isvc-f7674-predictor-79cc69864-jh7vj\" (UID: \"d7d3331c-2c7d-4880-ab5a-aa63b2107a19\") " pod="kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj"
Jul 02 13:12:23.214206 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:23.213973    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvvss\" (UniqueName: \"kubernetes.io/projected/d7d3331c-2c7d-4880-ab5a-aa63b2107a19-kube-api-access-pvvss\") pod \"error-404-isvc-f7674-predictor-79cc69864-jh7vj\" (UID: \"d7d3331c-2c7d-4880-ab5a-aa63b2107a19\") " pod="kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj"
Jul 02 13:12:23.214206 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:12:23.214085    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-f7674-predictor-serving-cert: secret "error-404-isvc-f7674-predictor-serving-cert" not found
Jul 02 13:12:23.214206 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:12:23.214166    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7d3331c-2c7d-4880-ab5a-aa63b2107a19-proxy-tls podName:d7d3331c-2c7d-4880-ab5a-aa63b2107a19 nodeName:}" failed. No retries permitted until 2026-07-02 13:12:23.714143696 +0000 UTC m=+2011.846969524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d7d3331c-2c7d-4880-ab5a-aa63b2107a19-proxy-tls") pod "error-404-isvc-f7674-predictor-79cc69864-jh7vj" (UID: "d7d3331c-2c7d-4880-ab5a-aa63b2107a19") : secret "error-404-isvc-f7674-predictor-serving-cert" not found
Jul 02 13:12:23.214757 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:23.214739    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7d3331c-2c7d-4880-ab5a-aa63b2107a19-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f7674-predictor-79cc69864-jh7vj\" (UID: \"d7d3331c-2c7d-4880-ab5a-aa63b2107a19\") " pod="kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj"
Jul 02 13:12:23.251314 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:23.251282    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8"]
Jul 02 13:12:23.251595 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:23.251536    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8" podUID="36d5af22-ebaa-484e-a59f-6f404f343543" containerName="kserve-container" containerID="cri-o://06ad1cf27d14560e3cd34078d8b539287124638e68d6a47c5e3b524de3f5519b" gracePeriod=30
Jul 02 13:12:23.251677 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:23.251581    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8" podUID="36d5af22-ebaa-484e-a59f-6f404f343543" containerName="kube-rbac-proxy" containerID="cri-o://227f4e4854d676893b6d42e9f959e41cdbde59a5761806cbf71830507e043add" gracePeriod=30
Jul 02 13:12:23.255363 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:23.255328    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z"
Jul 02 13:12:23.255823 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:23.255800    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvvss\" (UniqueName: \"kubernetes.io/projected/d7d3331c-2c7d-4880-ab5a-aa63b2107a19-kube-api-access-pvvss\") pod \"error-404-isvc-f7674-predictor-79cc69864-jh7vj\" (UID: \"d7d3331c-2c7d-4880-ab5a-aa63b2107a19\") " pod="kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj"
Jul 02 13:12:23.261699 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:23.261572    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:12:23.399728 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:23.399635    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z"]
Jul 02 13:12:23.401975 ip-10-0-137-127 kubenswrapper[2578]: W0702 13:12:23.401949    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11d14f5a_762b_4186_abe5_0d694152b0d3.slice/crio-cfb6c14cb4fbd36763d4d4205f170fec1a30eb1f83464bb98013e37597472f13 WatchSource:0}: Error finding container cfb6c14cb4fbd36763d4d4205f170fec1a30eb1f83464bb98013e37597472f13: Status 404 returned error can't find the container with id cfb6c14cb4fbd36763d4d4205f170fec1a30eb1f83464bb98013e37597472f13
Jul 02 13:12:23.719945 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:23.719909    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7d3331c-2c7d-4880-ab5a-aa63b2107a19-proxy-tls\") pod \"error-404-isvc-f7674-predictor-79cc69864-jh7vj\" (UID: \"d7d3331c-2c7d-4880-ab5a-aa63b2107a19\") " pod="kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj"
Jul 02 13:12:23.722327 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:23.722304    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7d3331c-2c7d-4880-ab5a-aa63b2107a19-proxy-tls\") pod \"error-404-isvc-f7674-predictor-79cc69864-jh7vj\" (UID: \"d7d3331c-2c7d-4880-ab5a-aa63b2107a19\") " pod="kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj"
Jul 02 13:12:23.749727 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:23.749690    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:12:23.973382 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:23.973283    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj"
Jul 02 13:12:24.121513 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:24.121478    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj"]
Jul 02 13:12:24.125379 ip-10-0-137-127 kubenswrapper[2578]: W0702 13:12:24.125346    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7d3331c_2c7d_4880_ab5a_aa63b2107a19.slice/crio-d0b8960ec99874bce6a679b521b86c7413cd3b84d61274a42b36ac83c4c683c4 WatchSource:0}: Error finding container d0b8960ec99874bce6a679b521b86c7413cd3b84d61274a42b36ac83c4c683c4: Status 404 returned error can't find the container with id d0b8960ec99874bce6a679b521b86c7413cd3b84d61274a42b36ac83c4c683c4
Jul 02 13:12:24.156959 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:24.156926    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z" event={"ID":"11d14f5a-762b-4186-abe5-0d694152b0d3","Type":"ContainerStarted","Data":"e3547ca691949d502723ad075f959716c8830ff64c3a087e9d9cc7bf630e2f88"}
Jul 02 13:12:24.157113 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:24.157086    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z"
Jul 02 13:12:24.157280 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:24.157209    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z"
Jul 02 13:12:24.157280 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:24.157248    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z" event={"ID":"11d14f5a-762b-4186-abe5-0d694152b0d3","Type":"ContainerStarted","Data":"4f6de2c3a2531289347cd0c19eb7e822c77aac51da9226988808b64be855f246"}
Jul 02 13:12:24.157380 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:24.157290    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z" event={"ID":"11d14f5a-762b-4186-abe5-0d694152b0d3","Type":"ContainerStarted","Data":"cfb6c14cb4fbd36763d4d4205f170fec1a30eb1f83464bb98013e37597472f13"}
Jul 02 13:12:24.158790 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:24.158758    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z" podUID="11d14f5a-762b-4186-abe5-0d694152b0d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 02 13:12:24.158894 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:24.158790    2578 generic.go:356] "Generic (PLEG): container finished" podID="36d5af22-ebaa-484e-a59f-6f404f343543" containerID="227f4e4854d676893b6d42e9f959e41cdbde59a5761806cbf71830507e043add" exitCode=2
Jul 02 13:12:24.158894 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:24.158854    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8" event={"ID":"36d5af22-ebaa-484e-a59f-6f404f343543","Type":"ContainerDied","Data":"227f4e4854d676893b6d42e9f959e41cdbde59a5761806cbf71830507e043add"}
Jul 02 13:12:24.159893 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:24.159876    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj" event={"ID":"d7d3331c-2c7d-4880-ab5a-aa63b2107a19","Type":"ContainerStarted","Data":"d0b8960ec99874bce6a679b521b86c7413cd3b84d61274a42b36ac83c4c683c4"}
Jul 02 13:12:24.189089 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:24.189046    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z" podStartSLOduration=2.189029272 podStartE2EDuration="2.189029272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:12:24.187546248 +0000 UTC m=+2012.320372111" watchObservedRunningTime="2026-07-02 13:12:24.189029272 +0000 UTC m=+2012.321855118"
Jul 02 13:12:24.986728 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:24.986687    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8" podUID="36d5af22-ebaa-484e-a59f-6f404f343543" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 02 13:12:24.992077 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:24.992054    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8" podUID="36d5af22-ebaa-484e-a59f-6f404f343543" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 02 13:12:25.166279 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:25.166242    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj" event={"ID":"d7d3331c-2c7d-4880-ab5a-aa63b2107a19","Type":"ContainerStarted","Data":"21a9e53ba8e55998d4640fd3b93470aed4cdf1ca51b947fd244023db8c321445"}
Jul 02 13:12:25.166460 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:25.166313    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj" event={"ID":"d7d3331c-2c7d-4880-ab5a-aa63b2107a19","Type":"ContainerStarted","Data":"c81f0d3d1525c3cd288b5f261d25c52ee55dc30e89bc3023d9b1ecb6d44e7a65"}
Jul 02 13:12:25.166522 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:25.166469    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z" podUID="11d14f5a-762b-4186-abe5-0d694152b0d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 02 13:12:25.191779 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:25.191725    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj" podStartSLOduration=2.191708913 podStartE2EDuration="2.191708913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:12:25.188901653 +0000 UTC m=+2013.321727498" watchObservedRunningTime="2026-07-02 13:12:25.191708913 +0000 UTC m=+2013.324534758"
Jul 02 13:12:26.169894 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:26.169863    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj"
Jul 02 13:12:26.170399 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:26.169920    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj"
Jul 02 13:12:26.171144 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:26.171119    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj" podUID="d7d3331c-2c7d-4880-ab5a-aa63b2107a19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 02 13:12:26.468827 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:26.468803    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5"
Jul 02 13:12:26.547726 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:26.547691    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59c38447-8a34-40df-9cbb-71e57dbb7e5a-proxy-tls\") pod \"59c38447-8a34-40df-9cbb-71e57dbb7e5a\" (UID: \"59c38447-8a34-40df-9cbb-71e57dbb7e5a\") "
Jul 02 13:12:26.547726 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:26.547733    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85k2m\" (UniqueName: \"kubernetes.io/projected/59c38447-8a34-40df-9cbb-71e57dbb7e5a-kube-api-access-85k2m\") pod \"59c38447-8a34-40df-9cbb-71e57dbb7e5a\" (UID: \"59c38447-8a34-40df-9cbb-71e57dbb7e5a\") "
Jul 02 13:12:26.547986 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:26.547771    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59c38447-8a34-40df-9cbb-71e57dbb7e5a-kube-rbac-proxy-sar-config\") pod \"59c38447-8a34-40df-9cbb-71e57dbb7e5a\" (UID: \"59c38447-8a34-40df-9cbb-71e57dbb7e5a\") "
Jul 02 13:12:26.548195 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:26.548154    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59c38447-8a34-40df-9cbb-71e57dbb7e5a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "59c38447-8a34-40df-9cbb-71e57dbb7e5a" (UID: "59c38447-8a34-40df-9cbb-71e57dbb7e5a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:12:26.549776 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:26.549745    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59c38447-8a34-40df-9cbb-71e57dbb7e5a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "59c38447-8a34-40df-9cbb-71e57dbb7e5a" (UID: "59c38447-8a34-40df-9cbb-71e57dbb7e5a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:12:26.549869 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:26.549798    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59c38447-8a34-40df-9cbb-71e57dbb7e5a-kube-api-access-85k2m" (OuterVolumeSpecName: "kube-api-access-85k2m") pod "59c38447-8a34-40df-9cbb-71e57dbb7e5a" (UID: "59c38447-8a34-40df-9cbb-71e57dbb7e5a"). InnerVolumeSpecName "kube-api-access-85k2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:12:26.649210 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:26.649157    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59c38447-8a34-40df-9cbb-71e57dbb7e5a-proxy-tls\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:12:26.649210 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:26.649203    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85k2m\" (UniqueName: \"kubernetes.io/projected/59c38447-8a34-40df-9cbb-71e57dbb7e5a-kube-api-access-85k2m\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:12:26.649210 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:26.649213    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59c38447-8a34-40df-9cbb-71e57dbb7e5a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:12:26.989171 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:26.989144    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8"
Jul 02 13:12:27.053395 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:27.053351    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36d5af22-ebaa-484e-a59f-6f404f343543-kube-rbac-proxy-sar-config\") pod \"36d5af22-ebaa-484e-a59f-6f404f343543\" (UID: \"36d5af22-ebaa-484e-a59f-6f404f343543\") "
Jul 02 13:12:27.053395 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:27.053398    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkg4n\" (UniqueName: \"kubernetes.io/projected/36d5af22-ebaa-484e-a59f-6f404f343543-kube-api-access-lkg4n\") pod \"36d5af22-ebaa-484e-a59f-6f404f343543\" (UID: \"36d5af22-ebaa-484e-a59f-6f404f343543\") "
Jul 02 13:12:27.053652 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:27.053435    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36d5af22-ebaa-484e-a59f-6f404f343543-proxy-tls\") pod \"36d5af22-ebaa-484e-a59f-6f404f343543\" (UID: \"36d5af22-ebaa-484e-a59f-6f404f343543\") "
Jul 02 13:12:27.053737 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:27.053714    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36d5af22-ebaa-484e-a59f-6f404f343543-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "36d5af22-ebaa-484e-a59f-6f404f343543" (UID: "36d5af22-ebaa-484e-a59f-6f404f343543"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:12:27.055509 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:27.055481    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36d5af22-ebaa-484e-a59f-6f404f343543-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "36d5af22-ebaa-484e-a59f-6f404f343543" (UID: "36d5af22-ebaa-484e-a59f-6f404f343543"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:12:27.055617 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:27.055509    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36d5af22-ebaa-484e-a59f-6f404f343543-kube-api-access-lkg4n" (OuterVolumeSpecName: "kube-api-access-lkg4n") pod "36d5af22-ebaa-484e-a59f-6f404f343543" (UID: "36d5af22-ebaa-484e-a59f-6f404f343543"). InnerVolumeSpecName "kube-api-access-lkg4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:12:27.154498 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:27.154398    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36d5af22-ebaa-484e-a59f-6f404f343543-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:12:27.154498 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:27.154442    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkg4n\" (UniqueName: \"kubernetes.io/projected/36d5af22-ebaa-484e-a59f-6f404f343543-kube-api-access-lkg4n\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:12:27.154498 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:27.154454    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36d5af22-ebaa-484e-a59f-6f404f343543-proxy-tls\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:12:27.175801 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:27.175769    2578 generic.go:356] "Generic (PLEG): container finished" podID="36d5af22-ebaa-484e-a59f-6f404f343543" containerID="06ad1cf27d14560e3cd34078d8b539287124638e68d6a47c5e3b524de3f5519b" exitCode=0
Jul 02 13:12:27.176222 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:27.175808    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8" event={"ID":"36d5af22-ebaa-484e-a59f-6f404f343543","Type":"ContainerDied","Data":"06ad1cf27d14560e3cd34078d8b539287124638e68d6a47c5e3b524de3f5519b"}
Jul 02 13:12:27.176222 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:27.175836    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8"
Jul 02 13:12:27.176222 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:27.175915    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8" event={"ID":"36d5af22-ebaa-484e-a59f-6f404f343543","Type":"ContainerDied","Data":"683f42cb1e50b2777c215ce12ab833159984b3af21ecba665ab66e54073a61db"}
Jul 02 13:12:27.176222 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:27.175953    2578 scope.go:117] "RemoveContainer" containerID="227f4e4854d676893b6d42e9f959e41cdbde59a5761806cbf71830507e043add"
Jul 02 13:12:27.177720 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:27.177695    2578 generic.go:356] "Generic (PLEG): container finished" podID="59c38447-8a34-40df-9cbb-71e57dbb7e5a" containerID="fd38296b53d41ae63554264b0b19fa0d5294d59d1008a47a91d1f5f4f2511dd7" exitCode=0
Jul 02 13:12:27.177836 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:27.177735    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5" event={"ID":"59c38447-8a34-40df-9cbb-71e57dbb7e5a","Type":"ContainerDied","Data":"fd38296b53d41ae63554264b0b19fa0d5294d59d1008a47a91d1f5f4f2511dd7"}
Jul 02 13:12:27.177836 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:27.177753    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5"
Jul 02 13:12:27.177836 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:27.177796    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5" event={"ID":"59c38447-8a34-40df-9cbb-71e57dbb7e5a","Type":"ContainerDied","Data":"fb1c1c64740aed8e205f141edbb2f91c1b769f0fd0f9d07dce418896965416e3"}
Jul 02 13:12:27.178255 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:27.178228    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj" podUID="d7d3331c-2c7d-4880-ab5a-aa63b2107a19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 02 13:12:27.184996 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:27.184905    2578 scope.go:117] "RemoveContainer" containerID="06ad1cf27d14560e3cd34078d8b539287124638e68d6a47c5e3b524de3f5519b"
Jul 02 13:12:27.192302 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:27.192288    2578 scope.go:117] "RemoveContainer" containerID="227f4e4854d676893b6d42e9f959e41cdbde59a5761806cbf71830507e043add"
Jul 02 13:12:27.192528 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:12:27.192509    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"227f4e4854d676893b6d42e9f959e41cdbde59a5761806cbf71830507e043add\": container with ID starting with 227f4e4854d676893b6d42e9f959e41cdbde59a5761806cbf71830507e043add not found: ID does not exist" containerID="227f4e4854d676893b6d42e9f959e41cdbde59a5761806cbf71830507e043add"
Jul 02 13:12:27.192588 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:27.192534    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"227f4e4854d676893b6d42e9f959e41cdbde59a5761806cbf71830507e043add"} err="failed to get container status \"227f4e4854d676893b6d42e9f959e41cdbde59a5761806cbf71830507e043add\": rpc error: code = NotFound desc = could not find container \"227f4e4854d676893b6d42e9f959e41cdbde59a5761806cbf71830507e043add\": container with ID starting with 227f4e4854d676893b6d42e9f959e41cdbde59a5761806cbf71830507e043add not found: ID does not exist"
Jul 02 13:12:27.192588 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:27.192550    2578 scope.go:117] "RemoveContainer" containerID="06ad1cf27d14560e3cd34078d8b539287124638e68d6a47c5e3b524de3f5519b"
Jul 02 13:12:27.192780 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:12:27.192763    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06ad1cf27d14560e3cd34078d8b539287124638e68d6a47c5e3b524de3f5519b\": container with ID starting with 06ad1cf27d14560e3cd34078d8b539287124638e68d6a47c5e3b524de3f5519b not found: ID does not exist" containerID="06ad1cf27d14560e3cd34078d8b539287124638e68d6a47c5e3b524de3f5519b"
Jul 02 13:12:27.192819 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:27.192787    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06ad1cf27d14560e3cd34078d8b539287124638e68d6a47c5e3b524de3f5519b"} err="failed to get container status \"06ad1cf27d14560e3cd34078d8b539287124638e68d6a47c5e3b524de3f5519b\": rpc error: code = NotFound desc = could not find container \"06ad1cf27d14560e3cd34078d8b539287124638e68d6a47c5e3b524de3f5519b\": container with ID starting with 06ad1cf27d14560e3cd34078d8b539287124638e68d6a47c5e3b524de3f5519b not found: ID does not exist"
Jul 02 13:12:27.192819 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:27.192803    2578 scope.go:117] "RemoveContainer" containerID="e51d97fb85bac058792fce166cde13f60133ad7c47212398bb8db5072e729f2f"
Jul 02 13:12:27.197958 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:27.197939    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8"]
Jul 02 13:12:27.200382 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:27.200364    2578 scope.go:117] "RemoveContainer" containerID="fd38296b53d41ae63554264b0b19fa0d5294d59d1008a47a91d1f5f4f2511dd7"
Jul 02 13:12:27.201933 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:27.201911    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5b6c7-predictor-757c7c7cd-26lr8"]
Jul 02 13:12:27.207656 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:27.207639    2578 scope.go:117] "RemoveContainer" containerID="e51d97fb85bac058792fce166cde13f60133ad7c47212398bb8db5072e729f2f"
Jul 02 13:12:27.207917 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:12:27.207897    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e51d97fb85bac058792fce166cde13f60133ad7c47212398bb8db5072e729f2f\": container with ID starting with e51d97fb85bac058792fce166cde13f60133ad7c47212398bb8db5072e729f2f not found: ID does not exist" containerID="e51d97fb85bac058792fce166cde13f60133ad7c47212398bb8db5072e729f2f"
Jul 02 13:12:27.207977 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:27.207922    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e51d97fb85bac058792fce166cde13f60133ad7c47212398bb8db5072e729f2f"} err="failed to get container status \"e51d97fb85bac058792fce166cde13f60133ad7c47212398bb8db5072e729f2f\": rpc error: code = NotFound desc = could not find container \"e51d97fb85bac058792fce166cde13f60133ad7c47212398bb8db5072e729f2f\": container with ID starting with e51d97fb85bac058792fce166cde13f60133ad7c47212398bb8db5072e729f2f not found: ID does not exist"
Jul 02 13:12:27.207977 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:27.207938    2578 scope.go:117] "RemoveContainer" containerID="fd38296b53d41ae63554264b0b19fa0d5294d59d1008a47a91d1f5f4f2511dd7"
Jul 02 13:12:27.208165 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:12:27.208149    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd38296b53d41ae63554264b0b19fa0d5294d59d1008a47a91d1f5f4f2511dd7\": container with ID starting with fd38296b53d41ae63554264b0b19fa0d5294d59d1008a47a91d1f5f4f2511dd7 not found: ID does not exist" containerID="fd38296b53d41ae63554264b0b19fa0d5294d59d1008a47a91d1f5f4f2511dd7"
Jul 02 13:12:27.208306 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:27.208171    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd38296b53d41ae63554264b0b19fa0d5294d59d1008a47a91d1f5f4f2511dd7"} err="failed to get container status \"fd38296b53d41ae63554264b0b19fa0d5294d59d1008a47a91d1f5f4f2511dd7\": rpc error: code = NotFound desc = could not find container \"fd38296b53d41ae63554264b0b19fa0d5294d59d1008a47a91d1f5f4f2511dd7\": container with ID starting with fd38296b53d41ae63554264b0b19fa0d5294d59d1008a47a91d1f5f4f2511dd7 not found: ID does not exist"
Jul 02 13:12:27.212420 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:27.212402    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5"]
Jul 02 13:12:27.218041 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:27.218023    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5b6c7-predictor-6954f8fdff-xx8t5"]
Jul 02 13:12:28.362834 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:28.362802    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36d5af22-ebaa-484e-a59f-6f404f343543" path="/var/lib/kubelet/pods/36d5af22-ebaa-484e-a59f-6f404f343543/volumes"
Jul 02 13:12:28.363257 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:28.363242    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59c38447-8a34-40df-9cbb-71e57dbb7e5a" path="/var/lib/kubelet/pods/59c38447-8a34-40df-9cbb-71e57dbb7e5a/volumes"
Jul 02 13:12:30.171475 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:30.171444    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z"
Jul 02 13:12:30.171925 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:30.171894    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z" podUID="11d14f5a-762b-4186-abe5-0d694152b0d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 02 13:12:32.183444 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:32.183414    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj"
Jul 02 13:12:32.184155 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:32.184118    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj" podUID="d7d3331c-2c7d-4880-ab5a-aa63b2107a19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 02 13:12:32.998456 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:32.998387    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:12:33.193697 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:33.193664    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:12:40.172398 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:40.172359    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z" podUID="11d14f5a-762b-4186-abe5-0d694152b0d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 02 13:12:42.184982 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:42.184944    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj" podUID="d7d3331c-2c7d-4880-ab5a-aa63b2107a19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 02 13:12:43.242462 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.242414    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:12:43.406781 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.406751    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf"]
Jul 02 13:12:43.407000 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.406976    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf" podUID="b6d34e5f-d9f1-4a6e-889f-90ea3304f93d" containerName="kserve-container" containerID="cri-o://122560edf39f71df2b6481767f0f25fc52b7545c7581d0fe302eeb633a9cf218" gracePeriod=30
Jul 02 13:12:43.407088 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.407037    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf" podUID="b6d34e5f-d9f1-4a6e-889f-90ea3304f93d" containerName="kube-rbac-proxy" containerID="cri-o://621170f28ceed5d957ca1b5d2b04094b1580c58a2e680f85f0272c5070ae0b85" gracePeriod=30
Jul 02 13:12:43.422251 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.422225    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:12:43.525412 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.525384    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn"]
Jul 02 13:12:43.526137 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.526113    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36d5af22-ebaa-484e-a59f-6f404f343543" containerName="kube-rbac-proxy"
Jul 02 13:12:43.526237 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.526141    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="36d5af22-ebaa-484e-a59f-6f404f343543" containerName="kube-rbac-proxy"
Jul 02 13:12:43.526237 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.526193    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59c38447-8a34-40df-9cbb-71e57dbb7e5a" containerName="kube-rbac-proxy"
Jul 02 13:12:43.526237 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.526203    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="59c38447-8a34-40df-9cbb-71e57dbb7e5a" containerName="kube-rbac-proxy"
Jul 02 13:12:43.531101 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.531015    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36d5af22-ebaa-484e-a59f-6f404f343543" containerName="kserve-container"
Jul 02 13:12:43.531101 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.531095    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="36d5af22-ebaa-484e-a59f-6f404f343543" containerName="kserve-container"
Jul 02 13:12:43.531101 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.531108    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59c38447-8a34-40df-9cbb-71e57dbb7e5a" containerName="kserve-container"
Jul 02 13:12:43.531354 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.531114    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="59c38447-8a34-40df-9cbb-71e57dbb7e5a" containerName="kserve-container"
Jul 02 13:12:43.531479 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.531452    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="36d5af22-ebaa-484e-a59f-6f404f343543" containerName="kserve-container"
Jul 02 13:12:43.531535 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.531482    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="36d5af22-ebaa-484e-a59f-6f404f343543" containerName="kube-rbac-proxy"
Jul 02 13:12:43.531535 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.531495    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="59c38447-8a34-40df-9cbb-71e57dbb7e5a" containerName="kube-rbac-proxy"
Jul 02 13:12:43.531535 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.531506    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="59c38447-8a34-40df-9cbb-71e57dbb7e5a" containerName="kserve-container"
Jul 02 13:12:43.535044 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.535020    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn"
Jul 02 13:12:43.541075 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.541051    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-99760-kube-rbac-proxy-sar-config\""
Jul 02 13:12:43.541472 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.541452    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-99760-predictor-serving-cert\""
Jul 02 13:12:43.552628 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.552602    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn"]
Jul 02 13:12:43.558255 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.558220    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:12:43.589211 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.589158    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa-proxy-tls\") pod \"success-200-isvc-99760-predictor-7f5586f59f-lnjjn\" (UID: \"e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa\") " pod="kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn"
Jul 02 13:12:43.589470 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.589444    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drw4r\" (UniqueName: \"kubernetes.io/projected/e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa-kube-api-access-drw4r\") pod \"success-200-isvc-99760-predictor-7f5586f59f-lnjjn\" (UID: \"e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa\") " pod="kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn"
Jul 02 13:12:43.589599 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.589538    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-99760-predictor-7f5586f59f-lnjjn\" (UID: \"e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa\") " pod="kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn"
Jul 02 13:12:43.627735 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.627702    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx"]
Jul 02 13:12:43.627967 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.627940    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx" podUID="233babba-5bbc-4832-9ba6-06063f0edd75" containerName="kserve-container" containerID="cri-o://275f67115a00f95a220b8efc40f309492f686d0a49bd23adb6eb3ed3e13ba7f9" gracePeriod=30
Jul 02 13:12:43.628053 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.627968    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx" podUID="233babba-5bbc-4832-9ba6-06063f0edd75" containerName="kube-rbac-proxy" containerID="cri-o://c22cdc945f713b76224eebfcb3a98d707fb8061de9532a40c4c03c974be0af82" gracePeriod=30
Jul 02 13:12:43.680424 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.680379    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:12:43.690448 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.690413    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drw4r\" (UniqueName: \"kubernetes.io/projected/e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa-kube-api-access-drw4r\") pod \"success-200-isvc-99760-predictor-7f5586f59f-lnjjn\" (UID: \"e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa\") " pod="kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn"
Jul 02 13:12:43.690631 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.690483    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-99760-predictor-7f5586f59f-lnjjn\" (UID: \"e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa\") " pod="kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn"
Jul 02 13:12:43.690631 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.690543    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa-proxy-tls\") pod \"success-200-isvc-99760-predictor-7f5586f59f-lnjjn\" (UID: \"e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa\") " pod="kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn"
Jul 02 13:12:43.691326 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.691296    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-99760-predictor-7f5586f59f-lnjjn\" (UID: \"e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa\") " pod="kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn"
Jul 02 13:12:43.693172 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.693147    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa-proxy-tls\") pod \"success-200-isvc-99760-predictor-7f5586f59f-lnjjn\" (UID: \"e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa\") " pod="kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn"
Jul 02 13:12:43.708326 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.708302    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drw4r\" (UniqueName: \"kubernetes.io/projected/e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa-kube-api-access-drw4r\") pod \"success-200-isvc-99760-predictor-7f5586f59f-lnjjn\" (UID: \"e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa\") " pod="kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn"
Jul 02 13:12:43.721631 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.721597    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr"]
Jul 02 13:12:43.725151 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.725135    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr"
Jul 02 13:12:43.728561 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.728542    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-99760-predictor-serving-cert\""
Jul 02 13:12:43.730535 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.730512    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-99760-kube-rbac-proxy-sar-config\""
Jul 02 13:12:43.740305 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.740278    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr"]
Jul 02 13:12:43.791763 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.791675    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdkg6\" (UniqueName: \"kubernetes.io/projected/f400839a-4d36-49d1-ba41-113472187487-kube-api-access-jdkg6\") pod \"error-404-isvc-99760-predictor-7d86666cd-zpgsr\" (UID: \"f400839a-4d36-49d1-ba41-113472187487\") " pod="kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr"
Jul 02 13:12:43.791763 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.791726    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f400839a-4d36-49d1-ba41-113472187487-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-99760-predictor-7d86666cd-zpgsr\" (UID: \"f400839a-4d36-49d1-ba41-113472187487\") " pod="kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr"
Jul 02 13:12:43.791994 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.791794    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f400839a-4d36-49d1-ba41-113472187487-proxy-tls\") pod \"error-404-isvc-99760-predictor-7d86666cd-zpgsr\" (UID: \"f400839a-4d36-49d1-ba41-113472187487\") " pod="kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr"
Jul 02 13:12:43.845573 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.845534    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn"
Jul 02 13:12:43.893337 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.892811    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f400839a-4d36-49d1-ba41-113472187487-proxy-tls\") pod \"error-404-isvc-99760-predictor-7d86666cd-zpgsr\" (UID: \"f400839a-4d36-49d1-ba41-113472187487\") " pod="kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr"
Jul 02 13:12:43.893337 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.892881    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdkg6\" (UniqueName: \"kubernetes.io/projected/f400839a-4d36-49d1-ba41-113472187487-kube-api-access-jdkg6\") pod \"error-404-isvc-99760-predictor-7d86666cd-zpgsr\" (UID: \"f400839a-4d36-49d1-ba41-113472187487\") " pod="kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr"
Jul 02 13:12:43.893337 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:12:43.892929    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-99760-predictor-serving-cert: secret "error-404-isvc-99760-predictor-serving-cert" not found
Jul 02 13:12:43.893337 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.892927    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f400839a-4d36-49d1-ba41-113472187487-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-99760-predictor-7d86666cd-zpgsr\" (UID: \"f400839a-4d36-49d1-ba41-113472187487\") " pod="kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr"
Jul 02 13:12:43.893337 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:12:43.892996    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f400839a-4d36-49d1-ba41-113472187487-proxy-tls podName:f400839a-4d36-49d1-ba41-113472187487 nodeName:}" failed. No retries permitted until 2026-07-02 13:12:44.392972978 +0000 UTC m=+2032.525798810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f400839a-4d36-49d1-ba41-113472187487-proxy-tls") pod "error-404-isvc-99760-predictor-7d86666cd-zpgsr" (UID: "f400839a-4d36-49d1-ba41-113472187487") : secret "error-404-isvc-99760-predictor-serving-cert" not found
Jul 02 13:12:43.893747 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.893704    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f400839a-4d36-49d1-ba41-113472187487-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-99760-predictor-7d86666cd-zpgsr\" (UID: \"f400839a-4d36-49d1-ba41-113472187487\") " pod="kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr"
Jul 02 13:12:43.895157 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.895077    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:12:43.917277 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:43.917248    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdkg6\" (UniqueName: \"kubernetes.io/projected/f400839a-4d36-49d1-ba41-113472187487-kube-api-access-jdkg6\") pod \"error-404-isvc-99760-predictor-7d86666cd-zpgsr\" (UID: \"f400839a-4d36-49d1-ba41-113472187487\") " pod="kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr"
Jul 02 13:12:44.002838 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:44.002803    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn"]
Jul 02 13:12:44.005754 ip-10-0-137-127 kubenswrapper[2578]: W0702 13:12:44.005722    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9aa7af2_62fc_41ec_bb5d_9e50722eb2aa.slice/crio-4fcee38cd220bef8d33e36ed4787797299b39228bd5195574c770422cfe4bb84 WatchSource:0}: Error finding container 4fcee38cd220bef8d33e36ed4787797299b39228bd5195574c770422cfe4bb84: Status 404 returned error can't find the container with id 4fcee38cd220bef8d33e36ed4787797299b39228bd5195574c770422cfe4bb84
Jul 02 13:12:44.250799 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:44.250758    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn" event={"ID":"e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa","Type":"ContainerStarted","Data":"62ef7426f416edf7834736623cd7ab585f8bbae596dc380ec29f37db0f846f0d"}
Jul 02 13:12:44.251301 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:44.250829    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn" event={"ID":"e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa","Type":"ContainerStarted","Data":"1e437054c4c9041121b054eaf5c82544331d1835888daf51ecf80f3357bc4588"}
Jul 02 13:12:44.251301 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:44.250862    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn"
Jul 02 13:12:44.251301 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:44.250890    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn" event={"ID":"e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa","Type":"ContainerStarted","Data":"4fcee38cd220bef8d33e36ed4787797299b39228bd5195574c770422cfe4bb84"}
Jul 02 13:12:44.252591 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:44.252568    2578 generic.go:356] "Generic (PLEG): container finished" podID="233babba-5bbc-4832-9ba6-06063f0edd75" containerID="c22cdc945f713b76224eebfcb3a98d707fb8061de9532a40c4c03c974be0af82" exitCode=2
Jul 02 13:12:44.252675 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:44.252625    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx" event={"ID":"233babba-5bbc-4832-9ba6-06063f0edd75","Type":"ContainerDied","Data":"c22cdc945f713b76224eebfcb3a98d707fb8061de9532a40c4c03c974be0af82"}
Jul 02 13:12:44.254300 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:44.254276    2578 generic.go:356] "Generic (PLEG): container finished" podID="b6d34e5f-d9f1-4a6e-889f-90ea3304f93d" containerID="621170f28ceed5d957ca1b5d2b04094b1580c58a2e680f85f0272c5070ae0b85" exitCode=2
Jul 02 13:12:44.254414 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:44.254305    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf" event={"ID":"b6d34e5f-d9f1-4a6e-889f-90ea3304f93d","Type":"ContainerDied","Data":"621170f28ceed5d957ca1b5d2b04094b1580c58a2e680f85f0272c5070ae0b85"}
Jul 02 13:12:44.275066 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:44.275018    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn" podStartSLOduration=1.275003081 podStartE2EDuration="1.275003081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:12:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:12:44.27236885 +0000 UTC m=+2032.405194695" watchObservedRunningTime="2026-07-02 13:12:44.275003081 +0000 UTC m=+2032.407828988"
Jul 02 13:12:44.397148 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:44.397062    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f400839a-4d36-49d1-ba41-113472187487-proxy-tls\") pod \"error-404-isvc-99760-predictor-7d86666cd-zpgsr\" (UID: \"f400839a-4d36-49d1-ba41-113472187487\") " pod="kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr"
Jul 02 13:12:44.399495 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:44.399474    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f400839a-4d36-49d1-ba41-113472187487-proxy-tls\") pod \"error-404-isvc-99760-predictor-7d86666cd-zpgsr\" (UID: \"f400839a-4d36-49d1-ba41-113472187487\") " pod="kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr"
Jul 02 13:12:44.413652 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:44.413621    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:12:44.637162 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:44.637118    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr"
Jul 02 13:12:44.740098 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:44.740049    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:12:44.779737 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:44.779705    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr"]
Jul 02 13:12:44.782999 ip-10-0-137-127 kubenswrapper[2578]: W0702 13:12:44.782965    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf400839a_4d36_49d1_ba41_113472187487.slice/crio-f6981185a21585547368647230ec4565b15645776f108d46b8aec54ab04f999d WatchSource:0}: Error finding container f6981185a21585547368647230ec4565b15645776f108d46b8aec54ab04f999d: Status 404 returned error can't find the container with id f6981185a21585547368647230ec4565b15645776f108d46b8aec54ab04f999d
Jul 02 13:12:45.261025 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:45.260988    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr" event={"ID":"f400839a-4d36-49d1-ba41-113472187487","Type":"ContainerStarted","Data":"c3e4009616a05c7d6e483d22d1122e6fa54a378ff0f4d0ae73712c4ef0f68f1b"}
Jul 02 13:12:45.261499 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:45.261060    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr" event={"ID":"f400839a-4d36-49d1-ba41-113472187487","Type":"ContainerStarted","Data":"7ab49ddc9cc8903d14a44fb39d9352ef9db53e948fe71d75a4eed70676d0ef9c"}
Jul 02 13:12:45.261499 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:45.261087    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr" event={"ID":"f400839a-4d36-49d1-ba41-113472187487","Type":"ContainerStarted","Data":"f6981185a21585547368647230ec4565b15645776f108d46b8aec54ab04f999d"}
Jul 02 13:12:45.261499 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:45.261420    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr"
Jul 02 13:12:45.261499 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:45.261475    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr"
Jul 02 13:12:45.261714 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:45.261507    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn"
Jul 02 13:12:45.262343 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:45.262312    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr" podUID="f400839a-4d36-49d1-ba41-113472187487" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 02 13:12:45.262447 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:45.262312    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn" podUID="e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 02 13:12:45.306640 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:45.306586    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr" podStartSLOduration=2.306573198 podStartE2EDuration="2.306573198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:12:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:12:45.306524554 +0000 UTC m=+2033.439350413" watchObservedRunningTime="2026-07-02 13:12:45.306573198 +0000 UTC m=+2033.439399043"
Jul 02 13:12:45.953811 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:45.953769    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf" podUID="b6d34e5f-d9f1-4a6e-889f-90ea3304f93d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 02 13:12:45.954006 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:45.953778    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx" podUID="233babba-5bbc-4832-9ba6-06063f0edd75" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 02 13:12:46.266198 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:46.266134    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn" podUID="e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 02 13:12:46.266603 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:46.266232    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr" podUID="f400839a-4d36-49d1-ba41-113472187487" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 02 13:12:47.272699 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:47.272666    2578 generic.go:356] "Generic (PLEG): container finished" podID="233babba-5bbc-4832-9ba6-06063f0edd75" containerID="275f67115a00f95a220b8efc40f309492f686d0a49bd23adb6eb3ed3e13ba7f9" exitCode=0
Jul 02 13:12:47.273162 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:47.272732    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx" event={"ID":"233babba-5bbc-4832-9ba6-06063f0edd75","Type":"ContainerDied","Data":"275f67115a00f95a220b8efc40f309492f686d0a49bd23adb6eb3ed3e13ba7f9"}
Jul 02 13:12:47.274324 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:47.274301    2578 generic.go:356] "Generic (PLEG): container finished" podID="b6d34e5f-d9f1-4a6e-889f-90ea3304f93d" containerID="122560edf39f71df2b6481767f0f25fc52b7545c7581d0fe302eeb633a9cf218" exitCode=0
Jul 02 13:12:47.274448 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:47.274328    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf" event={"ID":"b6d34e5f-d9f1-4a6e-889f-90ea3304f93d","Type":"ContainerDied","Data":"122560edf39f71df2b6481767f0f25fc52b7545c7581d0fe302eeb633a9cf218"}
Jul 02 13:12:47.398277 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:47.398247    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf"
Jul 02 13:12:47.400651 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:47.400631    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx"
Jul 02 13:12:47.524687 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:47.524656    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/233babba-5bbc-4832-9ba6-06063f0edd75-kube-rbac-proxy-sar-config\") pod \"233babba-5bbc-4832-9ba6-06063f0edd75\" (UID: \"233babba-5bbc-4832-9ba6-06063f0edd75\") "
Jul 02 13:12:47.524833 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:47.524697    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72b2c\" (UniqueName: \"kubernetes.io/projected/b6d34e5f-d9f1-4a6e-889f-90ea3304f93d-kube-api-access-72b2c\") pod \"b6d34e5f-d9f1-4a6e-889f-90ea3304f93d\" (UID: \"b6d34e5f-d9f1-4a6e-889f-90ea3304f93d\") "
Jul 02 13:12:47.524833 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:47.524750    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/233babba-5bbc-4832-9ba6-06063f0edd75-proxy-tls\") pod \"233babba-5bbc-4832-9ba6-06063f0edd75\" (UID: \"233babba-5bbc-4832-9ba6-06063f0edd75\") "
Jul 02 13:12:47.524833 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:47.524772    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v29k5\" (UniqueName: \"kubernetes.io/projected/233babba-5bbc-4832-9ba6-06063f0edd75-kube-api-access-v29k5\") pod \"233babba-5bbc-4832-9ba6-06063f0edd75\" (UID: \"233babba-5bbc-4832-9ba6-06063f0edd75\") "
Jul 02 13:12:47.524833 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:47.524810    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6d34e5f-d9f1-4a6e-889f-90ea3304f93d-kube-rbac-proxy-sar-config\") pod \"b6d34e5f-d9f1-4a6e-889f-90ea3304f93d\" (UID: \"b6d34e5f-d9f1-4a6e-889f-90ea3304f93d\") "
Jul 02 13:12:47.525002 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:47.524852    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6d34e5f-d9f1-4a6e-889f-90ea3304f93d-proxy-tls\") pod \"b6d34e5f-d9f1-4a6e-889f-90ea3304f93d\" (UID: \"b6d34e5f-d9f1-4a6e-889f-90ea3304f93d\") "
Jul 02 13:12:47.525216 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:47.525170    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/233babba-5bbc-4832-9ba6-06063f0edd75-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "233babba-5bbc-4832-9ba6-06063f0edd75" (UID: "233babba-5bbc-4832-9ba6-06063f0edd75"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:12:47.525351 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:47.525260    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6d34e5f-d9f1-4a6e-889f-90ea3304f93d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b6d34e5f-d9f1-4a6e-889f-90ea3304f93d" (UID: "b6d34e5f-d9f1-4a6e-889f-90ea3304f93d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:12:47.526899 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:47.526873    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/233babba-5bbc-4832-9ba6-06063f0edd75-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "233babba-5bbc-4832-9ba6-06063f0edd75" (UID: "233babba-5bbc-4832-9ba6-06063f0edd75"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:12:47.527007 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:47.526946    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/233babba-5bbc-4832-9ba6-06063f0edd75-kube-api-access-v29k5" (OuterVolumeSpecName: "kube-api-access-v29k5") pod "233babba-5bbc-4832-9ba6-06063f0edd75" (UID: "233babba-5bbc-4832-9ba6-06063f0edd75"). InnerVolumeSpecName "kube-api-access-v29k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:12:47.527236 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:47.527219    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d34e5f-d9f1-4a6e-889f-90ea3304f93d-kube-api-access-72b2c" (OuterVolumeSpecName: "kube-api-access-72b2c") pod "b6d34e5f-d9f1-4a6e-889f-90ea3304f93d" (UID: "b6d34e5f-d9f1-4a6e-889f-90ea3304f93d"). InnerVolumeSpecName "kube-api-access-72b2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:12:47.527301 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:47.527253    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6d34e5f-d9f1-4a6e-889f-90ea3304f93d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b6d34e5f-d9f1-4a6e-889f-90ea3304f93d" (UID: "b6d34e5f-d9f1-4a6e-889f-90ea3304f93d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:12:47.626040 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:47.626001    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6d34e5f-d9f1-4a6e-889f-90ea3304f93d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:12:47.626040 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:47.626034    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6d34e5f-d9f1-4a6e-889f-90ea3304f93d-proxy-tls\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:12:47.626040 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:47.626045    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/233babba-5bbc-4832-9ba6-06063f0edd75-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:12:47.626308 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:47.626056    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72b2c\" (UniqueName: \"kubernetes.io/projected/b6d34e5f-d9f1-4a6e-889f-90ea3304f93d-kube-api-access-72b2c\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:12:47.626308 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:47.626065    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/233babba-5bbc-4832-9ba6-06063f0edd75-proxy-tls\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:12:47.626308 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:47.626073    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v29k5\" (UniqueName: \"kubernetes.io/projected/233babba-5bbc-4832-9ba6-06063f0edd75-kube-api-access-v29k5\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:12:48.280307 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:48.280276    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx"
Jul 02 13:12:48.280715 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:48.280279    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx" event={"ID":"233babba-5bbc-4832-9ba6-06063f0edd75","Type":"ContainerDied","Data":"70bc1c5b8a6743d2f8427e3bafb8f5a802be9de56521a500e403e23cea28c5a7"}
Jul 02 13:12:48.280715 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:48.280432    2578 scope.go:117] "RemoveContainer" containerID="c22cdc945f713b76224eebfcb3a98d707fb8061de9532a40c4c03c974be0af82"
Jul 02 13:12:48.282040 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:48.282012    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf" event={"ID":"b6d34e5f-d9f1-4a6e-889f-90ea3304f93d","Type":"ContainerDied","Data":"0d8c0379607e2ed4ed0ecd3d0fe9cc865cde33442d34f8db97c7f03d0f88dbe1"}
Jul 02 13:12:48.282120 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:48.282038    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf"
Jul 02 13:12:48.288808 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:48.288789    2578 scope.go:117] "RemoveContainer" containerID="275f67115a00f95a220b8efc40f309492f686d0a49bd23adb6eb3ed3e13ba7f9"
Jul 02 13:12:48.297165 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:48.297150    2578 scope.go:117] "RemoveContainer" containerID="621170f28ceed5d957ca1b5d2b04094b1580c58a2e680f85f0272c5070ae0b85"
Jul 02 13:12:48.304507 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:48.304490    2578 scope.go:117] "RemoveContainer" containerID="122560edf39f71df2b6481767f0f25fc52b7545c7581d0fe302eeb633a9cf218"
Jul 02 13:12:48.307311 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:48.307291    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx"]
Jul 02 13:12:48.314164 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:48.314143    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-63f60-predictor-fc76d56f5-rrcpx"]
Jul 02 13:12:48.323262 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:48.323238    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf"]
Jul 02 13:12:48.325103 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:48.325083    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-63f60-predictor-58dc685fdb-6f6zf"]
Jul 02 13:12:48.362570 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:48.362539    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="233babba-5bbc-4832-9ba6-06063f0edd75" path="/var/lib/kubelet/pods/233babba-5bbc-4832-9ba6-06063f0edd75/volumes"
Jul 02 13:12:48.362994 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:48.362980    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6d34e5f-d9f1-4a6e-889f-90ea3304f93d" path="/var/lib/kubelet/pods/b6d34e5f-d9f1-4a6e-889f-90ea3304f93d/volumes"
Jul 02 13:12:50.171848 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:50.171813    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z" podUID="11d14f5a-762b-4186-abe5-0d694152b0d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 02 13:12:51.270528 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:51.270502    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr"
Jul 02 13:12:51.270968 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:51.270928    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr" podUID="f400839a-4d36-49d1-ba41-113472187487" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 02 13:12:51.272084 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:51.272067    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn"
Jul 02 13:12:51.272497 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:51.272475    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn" podUID="e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 02 13:12:52.184535 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:12:52.184493    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj" podUID="d7d3331c-2c7d-4880-ab5a-aa63b2107a19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 02 13:13:00.172138 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:13:00.172100    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z" podUID="11d14f5a-762b-4186-abe5-0d694152b0d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 02 13:13:01.271009 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:13:01.270970    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr" podUID="f400839a-4d36-49d1-ba41-113472187487" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 02 13:13:01.273259 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:13:01.273231    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn" podUID="e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 02 13:13:02.184207 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:13:02.184154    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj" podUID="d7d3331c-2c7d-4880-ab5a-aa63b2107a19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 02 13:13:10.171911 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:13:10.171869    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z" podUID="11d14f5a-762b-4186-abe5-0d694152b0d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 02 13:13:11.270984 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:13:11.270937    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr" podUID="f400839a-4d36-49d1-ba41-113472187487" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 02 13:13:11.272776 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:13:11.272743    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn" podUID="e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 02 13:13:12.184404 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:13:12.184309    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj" podUID="d7d3331c-2c7d-4880-ab5a-aa63b2107a19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 02 13:13:20.173362 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:13:20.173330    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z"
Jul 02 13:13:21.271716 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:13:21.271670    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr" podUID="f400839a-4d36-49d1-ba41-113472187487" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 02 13:13:21.272902 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:13:21.272872    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn" podUID="e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 02 13:13:22.185679 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:13:22.185648    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj"
Jul 02 13:13:31.271029 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:13:31.270990    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr" podUID="f400839a-4d36-49d1-ba41-113472187487" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 02 13:13:31.273818 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:13:31.273794    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn"
Jul 02 13:13:33.124461 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:13:33.124421    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:13:33.236952 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:13:33.236915    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:13:41.271323 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:13:41.271293    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr"
Jul 02 13:13:43.482780 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:13:43.482742    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:13:43.667065 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:13:43.667026    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:13:52.455495 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:13:52.455466    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95kwm_495e9fa8-b93f-42e0-95b5-813527df0993/ovn-acl-logging/0.log"
Jul 02 13:13:52.457888 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:13:52.457869    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95kwm_495e9fa8-b93f-42e0-95b5-813527df0993/ovn-acl-logging/0.log"
Jul 02 13:14:37.756041 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:14:37.756003    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:18:29.713632 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:18:29.713595    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:18:31.483663 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:18:31.483617    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:18:52.479726 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:18:52.479694    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95kwm_495e9fa8-b93f-42e0-95b5-813527df0993/ovn-acl-logging/0.log"
Jul 02 13:18:52.481230 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:18:52.481208    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95kwm_495e9fa8-b93f-42e0-95b5-813527df0993/ovn-acl-logging/0.log"
Jul 02 13:20:39.861191 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:20:39.861138    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:39.037273 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:21:39.037239    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:58.064075 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:21:58.064041    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:58.243481 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:21:58.243444    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn"]
Jul 02 13:21:58.243698 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:21:58.243672    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn" podUID="e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa" containerName="kserve-container" containerID="cri-o://1e437054c4c9041121b054eaf5c82544331d1835888daf51ecf80f3357bc4588" gracePeriod=30
Jul 02 13:21:58.243763 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:21:58.243700    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn" podUID="e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa" containerName="kube-rbac-proxy" containerID="cri-o://62ef7426f416edf7834736623cd7ab585f8bbae596dc380ec29f37db0f846f0d" gracePeriod=30
Jul 02 13:21:58.259859 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:21:58.259827    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:58.300961 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:21:58.300919    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr"]
Jul 02 13:21:58.301373 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:21:58.301323    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr" podUID="f400839a-4d36-49d1-ba41-113472187487" containerName="kserve-container" containerID="cri-o://7ab49ddc9cc8903d14a44fb39d9352ef9db53e948fe71d75a4eed70676d0ef9c" gracePeriod=30
Jul 02 13:21:58.304834 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:21:58.301599    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr" podUID="f400839a-4d36-49d1-ba41-113472187487" containerName="kube-rbac-proxy" containerID="cri-o://c3e4009616a05c7d6e483d22d1122e6fa54a378ff0f4d0ae73712c4ef0f68f1b" gracePeriod=30
Jul 02 13:21:58.355560 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:21:58.355529    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:58.385694 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:21:58.385658    2578 generic.go:356] "Generic (PLEG): container finished" podID="e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa" containerID="62ef7426f416edf7834736623cd7ab585f8bbae596dc380ec29f37db0f846f0d" exitCode=2
Jul 02 13:21:58.385857 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:21:58.385732    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn" event={"ID":"e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa","Type":"ContainerDied","Data":"62ef7426f416edf7834736623cd7ab585f8bbae596dc380ec29f37db0f846f0d"}
Jul 02 13:21:59.391966 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:21:59.391934    2578 generic.go:356] "Generic (PLEG): container finished" podID="f400839a-4d36-49d1-ba41-113472187487" containerID="c3e4009616a05c7d6e483d22d1122e6fa54a378ff0f4d0ae73712c4ef0f68f1b" exitCode=2
Jul 02 13:21:59.392392 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:21:59.392006    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr" event={"ID":"f400839a-4d36-49d1-ba41-113472187487","Type":"ContainerDied","Data":"c3e4009616a05c7d6e483d22d1122e6fa54a378ff0f4d0ae73712c4ef0f68f1b"}
Jul 02 13:22:01.267167 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:01.267121    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn" podUID="e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 02 13:22:01.267590 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:01.267123    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr" podUID="f400839a-4d36-49d1-ba41-113472187487" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Jul 02 13:22:01.271507 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:01.271481    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr" podUID="f400839a-4d36-49d1-ba41-113472187487" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 02 13:22:01.272695 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:01.272674    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn" podUID="e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 02 13:22:01.403236 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:01.403207    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn" event={"ID":"e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa","Type":"ContainerDied","Data":"1e437054c4c9041121b054eaf5c82544331d1835888daf51ecf80f3357bc4588"}
Jul 02 13:22:01.403389 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:01.403203    2578 generic.go:356] "Generic (PLEG): container finished" podID="e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa" containerID="1e437054c4c9041121b054eaf5c82544331d1835888daf51ecf80f3357bc4588" exitCode=0
Jul 02 13:22:01.527394 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:01.527370    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn"
Jul 02 13:22:01.614548 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:01.614517    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa-kube-rbac-proxy-sar-config\") pod \"e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa\" (UID: \"e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa\") "
Jul 02 13:22:01.614767 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:01.614595    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drw4r\" (UniqueName: \"kubernetes.io/projected/e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa-kube-api-access-drw4r\") pod \"e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa\" (UID: \"e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa\") "
Jul 02 13:22:01.614767 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:01.614633    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa-proxy-tls\") pod \"e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa\" (UID: \"e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa\") "
Jul 02 13:22:01.614944 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:01.614919    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa" (UID: "e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:22:01.617466 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:01.617434    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa" (UID: "e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:22:01.617466 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:01.617454    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa-kube-api-access-drw4r" (OuterVolumeSpecName: "kube-api-access-drw4r") pod "e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa" (UID: "e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa"). InnerVolumeSpecName "kube-api-access-drw4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:22:01.638411 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:01.638388    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr"
Jul 02 13:22:01.715403 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:01.715362    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f400839a-4d36-49d1-ba41-113472187487-proxy-tls\") pod \"f400839a-4d36-49d1-ba41-113472187487\" (UID: \"f400839a-4d36-49d1-ba41-113472187487\") "
Jul 02 13:22:01.715605 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:01.715425    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdkg6\" (UniqueName: \"kubernetes.io/projected/f400839a-4d36-49d1-ba41-113472187487-kube-api-access-jdkg6\") pod \"f400839a-4d36-49d1-ba41-113472187487\" (UID: \"f400839a-4d36-49d1-ba41-113472187487\") "
Jul 02 13:22:01.715605 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:01.715471    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f400839a-4d36-49d1-ba41-113472187487-kube-rbac-proxy-sar-config\") pod \"f400839a-4d36-49d1-ba41-113472187487\" (UID: \"f400839a-4d36-49d1-ba41-113472187487\") "
Jul 02 13:22:01.715727 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:01.715642    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drw4r\" (UniqueName: \"kubernetes.io/projected/e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa-kube-api-access-drw4r\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:22:01.715727 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:01.715679    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa-proxy-tls\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:22:01.715727 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:01.715690    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:22:01.715867 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:01.715804    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f400839a-4d36-49d1-ba41-113472187487-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f400839a-4d36-49d1-ba41-113472187487" (UID: "f400839a-4d36-49d1-ba41-113472187487"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:22:01.717532 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:01.717509    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f400839a-4d36-49d1-ba41-113472187487-kube-api-access-jdkg6" (OuterVolumeSpecName: "kube-api-access-jdkg6") pod "f400839a-4d36-49d1-ba41-113472187487" (UID: "f400839a-4d36-49d1-ba41-113472187487"). InnerVolumeSpecName "kube-api-access-jdkg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:22:01.717606 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:01.717526    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f400839a-4d36-49d1-ba41-113472187487-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f400839a-4d36-49d1-ba41-113472187487" (UID: "f400839a-4d36-49d1-ba41-113472187487"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:22:01.816759 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:01.816660    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdkg6\" (UniqueName: \"kubernetes.io/projected/f400839a-4d36-49d1-ba41-113472187487-kube-api-access-jdkg6\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:22:01.816759 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:01.816695    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f400839a-4d36-49d1-ba41-113472187487-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:22:01.816759 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:01.816708    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f400839a-4d36-49d1-ba41-113472187487-proxy-tls\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:22:02.409071 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:02.409040    2578 generic.go:356] "Generic (PLEG): container finished" podID="f400839a-4d36-49d1-ba41-113472187487" containerID="7ab49ddc9cc8903d14a44fb39d9352ef9db53e948fe71d75a4eed70676d0ef9c" exitCode=0
Jul 02 13:22:02.409501 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:02.409090    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr"
Jul 02 13:22:02.409501 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:02.409126    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr" event={"ID":"f400839a-4d36-49d1-ba41-113472187487","Type":"ContainerDied","Data":"7ab49ddc9cc8903d14a44fb39d9352ef9db53e948fe71d75a4eed70676d0ef9c"}
Jul 02 13:22:02.409501 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:02.409217    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr" event={"ID":"f400839a-4d36-49d1-ba41-113472187487","Type":"ContainerDied","Data":"f6981185a21585547368647230ec4565b15645776f108d46b8aec54ab04f999d"}
Jul 02 13:22:02.409501 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:02.409259    2578 scope.go:117] "RemoveContainer" containerID="c3e4009616a05c7d6e483d22d1122e6fa54a378ff0f4d0ae73712c4ef0f68f1b"
Jul 02 13:22:02.410990 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:02.410966    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn" event={"ID":"e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa","Type":"ContainerDied","Data":"4fcee38cd220bef8d33e36ed4787797299b39228bd5195574c770422cfe4bb84"}
Jul 02 13:22:02.411083 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:02.411031    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn"
Jul 02 13:22:02.417430 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:02.417406    2578 scope.go:117] "RemoveContainer" containerID="7ab49ddc9cc8903d14a44fb39d9352ef9db53e948fe71d75a4eed70676d0ef9c"
Jul 02 13:22:02.427034 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:02.427016    2578 scope.go:117] "RemoveContainer" containerID="c3e4009616a05c7d6e483d22d1122e6fa54a378ff0f4d0ae73712c4ef0f68f1b"
Jul 02 13:22:02.427385 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:22:02.427359    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3e4009616a05c7d6e483d22d1122e6fa54a378ff0f4d0ae73712c4ef0f68f1b\": container with ID starting with c3e4009616a05c7d6e483d22d1122e6fa54a378ff0f4d0ae73712c4ef0f68f1b not found: ID does not exist" containerID="c3e4009616a05c7d6e483d22d1122e6fa54a378ff0f4d0ae73712c4ef0f68f1b"
Jul 02 13:22:02.427446 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:02.427388    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3e4009616a05c7d6e483d22d1122e6fa54a378ff0f4d0ae73712c4ef0f68f1b"} err="failed to get container status \"c3e4009616a05c7d6e483d22d1122e6fa54a378ff0f4d0ae73712c4ef0f68f1b\": rpc error: code = NotFound desc = could not find container \"c3e4009616a05c7d6e483d22d1122e6fa54a378ff0f4d0ae73712c4ef0f68f1b\": container with ID starting with c3e4009616a05c7d6e483d22d1122e6fa54a378ff0f4d0ae73712c4ef0f68f1b not found: ID does not exist"
Jul 02 13:22:02.427446 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:02.427407    2578 scope.go:117] "RemoveContainer" containerID="7ab49ddc9cc8903d14a44fb39d9352ef9db53e948fe71d75a4eed70676d0ef9c"
Jul 02 13:22:02.427678 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:22:02.427656    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ab49ddc9cc8903d14a44fb39d9352ef9db53e948fe71d75a4eed70676d0ef9c\": container with ID starting with 7ab49ddc9cc8903d14a44fb39d9352ef9db53e948fe71d75a4eed70676d0ef9c not found: ID does not exist" containerID="7ab49ddc9cc8903d14a44fb39d9352ef9db53e948fe71d75a4eed70676d0ef9c"
Jul 02 13:22:02.427727 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:02.427688    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ab49ddc9cc8903d14a44fb39d9352ef9db53e948fe71d75a4eed70676d0ef9c"} err="failed to get container status \"7ab49ddc9cc8903d14a44fb39d9352ef9db53e948fe71d75a4eed70676d0ef9c\": rpc error: code = NotFound desc = could not find container \"7ab49ddc9cc8903d14a44fb39d9352ef9db53e948fe71d75a4eed70676d0ef9c\": container with ID starting with 7ab49ddc9cc8903d14a44fb39d9352ef9db53e948fe71d75a4eed70676d0ef9c not found: ID does not exist"
Jul 02 13:22:02.427727 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:02.427709    2578 scope.go:117] "RemoveContainer" containerID="62ef7426f416edf7834736623cd7ab585f8bbae596dc380ec29f37db0f846f0d"
Jul 02 13:22:02.427948 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:02.427931    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn"]
Jul 02 13:22:02.433104 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:02.433078    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-99760-predictor-7f5586f59f-lnjjn"]
Jul 02 13:22:02.435755 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:02.435739    2578 scope.go:117] "RemoveContainer" containerID="1e437054c4c9041121b054eaf5c82544331d1835888daf51ecf80f3357bc4588"
Jul 02 13:22:02.447716 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:02.447684    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr"]
Jul 02 13:22:02.449020 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:02.449002    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-99760-predictor-7d86666cd-zpgsr"]
Jul 02 13:22:04.362470 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:04.362437    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa" path="/var/lib/kubelet/pods/e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa/volumes"
Jul 02 13:22:04.362878 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:22:04.362864    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f400839a-4d36-49d1-ba41-113472187487" path="/var/lib/kubelet/pods/f400839a-4d36-49d1-ba41-113472187487/volumes"
Jul 02 13:23:52.504029 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:23:52.503947    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95kwm_495e9fa8-b93f-42e0-95b5-813527df0993/ovn-acl-logging/0.log"
Jul 02 13:23:52.506060 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:23:52.506040    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95kwm_495e9fa8-b93f-42e0-95b5-813527df0993/ovn-acl-logging/0.log"
Jul 02 13:27:38.877916 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:27:38.877884    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:27:44.106846 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:27:44.106812    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:27:54.713708 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:27:54.713677    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:28:46.399498 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:28:46.399460    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:28:52.526322 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:28:52.526294    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95kwm_495e9fa8-b93f-42e0-95b5-813527df0993/ovn-acl-logging/0.log"
Jul 02 13:28:52.529452 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:28:52.529433    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95kwm_495e9fa8-b93f-42e0-95b5-813527df0993/ovn-acl-logging/0.log"
Jul 02 13:29:52.198057 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:52.197969    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:29:52.334256 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:52.334230    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z"]
Jul 02 13:29:52.334491 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:52.334462    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z" podUID="11d14f5a-762b-4186-abe5-0d694152b0d3" containerName="kserve-container" containerID="cri-o://4f6de2c3a2531289347cd0c19eb7e822c77aac51da9226988808b64be855f246" gracePeriod=30
Jul 02 13:29:52.334641 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:52.334499    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z" podUID="11d14f5a-762b-4186-abe5-0d694152b0d3" containerName="kube-rbac-proxy" containerID="cri-o://e3547ca691949d502723ad075f959716c8830ff64c3a087e9d9cc7bf630e2f88" gracePeriod=30
Jul 02 13:29:52.358648 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:52.358619    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:29:52.394056 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:52.394026    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj"]
Jul 02 13:29:52.394325 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:52.394285    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj" podUID="d7d3331c-2c7d-4880-ab5a-aa63b2107a19" containerName="kserve-container" containerID="cri-o://c81f0d3d1525c3cd288b5f261d25c52ee55dc30e89bc3023d9b1ecb6d44e7a65" gracePeriod=30
Jul 02 13:29:52.394325 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:52.394316    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj" podUID="d7d3331c-2c7d-4880-ab5a-aa63b2107a19" containerName="kube-rbac-proxy" containerID="cri-o://21a9e53ba8e55998d4640fd3b93470aed4cdf1ca51b947fd244023db8c321445" gracePeriod=30
Jul 02 13:29:52.418888 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:52.418862    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:29:53.165882 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:53.165852    2578 generic.go:356] "Generic (PLEG): container finished" podID="d7d3331c-2c7d-4880-ab5a-aa63b2107a19" containerID="21a9e53ba8e55998d4640fd3b93470aed4cdf1ca51b947fd244023db8c321445" exitCode=2
Jul 02 13:29:53.166073 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:53.165933    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj" event={"ID":"d7d3331c-2c7d-4880-ab5a-aa63b2107a19","Type":"ContainerDied","Data":"21a9e53ba8e55998d4640fd3b93470aed4cdf1ca51b947fd244023db8c321445"}
Jul 02 13:29:53.167585 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:53.167563    2578 generic.go:356] "Generic (PLEG): container finished" podID="11d14f5a-762b-4186-abe5-0d694152b0d3" containerID="e3547ca691949d502723ad075f959716c8830ff64c3a087e9d9cc7bf630e2f88" exitCode=2
Jul 02 13:29:53.167708 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:53.167597    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z" event={"ID":"11d14f5a-762b-4186-abe5-0d694152b0d3","Type":"ContainerDied","Data":"e3547ca691949d502723ad075f959716c8830ff64c3a087e9d9cc7bf630e2f88"}
Jul 02 13:29:53.603760 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:53.603728    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:29:53.673195 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:53.673147    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:29:55.167252 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:55.167207    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z" podUID="11d14f5a-762b-4186-abe5-0d694152b0d3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Jul 02 13:29:55.179066 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:55.179033    2578 generic.go:356] "Generic (PLEG): container finished" podID="11d14f5a-762b-4186-abe5-0d694152b0d3" containerID="4f6de2c3a2531289347cd0c19eb7e822c77aac51da9226988808b64be855f246" exitCode=0
Jul 02 13:29:55.179244 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:55.179110    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z" event={"ID":"11d14f5a-762b-4186-abe5-0d694152b0d3","Type":"ContainerDied","Data":"4f6de2c3a2531289347cd0c19eb7e822c77aac51da9226988808b64be855f246"}
Jul 02 13:29:55.285359 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:55.285336    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z"
Jul 02 13:29:55.405866 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:55.405780    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxjzl\" (UniqueName: \"kubernetes.io/projected/11d14f5a-762b-4186-abe5-0d694152b0d3-kube-api-access-mxjzl\") pod \"11d14f5a-762b-4186-abe5-0d694152b0d3\" (UID: \"11d14f5a-762b-4186-abe5-0d694152b0d3\") "
Jul 02 13:29:55.405866 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:55.405834    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11d14f5a-762b-4186-abe5-0d694152b0d3-proxy-tls\") pod \"11d14f5a-762b-4186-abe5-0d694152b0d3\" (UID: \"11d14f5a-762b-4186-abe5-0d694152b0d3\") "
Jul 02 13:29:55.406051 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:55.405893    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11d14f5a-762b-4186-abe5-0d694152b0d3-kube-rbac-proxy-sar-config\") pod \"11d14f5a-762b-4186-abe5-0d694152b0d3\" (UID: \"11d14f5a-762b-4186-abe5-0d694152b0d3\") "
Jul 02 13:29:55.406299 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:55.406266    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11d14f5a-762b-4186-abe5-0d694152b0d3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "11d14f5a-762b-4186-abe5-0d694152b0d3" (UID: "11d14f5a-762b-4186-abe5-0d694152b0d3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:29:55.407923 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:55.407898    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11d14f5a-762b-4186-abe5-0d694152b0d3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "11d14f5a-762b-4186-abe5-0d694152b0d3" (UID: "11d14f5a-762b-4186-abe5-0d694152b0d3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:29:55.408020 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:55.407998    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d14f5a-762b-4186-abe5-0d694152b0d3-kube-api-access-mxjzl" (OuterVolumeSpecName: "kube-api-access-mxjzl") pod "11d14f5a-762b-4186-abe5-0d694152b0d3" (UID: "11d14f5a-762b-4186-abe5-0d694152b0d3"). InnerVolumeSpecName "kube-api-access-mxjzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:29:55.506657 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:55.506627    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxjzl\" (UniqueName: \"kubernetes.io/projected/11d14f5a-762b-4186-abe5-0d694152b0d3-kube-api-access-mxjzl\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:29:55.506657 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:55.506656    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11d14f5a-762b-4186-abe5-0d694152b0d3-proxy-tls\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:29:55.506845 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:55.506675    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11d14f5a-762b-4186-abe5-0d694152b0d3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:29:55.629784 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:55.629757    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj"
Jul 02 13:29:55.708016 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:55.707935    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7d3331c-2c7d-4880-ab5a-aa63b2107a19-kube-rbac-proxy-sar-config\") pod \"d7d3331c-2c7d-4880-ab5a-aa63b2107a19\" (UID: \"d7d3331c-2c7d-4880-ab5a-aa63b2107a19\") "
Jul 02 13:29:55.708016 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:55.707979    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7d3331c-2c7d-4880-ab5a-aa63b2107a19-proxy-tls\") pod \"d7d3331c-2c7d-4880-ab5a-aa63b2107a19\" (UID: \"d7d3331c-2c7d-4880-ab5a-aa63b2107a19\") "
Jul 02 13:29:55.708016 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:55.708013    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvvss\" (UniqueName: \"kubernetes.io/projected/d7d3331c-2c7d-4880-ab5a-aa63b2107a19-kube-api-access-pvvss\") pod \"d7d3331c-2c7d-4880-ab5a-aa63b2107a19\" (UID: \"d7d3331c-2c7d-4880-ab5a-aa63b2107a19\") "
Jul 02 13:29:55.708380 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:55.708350    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d3331c-2c7d-4880-ab5a-aa63b2107a19-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d7d3331c-2c7d-4880-ab5a-aa63b2107a19" (UID: "d7d3331c-2c7d-4880-ab5a-aa63b2107a19"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:29:55.710071 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:55.710050    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7d3331c-2c7d-4880-ab5a-aa63b2107a19-kube-api-access-pvvss" (OuterVolumeSpecName: "kube-api-access-pvvss") pod "d7d3331c-2c7d-4880-ab5a-aa63b2107a19" (UID: "d7d3331c-2c7d-4880-ab5a-aa63b2107a19"). InnerVolumeSpecName "kube-api-access-pvvss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:29:55.710162 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:55.710064    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7d3331c-2c7d-4880-ab5a-aa63b2107a19-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d7d3331c-2c7d-4880-ab5a-aa63b2107a19" (UID: "d7d3331c-2c7d-4880-ab5a-aa63b2107a19"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:29:55.808892 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:55.808856    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7d3331c-2c7d-4880-ab5a-aa63b2107a19-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:29:55.808892 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:55.808885    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7d3331c-2c7d-4880-ab5a-aa63b2107a19-proxy-tls\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:29:55.808892 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:55.808896    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvvss\" (UniqueName: \"kubernetes.io/projected/d7d3331c-2c7d-4880-ab5a-aa63b2107a19-kube-api-access-pvvss\") on node \"ip-10-0-137-127.ec2.internal\" DevicePath \"\""
Jul 02 13:29:56.185168 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:56.185137    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z"
Jul 02 13:29:56.185643 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:56.185163    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z" event={"ID":"11d14f5a-762b-4186-abe5-0d694152b0d3","Type":"ContainerDied","Data":"cfb6c14cb4fbd36763d4d4205f170fec1a30eb1f83464bb98013e37597472f13"}
Jul 02 13:29:56.185643 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:56.185263    2578 scope.go:117] "RemoveContainer" containerID="e3547ca691949d502723ad075f959716c8830ff64c3a087e9d9cc7bf630e2f88"
Jul 02 13:29:56.186850 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:56.186827    2578 generic.go:356] "Generic (PLEG): container finished" podID="d7d3331c-2c7d-4880-ab5a-aa63b2107a19" containerID="c81f0d3d1525c3cd288b5f261d25c52ee55dc30e89bc3023d9b1ecb6d44e7a65" exitCode=0
Jul 02 13:29:56.186941 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:56.186872    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj"
Jul 02 13:29:56.186987 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:56.186871    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj" event={"ID":"d7d3331c-2c7d-4880-ab5a-aa63b2107a19","Type":"ContainerDied","Data":"c81f0d3d1525c3cd288b5f261d25c52ee55dc30e89bc3023d9b1ecb6d44e7a65"}
Jul 02 13:29:56.187030 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:56.186994    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj" event={"ID":"d7d3331c-2c7d-4880-ab5a-aa63b2107a19","Type":"ContainerDied","Data":"d0b8960ec99874bce6a679b521b86c7413cd3b84d61274a42b36ac83c4c683c4"}
Jul 02 13:29:56.194325 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:56.194305    2578 scope.go:117] "RemoveContainer" containerID="4f6de2c3a2531289347cd0c19eb7e822c77aac51da9226988808b64be855f246"
Jul 02 13:29:56.201637 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:56.201618    2578 scope.go:117] "RemoveContainer" containerID="21a9e53ba8e55998d4640fd3b93470aed4cdf1ca51b947fd244023db8c321445"
Jul 02 13:29:56.208780 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:56.208755    2578 scope.go:117] "RemoveContainer" containerID="c81f0d3d1525c3cd288b5f261d25c52ee55dc30e89bc3023d9b1ecb6d44e7a65"
Jul 02 13:29:56.214440 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:56.214419    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z"]
Jul 02 13:29:56.216314 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:56.216301    2578 scope.go:117] "RemoveContainer" containerID="21a9e53ba8e55998d4640fd3b93470aed4cdf1ca51b947fd244023db8c321445"
Jul 02 13:29:56.216579 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:29:56.216552    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21a9e53ba8e55998d4640fd3b93470aed4cdf1ca51b947fd244023db8c321445\": container with ID starting with 21a9e53ba8e55998d4640fd3b93470aed4cdf1ca51b947fd244023db8c321445 not found: ID does not exist" containerID="21a9e53ba8e55998d4640fd3b93470aed4cdf1ca51b947fd244023db8c321445"
Jul 02 13:29:56.216649 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:56.216591    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21a9e53ba8e55998d4640fd3b93470aed4cdf1ca51b947fd244023db8c321445"} err="failed to get container status \"21a9e53ba8e55998d4640fd3b93470aed4cdf1ca51b947fd244023db8c321445\": rpc error: code = NotFound desc = could not find container \"21a9e53ba8e55998d4640fd3b93470aed4cdf1ca51b947fd244023db8c321445\": container with ID starting with 21a9e53ba8e55998d4640fd3b93470aed4cdf1ca51b947fd244023db8c321445 not found: ID does not exist"
Jul 02 13:29:56.216649 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:56.216617    2578 scope.go:117] "RemoveContainer" containerID="c81f0d3d1525c3cd288b5f261d25c52ee55dc30e89bc3023d9b1ecb6d44e7a65"
Jul 02 13:29:56.216844 ip-10-0-137-127 kubenswrapper[2578]: E0702 13:29:56.216828    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c81f0d3d1525c3cd288b5f261d25c52ee55dc30e89bc3023d9b1ecb6d44e7a65\": container with ID starting with c81f0d3d1525c3cd288b5f261d25c52ee55dc30e89bc3023d9b1ecb6d44e7a65 not found: ID does not exist" containerID="c81f0d3d1525c3cd288b5f261d25c52ee55dc30e89bc3023d9b1ecb6d44e7a65"
Jul 02 13:29:56.216884 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:56.216849    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c81f0d3d1525c3cd288b5f261d25c52ee55dc30e89bc3023d9b1ecb6d44e7a65"} err="failed to get container status \"c81f0d3d1525c3cd288b5f261d25c52ee55dc30e89bc3023d9b1ecb6d44e7a65\": rpc error: code = NotFound desc = could not find container \"c81f0d3d1525c3cd288b5f261d25c52ee55dc30e89bc3023d9b1ecb6d44e7a65\": container with ID starting with c81f0d3d1525c3cd288b5f261d25c52ee55dc30e89bc3023d9b1ecb6d44e7a65 not found: ID does not exist"
Jul 02 13:29:56.219707 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:56.219688    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f7674-predictor-84c5db9945-fr67z"]
Jul 02 13:29:56.239025 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:56.239006    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj"]
Jul 02 13:29:56.249783 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:56.249764    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f7674-predictor-79cc69864-jh7vj"]
Jul 02 13:29:56.361216 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:56.361162    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11d14f5a-762b-4186-abe5-0d694152b0d3" path="/var/lib/kubelet/pods/11d14f5a-762b-4186-abe5-0d694152b0d3/volumes"
Jul 02 13:29:56.361615 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:29:56.361601    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7d3331c-2c7d-4880-ab5a-aa63b2107a19" path="/var/lib/kubelet/pods/d7d3331c-2c7d-4880-ab5a-aa63b2107a19/volumes"
Jul 02 13:30:21.975544 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:21.975511    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:30:22.074741 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.074707    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-74fjg/must-gather-b9fgw"]
Jul 02 13:30:22.074997 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.074984    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="233babba-5bbc-4832-9ba6-06063f0edd75" containerName="kube-rbac-proxy"
Jul 02 13:30:22.075041 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.074998    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="233babba-5bbc-4832-9ba6-06063f0edd75" containerName="kube-rbac-proxy"
Jul 02 13:30:22.075041 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.075011    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6d34e5f-d9f1-4a6e-889f-90ea3304f93d" containerName="kserve-container"
Jul 02 13:30:22.075041 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.075017    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6d34e5f-d9f1-4a6e-889f-90ea3304f93d" containerName="kserve-container"
Jul 02 13:30:22.075041 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.075025    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11d14f5a-762b-4186-abe5-0d694152b0d3" containerName="kube-rbac-proxy"
Jul 02 13:30:22.075041 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.075031    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="11d14f5a-762b-4186-abe5-0d694152b0d3" containerName="kube-rbac-proxy"
Jul 02 13:30:22.075041 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.075037    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="233babba-5bbc-4832-9ba6-06063f0edd75" containerName="kserve-container"
Jul 02 13:30:22.075041 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.075043    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="233babba-5bbc-4832-9ba6-06063f0edd75" containerName="kserve-container"
Jul 02 13:30:22.075291 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.075050    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7d3331c-2c7d-4880-ab5a-aa63b2107a19" containerName="kserve-container"
Jul 02 13:30:22.075291 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.075055    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7d3331c-2c7d-4880-ab5a-aa63b2107a19" containerName="kserve-container"
Jul 02 13:30:22.075291 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.075060    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa" containerName="kserve-container"
Jul 02 13:30:22.075291 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.075065    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa" containerName="kserve-container"
Jul 02 13:30:22.075291 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.075072    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7d3331c-2c7d-4880-ab5a-aa63b2107a19" containerName="kube-rbac-proxy"
Jul 02 13:30:22.075291 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.075077    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7d3331c-2c7d-4880-ab5a-aa63b2107a19" containerName="kube-rbac-proxy"
Jul 02 13:30:22.075291 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.075083    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f400839a-4d36-49d1-ba41-113472187487" containerName="kserve-container"
Jul 02 13:30:22.075291 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.075088    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f400839a-4d36-49d1-ba41-113472187487" containerName="kserve-container"
Jul 02 13:30:22.075291 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.075093    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f400839a-4d36-49d1-ba41-113472187487" containerName="kube-rbac-proxy"
Jul 02 13:30:22.075291 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.075098    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f400839a-4d36-49d1-ba41-113472187487" containerName="kube-rbac-proxy"
Jul 02 13:30:22.075291 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.075109    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11d14f5a-762b-4186-abe5-0d694152b0d3" containerName="kserve-container"
Jul 02 13:30:22.075291 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.075114    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="11d14f5a-762b-4186-abe5-0d694152b0d3" containerName="kserve-container"
Jul 02 13:30:22.075291 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.075119    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa" containerName="kube-rbac-proxy"
Jul 02 13:30:22.075291 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.075124    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa" containerName="kube-rbac-proxy"
Jul 02 13:30:22.075291 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.075132    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6d34e5f-d9f1-4a6e-889f-90ea3304f93d" containerName="kube-rbac-proxy"
Jul 02 13:30:22.075291 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.075137    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6d34e5f-d9f1-4a6e-889f-90ea3304f93d" containerName="kube-rbac-proxy"
Jul 02 13:30:22.075291 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.075192    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6d34e5f-d9f1-4a6e-889f-90ea3304f93d" containerName="kube-rbac-proxy"
Jul 02 13:30:22.075291 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.075203    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f400839a-4d36-49d1-ba41-113472187487" containerName="kube-rbac-proxy"
Jul 02 13:30:22.075291 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.075213    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="233babba-5bbc-4832-9ba6-06063f0edd75" containerName="kube-rbac-proxy"
Jul 02 13:30:22.075291 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.075218    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="11d14f5a-762b-4186-abe5-0d694152b0d3" containerName="kube-rbac-proxy"
Jul 02 13:30:22.075291 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.075223    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="233babba-5bbc-4832-9ba6-06063f0edd75" containerName="kserve-container"
Jul 02 13:30:22.075291 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.075227    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7d3331c-2c7d-4880-ab5a-aa63b2107a19" containerName="kserve-container"
Jul 02 13:30:22.075291 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.075233    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f400839a-4d36-49d1-ba41-113472187487" containerName="kserve-container"
Jul 02 13:30:22.075291 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.075240    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6d34e5f-d9f1-4a6e-889f-90ea3304f93d" containerName="kserve-container"
Jul 02 13:30:22.075291 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.075245    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa" containerName="kserve-container"
Jul 02 13:30:22.075291 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.075250    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="11d14f5a-762b-4186-abe5-0d694152b0d3" containerName="kserve-container"
Jul 02 13:30:22.075291 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.075255    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9aa7af2-62fc-41ec-bb5d-9e50722eb2aa" containerName="kube-rbac-proxy"
Jul 02 13:30:22.075291 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.075261    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7d3331c-2c7d-4880-ab5a-aa63b2107a19" containerName="kube-rbac-proxy"
Jul 02 13:30:22.077916 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.077900    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-74fjg/must-gather-b9fgw"
Jul 02 13:30:22.081010 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.080988    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-74fjg\"/\"default-dockercfg-djlrx\""
Jul 02 13:30:22.081010 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.081005    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-74fjg\"/\"kube-root-ca.crt\""
Jul 02 13:30:22.081210 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.081081    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-74fjg\"/\"openshift-service-ca.crt\""
Jul 02 13:30:22.087746 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.087724    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-74fjg/must-gather-b9fgw"]
Jul 02 13:30:22.092206 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.092189    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:30:22.155060 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.155025    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:30:22.209906 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.209873    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/63f6d668-6a10-41c7-973c-387fc5a3df80-must-gather-output\") pod \"must-gather-b9fgw\" (UID: \"63f6d668-6a10-41c7-973c-387fc5a3df80\") " pod="openshift-must-gather-74fjg/must-gather-b9fgw"
Jul 02 13:30:22.210094 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.209920    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbpzw\" (UniqueName: \"kubernetes.io/projected/63f6d668-6a10-41c7-973c-387fc5a3df80-kube-api-access-hbpzw\") pod \"must-gather-b9fgw\" (UID: \"63f6d668-6a10-41c7-973c-387fc5a3df80\") " pod="openshift-must-gather-74fjg/must-gather-b9fgw"
Jul 02 13:30:22.311268 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.311237    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/63f6d668-6a10-41c7-973c-387fc5a3df80-must-gather-output\") pod \"must-gather-b9fgw\" (UID: \"63f6d668-6a10-41c7-973c-387fc5a3df80\") " pod="openshift-must-gather-74fjg/must-gather-b9fgw"
Jul 02 13:30:22.311439 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.311280    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbpzw\" (UniqueName: \"kubernetes.io/projected/63f6d668-6a10-41c7-973c-387fc5a3df80-kube-api-access-hbpzw\") pod \"must-gather-b9fgw\" (UID: \"63f6d668-6a10-41c7-973c-387fc5a3df80\") " pod="openshift-must-gather-74fjg/must-gather-b9fgw"
Jul 02 13:30:22.311624 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.311599    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/63f6d668-6a10-41c7-973c-387fc5a3df80-must-gather-output\") pod \"must-gather-b9fgw\" (UID: \"63f6d668-6a10-41c7-973c-387fc5a3df80\") " pod="openshift-must-gather-74fjg/must-gather-b9fgw"
Jul 02 13:30:22.319486 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.319457    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbpzw\" (UniqueName: \"kubernetes.io/projected/63f6d668-6a10-41c7-973c-387fc5a3df80-kube-api-access-hbpzw\") pod \"must-gather-b9fgw\" (UID: \"63f6d668-6a10-41c7-973c-387fc5a3df80\") " pod="openshift-must-gather-74fjg/must-gather-b9fgw"
Jul 02 13:30:22.385881 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.385847    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-74fjg/must-gather-b9fgw"
Jul 02 13:30:22.516339 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.516257    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-74fjg/must-gather-b9fgw"]
Jul 02 13:30:22.518359 ip-10-0-137-127 kubenswrapper[2578]: W0702 13:30:22.518332    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63f6d668_6a10_41c7_973c_387fc5a3df80.slice/crio-ae92229e7d3ee6f15689e401f5a42ecfaa581baaaf36554531229965f2b4c074 WatchSource:0}: Error finding container ae92229e7d3ee6f15689e401f5a42ecfaa581baaaf36554531229965f2b4c074: Status 404 returned error can't find the container with id ae92229e7d3ee6f15689e401f5a42ecfaa581baaaf36554531229965f2b4c074
Jul 02 13:30:22.520293 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:22.520278    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 13:30:23.290561 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:23.290522    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-74fjg/must-gather-b9fgw" event={"ID":"63f6d668-6a10-41c7-973c-387fc5a3df80","Type":"ContainerStarted","Data":"ae92229e7d3ee6f15689e401f5a42ecfaa581baaaf36554531229965f2b4c074"}
Jul 02 13:30:24.296580 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:24.296540    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-74fjg/must-gather-b9fgw" event={"ID":"63f6d668-6a10-41c7-973c-387fc5a3df80","Type":"ContainerStarted","Data":"94a97dc67210c647bfb56eaaf1bb6272e3412afbc9b575a15e50352a2e980772"}
Jul 02 13:30:24.297028 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:24.296596    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-74fjg/must-gather-b9fgw" event={"ID":"63f6d668-6a10-41c7-973c-387fc5a3df80","Type":"ContainerStarted","Data":"cf33813b804e1679355c09c30cc333f55895a74ebfb83ba8712e0f381adf2240"}
Jul 02 13:30:24.315441 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:24.315388    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-74fjg/must-gather-b9fgw" podStartSLOduration=1.635048407 podStartE2EDuration="2.315371434s" totalImagesPullingTime="680.323027ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:30:24.314349719 +0000 UTC m=+3092.447175567" watchObservedRunningTime="2026-07-02 13:30:24.315371434 +0000 UTC m=+3092.448197279"
Jul 02 13:30:24.668154 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:24.668078    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tsj7t_2718fba6-5e1c-4c48-a90c-fc26b0ccc385/global-pull-secret-syncer/0.log"
Jul 02 13:30:24.724903 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:24.724875    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log"
Jul 02 13:30:24.855264 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:24.855238    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-127.ec2.internal_59308479622ecc528e66f3883871894d/haproxy/0.log"
Jul 02 13:30:27.610833 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:27.610800    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:30:27.694805 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:27.694776    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f8kj_176a476f-a437-42c6-a32d-b6d23e4d2a9f/konnectivity-agent/0.log" limit=1024
Jul 02 13:30:29.231723 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:29.231695    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-cd8656b8-sb6pw_0785238a-fca2-4d58-a34c-d1ec38d13133/monitoring-plugin/0.log"
Jul 02 13:30:29.356079 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:29.356040    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9dsgh_1b45d14c-b4b5-4c87-bc99-ed18f33668ef/node-exporter/0.log"
Jul 02 13:30:29.383857 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:29.383820    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9dsgh_1b45d14c-b4b5-4c87-bc99-ed18f33668ef/kube-rbac-proxy/0.log"
Jul 02 13:30:29.408348 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:29.408317    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9dsgh_1b45d14c-b4b5-4c87-bc99-ed18f33668ef/init-textfile/0.log"
Jul 02 13:30:29.531303 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:29.531277    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-p7k4z_1d9e744f-62d0-46e6-a186-3b8d8e5efb94/kube-rbac-proxy-main/0.log"
Jul 02 13:30:29.558981 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:29.558953    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-p7k4z_1d9e744f-62d0-46e6-a186-3b8d8e5efb94/kube-rbac-proxy-self/0.log"
Jul 02 13:30:29.579872 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:29.579843    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-p7k4z_1d9e744f-62d0-46e6-a186-3b8d8e5efb94/openshift-state-metrics/0.log"
Jul 02 13:30:29.959857 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:29.959763    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c4844dbfd-jjrvk_67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8/thanos-query/0.log"
Jul 02 13:30:29.983925 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:29.983893    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c4844dbfd-jjrvk_67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8/kube-rbac-proxy-web/0.log"
Jul 02 13:30:30.011540 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:30.011509    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c4844dbfd-jjrvk_67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8/kube-rbac-proxy/0.log"
Jul 02 13:30:30.033148 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:30.033123    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c4844dbfd-jjrvk_67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8/prom-label-proxy/0.log"
Jul 02 13:30:30.053196 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:30.053152    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c4844dbfd-jjrvk_67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8/kube-rbac-proxy-rules/0.log"
Jul 02 13:30:30.083022 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:30.082988    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c4844dbfd-jjrvk_67d53376-3f77-4cb4-aa14-f6ad8e8b7ef8/kube-rbac-proxy-metrics/0.log"
Jul 02 13:30:33.107268 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:33.107235    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-74fjg/perf-node-gather-daemonset-tgpxf"]
Jul 02 13:30:33.111189 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:33.111152    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-74fjg/perf-node-gather-daemonset-tgpxf"
Jul 02 13:30:33.120463 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:33.120436    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-74fjg/perf-node-gather-daemonset-tgpxf"]
Jul 02 13:30:33.201846 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:33.201801    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtvx4\" (UniqueName: \"kubernetes.io/projected/95c73d3d-b91b-4197-8f1d-f1ff2b7e7de1-kube-api-access-mtvx4\") pod \"perf-node-gather-daemonset-tgpxf\" (UID: \"95c73d3d-b91b-4197-8f1d-f1ff2b7e7de1\") " pod="openshift-must-gather-74fjg/perf-node-gather-daemonset-tgpxf"
Jul 02 13:30:33.202028 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:33.201906    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/95c73d3d-b91b-4197-8f1d-f1ff2b7e7de1-lib-modules\") pod \"perf-node-gather-daemonset-tgpxf\" (UID: \"95c73d3d-b91b-4197-8f1d-f1ff2b7e7de1\") " pod="openshift-must-gather-74fjg/perf-node-gather-daemonset-tgpxf"
Jul 02 13:30:33.202028 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:33.201978    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95c73d3d-b91b-4197-8f1d-f1ff2b7e7de1-sys\") pod \"perf-node-gather-daemonset-tgpxf\" (UID: \"95c73d3d-b91b-4197-8f1d-f1ff2b7e7de1\") " pod="openshift-must-gather-74fjg/perf-node-gather-daemonset-tgpxf"
Jul 02 13:30:33.202028 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:33.202008    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/95c73d3d-b91b-4197-8f1d-f1ff2b7e7de1-proc\") pod \"perf-node-gather-daemonset-tgpxf\" (UID: \"95c73d3d-b91b-4197-8f1d-f1ff2b7e7de1\") " pod="openshift-must-gather-74fjg/perf-node-gather-daemonset-tgpxf"
Jul 02 13:30:33.202161 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:33.202050    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/95c73d3d-b91b-4197-8f1d-f1ff2b7e7de1-podres\") pod \"perf-node-gather-daemonset-tgpxf\" (UID: \"95c73d3d-b91b-4197-8f1d-f1ff2b7e7de1\") " pod="openshift-must-gather-74fjg/perf-node-gather-daemonset-tgpxf"
Jul 02 13:30:33.302859 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:33.302826    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95c73d3d-b91b-4197-8f1d-f1ff2b7e7de1-sys\") pod \"perf-node-gather-daemonset-tgpxf\" (UID: \"95c73d3d-b91b-4197-8f1d-f1ff2b7e7de1\") " pod="openshift-must-gather-74fjg/perf-node-gather-daemonset-tgpxf"
Jul 02 13:30:33.302859 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:33.302865    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/95c73d3d-b91b-4197-8f1d-f1ff2b7e7de1-proc\") pod \"perf-node-gather-daemonset-tgpxf\" (UID: \"95c73d3d-b91b-4197-8f1d-f1ff2b7e7de1\") " pod="openshift-must-gather-74fjg/perf-node-gather-daemonset-tgpxf"
Jul 02 13:30:33.303064 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:33.302899    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/95c73d3d-b91b-4197-8f1d-f1ff2b7e7de1-podres\") pod \"perf-node-gather-daemonset-tgpxf\" (UID: \"95c73d3d-b91b-4197-8f1d-f1ff2b7e7de1\") " pod="openshift-must-gather-74fjg/perf-node-gather-daemonset-tgpxf"
Jul 02 13:30:33.303064 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:33.302920    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtvx4\" (UniqueName: \"kubernetes.io/projected/95c73d3d-b91b-4197-8f1d-f1ff2b7e7de1-kube-api-access-mtvx4\") pod \"perf-node-gather-daemonset-tgpxf\" (UID: \"95c73d3d-b91b-4197-8f1d-f1ff2b7e7de1\") " pod="openshift-must-gather-74fjg/perf-node-gather-daemonset-tgpxf"
Jul 02 13:30:33.303064 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:33.302944    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/95c73d3d-b91b-4197-8f1d-f1ff2b7e7de1-lib-modules\") pod \"perf-node-gather-daemonset-tgpxf\" (UID: \"95c73d3d-b91b-4197-8f1d-f1ff2b7e7de1\") " pod="openshift-must-gather-74fjg/perf-node-gather-daemonset-tgpxf"
Jul 02 13:30:33.303064 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:33.302942    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95c73d3d-b91b-4197-8f1d-f1ff2b7e7de1-sys\") pod \"perf-node-gather-daemonset-tgpxf\" (UID: \"95c73d3d-b91b-4197-8f1d-f1ff2b7e7de1\") " pod="openshift-must-gather-74fjg/perf-node-gather-daemonset-tgpxf"
Jul 02 13:30:33.303064 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:33.303002    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/95c73d3d-b91b-4197-8f1d-f1ff2b7e7de1-proc\") pod \"perf-node-gather-daemonset-tgpxf\" (UID: \"95c73d3d-b91b-4197-8f1d-f1ff2b7e7de1\") " pod="openshift-must-gather-74fjg/perf-node-gather-daemonset-tgpxf"
Jul 02 13:30:33.303268 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:33.303067    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/95c73d3d-b91b-4197-8f1d-f1ff2b7e7de1-lib-modules\") pod \"perf-node-gather-daemonset-tgpxf\" (UID: \"95c73d3d-b91b-4197-8f1d-f1ff2b7e7de1\") " pod="openshift-must-gather-74fjg/perf-node-gather-daemonset-tgpxf"
Jul 02 13:30:33.303268 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:33.303075    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/95c73d3d-b91b-4197-8f1d-f1ff2b7e7de1-podres\") pod \"perf-node-gather-daemonset-tgpxf\" (UID: \"95c73d3d-b91b-4197-8f1d-f1ff2b7e7de1\") " pod="openshift-must-gather-74fjg/perf-node-gather-daemonset-tgpxf"
Jul 02 13:30:33.311624 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:33.311595    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtvx4\" (UniqueName: \"kubernetes.io/projected/95c73d3d-b91b-4197-8f1d-f1ff2b7e7de1-kube-api-access-mtvx4\") pod \"perf-node-gather-daemonset-tgpxf\" (UID: \"95c73d3d-b91b-4197-8f1d-f1ff2b7e7de1\") " pod="openshift-must-gather-74fjg/perf-node-gather-daemonset-tgpxf"
Jul 02 13:30:33.423857 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:33.423756    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-74fjg/perf-node-gather-daemonset-tgpxf"
Jul 02 13:30:33.597138 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:33.597104    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-74fjg/perf-node-gather-daemonset-tgpxf"]
Jul 02 13:30:33.995313 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:33.995285    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tg5m8_851415ab-7420-40c0-aa53-4e1d8899ee4c/dns/0.log"
Jul 02 13:30:34.018877 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:34.018843    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tg5m8_851415ab-7420-40c0-aa53-4e1d8899ee4c/kube-rbac-proxy/0.log"
Jul 02 13:30:34.039587 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:34.039562    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cjwdw_3a25dc59-8e44-4f73-8275-342792a291b6/dns-node-resolver/0.log"
Jul 02 13:30:34.364211 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:34.364108    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-74fjg/perf-node-gather-daemonset-tgpxf"
Jul 02 13:30:34.364775 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:34.364747    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-74fjg/perf-node-gather-daemonset-tgpxf" event={"ID":"95c73d3d-b91b-4197-8f1d-f1ff2b7e7de1","Type":"ContainerStarted","Data":"014bd924369eb316e4d37dd0e663657f9a79bff9bdc5595425677a009c0091d4"}
Jul 02 13:30:34.364936 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:34.364916    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-74fjg/perf-node-gather-daemonset-tgpxf" event={"ID":"95c73d3d-b91b-4197-8f1d-f1ff2b7e7de1","Type":"ContainerStarted","Data":"9bef0216dd1a6ac67eb2592aacef905e6df47a341e5c6b43e5665743de56699d"}
Jul 02 13:30:34.382138 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:34.382077    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-74fjg/perf-node-gather-daemonset-tgpxf" podStartSLOduration=1.382059041 podStartE2EDuration="1.382059041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:30:34.38077685 +0000 UTC m=+3102.513602696" watchObservedRunningTime="2026-07-02 13:30:34.382059041 +0000 UTC m=+3102.514884887"
Jul 02 13:30:34.598526 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:34.598484    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cp22n_9777c828-7acd-4d27-be6a-dbe5c3f6f18b/node-ca/0.log"
Jul 02 13:30:35.933649 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:35.933617    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-764zc_9dcd5dc8-2736-4029-b386-bcecfe1bea3d/serve-healthcheck-canary/0.log"
Jul 02 13:30:36.313653 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:36.313629    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4xx9d_e3a44725-2f0a-44ea-9a50-4f13d28b0bd3/kube-rbac-proxy/0.log"
Jul 02 13:30:36.333546 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:36.333527    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4xx9d_e3a44725-2f0a-44ea-9a50-4f13d28b0bd3/exporter/0.log"
Jul 02 13:30:36.355813 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:36.355787    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4xx9d_e3a44725-2f0a-44ea-9a50-4f13d28b0bd3/extractor/0.log"
Jul 02 13:30:38.584332 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:38.584301    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-69b5d94596-wdwkk_53050271-cef2-40e4-8f3b-d0b1bb9499ca/manager/0.log"
Jul 02 13:30:38.865345 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:38.865276    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-lqvhj_f8297d26-2bc0-4f2a-9ba3-73a29907cc5f/seaweedfs/0.log"
Jul 02 13:30:41.380198 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:41.380154    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-74fjg/perf-node-gather-daemonset-tgpxf"
Jul 02 13:30:44.326450 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:44.326423    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mtbbk_b1652685-279d-4a21-a047-736b5076b4e4/kube-multus-additional-cni-plugins/0.log"
Jul 02 13:30:44.348150 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:44.348119    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mtbbk_b1652685-279d-4a21-a047-736b5076b4e4/egress-router-binary-copy/0.log"
Jul 02 13:30:44.369226 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:44.369201    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mtbbk_b1652685-279d-4a21-a047-736b5076b4e4/cni-plugins/0.log"
Jul 02 13:30:44.388847 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:44.388821    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mtbbk_b1652685-279d-4a21-a047-736b5076b4e4/bond-cni-plugin/0.log"
Jul 02 13:30:44.413041 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:44.413018    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mtbbk_b1652685-279d-4a21-a047-736b5076b4e4/routeoverride-cni/0.log"
Jul 02 13:30:44.434031 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:44.434000    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mtbbk_b1652685-279d-4a21-a047-736b5076b4e4/whereabouts-cni-bincopy/0.log"
Jul 02 13:30:44.461733 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:44.461709    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mtbbk_b1652685-279d-4a21-a047-736b5076b4e4/whereabouts-cni/0.log"
Jul 02 13:30:44.690987 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:44.690916    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-kpfkc_7908d111-8ec6-4177-bd91-38966a57ac13/kube-multus/0.log"
Jul 02 13:30:44.828297 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:44.828265    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xsf2s_e2432aa2-93ec-46ce-afcd-e19f9dd03f8e/network-metrics-daemon/0.log"
Jul 02 13:30:44.846865 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:44.846825    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xsf2s_e2432aa2-93ec-46ce-afcd-e19f9dd03f8e/kube-rbac-proxy/0.log"
Jul 02 13:30:45.737492 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:45.737463    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95kwm_495e9fa8-b93f-42e0-95b5-813527df0993/ovn-controller/0.log"
Jul 02 13:30:45.816388 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:45.816356    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95kwm_495e9fa8-b93f-42e0-95b5-813527df0993/ovn-acl-logging/0.log"
Jul 02 13:30:45.816388 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:45.816386    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95kwm_495e9fa8-b93f-42e0-95b5-813527df0993/ovn-acl-logging/1.log"
Jul 02 13:30:45.897485 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:45.897400    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95kwm_495e9fa8-b93f-42e0-95b5-813527df0993/kube-rbac-proxy-node/0.log"
Jul 02 13:30:45.983430 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:45.983399    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95kwm_495e9fa8-b93f-42e0-95b5-813527df0993/kube-rbac-proxy-ovn-metrics/0.log"
Jul 02 13:30:46.021395 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:46.021371    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95kwm_495e9fa8-b93f-42e0-95b5-813527df0993/northd/0.log"
Jul 02 13:30:46.061765 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:46.061741    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95kwm_495e9fa8-b93f-42e0-95b5-813527df0993/nbdb/0.log"
Jul 02 13:30:46.099856 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:46.099825    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95kwm_495e9fa8-b93f-42e0-95b5-813527df0993/sbdb/0.log"
Jul 02 13:30:46.169763 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:46.169688    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95kwm_495e9fa8-b93f-42e0-95b5-813527df0993/ovnkube-controller/0.log"
Jul 02 13:30:47.710353 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:47.710319    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-4cw8v_cd215e6b-c112-412f-a031-30940e656f0d/network-check-target-container/0.log"
Jul 02 13:30:48.770539 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:48.770512    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-xplqk_b1a6f623-3064-4500-be97-92644ef17328/iptables-alerter/0.log"
Jul 02 13:30:49.330812 ip-10-0-137-127 kubenswrapper[2578]: I0702 13:30:49.330773    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-nj6tf_2b74229e-eeba-4f5d-b05a-1d67c83d450c/tuned/0.log"