Jul 28 17:22:29.596569 ip-10-0-131-233 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 17:22:29.921634 ip-10-0-131-233 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:22:29.921634 ip-10-0-131-233 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 17:22:29.921634 ip-10-0-131-233 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:22:29.921634 ip-10-0-131-233 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 17:22:29.922220 ip-10-0-131-233 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:22:29.923230 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.923145    2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 17:22:29.928016 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.927999    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:22:29.928055 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928019    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:22:29.928055 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928024    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:22:29.928055 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928029    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:22:29.928055 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928034    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:22:29.928055 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928039    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:22:29.928055 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928044    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:22:29.928055 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928048    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:22:29.928055 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928052    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:22:29.928055 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928056    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:22:29.928278 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928061    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:22:29.928278 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928066    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:22:29.928278 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928069    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:22:29.928278 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928074    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:22:29.928278 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928079    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:22:29.928278 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928083    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:22:29.928278 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928087    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:22:29.928278 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928091    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:22:29.928278 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928095    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:22:29.928278 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928099    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:22:29.928278 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928104    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:22:29.928278 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928108    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:22:29.928278 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928112    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:22:29.928278 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928115    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:22:29.928278 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928119    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:22:29.928278 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928123    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:22:29.928278 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928127    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:22:29.928278 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928131    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:22:29.928278 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928135    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:22:29.928278 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928138    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:22:29.928770 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928141    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:22:29.928770 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928145    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:22:29.928770 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928149    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:22:29.928770 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928153    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:22:29.928770 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928159    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:22:29.928770 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928164    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:22:29.928770 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928168    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:22:29.928770 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928172    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:22:29.928770 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928178    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:22:29.928770 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928184    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:22:29.928770 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928188    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:22:29.928770 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928192    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:22:29.928770 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928196    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:22:29.928770 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928202    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:22:29.928770 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928205    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:22:29.928770 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928209    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:22:29.928770 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928213    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:22:29.928770 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928217    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:22:29.928770 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928220    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:22:29.929235 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928224    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:22:29.929235 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928228    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:22:29.929235 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928231    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:22:29.929235 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928235    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:22:29.929235 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928238    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:22:29.929235 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928241    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:22:29.929235 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928245    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:22:29.929235 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928249    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:22:29.929235 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928253    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:22:29.929235 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928256    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:22:29.929235 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928260    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:22:29.929235 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928263    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:22:29.929235 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928267    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:22:29.929235 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928271    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:22:29.929235 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928274    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:22:29.929235 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928278    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:22:29.929235 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928281    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:22:29.929235 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928284    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:22:29.929235 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928288    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:22:29.929718 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928292    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:22:29.929718 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928296    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:22:29.929718 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928299    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:22:29.929718 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928304    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:22:29.929718 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928327    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:22:29.929718 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928332    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:22:29.929718 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928336    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:22:29.929718 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928341    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:22:29.929718 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928345    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:22:29.929718 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928350    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:22:29.929718 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928354    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:22:29.929718 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928358    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:22:29.929718 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928361    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:22:29.929718 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928365    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:22:29.929718 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928369    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:22:29.929718 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928372    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:22:29.929718 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928376    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:22:29.929718 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928379    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:22:29.929718 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928383    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:22:29.929718 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928386    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:22:29.930199 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928390    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:22:29.930199 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928393    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:22:29.930199 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928397    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:22:29.930199 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928400    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:22:29.930199 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928404    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:22:29.930199 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928407    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:22:29.930199 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928411    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:22:29.930199 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928414    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:22:29.930199 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928418    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:22:29.930199 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928422    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:22:29.930199 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.928425    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:22:29.930199 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929411    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:22:29.930199 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929420    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:22:29.930199 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929424    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:22:29.930199 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929428    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:22:29.930199 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929433    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:22:29.930199 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929437    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:22:29.930199 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929441    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:22:29.930199 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929445    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:22:29.930199 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929450    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:22:29.930697 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929454    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:22:29.930697 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929458    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:22:29.930697 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929462    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:22:29.930697 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929466    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:22:29.930697 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929470    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:22:29.930697 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929474    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:22:29.930697 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929478    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:22:29.930697 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929484    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:22:29.930697 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929489    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:22:29.930697 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929494    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:22:29.930697 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929497    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:22:29.930697 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929502    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:22:29.930697 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929505    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:22:29.930697 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929509    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:22:29.930697 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929512    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:22:29.930697 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929518    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:22:29.930697 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929523    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:22:29.930697 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929527    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:22:29.930697 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929531    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:22:29.931171 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929534    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:22:29.931171 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929538    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:22:29.931171 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929542    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:22:29.931171 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929547    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:22:29.931171 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929553    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:22:29.931171 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929557    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:22:29.931171 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929560    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:22:29.931171 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929564    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:22:29.931171 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929568    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:22:29.931171 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929572    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:22:29.931171 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929577    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:22:29.931171 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929581    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:22:29.931171 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929585    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:22:29.931171 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929589    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:22:29.931171 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929593    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:22:29.931171 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929598    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:22:29.931171 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929601    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:22:29.931171 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929604    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:22:29.931171 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929608    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:22:29.931649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929612    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:22:29.931649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929616    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:22:29.931649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929620    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:22:29.931649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929624    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:22:29.931649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929627    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:22:29.931649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929631    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:22:29.931649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929634    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:22:29.931649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929638    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:22:29.931649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929642    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:22:29.931649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929645    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:22:29.931649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929649    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:22:29.931649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929653    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:22:29.931649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929656    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:22:29.931649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929661    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:22:29.931649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929664    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:22:29.931649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929668    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:22:29.931649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929671    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:22:29.931649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929675    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:22:29.931649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929678    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:22:29.931649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929682    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:22:29.932147 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929686    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:22:29.932147 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929690    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:22:29.932147 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929694    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:22:29.932147 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929699    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:22:29.932147 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929703    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:22:29.932147 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929708    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:22:29.932147 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929712    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:22:29.932147 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929716    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:22:29.932147 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929719    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:22:29.932147 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929723    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:22:29.932147 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929729    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:22:29.932147 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929733    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:22:29.932147 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929737    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:22:29.932147 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929741    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:22:29.932147 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929744    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:22:29.932147 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929748    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:22:29.932147 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929752    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:22:29.932147 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929756    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:22:29.932147 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929759    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:22:29.932147 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929763    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:22:29.932649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929766    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:22:29.932649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929770    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:22:29.932649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929773    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:22:29.932649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929777    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:22:29.932649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929781    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:22:29.932649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929784    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:22:29.932649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929789    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:22:29.932649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929792    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:22:29.932649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929796    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:22:29.932649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929800    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:22:29.932649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929804    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:22:29.932649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.929808    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:22:29.932649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930310    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:22:29.932649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930330    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:22:29.932649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930335    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:22:29.932649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930339    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:22:29.932649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930343    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:22:29.932649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930349    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:22:29.932649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930353    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:22:29.932649 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930357    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:22:29.933148 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930361    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:22:29.933148 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930365    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:22:29.933148 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930369    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:22:29.933148 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930372    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:22:29.933148 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930376    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:22:29.933148 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930380    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:22:29.933148 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930384    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:22:29.933148 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930387    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:22:29.933148 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930391    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:22:29.933148 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930394    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:22:29.933148 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930397    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:22:29.933148 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930401    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:22:29.933148 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930405    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:22:29.933148 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930409    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:22:29.933148 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930412    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:22:29.933148 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930416    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:22:29.933148 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930419    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:22:29.933148 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930423    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:22:29.933148 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930426    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:22:29.933148 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930430    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:22:29.933679 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930434    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:22:29.933679 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930437    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:22:29.933679 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930441    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:22:29.933679 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930444    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:22:29.933679 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930448    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:22:29.933679 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930452    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:22:29.933679 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930456    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:22:29.933679 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930460    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:22:29.933679 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930464    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:22:29.933679 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930468    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:22:29.933679 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930472    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:22:29.933679 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930476    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:22:29.933679 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930480    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:22:29.933679 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930484    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:22:29.933679 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930488    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:22:29.933679 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930491    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:22:29.933679 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930495    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:22:29.933679 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930498    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:22:29.933679 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930502    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:22:29.933679 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930506    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:22:29.934162 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930510    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:22:29.934162 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930513    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:22:29.934162 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930517    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:22:29.934162 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930521    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:22:29.934162 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930525    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:22:29.934162 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930528    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:22:29.934162 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930532    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:22:29.934162 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930536    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:22:29.934162 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930539    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:22:29.934162 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930544    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:22:29.934162 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930548    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:22:29.934162 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930551    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:22:29.934162 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930555    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:22:29.934162 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930558    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:22:29.934162 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930562    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:22:29.934162 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930565    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:22:29.934162 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930569    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:22:29.934162 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930573    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:22:29.934162 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930576    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:22:29.934644 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930580    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:22:29.934644 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930584    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:22:29.934644 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930588    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:22:29.934644 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930592    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:22:29.934644 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930596    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:22:29.934644 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930600    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:22:29.934644 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930604    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:22:29.934644 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930607    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:22:29.934644 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930611    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:22:29.934644 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930615    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:22:29.934644 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930620    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:22:29.934644 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930625    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:22:29.934644 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930628    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:22:29.934644 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930632    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:22:29.934644 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930636    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:22:29.934644 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930639    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:22:29.934644 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930646    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:22:29.934644 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930651    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:22:29.934644 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930656    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:22:29.935133 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930661    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:22:29.935133 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930666    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:22:29.935133 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930670    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:22:29.935133 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930674    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:22:29.935133 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930678    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:22:29.935133 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930682    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:22:29.935133 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930686    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:22:29.935133 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930690    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:22:29.935133 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930694    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:22:29.935133 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930697    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:22:29.935133 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930701    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:22:29.935133 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930705    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:22:29.935133 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.930709    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:22:29.935133 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930801    2566 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 17:22:29.935133 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930810    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 17:22:29.935133 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930818    2566 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 17:22:29.935133 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930824    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 17:22:29.935133 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930830    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 17:22:29.935133 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930835    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 17:22:29.935133 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930841    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 17:22:29.935133 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930847    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 17:22:29.935737 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930851    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 17:22:29.935737 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930855    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 17:22:29.935737 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930860    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 17:22:29.935737 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930864    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 17:22:29.935737 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930869    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 17:22:29.935737 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930873    2566 flags.go:64] FLAG: --cgroup-root=""
Jul 28 17:22:29.935737 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930878    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 17:22:29.935737 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930882    2566 flags.go:64] FLAG: --client-ca-file=""
Jul 28 17:22:29.935737 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930886    2566 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 17:22:29.935737 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930890    2566 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 17:22:29.935737 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930897    2566 flags.go:64] FLAG: --cluster-domain=""
Jul 28 17:22:29.935737 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930901    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 17:22:29.935737 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930906    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 17:22:29.935737 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930911    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 17:22:29.935737 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930915    2566 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 17:22:29.935737 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930921    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 17:22:29.935737 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930925    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 17:22:29.935737 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930929    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 17:22:29.935737 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930933    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 17:22:29.935737 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930938    2566 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 17:22:29.935737 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930942    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 17:22:29.935737 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930946    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 17:22:29.935737 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930951    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 17:22:29.935737 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930955    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 17:22:29.936302 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930961    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 17:22:29.936302 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930965    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 17:22:29.936302 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930973    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 17:22:29.936302 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930977    2566 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 17:22:29.936302 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930981    2566 flags.go:64] FLAG: --enable-server="true"
Jul 28 17:22:29.936302 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930985    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 17:22:29.936302 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930992    2566 flags.go:64] FLAG: --event-burst="100"
Jul 28 17:22:29.936302 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.930996    2566 flags.go:64] FLAG: --event-qps="50"
Jul 28 17:22:29.936302 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931000    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 17:22:29.936302 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931004    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 17:22:29.936302 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931008    2566 flags.go:64] FLAG: --eviction-hard=""
Jul 28 17:22:29.936302 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931014    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 17:22:29.936302 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931018    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 17:22:29.936302 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931022    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 17:22:29.936302 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931026    2566 flags.go:64] FLAG: --eviction-soft=""
Jul 28 17:22:29.936302 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931031    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 17:22:29.936302 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931035    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 17:22:29.936302 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931039    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 17:22:29.936302 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931043    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 17:22:29.936302 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931048    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 17:22:29.936302 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931052    2566 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 17:22:29.936302 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931056    2566 flags.go:64] FLAG: --feature-gates=""
Jul 28 17:22:29.936302 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931061    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 17:22:29.936302 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931066    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 17:22:29.936302 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931071    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 17:22:29.936987 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931075    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 17:22:29.936987 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931080    2566 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 17:22:29.936987 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931084    2566 flags.go:64] FLAG: --help="false"
Jul 28 17:22:29.936987 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931088    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-131-233.ec2.internal"
Jul 28 17:22:29.936987 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931092    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 17:22:29.936987 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931097    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 17:22:29.936987 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931101    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 17:22:29.936987 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931106    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 17:22:29.936987 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931111    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 17:22:29.936987 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931117    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 17:22:29.936987 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931121    2566 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 17:22:29.936987 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931125    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 17:22:29.936987 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931130    2566 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 17:22:29.936987 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931134    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 17:22:29.936987 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931138    2566 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 17:22:29.936987 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931143    2566 flags.go:64] FLAG: --kube-reserved=""
Jul 28 17:22:29.936987 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931147    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 17:22:29.936987 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931152    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 17:22:29.936987 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931156    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 17:22:29.936987 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931160    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 17:22:29.936987 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931163    2566 flags.go:64] FLAG: --lock-file=""
Jul 28 17:22:29.936987 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931168    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 17:22:29.936987 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931172    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 17:22:29.936987 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931177    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 17:22:29.937592 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931184    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 17:22:29.937592 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931189    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 17:22:29.937592 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931193    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 17:22:29.937592 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931197    2566 flags.go:64] FLAG: --logging-format="text"
Jul 28 17:22:29.937592 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931201    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 17:22:29.937592 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931205    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 17:22:29.937592 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931209    2566 flags.go:64] FLAG: --manifest-url=""
Jul 28 17:22:29.937592 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931213    2566 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 17:22:29.937592 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931219    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 17:22:29.937592 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931223    2566 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 17:22:29.937592 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931228    2566 flags.go:64] FLAG: --max-pods="110"
Jul 28 17:22:29.937592 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931233    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 17:22:29.937592 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931237    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 17:22:29.937592 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931241    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 17:22:29.937592 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931246    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 17:22:29.937592 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931250    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 17:22:29.937592 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931254    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 17:22:29.937592 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931260    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 17:22:29.937592 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931270    2566 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 17:22:29.937592 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931274    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 17:22:29.937592 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931278    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 17:22:29.937592 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931283    2566 flags.go:64] FLAG: --pod-cidr=""
Jul 28 17:22:29.937592 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931287    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 17:22:29.938154 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931294    2566 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 17:22:29.938154 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931299    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 17:22:29.938154 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931303    2566 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 17:22:29.938154 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931308    2566 flags.go:64] FLAG: --port="10250"
Jul 28 17:22:29.938154 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931312    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 17:22:29.938154 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931331    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-035ba790b1d92cd4e"
Jul 28 17:22:29.938154 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931336    2566 flags.go:64] FLAG: --qos-reserved=""
Jul 28 17:22:29.938154 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931341    2566 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 17:22:29.938154 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931346    2566 flags.go:64] FLAG: --register-node="true"
Jul 28 17:22:29.938154 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931350    2566 flags.go:64] FLAG: --register-with-taints=""
Jul 28 17:22:29.938154 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931355    2566 flags.go:64] FLAG: --registry-burst="10"
Jul 28 17:22:29.938154 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931359    2566 flags.go:64] FLAG: --registry-qps="5"
Jul 28 17:22:29.938154 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931363    2566 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 17:22:29.938154 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931368    2566 flags.go:64] FLAG: --reserved-memory=""
Jul 28 17:22:29.938154 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931373    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 17:22:29.938154 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931377    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 17:22:29.938154 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931381    2566 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 17:22:29.938154 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931386    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 17:22:29.938154 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931390    2566 flags.go:64] FLAG: --runonce="false"
Jul 28 17:22:29.938154 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931395    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 17:22:29.938154 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931399    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 17:22:29.938154 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931403    2566 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 17:22:29.938154 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931407    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 17:22:29.938154 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931411    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 17:22:29.938154 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931416    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 17:22:29.938154 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931420    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 17:22:29.938819 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931424    2566 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 17:22:29.938819 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931430    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 17:22:29.938819 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931434    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 17:22:29.938819 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931438    2566 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 17:22:29.938819 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931442    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 17:22:29.938819 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931446    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 17:22:29.938819 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931450    2566 flags.go:64] FLAG: --system-cgroups=""
Jul 28 17:22:29.938819 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931455    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 17:22:29.938819 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931462    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 17:22:29.938819 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931466    2566 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 17:22:29.938819 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931470    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 17:22:29.938819 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931477    2566 flags.go:64] FLAG: --tls-min-version=""
Jul 28 17:22:29.938819 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931480    2566 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 17:22:29.938819 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931486    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 17:22:29.938819 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931490    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 17:22:29.938819 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931495    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 17:22:29.938819 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931499    2566 flags.go:64] FLAG: --v="2"
Jul 28 17:22:29.938819 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931505    2566 flags.go:64] FLAG: --version="false"
Jul 28 17:22:29.938819 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931509    2566 flags.go:64] FLAG: --vmodule=""
Jul 28 17:22:29.938819 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931515    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 17:22:29.938819 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.931520    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 17:22:29.938819 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931680    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:22:29.938819 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931686    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:22:29.938819 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931691    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:22:29.939408 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931695    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:22:29.939408 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931699    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:22:29.939408 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931703    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:22:29.939408 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931707    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:22:29.939408 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931711    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:22:29.939408 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931714    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:22:29.939408 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931718    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:22:29.939408 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931722    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:22:29.939408 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931726    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:22:29.939408 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931731    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:22:29.939408 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931735    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:22:29.939408 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931738    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:22:29.939408 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931742    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:22:29.939408 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931747    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:22:29.939408 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931750    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:22:29.939408 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931754    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:22:29.939408 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931758    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:22:29.939408 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931762    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:22:29.939408 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931766    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:22:29.939408 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931770    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:22:29.940047 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931773    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:22:29.940047 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931780    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:22:29.940047 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931784    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:22:29.940047 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931789    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:22:29.940047 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931794    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:22:29.940047 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931799    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:22:29.940047 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931803    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:22:29.940047 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931831    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:22:29.940047 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931837    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:22:29.940047 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931842    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:22:29.940047 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931846    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:22:29.940047 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931850    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:22:29.940047 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931854    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:22:29.940047 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931857    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:22:29.940047 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931861    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:22:29.940047 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931865    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:22:29.940047 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931869    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:22:29.940047 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931872    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:22:29.940047 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931876    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:22:29.940885 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931880    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:22:29.940885 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931883    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:22:29.940885 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931889    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:22:29.940885 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931892    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:22:29.940885 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931896    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:22:29.940885 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931900    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:22:29.940885 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931903    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:22:29.940885 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931907    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:22:29.940885 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931911    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:22:29.940885 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931915    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:22:29.940885 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931919    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:22:29.940885 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931923    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:22:29.940885 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931929    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:22:29.940885 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931932    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:22:29.940885 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931938    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:22:29.940885 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931942    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:22:29.940885 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931946    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:22:29.940885 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931949    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:22:29.940885 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931953    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:22:29.941472 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931956    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:22:29.941472 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931960    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:22:29.941472 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931964    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:22:29.941472 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931969    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:22:29.941472 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931972    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:22:29.941472 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931975    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:22:29.941472 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931979    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:22:29.941472 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931983    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:22:29.941472 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931987    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:22:29.941472 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931990    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:22:29.941472 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931994    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:22:29.941472 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.931997    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:22:29.941472 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.932000    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:22:29.941472 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.932004    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:22:29.941472 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.932008    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:22:29.941472 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.932013    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:22:29.941472 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.932017    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:22:29.941472 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.932020    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:22:29.941472 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.932024    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:22:29.941472 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.932028    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:22:29.941972 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.932032    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:22:29.941972 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.932036    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:22:29.941972 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.932039    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:22:29.941972 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.932043    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:22:29.941972 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.932047    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:22:29.941972 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.932050    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:22:29.941972 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.932054    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:22:29.941972 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.932060    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:22:29.941972 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.932063    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:22:29.941972 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.932067    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:22:29.941972 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.932070    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:22:29.941972 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.932074    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:22:29.941972 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.932078    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:22:29.941972 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.932082    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:22:29.941972 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.932085    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:22:29.941972 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.932089    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:22:29.941972 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.932093    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:22:29.941972 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.932096    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:22:29.942423 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.932931    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:22:29.942423 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.940518    2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 17:22:29.942423 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.940539    2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 17:22:29.942423 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.941211    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 17:22:29.942423 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.941255    2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 17:22:29.942423 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941543    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:22:29.942423 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941557    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:22:29.942423 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941561    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:22:29.942423 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941564    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:22:29.942423 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941567    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:22:29.942423 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941571    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:22:29.942423 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941574    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:22:29.942423 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941577    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:22:29.942423 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941580    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:22:29.942779 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941582    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:22:29.942779 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941585    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:22:29.942779 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941588    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:22:29.942779 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941592    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:22:29.942779 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941596    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:22:29.942779 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941600    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:22:29.942779 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941603    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:22:29.942779 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941606    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:22:29.942779 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941609    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:22:29.942779 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941612    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:22:29.942779 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941614    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:22:29.942779 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941618    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:22:29.942779 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941622    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:22:29.942779 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941625    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:22:29.942779 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941628    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:22:29.942779 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941631    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:22:29.942779 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941634    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:22:29.942779 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941637    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:22:29.942779 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941640    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:22:29.943232 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941643    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:22:29.943232 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941646    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:22:29.943232 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941650    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:22:29.943232 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941653    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:22:29.943232 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941657    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:22:29.943232 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941661    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:22:29.943232 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941665    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:22:29.943232 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941667    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:22:29.943232 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941671    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:22:29.943232 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941673    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:22:29.943232 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941676    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:22:29.943232 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941679    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:22:29.943232 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941683    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:22:29.943232 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941685    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:22:29.943232 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941688    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:22:29.943232 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941691    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:22:29.943232 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941694    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:22:29.943232 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941697    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:22:29.943232 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941700    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:22:29.943713 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941703    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:22:29.943713 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941706    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:22:29.943713 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941708    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:22:29.943713 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941711    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:22:29.943713 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941714    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:22:29.943713 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941716    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:22:29.943713 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941719    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:22:29.943713 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941721    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:22:29.943713 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941724    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:22:29.943713 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941726    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:22:29.943713 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941729    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:22:29.943713 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941732    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:22:29.943713 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941734    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:22:29.943713 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941737    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:22:29.943713 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941740    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:22:29.943713 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941743    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:22:29.943713 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941745    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:22:29.943713 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941748    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:22:29.943713 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941751    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:22:29.943713 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941754    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:22:29.944233 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941757    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:22:29.944233 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941759    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:22:29.944233 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941762    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:22:29.944233 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941765    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:22:29.944233 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941767    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:22:29.944233 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941770    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:22:29.944233 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941772    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:22:29.944233 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941775    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:22:29.944233 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941777    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:22:29.944233 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941780    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:22:29.944233 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941782    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:22:29.944233 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941785    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:22:29.944233 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941788    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:22:29.944233 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941790    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:22:29.944233 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941793    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:22:29.944233 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941796    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:22:29.944233 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941798    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:22:29.944233 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941801    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:22:29.944233 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941803    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:22:29.944233 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941806    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:22:29.944740 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941809    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:22:29.944740 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941811    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:22:29.944740 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941814    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:22:29.944740 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941817    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:22:29.944740 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941820    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:22:29.944740 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941823    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:22:29.944740 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941825    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:22:29.944740 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941828    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:22:29.944740 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941831    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:22:29.944740 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941833    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:22:29.944740 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941836    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:22:29.944740 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941839    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:22:29.944740 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.941844    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:22:29.944740 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941964    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:22:29.944740 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941968    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:22:29.945109 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941971    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:22:29.945109 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941974    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:22:29.945109 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941977    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:22:29.945109 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941980    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:22:29.945109 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941984    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:22:29.945109 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941987    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:22:29.945109 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941990    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:22:29.945109 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941993    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:22:29.945109 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941996    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:22:29.945109 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.941998    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:22:29.945109 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942003    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:22:29.945109 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942006    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:22:29.945109 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942008    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:22:29.945109 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942011    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:22:29.945109 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942015    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:22:29.945109 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942018    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:22:29.945109 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942021    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:22:29.945109 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942024    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:22:29.945109 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942027    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:22:29.945109 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942029    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:22:29.945676 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942033    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:22:29.945676 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942035    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:22:29.945676 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942038    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:22:29.945676 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942041    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:22:29.945676 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942044    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:22:29.945676 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942046    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:22:29.945676 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942048    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:22:29.945676 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942051    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:22:29.945676 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942054    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:22:29.945676 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942057    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:22:29.945676 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942061    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:22:29.945676 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942064    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:22:29.945676 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942066    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:22:29.945676 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942069    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:22:29.945676 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942071    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:22:29.945676 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942074    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:22:29.945676 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942077    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:22:29.945676 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942079    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:22:29.945676 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942082    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:22:29.945676 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942084    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:22:29.946168 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942087    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:22:29.946168 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942090    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:22:29.946168 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942092    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:22:29.946168 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942095    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:22:29.946168 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942098    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:22:29.946168 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942100    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:22:29.946168 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942103    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:22:29.946168 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942105    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:22:29.946168 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942108    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:22:29.946168 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942110    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:22:29.946168 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942113    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:22:29.946168 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942115    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:22:29.946168 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942118    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:22:29.946168 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942120    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:22:29.946168 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942123    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:22:29.946168 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942126    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:22:29.946168 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942128    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:22:29.946168 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942131    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:22:29.946168 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942133    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:22:29.946168 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942136    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:22:29.946714 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942138    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:22:29.946714 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942141    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:22:29.946714 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942143    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:22:29.946714 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942146    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:22:29.946714 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942148    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:22:29.946714 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942151    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:22:29.946714 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942154    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:22:29.946714 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942156    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:22:29.946714 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942159    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:22:29.946714 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942161    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:22:29.946714 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942165    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:22:29.946714 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942168    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:22:29.946714 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942171    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:22:29.946714 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942174    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:22:29.946714 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942176    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:22:29.946714 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942179    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:22:29.946714 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942183    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:22:29.946714 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942186    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:22:29.946714 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942188    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:22:29.947188 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942191    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:22:29.947188 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942193    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:22:29.947188 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942196    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:22:29.947188 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942198    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:22:29.947188 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942201    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:22:29.947188 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942204    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:22:29.947188 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942206    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:22:29.947188 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942209    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:22:29.947188 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942211    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:22:29.947188 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942214    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:22:29.947188 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942216    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:22:29.947188 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942219    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:22:29.947188 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942222    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:22:29.947188 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942224    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:22:29.947188 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942227    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:22:29.947188 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942229    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:22:29.947188 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942232    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:22:29.947188 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:29.942234    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:22:29.947731 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.942239    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:22:29.947731 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.942352    2566 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 17:22:29.947731 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.946043    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 17:22:29.947731 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.947231    2566 server.go:1019] "Starting client certificate rotation"
Jul 28 17:22:29.947731 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.947360    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:22:29.947731 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.947400    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:22:29.949349 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.949314    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:22:29.950464 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.950448    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:22:29.961220 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.961201    2566 log.go:25] "Validated CRI v1 runtime API"
Jul 28 17:22:29.963183 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.963168    2566 log.go:25] "Validated CRI v1 image API"
Jul 28 17:22:29.964013 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.963995    2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 17:22:29.966130 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.966105    2566 fs.go:135] Filesystem UUIDs: map[6058b563-8571-4cf8-adef-39732b8216fe:/dev/nvme0n1p3 79e72641-75e8-4bfb-8b16-1cbb5dd9009d:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 28 17:22:29.966203 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.966128    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 17:22:29.971925 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.971809    2566 manager.go:216] Machine: {Timestamp:2026-07-28 17:22:29.970721787 +0000 UTC m=+0.280887302 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3106051 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22adee1c30dad51cf094d0b285e29e SystemUUID:ec22adee-1c30-dad5-1cf0-94d0b285e29e BootID:0af943b1-0aad-4d3b-a52e-b6ab66389e64 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:74:47:a1:db:d5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:74:47:a1:db:d5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:72:9f:15:05:24 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 17:22:29.971925 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.971916    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 17:22:29.972074 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.971985    2566 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 17:22:29.973297 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.973268    2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 17:22:29.973470 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.973299    2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-233.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 17:22:29.974036 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.974023    2566 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 17:22:29.974036 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.974037    2566 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 17:22:29.974110 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.974044    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:22:29.974110 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.974060    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:22:29.974834 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.974824    2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 17:22:29.976763 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.976742    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:22:29.977445 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.977434    2566 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:22:29.977550 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.977541    2566 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 17:22:29.977583 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.977569    2566 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 17:22:29.977583 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.977578    2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 17:22:29.977634 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.977589    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 17:22:29.977634 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.977608    2566 kubelet.go:398] "Adding apiserver pod source"
Jul 28 17:22:29.977634 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.977627    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 17:22:29.978228 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.978212    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:22:29.978313 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.978233    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:22:29.982215 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.982186    2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 17:22:29.983981 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.983966    2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 17:22:29.984056 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.983995    2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 17:22:29.985379 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.985367    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 17:22:29.985438 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.985383    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 17:22:29.985438 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.985389    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 17:22:29.985438 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.985395    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 17:22:29.985438 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.985404    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 17:22:29.985438 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.985409    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 17:22:29.985438 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.985417    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 17:22:29.985438 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.985422    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 17:22:29.985438 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.985429    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 17:22:29.985438 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.985435    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 17:22:29.985668 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.985443    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 17:22:29.985668 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.985449    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 17:22:29.986035 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.986026    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 17:22:29.986035 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.986035    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 17:22:29.989065 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.989048    2566 server.go:1268] "Started kubelet"
Jul 28 17:22:29.989268 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.989218    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 17:22:29.989268 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.989209    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 17:22:29.989268 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:29.989249    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 17:22:29.989453 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.989283    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 17:22:29.989453 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:29.989333    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-233.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 17:22:29.989453 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.989407    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-233.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 17:22:29.989916 ip-10-0-131-233 systemd[1]: Started Kubernetes Kubelet.
Jul 28 17:22:29.991231 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.991214    2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 17:22:29.991660 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.991641    2566 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 17:22:29.993798 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:29.993780    2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 17:22:29.995259 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:29.994511    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-233.ec2.internal.18c683eb8c0a4042  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-233.ec2.internal,UID:ip-10-0-131-233.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-233.ec2.internal,},FirstTimestamp:2026-07-28 17:22:29.989015618 +0000 UTC m=+0.299181114,LastTimestamp:2026-07-28 17:22:29.989015618 +0000 UTC m=+0.299181114,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-233.ec2.internal,}"
Jul 28 17:22:29.997015 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.997001    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 17:22:29.997074 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.997015    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 17:22:29.997127 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:29.997080    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-233.ec2.internal\" not found"
Jul 28 17:22:29.997649 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.997628    2566 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 17:22:29.997649 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.997628    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 17:22:29.997764 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.997653    2566 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 17:22:29.997823 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.997756    2566 factory.go:55] Registering systemd factory
Jul 28 17:22:29.997823 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.997780    2566 factory.go:223] Registration of the systemd container factory successfully
Jul 28 17:22:29.997823 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.997785    2566 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 17:22:29.997823 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.997794    2566 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 17:22:29.998013 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.997991    2566 factory.go:153] Registering CRI-O factory
Jul 28 17:22:29.998013 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.998003    2566 factory.go:223] Registration of the crio container factory successfully
Jul 28 17:22:29.998094 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.998051    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 17:22:29.998094 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.998078    2566 factory.go:103] Registering Raw factory
Jul 28 17:22:29.998094 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.998093    2566 manager.go:1196] Started watching for new ooms in manager
Jul 28 17:22:29.998734 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:29.998717    2566 manager.go:349] Starting recovery of all containers
Jul 28 17:22:30.000609 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.000586    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b4qvv"
Jul 28 17:22:30.001037 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:30.001016    2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-233.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 17:22:30.001276 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:30.001236    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 17:22:30.011262 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.011240    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b4qvv"
Jul 28 17:22:30.013237 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.013218    2566 manager.go:354] Recovery completed
Jul 28 17:22:30.015272 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:30.015251    2566 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 28 17:22:30.018180 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.018169    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:22:30.020197 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.020093    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-233.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:22:30.020271 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.020215    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-233.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:22:30.020271 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.020232    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-233.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:22:30.020778 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.020766    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 17:22:30.020778 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.020777    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 17:22:30.020895 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.020784    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:22:30.020895 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.020794    2566 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:22:30.021457 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.021442    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-233.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:22:30.021516 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.021469    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-233.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:22:30.021516 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.021482    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-233.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:22:30.022144 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:30.022085    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-233.ec2.internal.18c683eb8de60d4b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-233.ec2.internal,UID:ip-10-0-131-233.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-233.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-233.ec2.internal,},FirstTimestamp:2026-07-28 17:22:30.020197707 +0000 UTC m=+0.330363208,LastTimestamp:2026-07-28 17:22:30.020197707 +0000 UTC m=+0.330363208,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-233.ec2.internal,}"
Jul 28 17:22:30.023208 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.023195    2566 policy_none.go:49] "None policy: Start"
Jul 28 17:22:30.023257 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.023212    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 17:22:30.023257 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.023222    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 17:22:30.025196 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.025185    2566 policy_none.go:47] "Start"
Jul 28 17:22:30.063829 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.063812    2566 manager.go:337] "Starting Device Plugin manager"
Jul 28 17:22:30.063904 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:30.063844    2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 17:22:30.063904 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.063855    2566 server.go:87] "Starting device plugin registration server"
Jul 28 17:22:30.064048 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.064038    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 17:22:30.064078 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.064050    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 17:22:30.064139 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.064126    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 17:22:30.064215 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.064201    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 17:22:30.064258 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.064215    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 17:22:30.064672 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:30.064651    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 17:22:30.064763 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:30.064710    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-233.ec2.internal\" not found"
Jul 28 17:22:30.091515 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.091491    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 17:22:30.100036 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.092544    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 17:22:30.100036 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.092579    2566 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 17:22:30.100036 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.092598    2566 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 17:22:30.100036 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:30.092633    2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 17:22:30.100036 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.094719    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:22:30.164982 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.164964    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:22:30.165685 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.165665    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-233.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:22:30.165767 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.165694    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-233.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:22:30.165767 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.165707    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-233.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:22:30.165767 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.165727    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-233.ec2.internal"
Jul 28 17:22:30.171865 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.171829    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-233.ec2.internal"
Jul 28 17:22:30.171865 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:30.171852    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-233.ec2.internal\": node \"ip-10-0-131-233.ec2.internal\" not found"
Jul 28 17:22:30.182713 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:30.182698    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-233.ec2.internal\" not found"
Jul 28 17:22:30.193536 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.193510    2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-233.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-233.ec2.internal"]
Jul 28 17:22:30.193587 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.193569    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:22:30.194222 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.194208    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-233.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:22:30.194280 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.194236    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-233.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:22:30.194280 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.194258    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-233.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:22:30.196565 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.196553    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:22:30.196676 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.196663    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-233.ec2.internal"
Jul 28 17:22:30.196751 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.196694    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:22:30.197145 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.197129    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-233.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:22:30.197201 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.197153    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-233.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:22:30.197201 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.197164    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-233.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:22:30.197698 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.197683    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-233.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:22:30.197801 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.197705    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-233.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:22:30.197801 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.197719    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-233.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:22:30.198957 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.198941    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f88513b366a558c56d55360985f15ee-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-233.ec2.internal\" (UID: \"7f88513b366a558c56d55360985f15ee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-233.ec2.internal"
Jul 28 17:22:30.199017 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.198966    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f88513b366a558c56d55360985f15ee-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-233.ec2.internal\" (UID: \"7f88513b366a558c56d55360985f15ee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-233.ec2.internal"
Jul 28 17:22:30.199017 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.198981    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a2d32026394497da64d771aa0e11975c-config\") pod \"kube-apiserver-proxy-ip-10-0-131-233.ec2.internal\" (UID: \"a2d32026394497da64d771aa0e11975c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-233.ec2.internal"
Jul 28 17:22:30.199884 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.199869    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-233.ec2.internal"
Jul 28 17:22:30.199930 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.199898    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:22:30.200524 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.200505    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-233.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:22:30.200602 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.200539    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-233.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:22:30.200602 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.200553    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-233.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:22:30.227850 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:30.227830    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-233.ec2.internal\" not found" node="ip-10-0-131-233.ec2.internal"
Jul 28 17:22:30.232219 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:30.232205    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-233.ec2.internal\" not found" node="ip-10-0-131-233.ec2.internal"
Jul 28 17:22:30.283716 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:30.283698    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-233.ec2.internal\" not found"
Jul 28 17:22:30.299073 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.299054    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a2d32026394497da64d771aa0e11975c-config\") pod \"kube-apiserver-proxy-ip-10-0-131-233.ec2.internal\" (UID: \"a2d32026394497da64d771aa0e11975c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-233.ec2.internal"
Jul 28 17:22:30.299129 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.299088    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f88513b366a558c56d55360985f15ee-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-233.ec2.internal\" (UID: \"7f88513b366a558c56d55360985f15ee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-233.ec2.internal"
Jul 28 17:22:30.299129 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.299104    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f88513b366a558c56d55360985f15ee-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-233.ec2.internal\" (UID: \"7f88513b366a558c56d55360985f15ee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-233.ec2.internal"
Jul 28 17:22:30.299194 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.299148    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a2d32026394497da64d771aa0e11975c-config\") pod \"kube-apiserver-proxy-ip-10-0-131-233.ec2.internal\" (UID: \"a2d32026394497da64d771aa0e11975c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-233.ec2.internal"
Jul 28 17:22:30.299194 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.299163    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f88513b366a558c56d55360985f15ee-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-233.ec2.internal\" (UID: \"7f88513b366a558c56d55360985f15ee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-233.ec2.internal"
Jul 28 17:22:30.299263 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.299228    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f88513b366a558c56d55360985f15ee-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-233.ec2.internal\" (UID: \"7f88513b366a558c56d55360985f15ee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-233.ec2.internal"
Jul 28 17:22:30.384083 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:30.384063    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-233.ec2.internal\" not found"
Jul 28 17:22:30.484759 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:30.484711    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-233.ec2.internal\" not found"
Jul 28 17:22:30.530420 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.530393    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-233.ec2.internal"
Jul 28 17:22:30.535557 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.535437    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-233.ec2.internal"
Jul 28 17:22:30.585055 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:30.585033    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-233.ec2.internal\" not found"
Jul 28 17:22:30.685913 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:30.685875    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-233.ec2.internal\" not found"
Jul 28 17:22:30.787474 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:30.787384    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-233.ec2.internal\" not found"
Jul 28 17:22:30.887919 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:30.887889    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-233.ec2.internal\" not found"
Jul 28 17:22:30.946649 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.946610    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 17:22:30.947164 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:30.946812    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 17:22:30.988594 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:30.988563    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-233.ec2.internal\" not found"
Jul 28 17:22:31.018487 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:31.018450    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 17:17:30 +0000 UTC" deadline="2028-01-14 08:19:32.719138436 +0000 UTC"
Jul 28 17:22:31.018487 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:31.018483    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12830h57m1.70066002s"
Jul 28 17:22:31.088798 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:31.088730    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-233.ec2.internal\" not found"
Jul 28 17:22:31.117090 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:31.117059    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2d32026394497da64d771aa0e11975c.slice/crio-763776d3943ba3757a69a062f86ffd744529db66a3e8905b59cda68ee5d8579e WatchSource:0}: Error finding container 763776d3943ba3757a69a062f86ffd744529db66a3e8905b59cda68ee5d8579e: Status 404 returned error can't find the container with id 763776d3943ba3757a69a062f86ffd744529db66a3e8905b59cda68ee5d8579e
Jul 28 17:22:31.117592 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:31.117576    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f88513b366a558c56d55360985f15ee.slice/crio-6f075ba5dfcc9f25593df7aa9e79fa13a069a31fe5a7590d47d5b69e756d406f WatchSource:0}: Error finding container 6f075ba5dfcc9f25593df7aa9e79fa13a069a31fe5a7590d47d5b69e756d406f: Status 404 returned error can't find the container with id 6f075ba5dfcc9f25593df7aa9e79fa13a069a31fe5a7590d47d5b69e756d406f
Jul 28 17:22:31.121915 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:31.121895    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:22:31.182131 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:31.182102    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:22:31.189217 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:31.189193    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-233.ec2.internal\" not found"
Jul 28 17:22:31.295798 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:31.295769    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-233.ec2.internal\" not found"
Jul 28 17:22:31.395965 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:31.395898    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-233.ec2.internal\" not found"
Jul 28 17:22:31.496712 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:31.496674    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-233.ec2.internal\" not found"
Jul 28 17:22:31.543010 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:31.542986    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:22:31.597663 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:31.597628    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-233.ec2.internal"
Jul 28 17:22:31.602367 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:31.602342    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:22:31.615227 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:31.615193    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:22:31.616306 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:31.616284    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-233.ec2.internal"
Jul 28 17:22:31.624434 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:31.624411    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:22:31.980240 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:31.980202    2566 apiserver.go:52] "Watching apiserver"
Jul 28 17:22:31.989292 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:31.989264    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 17:22:31.989865 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:31.989840    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-vf9hs","kube-system/global-pull-secret-syncer-dpv42","kube-system/konnectivity-agent-xsptw","kube-system/kube-apiserver-proxy-ip-10-0-131-233.ec2.internal","openshift-dns/node-resolver-s7j5k","openshift-multus/multus-additional-cni-plugins-48d52","openshift-multus/network-metrics-daemon-znptx","openshift-network-diagnostics/network-check-target-rvrql","openshift-ovn-kubernetes/ovnkube-node-5sdnx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jv7m","openshift-cluster-node-tuning-operator/tuned-lqf4t","openshift-image-registry/node-ca-5bhch","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-233.ec2.internal","openshift-multus/multus-xnpc6"]
Jul 28 17:22:31.992809 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:31.992780    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jv7m"
Jul 28 17:22:31.995043 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:31.995020    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:31.995776 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:31.995753    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 17:22:31.995891 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:31.995780    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 17:22:31.995891 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:31.995781    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 17:22:31.995998 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:31.995925    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lgs4h\""
Jul 28 17:22:31.997388 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:31.997351    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xsptw"
Jul 28 17:22:31.997856 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:31.997839    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 17:22:31.997959 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:31.997883    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 17:22:31.998031 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:31.998007    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:22:31.998982 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:31.998211    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-55rpk\""
Jul 28 17:22:31.999876 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:31.999844    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 17:22:32.000139 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.000122    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 17:22:32.000489 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.000453    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.001010 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.000988    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-n7j44\""
Jul 28 17:22:32.003700 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.002869    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s7j5k"
Jul 28 17:22:32.003934 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.003908    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 17:22:32.003934 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.003918    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 17:22:32.004052 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.004035    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 17:22:32.004143 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.004036    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hldxt\""
Jul 28 17:22:32.004217 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.004178    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 17:22:32.005413 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.005382    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-48d52"
Jul 28 17:22:32.006862 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.006561    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 17:22:32.006862 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.006561    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 17:22:32.007030 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.006883    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cfbqm\""
Jul 28 17:22:32.008553 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.008480    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.011099 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.011073    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-host-run-multus-certs\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.011209 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.011116    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/330381c0-df2c-4d4b-8865-09d4488c075e-etc-tuned\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.011209 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.011144    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-host-run-netns\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.011209 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.011170    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-host-var-lib-kubelet\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.011209 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.011198    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/330381c0-df2c-4d4b-8865-09d4488c075e-etc-kubernetes\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.011454 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.011222    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/330381c0-df2c-4d4b-8865-09d4488c075e-etc-systemd\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.011454 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.011247    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-cnibin\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.011454 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.011282    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/795058ed-0974-4328-b462-32e755000a70-cni-binary-copy\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.011454 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.011308    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-multus-conf-dir\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.011454 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.011333    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94fr2\" (UniqueName: \"kubernetes.io/projected/795058ed-0974-4328-b462-32e755000a70-kube-api-access-94fr2\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.011454 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.011373    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/22ea97a4-e795-4d89-8257-fe0baf30b83e-etc-selinux\") pod \"aws-ebs-csi-driver-node-6jv7m\" (UID: \"22ea97a4-e795-4d89-8257-fe0baf30b83e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jv7m"
Jul 28 17:22:32.011454 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.011399    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/330381c0-df2c-4d4b-8865-09d4488c075e-etc-modprobe-d\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.011454 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.011423    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-hostroot\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.011454 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.011455    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/795058ed-0974-4328-b462-32e755000a70-multus-daemon-config\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.011882 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.011484    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/22ea97a4-e795-4d89-8257-fe0baf30b83e-registration-dir\") pod \"aws-ebs-csi-driver-node-6jv7m\" (UID: \"22ea97a4-e795-4d89-8257-fe0baf30b83e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jv7m"
Jul 28 17:22:32.011882 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.011509    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/330381c0-df2c-4d4b-8865-09d4488c075e-var-lib-kubelet\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.011882 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.011536    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-multus-cni-dir\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.011882 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.011564    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-os-release\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.011882 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.011592    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/330381c0-df2c-4d4b-8865-09d4488c075e-etc-sysctl-d\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.011882 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.011615    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/330381c0-df2c-4d4b-8865-09d4488c075e-tmp\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.011882 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.011639    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/51a23dc6-82ad-4ab6-a528-82ad3146e7c6-agent-certs\") pod \"konnectivity-agent-xsptw\" (UID: \"51a23dc6-82ad-4ab6-a528-82ad3146e7c6\") " pod="kube-system/konnectivity-agent-xsptw"
Jul 28 17:22:32.011882 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.011666    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-host-var-lib-cni-multus\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.011882 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.011691    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-etc-kubernetes\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.011882 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.011716    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/22ea97a4-e795-4d89-8257-fe0baf30b83e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6jv7m\" (UID: \"22ea97a4-e795-4d89-8257-fe0baf30b83e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jv7m"
Jul 28 17:22:32.011882 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.011742    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/22ea97a4-e795-4d89-8257-fe0baf30b83e-device-dir\") pod \"aws-ebs-csi-driver-node-6jv7m\" (UID: \"22ea97a4-e795-4d89-8257-fe0baf30b83e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jv7m"
Jul 28 17:22:32.011882 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.011771    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/330381c0-df2c-4d4b-8865-09d4488c075e-etc-sysconfig\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.011882 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.011797    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/330381c0-df2c-4d4b-8865-09d4488c075e-etc-sysctl-conf\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.011882 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.011819    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/330381c0-df2c-4d4b-8865-09d4488c075e-run\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.011882 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.011842    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/330381c0-df2c-4d4b-8865-09d4488c075e-sys\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.011882 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.011867    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmmbq\" (UniqueName: \"kubernetes.io/projected/330381c0-df2c-4d4b-8865-09d4488c075e-kube-api-access-rmmbq\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.012646 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.011914    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-system-cni-dir\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.012646 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.011939    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/22ea97a4-e795-4d89-8257-fe0baf30b83e-sys-fs\") pod \"aws-ebs-csi-driver-node-6jv7m\" (UID: \"22ea97a4-e795-4d89-8257-fe0baf30b83e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jv7m"
Jul 28 17:22:32.012646 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.011967    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdz4g\" (UniqueName: \"kubernetes.io/projected/22ea97a4-e795-4d89-8257-fe0baf30b83e-kube-api-access-rdz4g\") pod \"aws-ebs-csi-driver-node-6jv7m\" (UID: \"22ea97a4-e795-4d89-8257-fe0baf30b83e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jv7m"
Jul 28 17:22:32.012646 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.011992    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/330381c0-df2c-4d4b-8865-09d4488c075e-host\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.012646 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.012045    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-multus-socket-dir-parent\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.012646 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.012072    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/22ea97a4-e795-4d89-8257-fe0baf30b83e-socket-dir\") pod \"aws-ebs-csi-driver-node-6jv7m\" (UID: \"22ea97a4-e795-4d89-8257-fe0baf30b83e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jv7m"
Jul 28 17:22:32.012646 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.012099    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/330381c0-df2c-4d4b-8865-09d4488c075e-lib-modules\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.012646 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.012125    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/51a23dc6-82ad-4ab6-a528-82ad3146e7c6-konnectivity-ca\") pod \"konnectivity-agent-xsptw\" (UID: \"51a23dc6-82ad-4ab6-a528-82ad3146e7c6\") " pod="kube-system/konnectivity-agent-xsptw"
Jul 28 17:22:32.012646 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.012151    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-host-run-k8s-cni-cncf-io\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.012646 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.012177    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-host-var-lib-cni-bin\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.013241 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.012734    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvrql"
Jul 28 17:22:32.013241 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:32.012806    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvrql" podUID="a743f0d2-5033-4955-bbca-1e7327255603"
Jul 28 17:22:32.014226 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.014198    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-52j9s\""
Jul 28 17:22:32.014466 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.014395    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 17:22:32.014567 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.014482    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpv42"
Jul 28 17:22:32.014641 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:32.014571    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpv42" podUID="3b881ad9-ece5-40ab-b106-32e6d520c37e"
Jul 28 17:22:32.015362 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.014765    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 17:22:32.015362 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.015031    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 17:22:32.015362 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.015307    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 17:22:32.015975 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.015953    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hn7jw\""
Jul 28 17:22:32.016269 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.016248    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 17:22:32.016360 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.016278    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 17:22:32.016462 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.016352    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 17:22:32.018964 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.018646    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 17:22:32.019781 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.019276    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vf9hs"
Jul 28 17:22:32.021436 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.021416    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:22:32.022632 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.022612    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znptx"
Jul 28 17:22:32.022739 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.022642    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 17:22:32.022739 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:32.022694    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-znptx" podUID="dedc8b78-7469-4e8d-a7b8-b11c741186d7"
Jul 28 17:22:32.023077 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.022863    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:22:32.023077 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.022867    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rpt4q\""
Jul 28 17:22:32.023246 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.023151    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 17:22:32.025474 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.025449    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5bhch"
Jul 28 17:22:32.028173 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.028012    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 17:22:32.028634 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.028290    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7x44p\""
Jul 28 17:22:32.028634 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.028433    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 17:22:32.028634 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.028566    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 17:22:32.035069 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.035045    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cbr9b"
Jul 28 17:22:32.044220 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.044200    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cbr9b"
Jul 28 17:22:32.097777 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.097714    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-233.ec2.internal" event={"ID":"7f88513b366a558c56d55360985f15ee","Type":"ContainerStarted","Data":"6f075ba5dfcc9f25593df7aa9e79fa13a069a31fe5a7590d47d5b69e756d406f"}
Jul 28 17:22:32.098896 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.098877    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 17:22:32.099047 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.099027    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-233.ec2.internal" event={"ID":"a2d32026394497da64d771aa0e11975c","Type":"ContainerStarted","Data":"763776d3943ba3757a69a062f86ffd744529db66a3e8905b59cda68ee5d8579e"}
Jul 28 17:22:32.113388 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.113362    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-var-lib-openvswitch\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.113501 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.113403    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-host-run-ovn-kubernetes\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.113501 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.113431    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/630e1b18-a044-454e-9be4-359de1817e2b-host\") pod \"node-ca-5bhch\" (UID: \"630e1b18-a044-454e-9be4-359de1817e2b\") " pod="openshift-image-registry/node-ca-5bhch"
Jul 28 17:22:32.113501 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.113452    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sxr7\" (UniqueName: \"kubernetes.io/projected/3acbd8e1-dd56-4d3f-9cec-54aed9951ef7-kube-api-access-9sxr7\") pod \"iptables-alerter-vf9hs\" (UID: \"3acbd8e1-dd56-4d3f-9cec-54aed9951ef7\") " pod="openshift-network-operator/iptables-alerter-vf9hs"
Jul 28 17:22:32.113501 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.113475    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/330381c0-df2c-4d4b-8865-09d4488c075e-etc-systemd\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.113501 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.113494    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-cnibin\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.113767 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.113515    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/795058ed-0974-4328-b462-32e755000a70-cni-binary-copy\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.113767 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.113552    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-multus-conf-dir\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.113767 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.113588    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94fr2\" (UniqueName: \"kubernetes.io/projected/795058ed-0974-4328-b462-32e755000a70-kube-api-access-94fr2\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.113767 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.113615    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-host-slash\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.113767 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.113650    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-cnibin\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.113767 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.113650    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/330381c0-df2c-4d4b-8865-09d4488c075e-etc-systemd\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.113767 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.113654    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-hostroot\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.113767 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.113710    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-hostroot\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.113767 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.113713    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-multus-conf-dir\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.113767 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.113728    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3acbd8e1-dd56-4d3f-9cec-54aed9951ef7-iptables-alerter-script\") pod \"iptables-alerter-vf9hs\" (UID: \"3acbd8e1-dd56-4d3f-9cec-54aed9951ef7\") " pod="openshift-network-operator/iptables-alerter-vf9hs"
Jul 28 17:22:32.114342 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.113775    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-run-systemd\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.114342 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.113803    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-host-cni-bin\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.114342 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.113846    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/330381c0-df2c-4d4b-8865-09d4488c075e-var-lib-kubelet\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.114342 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.113872    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-os-release\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.114342 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.113889    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/330381c0-df2c-4d4b-8865-09d4488c075e-var-lib-kubelet\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.114342 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.113897    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/15fc65f5-e122-489d-b56c-1999d101223f-os-release\") pod \"multus-additional-cni-plugins-48d52\" (UID: \"15fc65f5-e122-489d-b56c-1999d101223f\") " pod="openshift-multus/multus-additional-cni-plugins-48d52"
Jul 28 17:22:32.114342 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.113924    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15fc65f5-e122-489d-b56c-1999d101223f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-48d52\" (UID: \"15fc65f5-e122-489d-b56c-1999d101223f\") " pod="openshift-multus/multus-additional-cni-plugins-48d52"
Jul 28 17:22:32.114342 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.113936    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-os-release\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.114342 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.113965    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-etc-openvswitch\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.114342 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114007    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-env-overrides\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.114342 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114031    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-ovnkube-script-lib\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.114342 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114057    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/630e1b18-a044-454e-9be4-359de1817e2b-serviceca\") pod \"node-ca-5bhch\" (UID: \"630e1b18-a044-454e-9be4-359de1817e2b\") " pod="openshift-image-registry/node-ca-5bhch"
Jul 28 17:22:32.114342 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114110    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/330381c0-df2c-4d4b-8865-09d4488c075e-etc-sysctl-d\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.114342 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114159    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/51a23dc6-82ad-4ab6-a528-82ad3146e7c6-agent-certs\") pod \"konnectivity-agent-xsptw\" (UID: \"51a23dc6-82ad-4ab6-a528-82ad3146e7c6\") " pod="kube-system/konnectivity-agent-xsptw"
Jul 28 17:22:32.114342 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114186    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-host-var-lib-cni-multus\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.114342 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114207    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/22ea97a4-e795-4d89-8257-fe0baf30b83e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6jv7m\" (UID: \"22ea97a4-e795-4d89-8257-fe0baf30b83e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jv7m"
Jul 28 17:22:32.114342 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114232    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j94fw\" (UniqueName: \"kubernetes.io/projected/15fc65f5-e122-489d-b56c-1999d101223f-kube-api-access-j94fw\") pod \"multus-additional-cni-plugins-48d52\" (UID: \"15fc65f5-e122-489d-b56c-1999d101223f\") " pod="openshift-multus/multus-additional-cni-plugins-48d52"
Jul 28 17:22:32.115150 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114260    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b881ad9-ece5-40ab-b106-32e6d520c37e-original-pull-secret\") pod \"global-pull-secret-syncer-dpv42\" (UID: \"3b881ad9-ece5-40ab-b106-32e6d520c37e\") " pod="kube-system/global-pull-secret-syncer-dpv42"
Jul 28 17:22:32.115150 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114268    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/330381c0-df2c-4d4b-8865-09d4488c075e-etc-sysctl-d\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.115150 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114284    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-host-var-lib-cni-multus\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.115150 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114289    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/330381c0-df2c-4d4b-8865-09d4488c075e-etc-sysconfig\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.115150 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114283    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/795058ed-0974-4328-b462-32e755000a70-cni-binary-copy\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.115150 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114352    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/330381c0-df2c-4d4b-8865-09d4488c075e-etc-sysctl-conf\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.115150 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114357    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/22ea97a4-e795-4d89-8257-fe0baf30b83e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6jv7m\" (UID: \"22ea97a4-e795-4d89-8257-fe0baf30b83e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jv7m"
Jul 28 17:22:32.115150 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114407    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/330381c0-df2c-4d4b-8865-09d4488c075e-run\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.115150 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114415    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/330381c0-df2c-4d4b-8865-09d4488c075e-etc-sysconfig\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.115150 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114434    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/330381c0-df2c-4d4b-8865-09d4488c075e-sys\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.115150 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114460    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdz4g\" (UniqueName: \"kubernetes.io/projected/22ea97a4-e795-4d89-8257-fe0baf30b83e-kube-api-access-rdz4g\") pod \"aws-ebs-csi-driver-node-6jv7m\" (UID: \"22ea97a4-e795-4d89-8257-fe0baf30b83e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jv7m"
Jul 28 17:22:32.115150 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114475    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/330381c0-df2c-4d4b-8865-09d4488c075e-run\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.115150 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114482    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/330381c0-df2c-4d4b-8865-09d4488c075e-etc-sysctl-conf\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.115150 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114488    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfwf2\" (UniqueName: \"kubernetes.io/projected/a743f0d2-5033-4955-bbca-1e7327255603-kube-api-access-lfwf2\") pod \"network-check-target-rvrql\" (UID: \"a743f0d2-5033-4955-bbca-1e7327255603\") " pod="openshift-network-diagnostics/network-check-target-rvrql"
Jul 28 17:22:32.115150 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114523    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/330381c0-df2c-4d4b-8865-09d4488c075e-sys\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.115150 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114556    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-host-run-netns\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.115150 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114587    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-ovnkube-config\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.115988 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114613    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-ovn-node-metrics-cert\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.115988 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114629    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 17:22:32.115988 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114642    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8489\" (UniqueName: \"kubernetes.io/projected/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-kube-api-access-w8489\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.115988 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114714    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/51a23dc6-82ad-4ab6-a528-82ad3146e7c6-konnectivity-ca\") pod \"konnectivity-agent-xsptw\" (UID: \"51a23dc6-82ad-4ab6-a528-82ad3146e7c6\") " pod="kube-system/konnectivity-agent-xsptw"
Jul 28 17:22:32.115988 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114732    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-host-var-lib-cni-bin\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.115988 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114748    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/330381c0-df2c-4d4b-8865-09d4488c075e-etc-tuned\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.115988 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114765    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-host-run-netns\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.115988 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114798    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-host-var-lib-kubelet\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.115988 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114817    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-host-cni-netd\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.115988 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114826    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-host-var-lib-cni-bin\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.115988 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114852    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-host-var-lib-kubelet\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.115988 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114874    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-host-run-netns\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.115988 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114883    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3b881ad9-ece5-40ab-b106-32e6d520c37e-kubelet-config\") pod \"global-pull-secret-syncer-dpv42\" (UID: \"3b881ad9-ece5-40ab-b106-32e6d520c37e\") " pod="kube-system/global-pull-secret-syncer-dpv42"
Jul 28 17:22:32.115988 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114928    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/330381c0-df2c-4d4b-8865-09d4488c075e-etc-kubernetes\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.115988 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114956    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/22ea97a4-e795-4d89-8257-fe0baf30b83e-etc-selinux\") pod \"aws-ebs-csi-driver-node-6jv7m\" (UID: \"22ea97a4-e795-4d89-8257-fe0baf30b83e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jv7m"
Jul 28 17:22:32.115988 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.114985    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/091666fc-49ce-46bc-9a7d-5815676ea342-hosts-file\") pod \"node-resolver-s7j5k\" (UID: \"091666fc-49ce-46bc-9a7d-5815676ea342\") " pod="openshift-dns/node-resolver-s7j5k"
Jul 28 17:22:32.115988 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.115013    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/15fc65f5-e122-489d-b56c-1999d101223f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-48d52\" (UID: \"15fc65f5-e122-489d-b56c-1999d101223f\") " pod="openshift-multus/multus-additional-cni-plugins-48d52"
Jul 28 17:22:32.116911 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.115019    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/330381c0-df2c-4d4b-8865-09d4488c075e-etc-kubernetes\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.116911 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.115035    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3acbd8e1-dd56-4d3f-9cec-54aed9951ef7-host-slash\") pod \"iptables-alerter-vf9hs\" (UID: \"3acbd8e1-dd56-4d3f-9cec-54aed9951ef7\") " pod="openshift-network-operator/iptables-alerter-vf9hs"
Jul 28 17:22:32.116911 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.115061    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-node-log\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.116911 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.115086    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/330381c0-df2c-4d4b-8865-09d4488c075e-etc-modprobe-d\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.116911 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.115118    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/795058ed-0974-4328-b462-32e755000a70-multus-daemon-config\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.116911 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.115158    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/22ea97a4-e795-4d89-8257-fe0baf30b83e-etc-selinux\") pod \"aws-ebs-csi-driver-node-6jv7m\" (UID: \"22ea97a4-e795-4d89-8257-fe0baf30b83e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jv7m"
Jul 28 17:22:32.116911 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.115198    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/22ea97a4-e795-4d89-8257-fe0baf30b83e-registration-dir\") pod \"aws-ebs-csi-driver-node-6jv7m\" (UID: \"22ea97a4-e795-4d89-8257-fe0baf30b83e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jv7m"
Jul 28 17:22:32.116911 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.115206    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/330381c0-df2c-4d4b-8865-09d4488c075e-etc-modprobe-d\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.116911 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.115236    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dedc8b78-7469-4e8d-a7b8-b11c741186d7-metrics-certs\") pod \"network-metrics-daemon-znptx\" (UID: \"dedc8b78-7469-4e8d-a7b8-b11c741186d7\") " pod="openshift-multus/network-metrics-daemon-znptx"
Jul 28 17:22:32.116911 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.115285    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/22ea97a4-e795-4d89-8257-fe0baf30b83e-registration-dir\") pod \"aws-ebs-csi-driver-node-6jv7m\" (UID: \"22ea97a4-e795-4d89-8257-fe0baf30b83e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jv7m"
Jul 28 17:22:32.116911 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.115351    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-multus-cni-dir\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.116911 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.115403    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/091666fc-49ce-46bc-9a7d-5815676ea342-tmp-dir\") pod \"node-resolver-s7j5k\" (UID: \"091666fc-49ce-46bc-9a7d-5815676ea342\") " pod="openshift-dns/node-resolver-s7j5k"
Jul 28 17:22:32.116911 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.115448    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/51a23dc6-82ad-4ab6-a528-82ad3146e7c6-konnectivity-ca\") pod \"konnectivity-agent-xsptw\" (UID: \"51a23dc6-82ad-4ab6-a528-82ad3146e7c6\") " pod="kube-system/konnectivity-agent-xsptw"
Jul 28 17:22:32.116911 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.115451    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z656q\" (UniqueName: \"kubernetes.io/projected/dedc8b78-7469-4e8d-a7b8-b11c741186d7-kube-api-access-z656q\") pod \"network-metrics-daemon-znptx\" (UID: \"dedc8b78-7469-4e8d-a7b8-b11c741186d7\") " pod="openshift-multus/network-metrics-daemon-znptx"
Jul 28 17:22:32.116911 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.115512    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/22ea97a4-e795-4d89-8257-fe0baf30b83e-sys-fs\") pod \"aws-ebs-csi-driver-node-6jv7m\" (UID: \"22ea97a4-e795-4d89-8257-fe0baf30b83e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jv7m"
Jul 28 17:22:32.116911 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.115563    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ltgj\" (UniqueName: \"kubernetes.io/projected/630e1b18-a044-454e-9be4-359de1817e2b-kube-api-access-8ltgj\") pod \"node-ca-5bhch\" (UID: \"630e1b18-a044-454e-9be4-359de1817e2b\") " pod="openshift-image-registry/node-ca-5bhch"
Jul 28 17:22:32.116911 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.115582    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-multus-cni-dir\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.117794 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.115603    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/22ea97a4-e795-4d89-8257-fe0baf30b83e-sys-fs\") pod \"aws-ebs-csi-driver-node-6jv7m\" (UID: \"22ea97a4-e795-4d89-8257-fe0baf30b83e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jv7m"
Jul 28 17:22:32.117794 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.115607    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/330381c0-df2c-4d4b-8865-09d4488c075e-tmp\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.117794 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.115649    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-etc-kubernetes\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.117794 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.115675    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/22ea97a4-e795-4d89-8257-fe0baf30b83e-device-dir\") pod \"aws-ebs-csi-driver-node-6jv7m\" (UID: \"22ea97a4-e795-4d89-8257-fe0baf30b83e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jv7m"
Jul 28 17:22:32.117794 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.115702    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-host-kubelet\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.117794 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.115717    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-etc-kubernetes\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.117794 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.115731    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-run-openvswitch\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.117794 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.115759    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/22ea97a4-e795-4d89-8257-fe0baf30b83e-device-dir\") pod \"aws-ebs-csi-driver-node-6jv7m\" (UID: \"22ea97a4-e795-4d89-8257-fe0baf30b83e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jv7m"
Jul 28 17:22:32.117794 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.115758    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmmbq\" (UniqueName: \"kubernetes.io/projected/330381c0-df2c-4d4b-8865-09d4488c075e-kube-api-access-rmmbq\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.117794 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.115808    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-system-cni-dir\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.117794 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.115828    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/15fc65f5-e122-489d-b56c-1999d101223f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-48d52\" (UID: \"15fc65f5-e122-489d-b56c-1999d101223f\") " pod="openshift-multus/multus-additional-cni-plugins-48d52"
Jul 28 17:22:32.117794 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.115861    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-systemd-units\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.117794 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.115912    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-system-cni-dir\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.117794 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.115916    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-log-socket\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.117794 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.115951    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3b881ad9-ece5-40ab-b106-32e6d520c37e-dbus\") pod \"global-pull-secret-syncer-dpv42\" (UID: \"3b881ad9-ece5-40ab-b106-32e6d520c37e\") " pod="kube-system/global-pull-secret-syncer-dpv42"
Jul 28 17:22:32.117794 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.115987    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/330381c0-df2c-4d4b-8865-09d4488c075e-host\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.117794 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.116015    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-multus-socket-dir-parent\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.118664 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.116053    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/22ea97a4-e795-4d89-8257-fe0baf30b83e-socket-dir\") pod \"aws-ebs-csi-driver-node-6jv7m\" (UID: \"22ea97a4-e795-4d89-8257-fe0baf30b83e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jv7m"
Jul 28 17:22:32.118664 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.116078    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15fc65f5-e122-489d-b56c-1999d101223f-cnibin\") pod \"multus-additional-cni-plugins-48d52\" (UID: \"15fc65f5-e122-489d-b56c-1999d101223f\") " pod="openshift-multus/multus-additional-cni-plugins-48d52"
Jul 28 17:22:32.118664 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.116081    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/795058ed-0974-4328-b462-32e755000a70-multus-daemon-config\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.118664 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.116077    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/330381c0-df2c-4d4b-8865-09d4488c075e-host\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.118664 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.116106    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15fc65f5-e122-489d-b56c-1999d101223f-cni-binary-copy\") pod \"multus-additional-cni-plugins-48d52\" (UID: \"15fc65f5-e122-489d-b56c-1999d101223f\") " pod="openshift-multus/multus-additional-cni-plugins-48d52"
Jul 28 17:22:32.118664 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.116113    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-multus-socket-dir-parent\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.118664 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.116137    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-run-ovn\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.118664 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.116183    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.118664 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.116190    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/22ea97a4-e795-4d89-8257-fe0baf30b83e-socket-dir\") pod \"aws-ebs-csi-driver-node-6jv7m\" (UID: \"22ea97a4-e795-4d89-8257-fe0baf30b83e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jv7m"
Jul 28 17:22:32.118664 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.116217    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/330381c0-df2c-4d4b-8865-09d4488c075e-lib-modules\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.118664 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.116243    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-host-run-k8s-cni-cncf-io\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.118664 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.116279    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-host-run-multus-certs\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.118664 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.116308    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjtkc\" (UniqueName: \"kubernetes.io/projected/091666fc-49ce-46bc-9a7d-5815676ea342-kube-api-access-zjtkc\") pod \"node-resolver-s7j5k\" (UID: \"091666fc-49ce-46bc-9a7d-5815676ea342\") " pod="openshift-dns/node-resolver-s7j5k"
Jul 28 17:22:32.118664 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.116333    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15fc65f5-e122-489d-b56c-1999d101223f-system-cni-dir\") pod \"multus-additional-cni-plugins-48d52\" (UID: \"15fc65f5-e122-489d-b56c-1999d101223f\") " pod="openshift-multus/multus-additional-cni-plugins-48d52"
Jul 28 17:22:32.118664 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.116375    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-host-run-multus-certs\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.118664 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.116413    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/330381c0-df2c-4d4b-8865-09d4488c075e-lib-modules\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.118664 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.116471    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/795058ed-0974-4328-b462-32e755000a70-host-run-k8s-cni-cncf-io\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.119218 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.118570    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/330381c0-df2c-4d4b-8865-09d4488c075e-etc-tuned\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.119218 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.118610    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/330381c0-df2c-4d4b-8865-09d4488c075e-tmp\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.119218 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.118886    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/51a23dc6-82ad-4ab6-a528-82ad3146e7c6-agent-certs\") pod \"konnectivity-agent-xsptw\" (UID: \"51a23dc6-82ad-4ab6-a528-82ad3146e7c6\") " pod="kube-system/konnectivity-agent-xsptw"
Jul 28 17:22:32.123877 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.123856    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94fr2\" (UniqueName: \"kubernetes.io/projected/795058ed-0974-4328-b462-32e755000a70-kube-api-access-94fr2\") pod \"multus-xnpc6\" (UID: \"795058ed-0974-4328-b462-32e755000a70\") " pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.124210 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.124186    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdz4g\" (UniqueName: \"kubernetes.io/projected/22ea97a4-e795-4d89-8257-fe0baf30b83e-kube-api-access-rdz4g\") pod \"aws-ebs-csi-driver-node-6jv7m\" (UID: \"22ea97a4-e795-4d89-8257-fe0baf30b83e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jv7m"
Jul 28 17:22:32.124210 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.124186    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmmbq\" (UniqueName: \"kubernetes.io/projected/330381c0-df2c-4d4b-8865-09d4488c075e-kube-api-access-rmmbq\") pod \"tuned-lqf4t\" (UID: \"330381c0-df2c-4d4b-8865-09d4488c075e\") " pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.217520 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.217480    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-host-slash\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.217520 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.217521    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3acbd8e1-dd56-4d3f-9cec-54aed9951ef7-iptables-alerter-script\") pod \"iptables-alerter-vf9hs\" (UID: \"3acbd8e1-dd56-4d3f-9cec-54aed9951ef7\") " pod="openshift-network-operator/iptables-alerter-vf9hs"
Jul 28 17:22:32.217726 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.217540    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-run-systemd\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.217726 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.217560    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-host-cni-bin\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.217726 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.217560    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-host-slash\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.217726 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.217633    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-run-systemd\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.217726 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.217644    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-host-cni-bin\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.217726 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.217692    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/15fc65f5-e122-489d-b56c-1999d101223f-os-release\") pod \"multus-additional-cni-plugins-48d52\" (UID: \"15fc65f5-e122-489d-b56c-1999d101223f\") " pod="openshift-multus/multus-additional-cni-plugins-48d52"
Jul 28 17:22:32.218028 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.217734    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15fc65f5-e122-489d-b56c-1999d101223f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-48d52\" (UID: \"15fc65f5-e122-489d-b56c-1999d101223f\") " pod="openshift-multus/multus-additional-cni-plugins-48d52"
Jul 28 17:22:32.218028 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.217764    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-etc-openvswitch\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.218028 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.217736    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/15fc65f5-e122-489d-b56c-1999d101223f-os-release\") pod \"multus-additional-cni-plugins-48d52\" (UID: \"15fc65f5-e122-489d-b56c-1999d101223f\") " pod="openshift-multus/multus-additional-cni-plugins-48d52"
Jul 28 17:22:32.218028 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.217789    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-env-overrides\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.218028 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.217841    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-ovnkube-script-lib\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.218028 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.217854    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-etc-openvswitch\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.218028 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.217879    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/630e1b18-a044-454e-9be4-359de1817e2b-serviceca\") pod \"node-ca-5bhch\" (UID: \"630e1b18-a044-454e-9be4-359de1817e2b\") " pod="openshift-image-registry/node-ca-5bhch"
Jul 28 17:22:32.218028 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.217911    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j94fw\" (UniqueName: \"kubernetes.io/projected/15fc65f5-e122-489d-b56c-1999d101223f-kube-api-access-j94fw\") pod \"multus-additional-cni-plugins-48d52\" (UID: \"15fc65f5-e122-489d-b56c-1999d101223f\") " pod="openshift-multus/multus-additional-cni-plugins-48d52"
Jul 28 17:22:32.218028 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.217919    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15fc65f5-e122-489d-b56c-1999d101223f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-48d52\" (UID: \"15fc65f5-e122-489d-b56c-1999d101223f\") " pod="openshift-multus/multus-additional-cni-plugins-48d52"
Jul 28 17:22:32.218028 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.217941    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b881ad9-ece5-40ab-b106-32e6d520c37e-original-pull-secret\") pod \"global-pull-secret-syncer-dpv42\" (UID: \"3b881ad9-ece5-40ab-b106-32e6d520c37e\") " pod="kube-system/global-pull-secret-syncer-dpv42"
Jul 28 17:22:32.218028 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.217973    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfwf2\" (UniqueName: \"kubernetes.io/projected/a743f0d2-5033-4955-bbca-1e7327255603-kube-api-access-lfwf2\") pod \"network-check-target-rvrql\" (UID: \"a743f0d2-5033-4955-bbca-1e7327255603\") " pod="openshift-network-diagnostics/network-check-target-rvrql"
Jul 28 17:22:32.218028 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.217999    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-host-run-netns\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.218667 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:32.218099    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:22:32.218667 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.218162    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3acbd8e1-dd56-4d3f-9cec-54aed9951ef7-iptables-alerter-script\") pod \"iptables-alerter-vf9hs\" (UID: \"3acbd8e1-dd56-4d3f-9cec-54aed9951ef7\") " pod="openshift-network-operator/iptables-alerter-vf9hs"
Jul 28 17:22:32.218667 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:32.218174    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b881ad9-ece5-40ab-b106-32e6d520c37e-original-pull-secret podName:3b881ad9-ece5-40ab-b106-32e6d520c37e nodeName:}" failed. No retries permitted until 2026-07-28 17:22:32.718150557 +0000 UTC m=+3.028316060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b881ad9-ece5-40ab-b106-32e6d520c37e-original-pull-secret") pod "global-pull-secret-syncer-dpv42" (UID: "3b881ad9-ece5-40ab-b106-32e6d520c37e") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:22:32.218667 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.218217    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-ovnkube-config\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.218667 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.218246    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-ovn-node-metrics-cert\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.218667 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.218266    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-host-run-netns\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.218667 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.218271    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8489\" (UniqueName: \"kubernetes.io/projected/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-kube-api-access-w8489\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.218667 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.218284    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-env-overrides\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.218667 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.218432    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/630e1b18-a044-454e-9be4-359de1817e2b-serviceca\") pod \"node-ca-5bhch\" (UID: \"630e1b18-a044-454e-9be4-359de1817e2b\") " pod="openshift-image-registry/node-ca-5bhch"
Jul 28 17:22:32.218667 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.218444    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-host-cni-netd\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.218667 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.218519    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-host-cni-netd\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.218667 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.218525    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3b881ad9-ece5-40ab-b106-32e6d520c37e-kubelet-config\") pod \"global-pull-secret-syncer-dpv42\" (UID: \"3b881ad9-ece5-40ab-b106-32e6d520c37e\") " pod="kube-system/global-pull-secret-syncer-dpv42"
Jul 28 17:22:32.218667 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.218583    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/091666fc-49ce-46bc-9a7d-5815676ea342-hosts-file\") pod \"node-resolver-s7j5k\" (UID: \"091666fc-49ce-46bc-9a7d-5815676ea342\") " pod="openshift-dns/node-resolver-s7j5k"
Jul 28 17:22:32.218667 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.218587    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3b881ad9-ece5-40ab-b106-32e6d520c37e-kubelet-config\") pod \"global-pull-secret-syncer-dpv42\" (UID: \"3b881ad9-ece5-40ab-b106-32e6d520c37e\") " pod="kube-system/global-pull-secret-syncer-dpv42"
Jul 28 17:22:32.218667 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.218628    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/15fc65f5-e122-489d-b56c-1999d101223f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-48d52\" (UID: \"15fc65f5-e122-489d-b56c-1999d101223f\") " pod="openshift-multus/multus-additional-cni-plugins-48d52"
Jul 28 17:22:32.218667 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.218642    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/091666fc-49ce-46bc-9a7d-5815676ea342-hosts-file\") pod \"node-resolver-s7j5k\" (UID: \"091666fc-49ce-46bc-9a7d-5815676ea342\") " pod="openshift-dns/node-resolver-s7j5k"
Jul 28 17:22:32.218667 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.218660    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3acbd8e1-dd56-4d3f-9cec-54aed9951ef7-host-slash\") pod \"iptables-alerter-vf9hs\" (UID: \"3acbd8e1-dd56-4d3f-9cec-54aed9951ef7\") " pod="openshift-network-operator/iptables-alerter-vf9hs"
Jul 28 17:22:32.219544 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.218687    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-node-log\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.219544 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.218718    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dedc8b78-7469-4e8d-a7b8-b11c741186d7-metrics-certs\") pod \"network-metrics-daemon-znptx\" (UID: \"dedc8b78-7469-4e8d-a7b8-b11c741186d7\") " pod="openshift-multus/network-metrics-daemon-znptx"
Jul 28 17:22:32.219544 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.218745    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/091666fc-49ce-46bc-9a7d-5815676ea342-tmp-dir\") pod \"node-resolver-s7j5k\" (UID: \"091666fc-49ce-46bc-9a7d-5815676ea342\") " pod="openshift-dns/node-resolver-s7j5k"
Jul 28 17:22:32.219544 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.218747    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-ovnkube-script-lib\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.219544 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.218772    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z656q\" (UniqueName: \"kubernetes.io/projected/dedc8b78-7469-4e8d-a7b8-b11c741186d7-kube-api-access-z656q\") pod \"network-metrics-daemon-znptx\" (UID: \"dedc8b78-7469-4e8d-a7b8-b11c741186d7\") " pod="openshift-multus/network-metrics-daemon-znptx"
Jul 28 17:22:32.219544 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.218786    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-node-log\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.219544 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.218791    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-ovnkube-config\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.219544 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.218801    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ltgj\" (UniqueName: \"kubernetes.io/projected/630e1b18-a044-454e-9be4-359de1817e2b-kube-api-access-8ltgj\") pod \"node-ca-5bhch\" (UID: \"630e1b18-a044-454e-9be4-359de1817e2b\") " pod="openshift-image-registry/node-ca-5bhch"
Jul 28 17:22:32.219544 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.218815    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3acbd8e1-dd56-4d3f-9cec-54aed9951ef7-host-slash\") pod \"iptables-alerter-vf9hs\" (UID: \"3acbd8e1-dd56-4d3f-9cec-54aed9951ef7\") " pod="openshift-network-operator/iptables-alerter-vf9hs"
Jul 28 17:22:32.219544 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.218840    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-host-kubelet\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.219544 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.218866    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-run-openvswitch\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.219544 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.218895    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/15fc65f5-e122-489d-b56c-1999d101223f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-48d52\" (UID: \"15fc65f5-e122-489d-b56c-1999d101223f\") " pod="openshift-multus/multus-additional-cni-plugins-48d52"
Jul 28 17:22:32.219544 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:32.218902    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:22:32.219544 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.218922    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-systemd-units\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.219544 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.218945    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-log-socket\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.219544 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:32.218966    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dedc8b78-7469-4e8d-a7b8-b11c741186d7-metrics-certs podName:dedc8b78-7469-4e8d-a7b8-b11c741186d7 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:32.718946345 +0000 UTC m=+3.029111844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dedc8b78-7469-4e8d-a7b8-b11c741186d7-metrics-certs") pod "network-metrics-daemon-znptx" (UID: "dedc8b78-7469-4e8d-a7b8-b11c741186d7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:22:32.219544 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.218988    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-log-socket\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.220189 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.218998    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3b881ad9-ece5-40ab-b106-32e6d520c37e-dbus\") pod \"global-pull-secret-syncer-dpv42\" (UID: \"3b881ad9-ece5-40ab-b106-32e6d520c37e\") " pod="kube-system/global-pull-secret-syncer-dpv42"
Jul 28 17:22:32.220189 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.219044    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-systemd-units\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.220189 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.219068    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/091666fc-49ce-46bc-9a7d-5815676ea342-tmp-dir\") pod \"node-resolver-s7j5k\" (UID: \"091666fc-49ce-46bc-9a7d-5815676ea342\") " pod="openshift-dns/node-resolver-s7j5k"
Jul 28 17:22:32.220189 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.219084    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-host-kubelet\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.220189 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.219084    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3b881ad9-ece5-40ab-b106-32e6d520c37e-dbus\") pod \"global-pull-secret-syncer-dpv42\" (UID: \"3b881ad9-ece5-40ab-b106-32e6d520c37e\") " pod="kube-system/global-pull-secret-syncer-dpv42"
Jul 28 17:22:32.220189 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.219119    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-run-openvswitch\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.220189 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.219151    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15fc65f5-e122-489d-b56c-1999d101223f-cnibin\") pod \"multus-additional-cni-plugins-48d52\" (UID: \"15fc65f5-e122-489d-b56c-1999d101223f\") " pod="openshift-multus/multus-additional-cni-plugins-48d52"
Jul 28 17:22:32.220189 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.219188    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15fc65f5-e122-489d-b56c-1999d101223f-cni-binary-copy\") pod \"multus-additional-cni-plugins-48d52\" (UID: \"15fc65f5-e122-489d-b56c-1999d101223f\") " pod="openshift-multus/multus-additional-cni-plugins-48d52"
Jul 28 17:22:32.220189 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.219215    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-run-ovn\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.220189 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.219216    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15fc65f5-e122-489d-b56c-1999d101223f-cnibin\") pod \"multus-additional-cni-plugins-48d52\" (UID: \"15fc65f5-e122-489d-b56c-1999d101223f\") " pod="openshift-multus/multus-additional-cni-plugins-48d52"
Jul 28 17:22:32.220189 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.219249    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-run-ovn\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.220189 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.219249    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.220189 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.219284    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.220189 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.219301    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjtkc\" (UniqueName: \"kubernetes.io/projected/091666fc-49ce-46bc-9a7d-5815676ea342-kube-api-access-zjtkc\") pod \"node-resolver-s7j5k\" (UID: \"091666fc-49ce-46bc-9a7d-5815676ea342\") " pod="openshift-dns/node-resolver-s7j5k"
Jul 28 17:22:32.220189 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.219350    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15fc65f5-e122-489d-b56c-1999d101223f-system-cni-dir\") pod \"multus-additional-cni-plugins-48d52\" (UID: \"15fc65f5-e122-489d-b56c-1999d101223f\") " pod="openshift-multus/multus-additional-cni-plugins-48d52"
Jul 28 17:22:32.220189 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.219379    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-var-lib-openvswitch\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.220189 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.219404    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-host-run-ovn-kubernetes\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.220841 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.219409    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15fc65f5-e122-489d-b56c-1999d101223f-system-cni-dir\") pod \"multus-additional-cni-plugins-48d52\" (UID: \"15fc65f5-e122-489d-b56c-1999d101223f\") " pod="openshift-multus/multus-additional-cni-plugins-48d52"
Jul 28 17:22:32.220841 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.219433    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/630e1b18-a044-454e-9be4-359de1817e2b-host\") pod \"node-ca-5bhch\" (UID: \"630e1b18-a044-454e-9be4-359de1817e2b\") " pod="openshift-image-registry/node-ca-5bhch"
Jul 28 17:22:32.220841 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.219457    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-var-lib-openvswitch\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.220841 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.219461    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sxr7\" (UniqueName: \"kubernetes.io/projected/3acbd8e1-dd56-4d3f-9cec-54aed9951ef7-kube-api-access-9sxr7\") pod \"iptables-alerter-vf9hs\" (UID: \"3acbd8e1-dd56-4d3f-9cec-54aed9951ef7\") " pod="openshift-network-operator/iptables-alerter-vf9hs"
Jul 28 17:22:32.220841 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.219499    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-host-run-ovn-kubernetes\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.220841 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.219500    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/15fc65f5-e122-489d-b56c-1999d101223f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-48d52\" (UID: \"15fc65f5-e122-489d-b56c-1999d101223f\") " pod="openshift-multus/multus-additional-cni-plugins-48d52"
Jul 28 17:22:32.220841 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.219531    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/630e1b18-a044-454e-9be4-359de1817e2b-host\") pod \"node-ca-5bhch\" (UID: \"630e1b18-a044-454e-9be4-359de1817e2b\") " pod="openshift-image-registry/node-ca-5bhch"
Jul 28 17:22:32.220841 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.219774    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/15fc65f5-e122-489d-b56c-1999d101223f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-48d52\" (UID: \"15fc65f5-e122-489d-b56c-1999d101223f\") " pod="openshift-multus/multus-additional-cni-plugins-48d52"
Jul 28 17:22:32.220841 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.219830    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15fc65f5-e122-489d-b56c-1999d101223f-cni-binary-copy\") pod \"multus-additional-cni-plugins-48d52\" (UID: \"15fc65f5-e122-489d-b56c-1999d101223f\") " pod="openshift-multus/multus-additional-cni-plugins-48d52"
Jul 28 17:22:32.221161 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.221143    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-ovn-node-metrics-cert\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.227421 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:32.227398    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:22:32.227421 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:32.227426    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:22:32.227566 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:32.227437    2566 projected.go:196] Error preparing data for projected volume kube-api-access-lfwf2 for pod openshift-network-diagnostics/network-check-target-rvrql: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:22:32.227566 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:32.227492    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a743f0d2-5033-4955-bbca-1e7327255603-kube-api-access-lfwf2 podName:a743f0d2-5033-4955-bbca-1e7327255603 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:32.727480743 +0000 UTC m=+3.037646231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lfwf2" (UniqueName: "kubernetes.io/projected/a743f0d2-5033-4955-bbca-1e7327255603-kube-api-access-lfwf2") pod "network-check-target-rvrql" (UID: "a743f0d2-5033-4955-bbca-1e7327255603") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:22:32.230489 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.230422    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j94fw\" (UniqueName: \"kubernetes.io/projected/15fc65f5-e122-489d-b56c-1999d101223f-kube-api-access-j94fw\") pod \"multus-additional-cni-plugins-48d52\" (UID: \"15fc65f5-e122-489d-b56c-1999d101223f\") " pod="openshift-multus/multus-additional-cni-plugins-48d52"
Jul 28 17:22:32.230857 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.230835    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ltgj\" (UniqueName: \"kubernetes.io/projected/630e1b18-a044-454e-9be4-359de1817e2b-kube-api-access-8ltgj\") pod \"node-ca-5bhch\" (UID: \"630e1b18-a044-454e-9be4-359de1817e2b\") " pod="openshift-image-registry/node-ca-5bhch"
Jul 28 17:22:32.231225 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.231207    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sxr7\" (UniqueName: \"kubernetes.io/projected/3acbd8e1-dd56-4d3f-9cec-54aed9951ef7-kube-api-access-9sxr7\") pod \"iptables-alerter-vf9hs\" (UID: \"3acbd8e1-dd56-4d3f-9cec-54aed9951ef7\") " pod="openshift-network-operator/iptables-alerter-vf9hs"
Jul 28 17:22:32.231300 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.231261    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjtkc\" (UniqueName: \"kubernetes.io/projected/091666fc-49ce-46bc-9a7d-5815676ea342-kube-api-access-zjtkc\") pod \"node-resolver-s7j5k\" (UID: \"091666fc-49ce-46bc-9a7d-5815676ea342\") " pod="openshift-dns/node-resolver-s7j5k"
Jul 28 17:22:32.232423 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.232401    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8489\" (UniqueName: \"kubernetes.io/projected/6e018f9d-b031-40ad-9846-f1fb5b4e21e8-kube-api-access-w8489\") pod \"ovnkube-node-5sdnx\" (UID: \"6e018f9d-b031-40ad-9846-f1fb5b4e21e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.232526 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.232490    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z656q\" (UniqueName: \"kubernetes.io/projected/dedc8b78-7469-4e8d-a7b8-b11c741186d7-kube-api-access-z656q\") pod \"network-metrics-daemon-znptx\" (UID: \"dedc8b78-7469-4e8d-a7b8-b11c741186d7\") " pod="openshift-multus/network-metrics-daemon-znptx"
Jul 28 17:22:32.307664 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.307631    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jv7m"
Jul 28 17:22:32.316359 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.316335    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lqf4t"
Jul 28 17:22:32.317478 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.317456    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:22:32.326234 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.326215    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xsptw"
Jul 28 17:22:32.336642 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.336624    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xnpc6"
Jul 28 17:22:32.343812 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.343792    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s7j5k"
Jul 28 17:22:32.351829 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.351807    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-48d52"
Jul 28 17:22:32.359831 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.359805    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:32.368580 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.368560    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vf9hs"
Jul 28 17:22:32.375242 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.375223    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5bhch"
Jul 28 17:22:32.724385 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.724226    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b881ad9-ece5-40ab-b106-32e6d520c37e-original-pull-secret\") pod \"global-pull-secret-syncer-dpv42\" (UID: \"3b881ad9-ece5-40ab-b106-32e6d520c37e\") " pod="kube-system/global-pull-secret-syncer-dpv42"
Jul 28 17:22:32.724385 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.724295    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dedc8b78-7469-4e8d-a7b8-b11c741186d7-metrics-certs\") pod \"network-metrics-daemon-znptx\" (UID: \"dedc8b78-7469-4e8d-a7b8-b11c741186d7\") " pod="openshift-multus/network-metrics-daemon-znptx"
Jul 28 17:22:32.724614 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:32.724453    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:22:32.724614 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:32.724523    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dedc8b78-7469-4e8d-a7b8-b11c741186d7-metrics-certs podName:dedc8b78-7469-4e8d-a7b8-b11c741186d7 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:33.724504305 +0000 UTC m=+4.034669791 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dedc8b78-7469-4e8d-a7b8-b11c741186d7-metrics-certs") pod "network-metrics-daemon-znptx" (UID: "dedc8b78-7469-4e8d-a7b8-b11c741186d7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:22:32.724614 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:32.724454    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:22:32.724747 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:32.724640    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b881ad9-ece5-40ab-b106-32e6d520c37e-original-pull-secret podName:3b881ad9-ece5-40ab-b106-32e6d520c37e nodeName:}" failed. No retries permitted until 2026-07-28 17:22:33.724613879 +0000 UTC m=+4.034779376 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b881ad9-ece5-40ab-b106-32e6d520c37e-original-pull-secret") pod "global-pull-secret-syncer-dpv42" (UID: "3b881ad9-ece5-40ab-b106-32e6d520c37e") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:22:32.824934 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:32.824889    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfwf2\" (UniqueName: \"kubernetes.io/projected/a743f0d2-5033-4955-bbca-1e7327255603-kube-api-access-lfwf2\") pod \"network-check-target-rvrql\" (UID: \"a743f0d2-5033-4955-bbca-1e7327255603\") " pod="openshift-network-diagnostics/network-check-target-rvrql"
Jul 28 17:22:32.825103 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:32.825067    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:22:32.825103 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:32.825097    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:22:32.825191 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:32.825111    2566 projected.go:196] Error preparing data for projected volume kube-api-access-lfwf2 for pod openshift-network-diagnostics/network-check-target-rvrql: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:22:32.825191 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:32.825178    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a743f0d2-5033-4955-bbca-1e7327255603-kube-api-access-lfwf2 podName:a743f0d2-5033-4955-bbca-1e7327255603 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:33.825158657 +0000 UTC m=+4.135324168 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lfwf2" (UniqueName: "kubernetes.io/projected/a743f0d2-5033-4955-bbca-1e7327255603-kube-api-access-lfwf2") pod "network-check-target-rvrql" (UID: "a743f0d2-5033-4955-bbca-1e7327255603") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:22:33.039594 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:33.039556    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod091666fc_49ce_46bc_9a7d_5815676ea342.slice/crio-c15b700f6ca0e23c953dff885adc3b3de21c011669f36a0400929b58c77a6ce2 WatchSource:0}: Error finding container c15b700f6ca0e23c953dff885adc3b3de21c011669f36a0400929b58c77a6ce2: Status 404 returned error can't find the container with id c15b700f6ca0e23c953dff885adc3b3de21c011669f36a0400929b58c77a6ce2
Jul 28 17:22:33.042193 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:33.042151    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod795058ed_0974_4328_b462_32e755000a70.slice/crio-b63e340964e5b78ff850028aa19df96e89f9f1a1873e94b556cb168b79d5c020 WatchSource:0}: Error finding container b63e340964e5b78ff850028aa19df96e89f9f1a1873e94b556cb168b79d5c020: Status 404 returned error can't find the container with id b63e340964e5b78ff850028aa19df96e89f9f1a1873e94b556cb168b79d5c020
Jul 28 17:22:33.044821 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:33.044789    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22ea97a4_e795_4d89_8257_fe0baf30b83e.slice/crio-f497448c98c32122fbe577cfca11bc3488c1d8bb3f3e4b44f0b2764ae6a44822 WatchSource:0}: Error finding container f497448c98c32122fbe577cfca11bc3488c1d8bb3f3e4b44f0b2764ae6a44822: Status 404 returned error can't find the container with id f497448c98c32122fbe577cfca11bc3488c1d8bb3f3e4b44f0b2764ae6a44822
Jul 28 17:22:33.045812 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:33.045507    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e018f9d_b031_40ad_9846_f1fb5b4e21e8.slice/crio-4734762b1a5bbdd66a28b814caeeb82bd5e3312de78edd7732ab029b32c0435d WatchSource:0}: Error finding container 4734762b1a5bbdd66a28b814caeeb82bd5e3312de78edd7732ab029b32c0435d: Status 404 returned error can't find the container with id 4734762b1a5bbdd66a28b814caeeb82bd5e3312de78edd7732ab029b32c0435d
Jul 28 17:22:33.047446 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:33.047410    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15fc65f5_e122_489d_b56c_1999d101223f.slice/crio-f0f832fc22e03991830d9b9bea18fcae7a5a783c8b4f59e2c9ee29c5924815c8 WatchSource:0}: Error finding container f0f832fc22e03991830d9b9bea18fcae7a5a783c8b4f59e2c9ee29c5924815c8: Status 404 returned error can't find the container with id f0f832fc22e03991830d9b9bea18fcae7a5a783c8b4f59e2c9ee29c5924815c8
Jul 28 17:22:33.047877 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:33.047830    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:17:32 +0000 UTC" deadline="2028-04-14 00:19:10.795039194 +0000 UTC"
Jul 28 17:22:33.047967 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:33.047876    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15006h56m37.747166296s"
Jul 28 17:22:33.049217 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:33.049060    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod630e1b18_a044_454e_9be4_359de1817e2b.slice/crio-68a64b523cdaf982aa7e8937480f016b698ff1bb24d38e460e3273dbe476202b WatchSource:0}: Error finding container 68a64b523cdaf982aa7e8937480f016b698ff1bb24d38e460e3273dbe476202b: Status 404 returned error can't find the container with id 68a64b523cdaf982aa7e8937480f016b698ff1bb24d38e460e3273dbe476202b
Jul 28 17:22:33.050143 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:33.049881    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330381c0_df2c_4d4b_8865_09d4488c075e.slice/crio-42895367295e9b75d814a7f134a8f066f8d63707a148b85761155ff58177310f WatchSource:0}: Error finding container 42895367295e9b75d814a7f134a8f066f8d63707a148b85761155ff58177310f: Status 404 returned error can't find the container with id 42895367295e9b75d814a7f134a8f066f8d63707a148b85761155ff58177310f
Jul 28 17:22:33.073545 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:33.073518    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a23dc6_82ad_4ab6_a528_82ad3146e7c6.slice/crio-a54bee94858ee91564ae6b211ede4e9206fa7ce8a6a1a9e1421f8fbdf82f6793 WatchSource:0}: Error finding container a54bee94858ee91564ae6b211ede4e9206fa7ce8a6a1a9e1421f8fbdf82f6793: Status 404 returned error can't find the container with id a54bee94858ee91564ae6b211ede4e9206fa7ce8a6a1a9e1421f8fbdf82f6793
Jul 28 17:22:33.075269 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:22:33.075246    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3acbd8e1_dd56_4d3f_9cec_54aed9951ef7.slice/crio-fdd9caed20610b8dcc41c4053d21586dec871bc99c6defd369f3dd0b9d93e5a7 WatchSource:0}: Error finding container fdd9caed20610b8dcc41c4053d21586dec871bc99c6defd369f3dd0b9d93e5a7: Status 404 returned error can't find the container with id fdd9caed20610b8dcc41c4053d21586dec871bc99c6defd369f3dd0b9d93e5a7
Jul 28 17:22:33.093328 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:33.093306    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znptx"
Jul 28 17:22:33.093444 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:33.093430    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-znptx" podUID="dedc8b78-7469-4e8d-a7b8-b11c741186d7"
Jul 28 17:22:33.101866 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:33.101843    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lqf4t" event={"ID":"330381c0-df2c-4d4b-8865-09d4488c075e","Type":"ContainerStarted","Data":"42895367295e9b75d814a7f134a8f066f8d63707a148b85761155ff58177310f"}
Jul 28 17:22:33.103915 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:33.103885    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5bhch" event={"ID":"630e1b18-a044-454e-9be4-359de1817e2b","Type":"ContainerStarted","Data":"68a64b523cdaf982aa7e8937480f016b698ff1bb24d38e460e3273dbe476202b"}
Jul 28 17:22:33.106353 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:33.106309    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-48d52" event={"ID":"15fc65f5-e122-489d-b56c-1999d101223f","Type":"ContainerStarted","Data":"f0f832fc22e03991830d9b9bea18fcae7a5a783c8b4f59e2c9ee29c5924815c8"}
Jul 28 17:22:33.108902 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:33.108880    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jv7m" event={"ID":"22ea97a4-e795-4d89-8257-fe0baf30b83e","Type":"ContainerStarted","Data":"f497448c98c32122fbe577cfca11bc3488c1d8bb3f3e4b44f0b2764ae6a44822"}
Jul 28 17:22:33.111727 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:33.111703    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xnpc6" event={"ID":"795058ed-0974-4328-b462-32e755000a70","Type":"ContainerStarted","Data":"b63e340964e5b78ff850028aa19df96e89f9f1a1873e94b556cb168b79d5c020"}
Jul 28 17:22:33.113823 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:33.113800    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s7j5k" event={"ID":"091666fc-49ce-46bc-9a7d-5815676ea342","Type":"ContainerStarted","Data":"c15b700f6ca0e23c953dff885adc3b3de21c011669f36a0400929b58c77a6ce2"}
Jul 28 17:22:33.115424 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:33.115397    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx" event={"ID":"6e018f9d-b031-40ad-9846-f1fb5b4e21e8","Type":"ContainerStarted","Data":"4734762b1a5bbdd66a28b814caeeb82bd5e3312de78edd7732ab029b32c0435d"}
Jul 28 17:22:33.116663 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:33.116637    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vf9hs" event={"ID":"3acbd8e1-dd56-4d3f-9cec-54aed9951ef7","Type":"ContainerStarted","Data":"fdd9caed20610b8dcc41c4053d21586dec871bc99c6defd369f3dd0b9d93e5a7"}
Jul 28 17:22:33.117663 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:33.117644    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xsptw" event={"ID":"51a23dc6-82ad-4ab6-a528-82ad3146e7c6","Type":"ContainerStarted","Data":"a54bee94858ee91564ae6b211ede4e9206fa7ce8a6a1a9e1421f8fbdf82f6793"}
Jul 28 17:22:33.733892 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:33.732975    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dedc8b78-7469-4e8d-a7b8-b11c741186d7-metrics-certs\") pod \"network-metrics-daemon-znptx\" (UID: \"dedc8b78-7469-4e8d-a7b8-b11c741186d7\") " pod="openshift-multus/network-metrics-daemon-znptx"
Jul 28 17:22:33.733892 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:33.733061    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b881ad9-ece5-40ab-b106-32e6d520c37e-original-pull-secret\") pod \"global-pull-secret-syncer-dpv42\" (UID: \"3b881ad9-ece5-40ab-b106-32e6d520c37e\") " pod="kube-system/global-pull-secret-syncer-dpv42"
Jul 28 17:22:33.733892 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:33.733239    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:22:33.733892 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:33.733303    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b881ad9-ece5-40ab-b106-32e6d520c37e-original-pull-secret podName:3b881ad9-ece5-40ab-b106-32e6d520c37e nodeName:}" failed. No retries permitted until 2026-07-28 17:22:35.733284359 +0000 UTC m=+6.043449846 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b881ad9-ece5-40ab-b106-32e6d520c37e-original-pull-secret") pod "global-pull-secret-syncer-dpv42" (UID: "3b881ad9-ece5-40ab-b106-32e6d520c37e") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:22:33.733892 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:33.733744    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:22:33.733892 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:33.733800    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dedc8b78-7469-4e8d-a7b8-b11c741186d7-metrics-certs podName:dedc8b78-7469-4e8d-a7b8-b11c741186d7 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:35.733782974 +0000 UTC m=+6.043948462 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dedc8b78-7469-4e8d-a7b8-b11c741186d7-metrics-certs") pod "network-metrics-daemon-znptx" (UID: "dedc8b78-7469-4e8d-a7b8-b11c741186d7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:22:33.833975 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:33.833935    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfwf2\" (UniqueName: \"kubernetes.io/projected/a743f0d2-5033-4955-bbca-1e7327255603-kube-api-access-lfwf2\") pod \"network-check-target-rvrql\" (UID: \"a743f0d2-5033-4955-bbca-1e7327255603\") " pod="openshift-network-diagnostics/network-check-target-rvrql"
Jul 28 17:22:33.834235 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:33.834207    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:22:33.834318 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:33.834244    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:22:33.834318 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:33.834260    2566 projected.go:196] Error preparing data for projected volume kube-api-access-lfwf2 for pod openshift-network-diagnostics/network-check-target-rvrql: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:22:33.834480 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:33.834324    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a743f0d2-5033-4955-bbca-1e7327255603-kube-api-access-lfwf2 podName:a743f0d2-5033-4955-bbca-1e7327255603 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:35.834304809 +0000 UTC m=+6.144470296 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lfwf2" (UniqueName: "kubernetes.io/projected/a743f0d2-5033-4955-bbca-1e7327255603-kube-api-access-lfwf2") pod "network-check-target-rvrql" (UID: "a743f0d2-5033-4955-bbca-1e7327255603") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:22:34.048754 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:34.048660    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:17:32 +0000 UTC" deadline="2028-04-15 00:32:08.287802888 +0000 UTC"
Jul 28 17:22:34.048754 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:34.048705    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15031h9m34.239103216s"
Jul 28 17:22:34.095096 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:34.095064    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpv42"
Jul 28 17:22:34.095261 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:34.095110    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvrql"
Jul 28 17:22:34.095261 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:34.095205    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpv42" podUID="3b881ad9-ece5-40ab-b106-32e6d520c37e"
Jul 28 17:22:34.095449 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:34.095369    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvrql" podUID="a743f0d2-5033-4955-bbca-1e7327255603"
Jul 28 17:22:34.128345 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:34.128314    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-233.ec2.internal" event={"ID":"a2d32026394497da64d771aa0e11975c","Type":"ContainerStarted","Data":"fd43c54badd2b80c1853d410365c6d35a4f7cb59d115a00a4781486c1fc326a8"}
Jul 28 17:22:34.139861 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:34.139831    2566 generic.go:356] "Generic (PLEG): container finished" podID="7f88513b366a558c56d55360985f15ee" containerID="52421d3f03e443667d15ffbe7b55b883324f2f5f3820387f9bce7e12607b0795" exitCode=0
Jul 28 17:22:34.139992 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:34.139883    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-233.ec2.internal" event={"ID":"7f88513b366a558c56d55360985f15ee","Type":"ContainerDied","Data":"52421d3f03e443667d15ffbe7b55b883324f2f5f3820387f9bce7e12607b0795"}
Jul 28 17:22:34.205822 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:34.205759    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-233.ec2.internal" podStartSLOduration=2.2057375 podStartE2EDuration="2.2057375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:22:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:22:34.175541518 +0000 UTC m=+4.485707029" watchObservedRunningTime="2026-07-28 17:22:34.2057375 +0000 UTC m=+4.515903009"
Jul 28 17:22:35.093899 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:35.093748    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znptx"
Jul 28 17:22:35.094417 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:35.093909    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-znptx" podUID="dedc8b78-7469-4e8d-a7b8-b11c741186d7"
Jul 28 17:22:35.166461 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:35.166416    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-233.ec2.internal" event={"ID":"7f88513b366a558c56d55360985f15ee","Type":"ContainerStarted","Data":"98075fb882d533af0e109664382c702ad093f3128b566c98d233da25af55f49c"}
Jul 28 17:22:35.749203 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:35.749159    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b881ad9-ece5-40ab-b106-32e6d520c37e-original-pull-secret\") pod \"global-pull-secret-syncer-dpv42\" (UID: \"3b881ad9-ece5-40ab-b106-32e6d520c37e\") " pod="kube-system/global-pull-secret-syncer-dpv42"
Jul 28 17:22:35.749438 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:35.749300    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dedc8b78-7469-4e8d-a7b8-b11c741186d7-metrics-certs\") pod \"network-metrics-daemon-znptx\" (UID: \"dedc8b78-7469-4e8d-a7b8-b11c741186d7\") " pod="openshift-multus/network-metrics-daemon-znptx"
Jul 28 17:22:35.749511 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:35.749457    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:22:35.749586 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:35.749569    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dedc8b78-7469-4e8d-a7b8-b11c741186d7-metrics-certs podName:dedc8b78-7469-4e8d-a7b8-b11c741186d7 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:39.749508615 +0000 UTC m=+10.059674116 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dedc8b78-7469-4e8d-a7b8-b11c741186d7-metrics-certs") pod "network-metrics-daemon-znptx" (UID: "dedc8b78-7469-4e8d-a7b8-b11c741186d7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:22:35.750040 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:35.750015    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:22:35.750149 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:35.750082    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b881ad9-ece5-40ab-b106-32e6d520c37e-original-pull-secret podName:3b881ad9-ece5-40ab-b106-32e6d520c37e nodeName:}" failed. No retries permitted until 2026-07-28 17:22:39.750064504 +0000 UTC m=+10.060229990 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b881ad9-ece5-40ab-b106-32e6d520c37e-original-pull-secret") pod "global-pull-secret-syncer-dpv42" (UID: "3b881ad9-ece5-40ab-b106-32e6d520c37e") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:22:35.849997 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:35.849954    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfwf2\" (UniqueName: \"kubernetes.io/projected/a743f0d2-5033-4955-bbca-1e7327255603-kube-api-access-lfwf2\") pod \"network-check-target-rvrql\" (UID: \"a743f0d2-5033-4955-bbca-1e7327255603\") " pod="openshift-network-diagnostics/network-check-target-rvrql"
Jul 28 17:22:35.850227 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:35.850207    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:22:35.850299 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:35.850236    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:22:35.850299 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:35.850249    2566 projected.go:196] Error preparing data for projected volume kube-api-access-lfwf2 for pod openshift-network-diagnostics/network-check-target-rvrql: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:22:35.850429 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:35.850314    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a743f0d2-5033-4955-bbca-1e7327255603-kube-api-access-lfwf2 podName:a743f0d2-5033-4955-bbca-1e7327255603 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:39.850294618 +0000 UTC m=+10.160460120 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lfwf2" (UniqueName: "kubernetes.io/projected/a743f0d2-5033-4955-bbca-1e7327255603-kube-api-access-lfwf2") pod "network-check-target-rvrql" (UID: "a743f0d2-5033-4955-bbca-1e7327255603") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:22:36.094222 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:36.093676    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvrql"
Jul 28 17:22:36.094222 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:36.093813    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvrql" podUID="a743f0d2-5033-4955-bbca-1e7327255603"
Jul 28 17:22:36.094222 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:36.093938    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpv42"
Jul 28 17:22:36.094222 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:36.094024    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpv42" podUID="3b881ad9-ece5-40ab-b106-32e6d520c37e"
Jul 28 17:22:37.093591 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:37.093530    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znptx"
Jul 28 17:22:37.093776 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:37.093684    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-znptx" podUID="dedc8b78-7469-4e8d-a7b8-b11c741186d7"
Jul 28 17:22:38.093895 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:38.093336    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpv42"
Jul 28 17:22:38.093895 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:38.093501    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpv42" podUID="3b881ad9-ece5-40ab-b106-32e6d520c37e"
Jul 28 17:22:38.094454 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:38.093946    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvrql"
Jul 28 17:22:38.096152 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:38.094930    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvrql" podUID="a743f0d2-5033-4955-bbca-1e7327255603"
Jul 28 17:22:39.093444 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:39.093376    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znptx"
Jul 28 17:22:39.093628 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:39.093518    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-znptx" podUID="dedc8b78-7469-4e8d-a7b8-b11c741186d7"
Jul 28 17:22:39.782249 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:39.782209    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dedc8b78-7469-4e8d-a7b8-b11c741186d7-metrics-certs\") pod \"network-metrics-daemon-znptx\" (UID: \"dedc8b78-7469-4e8d-a7b8-b11c741186d7\") " pod="openshift-multus/network-metrics-daemon-znptx"
Jul 28 17:22:39.782753 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:39.782297    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b881ad9-ece5-40ab-b106-32e6d520c37e-original-pull-secret\") pod \"global-pull-secret-syncer-dpv42\" (UID: \"3b881ad9-ece5-40ab-b106-32e6d520c37e\") " pod="kube-system/global-pull-secret-syncer-dpv42"
Jul 28 17:22:39.782753 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:39.782450    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:22:39.782753 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:39.782520    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b881ad9-ece5-40ab-b106-32e6d520c37e-original-pull-secret podName:3b881ad9-ece5-40ab-b106-32e6d520c37e nodeName:}" failed. No retries permitted until 2026-07-28 17:22:47.782499785 +0000 UTC m=+18.092665272 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b881ad9-ece5-40ab-b106-32e6d520c37e-original-pull-secret") pod "global-pull-secret-syncer-dpv42" (UID: "3b881ad9-ece5-40ab-b106-32e6d520c37e") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:22:39.782994 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:39.782971    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:22:39.783121 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:39.783035    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dedc8b78-7469-4e8d-a7b8-b11c741186d7-metrics-certs podName:dedc8b78-7469-4e8d-a7b8-b11c741186d7 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:47.783017426 +0000 UTC m=+18.093182910 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dedc8b78-7469-4e8d-a7b8-b11c741186d7-metrics-certs") pod "network-metrics-daemon-znptx" (UID: "dedc8b78-7469-4e8d-a7b8-b11c741186d7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:22:39.883523 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:39.883485    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfwf2\" (UniqueName: \"kubernetes.io/projected/a743f0d2-5033-4955-bbca-1e7327255603-kube-api-access-lfwf2\") pod \"network-check-target-rvrql\" (UID: \"a743f0d2-5033-4955-bbca-1e7327255603\") " pod="openshift-network-diagnostics/network-check-target-rvrql"
Jul 28 17:22:39.883708 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:39.883643    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:22:39.883708 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:39.883662    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:22:39.883708 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:39.883674    2566 projected.go:196] Error preparing data for projected volume kube-api-access-lfwf2 for pod openshift-network-diagnostics/network-check-target-rvrql: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:22:39.883829 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:39.883734    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a743f0d2-5033-4955-bbca-1e7327255603-kube-api-access-lfwf2 podName:a743f0d2-5033-4955-bbca-1e7327255603 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:47.883715851 +0000 UTC m=+18.193881360 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lfwf2" (UniqueName: "kubernetes.io/projected/a743f0d2-5033-4955-bbca-1e7327255603-kube-api-access-lfwf2") pod "network-check-target-rvrql" (UID: "a743f0d2-5033-4955-bbca-1e7327255603") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:22:40.094782 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:40.094283    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpv42"
Jul 28 17:22:40.094782 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:40.094401    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpv42" podUID="3b881ad9-ece5-40ab-b106-32e6d520c37e"
Jul 28 17:22:40.094782 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:40.094481    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvrql"
Jul 28 17:22:40.094782 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:40.094567    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvrql" podUID="a743f0d2-5033-4955-bbca-1e7327255603"
Jul 28 17:22:41.093663 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:41.093616    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znptx"
Jul 28 17:22:41.094132 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:41.093762    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-znptx" podUID="dedc8b78-7469-4e8d-a7b8-b11c741186d7"
Jul 28 17:22:42.096311 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:42.096004    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpv42"
Jul 28 17:22:42.096311 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:42.096135    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpv42" podUID="3b881ad9-ece5-40ab-b106-32e6d520c37e"
Jul 28 17:22:42.096311 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:42.096211    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvrql"
Jul 28 17:22:42.096311 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:42.096277    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvrql" podUID="a743f0d2-5033-4955-bbca-1e7327255603"
Jul 28 17:22:43.093467 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:43.093429    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znptx"
Jul 28 17:22:43.093658 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:43.093572    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-znptx" podUID="dedc8b78-7469-4e8d-a7b8-b11c741186d7"
Jul 28 17:22:44.093348 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:44.093303    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvrql"
Jul 28 17:22:44.093348 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:44.093337    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpv42"
Jul 28 17:22:44.093828 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:44.093451    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvrql" podUID="a743f0d2-5033-4955-bbca-1e7327255603"
Jul 28 17:22:44.093828 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:44.093510    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpv42" podUID="3b881ad9-ece5-40ab-b106-32e6d520c37e"
Jul 28 17:22:45.092892 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:45.092858    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znptx"
Jul 28 17:22:45.093060 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:45.092971    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-znptx" podUID="dedc8b78-7469-4e8d-a7b8-b11c741186d7"
Jul 28 17:22:46.093683 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:46.093649    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpv42"
Jul 28 17:22:46.093683 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:46.093671    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvrql"
Jul 28 17:22:46.094227 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:46.093784    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpv42" podUID="3b881ad9-ece5-40ab-b106-32e6d520c37e"
Jul 28 17:22:46.094227 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:46.093864    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvrql" podUID="a743f0d2-5033-4955-bbca-1e7327255603"
Jul 28 17:22:47.093354 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:47.093309    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znptx"
Jul 28 17:22:47.093596 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:47.093437    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-znptx" podUID="dedc8b78-7469-4e8d-a7b8-b11c741186d7"
Jul 28 17:22:47.850501 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:47.850466    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dedc8b78-7469-4e8d-a7b8-b11c741186d7-metrics-certs\") pod \"network-metrics-daemon-znptx\" (UID: \"dedc8b78-7469-4e8d-a7b8-b11c741186d7\") " pod="openshift-multus/network-metrics-daemon-znptx"
Jul 28 17:22:47.850933 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:47.850536    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b881ad9-ece5-40ab-b106-32e6d520c37e-original-pull-secret\") pod \"global-pull-secret-syncer-dpv42\" (UID: \"3b881ad9-ece5-40ab-b106-32e6d520c37e\") " pod="kube-system/global-pull-secret-syncer-dpv42"
Jul 28 17:22:47.850933 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:47.850640    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:22:47.850933 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:47.850674    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:22:47.850933 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:47.850731    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dedc8b78-7469-4e8d-a7b8-b11c741186d7-metrics-certs podName:dedc8b78-7469-4e8d-a7b8-b11c741186d7 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:03.850709208 +0000 UTC m=+34.160874706 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dedc8b78-7469-4e8d-a7b8-b11c741186d7-metrics-certs") pod "network-metrics-daemon-znptx" (UID: "dedc8b78-7469-4e8d-a7b8-b11c741186d7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:22:47.850933 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:47.850752    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b881ad9-ece5-40ab-b106-32e6d520c37e-original-pull-secret podName:3b881ad9-ece5-40ab-b106-32e6d520c37e nodeName:}" failed. No retries permitted until 2026-07-28 17:23:03.850742117 +0000 UTC m=+34.160907616 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b881ad9-ece5-40ab-b106-32e6d520c37e-original-pull-secret") pod "global-pull-secret-syncer-dpv42" (UID: "3b881ad9-ece5-40ab-b106-32e6d520c37e") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:22:47.951186 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:47.951156    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfwf2\" (UniqueName: \"kubernetes.io/projected/a743f0d2-5033-4955-bbca-1e7327255603-kube-api-access-lfwf2\") pod \"network-check-target-rvrql\" (UID: \"a743f0d2-5033-4955-bbca-1e7327255603\") " pod="openshift-network-diagnostics/network-check-target-rvrql"
Jul 28 17:22:47.951360 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:47.951285    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:22:47.951360 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:47.951302    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:22:47.951360 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:47.951311    2566 projected.go:196] Error preparing data for projected volume kube-api-access-lfwf2 for pod openshift-network-diagnostics/network-check-target-rvrql: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:22:47.951529 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:47.951373    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a743f0d2-5033-4955-bbca-1e7327255603-kube-api-access-lfwf2 podName:a743f0d2-5033-4955-bbca-1e7327255603 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:03.951360016 +0000 UTC m=+34.261525512 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lfwf2" (UniqueName: "kubernetes.io/projected/a743f0d2-5033-4955-bbca-1e7327255603-kube-api-access-lfwf2") pod "network-check-target-rvrql" (UID: "a743f0d2-5033-4955-bbca-1e7327255603") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:22:48.093051 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:48.093001    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpv42"
Jul 28 17:22:48.093051 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:48.093046    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvrql"
Jul 28 17:22:48.093253 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:48.093140    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpv42" podUID="3b881ad9-ece5-40ab-b106-32e6d520c37e"
Jul 28 17:22:48.093253 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:48.093215    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvrql" podUID="a743f0d2-5033-4955-bbca-1e7327255603"
Jul 28 17:22:49.093759 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:49.093724    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znptx"
Jul 28 17:22:49.094202 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:49.093851    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-znptx" podUID="dedc8b78-7469-4e8d-a7b8-b11c741186d7"
Jul 28 17:22:50.093810 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:50.093779    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpv42"
Jul 28 17:22:50.094209 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:50.093900    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpv42" podUID="3b881ad9-ece5-40ab-b106-32e6d520c37e"
Jul 28 17:22:50.094209 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:50.093958    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvrql"
Jul 28 17:22:50.094209 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:50.094048    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvrql" podUID="a743f0d2-5033-4955-bbca-1e7327255603"
Jul 28 17:22:51.093759 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:51.093721    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znptx"
Jul 28 17:22:51.093934 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:51.093841    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-znptx" podUID="dedc8b78-7469-4e8d-a7b8-b11c741186d7"
Jul 28 17:22:52.093649 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:52.093622    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvrql"
Jul 28 17:22:52.093774 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:52.093750    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvrql" podUID="a743f0d2-5033-4955-bbca-1e7327255603"
Jul 28 17:22:52.094250 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:52.094205    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpv42"
Jul 28 17:22:52.094590 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:52.094385    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpv42" podUID="3b881ad9-ece5-40ab-b106-32e6d520c37e"
Jul 28 17:22:52.203343 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:52.201920    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xsptw" event={"ID":"51a23dc6-82ad-4ab6-a528-82ad3146e7c6","Type":"ContainerStarted","Data":"c078d1ffb44042d706a5505d43db85f8ba1841415681a535544c9da429830462"}
Jul 28 17:22:52.204202 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:52.204175    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lqf4t" event={"ID":"330381c0-df2c-4d4b-8865-09d4488c075e","Type":"ContainerStarted","Data":"ca112fa79094c08a4295824e53d73387ebd5ccaee4347bda6254759459e9329f"}
Jul 28 17:22:52.206914 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:52.206846    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5bhch" event={"ID":"630e1b18-a044-454e-9be4-359de1817e2b","Type":"ContainerStarted","Data":"5d86c62072a45280acff2c3e8d1cf58c503cd4307a4c0d79921de6e313db8d05"}
Jul 28 17:22:52.208899 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:52.208856    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-48d52" event={"ID":"15fc65f5-e122-489d-b56c-1999d101223f","Type":"ContainerStarted","Data":"f0fd65bf44d0c14e78715533e81cb02e7a6657d3bdcefca36ab884602a437392"}
Jul 28 17:22:52.211423 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:52.211204    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jv7m" event={"ID":"22ea97a4-e795-4d89-8257-fe0baf30b83e","Type":"ContainerStarted","Data":"403448d827332c83c3b23bd7587f11cfeb3a38946972f207ad2077f2ff9dfba3"}
Jul 28 17:22:52.213612 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:52.213545    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xnpc6" event={"ID":"795058ed-0974-4328-b462-32e755000a70","Type":"ContainerStarted","Data":"814bcb625317d4098881ac290f5fc21e473f140c841f9b0e0c4424de871867b2"}
Jul 28 17:22:52.215902 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:52.215823    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s7j5k" event={"ID":"091666fc-49ce-46bc-9a7d-5815676ea342","Type":"ContainerStarted","Data":"49573a011a9c9f307f4c7a1a55830d3a3942b4ac2a05c9e9868f0ad766550243"}
Jul 28 17:22:52.221024 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:52.220801    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx" event={"ID":"6e018f9d-b031-40ad-9846-f1fb5b4e21e8","Type":"ContainerStarted","Data":"0dd89bc52b68d6c084de3a93256f6ee6778ac5dfb85da0d303a09793c6c2c21d"}
Jul 28 17:22:52.227909 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:52.223105    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-233.ec2.internal" podStartSLOduration=20.223083627 podStartE2EDuration="20.223083627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:22:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:22:35.186587522 +0000 UTC m=+5.496753031" watchObservedRunningTime="2026-07-28 17:22:52.223083627 +0000 UTC m=+22.533249133"
Jul 28 17:22:52.227909 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:52.223463    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xsptw" podStartSLOduration=7.126935442 podStartE2EDuration="22.22345114s" totalImagesPullingTime="15.096515698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:22:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:22:52.222200239 +0000 UTC m=+22.532365744" watchObservedRunningTime="2026-07-28 17:22:52.22345114 +0000 UTC m=+22.533616646"
Jul 28 17:22:52.241721 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:52.241486    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5bhch" podStartSLOduration=3.318796315 podStartE2EDuration="22.241443228s" totalImagesPullingTime="18.922646913s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:22:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:22:52.240447249 +0000 UTC m=+22.550612777" watchObservedRunningTime="2026-07-28 17:22:52.241443228 +0000 UTC m=+22.551608734"
Jul 28 17:22:52.285001 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:52.284595    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lqf4t" podStartSLOduration=3.350489795 podStartE2EDuration="22.284580373s" totalImagesPullingTime="18.934090578s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:22:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:22:52.264491257 +0000 UTC m=+22.574656763" watchObservedRunningTime="2026-07-28 17:22:52.284580373 +0000 UTC m=+22.594745878"
Jul 28 17:22:52.305647 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:52.305607    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xnpc6" podStartSLOduration=3.279652683 podStartE2EDuration="22.305591935s" totalImagesPullingTime="19.025939252s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:22:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:22:52.284846267 +0000 UTC m=+22.595011769" watchObservedRunningTime="2026-07-28 17:22:52.305591935 +0000 UTC m=+22.615757439"
Jul 28 17:22:52.305849 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:52.305828    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-s7j5k" podStartSLOduration=3.358274324 podStartE2EDuration="22.305821044s" totalImagesPullingTime="18.94754672s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:22:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:22:52.305404484 +0000 UTC m=+22.615570010" watchObservedRunningTime="2026-07-28 17:22:52.305821044 +0000 UTC m=+22.615986548"
Jul 28 17:22:53.093619 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:53.093594    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znptx"
Jul 28 17:22:53.093781 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:53.093691    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-znptx" podUID="dedc8b78-7469-4e8d-a7b8-b11c741186d7"
Jul 28 17:22:53.224583 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:53.224548    2566 generic.go:356] "Generic (PLEG): container finished" podID="15fc65f5-e122-489d-b56c-1999d101223f" containerID="f0fd65bf44d0c14e78715533e81cb02e7a6657d3bdcefca36ab884602a437392" exitCode=0
Jul 28 17:22:53.225358 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:53.224635    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-48d52" event={"ID":"15fc65f5-e122-489d-b56c-1999d101223f","Type":"ContainerDied","Data":"f0fd65bf44d0c14e78715533e81cb02e7a6657d3bdcefca36ab884602a437392"}
Jul 28 17:22:53.227681 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:53.227657    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx" event={"ID":"6e018f9d-b031-40ad-9846-f1fb5b4e21e8","Type":"ContainerStarted","Data":"2f543d9be464165b703dcbd206e76f8bfb97098164bb7df7961521df71e26340"}
Jul 28 17:22:53.227796 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:53.227726    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx" event={"ID":"6e018f9d-b031-40ad-9846-f1fb5b4e21e8","Type":"ContainerStarted","Data":"3b93b3acc77316217c9868bc19df201f5b76838e9f013cd7a9a362abd03b7563"}
Jul 28 17:22:53.227796 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:53.227760    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx" event={"ID":"6e018f9d-b031-40ad-9846-f1fb5b4e21e8","Type":"ContainerStarted","Data":"de1c9df2fb933418fde87e75e2cbd1204b3fd054042b44ec10c04872fcd37ea9"}
Jul 28 17:22:53.227879 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:53.227797    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx" event={"ID":"6e018f9d-b031-40ad-9846-f1fb5b4e21e8","Type":"ContainerStarted","Data":"933caa42a7427f5b298f63897da1ddef9b75bd7764b94567f93896de3ff52f56"}
Jul 28 17:22:53.227879 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:53.227824    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx" event={"ID":"6e018f9d-b031-40ad-9846-f1fb5b4e21e8","Type":"ContainerStarted","Data":"bc1bece656846917f001985e6556d8a8d8d0ec7e1e52167bf8aecc6cbe4fe867"}
Jul 28 17:22:53.229069 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:53.229045    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vf9hs" event={"ID":"3acbd8e1-dd56-4d3f-9cec-54aed9951ef7","Type":"ContainerStarted","Data":"e90a1d2a18d27447ff1ea1cd930678678757150551d149d92219bf8e919c4a08"}
Jul 28 17:22:53.271704 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:53.271625    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vf9hs" podStartSLOduration=4.354189244 podStartE2EDuration="23.271612297s" totalImagesPullingTime="18.917423053s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:22:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:22:53.271410854 +0000 UTC m=+23.581576362" watchObservedRunningTime="2026-07-28 17:22:53.271612297 +0000 UTC m=+23.581777801"
Jul 28 17:22:54.092935 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:54.092870    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvrql"
Jul 28 17:22:54.092935 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:54.092889    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpv42"
Jul 28 17:22:54.093191 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:54.092991    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvrql" podUID="a743f0d2-5033-4955-bbca-1e7327255603"
Jul 28 17:22:54.093284 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:54.093260    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpv42" podUID="3b881ad9-ece5-40ab-b106-32e6d520c37e"
Jul 28 17:22:55.093100 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:55.093028    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znptx"
Jul 28 17:22:55.093532 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:55.093144    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-znptx" podUID="dedc8b78-7469-4e8d-a7b8-b11c741186d7"
Jul 28 17:22:55.237289 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:55.237249    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx" event={"ID":"6e018f9d-b031-40ad-9846-f1fb5b4e21e8","Type":"ContainerStarted","Data":"8b6a308d69fd7481675a308ff9f32e033dd35b4393ca6725a1e301a0228849a8"}
Jul 28 17:22:55.959170 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:55.959121    2566 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 500 Internal Server Error; artifact err: provided artifact is a container image" image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d1d940b941cc3129d8c30028e2371b5741771afc1860c872c66cdc8ebda6c0fe"
Jul 28 17:22:55.959338 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:55.959246    2566 kuberuntime_manager.go:1459] "Unhandled Error" err="container csi-node-driver-registrar start failed in pod aws-ebs-csi-driver-node-6jv7m_openshift-cluster-csi-drivers(22ea97a4-e795-4d89-8257-fe0baf30b83e): ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 500 Internal Server Error; artifact err: provided artifact is a container image" logger="UnhandledError"
Jul 28 17:22:56.093832 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:56.093760    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpv42"
Jul 28 17:22:56.094295 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:56.093900    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpv42" podUID="3b881ad9-ece5-40ab-b106-32e6d520c37e"
Jul 28 17:22:56.094295 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:56.093957    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvrql"
Jul 28 17:22:56.094295 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:56.094075    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvrql" podUID="a743f0d2-5033-4955-bbca-1e7327255603"
Jul 28 17:22:56.440152 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:56.440120    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xsptw"
Jul 28 17:22:56.440795 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:56.440771    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xsptw"
Jul 28 17:22:57.093497 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:57.093462    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znptx"
Jul 28 17:22:57.093698 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:57.093589    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-znptx" podUID="dedc8b78-7469-4e8d-a7b8-b11c741186d7"
Jul 28 17:22:58.093760 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:58.093582    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvrql"
Jul 28 17:22:58.094576 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:58.093631    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpv42"
Jul 28 17:22:58.094576 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:58.093852    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvrql" podUID="a743f0d2-5033-4955-bbca-1e7327255603"
Jul 28 17:22:58.094576 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:58.093900    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpv42" podUID="3b881ad9-ece5-40ab-b106-32e6d520c37e"
Jul 28 17:22:58.247488 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:58.247451    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx" event={"ID":"6e018f9d-b031-40ad-9846-f1fb5b4e21e8","Type":"ContainerStarted","Data":"f38ce4892da7af7f9a203090977c84431aabb768997a6acba959502369553ee3"}
Jul 28 17:22:58.247765 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:58.247679    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:58.247765 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:58.247744    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:58.249313 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:58.249291    2566 generic.go:356] "Generic (PLEG): container finished" podID="15fc65f5-e122-489d-b56c-1999d101223f" containerID="ccb54b99511d79760bf365d2dd0558e150e018c67e489350c9161eaa8069e71a" exitCode=0
Jul 28 17:22:58.249423 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:58.249337    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-48d52" event={"ID":"15fc65f5-e122-489d-b56c-1999d101223f","Type":"ContainerDied","Data":"ccb54b99511d79760bf365d2dd0558e150e018c67e489350c9161eaa8069e71a"}
Jul 28 17:22:58.262907 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:58.262887    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:58.281563 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:58.281527    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx" podStartSLOduration=9.266551427 podStartE2EDuration="28.281516689s" totalImagesPullingTime="19.014965262s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:22:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:22:58.2812792 +0000 UTC m=+28.591444705" watchObservedRunningTime="2026-07-28 17:22:58.281516689 +0000 UTC m=+28.591682193"
Jul 28 17:22:58.432809 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:58.432648    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 500 Internal Server Error; artifact err: provided artifact is a container image\"" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jv7m" podUID="22ea97a4-e795-4d89-8257-fe0baf30b83e"
Jul 28 17:22:59.093638 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:59.093611    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znptx"
Jul 28 17:22:59.093844 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:59.093760    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-znptx" podUID="dedc8b78-7469-4e8d-a7b8-b11c741186d7"
Jul 28 17:22:59.253077 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:59.253032    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jv7m" event={"ID":"22ea97a4-e795-4d89-8257-fe0baf30b83e","Type":"ContainerStarted","Data":"af9a64ced28130836162a9b82a3865988c9fce04f07f2317b51c93fdd89d087a"}
Jul 28 17:22:59.253402 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:59.253382    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:59.254428 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:59.254314    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d1d940b941cc3129d8c30028e2371b5741771afc1860c872c66cdc8ebda6c0fe\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 500 Internal Server Error; artifact err: provided artifact is a container image\"" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jv7m" podUID="22ea97a4-e795-4d89-8257-fe0baf30b83e"
Jul 28 17:22:59.268512 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:59.268487    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:22:59.337018 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:59.336988    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dpv42"]
Jul 28 17:22:59.337166 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:59.337111    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpv42"
Jul 28 17:22:59.337215 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:59.337187    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpv42" podUID="3b881ad9-ece5-40ab-b106-32e6d520c37e"
Jul 28 17:22:59.342293 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:59.342230    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-znptx"]
Jul 28 17:22:59.342440 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:59.342343    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znptx"
Jul 28 17:22:59.342505 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:59.342442    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-znptx" podUID="dedc8b78-7469-4e8d-a7b8-b11c741186d7"
Jul 28 17:22:59.342862 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:59.342841    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rvrql"]
Jul 28 17:22:59.342947 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:22:59.342935    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvrql"
Jul 28 17:22:59.343043 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:22:59.343024    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvrql" podUID="a743f0d2-5033-4955-bbca-1e7327255603"
Jul 28 17:23:00.257168 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:00.257125    2566 generic.go:356] "Generic (PLEG): container finished" podID="15fc65f5-e122-489d-b56c-1999d101223f" containerID="e744974f251434a964be74485d3794352744d2cbbe3efe9868b9e3cf4b679480" exitCode=0
Jul 28 17:23:00.257539 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:00.257214    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-48d52" event={"ID":"15fc65f5-e122-489d-b56c-1999d101223f","Type":"ContainerDied","Data":"e744974f251434a964be74485d3794352744d2cbbe3efe9868b9e3cf4b679480"}
Jul 28 17:23:00.258520 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:00.258496    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d1d940b941cc3129d8c30028e2371b5741771afc1860c872c66cdc8ebda6c0fe\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 500 Internal Server Error; artifact err: provided artifact is a container image\"" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jv7m" podUID="22ea97a4-e795-4d89-8257-fe0baf30b83e"
Jul 28 17:23:00.350344 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:00.350300    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xsptw"
Jul 28 17:23:00.350494 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:00.350442    2566 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 28 17:23:00.350837 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:00.350818    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xsptw"
Jul 28 17:23:01.093633 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:01.093598    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvrql"
Jul 28 17:23:01.093633 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:01.093616    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znptx"
Jul 28 17:23:01.093814 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:01.093700    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvrql" podUID="a743f0d2-5033-4955-bbca-1e7327255603"
Jul 28 17:23:01.093850 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:01.093815    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-znptx" podUID="dedc8b78-7469-4e8d-a7b8-b11c741186d7"
Jul 28 17:23:01.093884 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:01.093873    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpv42"
Jul 28 17:23:01.093937 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:01.093923    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpv42" podUID="3b881ad9-ece5-40ab-b106-32e6d520c37e"
Jul 28 17:23:02.264822 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:02.264622    2566 generic.go:356] "Generic (PLEG): container finished" podID="15fc65f5-e122-489d-b56c-1999d101223f" containerID="98bb5a56bb59973ac8782ba979b685978cb2c11b1d8a8785867d6375fdbf44c7" exitCode=0
Jul 28 17:23:02.264822 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:02.264704    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-48d52" event={"ID":"15fc65f5-e122-489d-b56c-1999d101223f","Type":"ContainerDied","Data":"98bb5a56bb59973ac8782ba979b685978cb2c11b1d8a8785867d6375fdbf44c7"}
Jul 28 17:23:03.093231 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:03.093198    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvrql"
Jul 28 17:23:03.093231 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:03.093221    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpv42"
Jul 28 17:23:03.093509 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:03.093315    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znptx"
Jul 28 17:23:03.093509 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:03.093336    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvrql" podUID="a743f0d2-5033-4955-bbca-1e7327255603"
Jul 28 17:23:03.093509 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:03.093419    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpv42" podUID="3b881ad9-ece5-40ab-b106-32e6d520c37e"
Jul 28 17:23:03.093668 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:03.093518    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-znptx" podUID="dedc8b78-7469-4e8d-a7b8-b11c741186d7"
Jul 28 17:23:03.870488 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:03.870447    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b881ad9-ece5-40ab-b106-32e6d520c37e-original-pull-secret\") pod \"global-pull-secret-syncer-dpv42\" (UID: \"3b881ad9-ece5-40ab-b106-32e6d520c37e\") " pod="kube-system/global-pull-secret-syncer-dpv42"
Jul 28 17:23:03.870946 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:03.870513    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dedc8b78-7469-4e8d-a7b8-b11c741186d7-metrics-certs\") pod \"network-metrics-daemon-znptx\" (UID: \"dedc8b78-7469-4e8d-a7b8-b11c741186d7\") " pod="openshift-multus/network-metrics-daemon-znptx"
Jul 28 17:23:03.870946 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:03.870628    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:23:03.870946 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:03.870690    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dedc8b78-7469-4e8d-a7b8-b11c741186d7-metrics-certs podName:dedc8b78-7469-4e8d-a7b8-b11c741186d7 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:35.870671077 +0000 UTC m=+66.180836579 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dedc8b78-7469-4e8d-a7b8-b11c741186d7-metrics-certs") pod "network-metrics-daemon-znptx" (UID: "dedc8b78-7469-4e8d-a7b8-b11c741186d7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:23:03.870946 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:03.870626    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:23:03.870946 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:03.870766    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b881ad9-ece5-40ab-b106-32e6d520c37e-original-pull-secret podName:3b881ad9-ece5-40ab-b106-32e6d520c37e nodeName:}" failed. No retries permitted until 2026-07-28 17:23:35.87075307 +0000 UTC m=+66.180918563 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b881ad9-ece5-40ab-b106-32e6d520c37e-original-pull-secret") pod "global-pull-secret-syncer-dpv42" (UID: "3b881ad9-ece5-40ab-b106-32e6d520c37e") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:23:03.971712 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:03.971679    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfwf2\" (UniqueName: \"kubernetes.io/projected/a743f0d2-5033-4955-bbca-1e7327255603-kube-api-access-lfwf2\") pod \"network-check-target-rvrql\" (UID: \"a743f0d2-5033-4955-bbca-1e7327255603\") " pod="openshift-network-diagnostics/network-check-target-rvrql"
Jul 28 17:23:03.971908 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:03.971834    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:23:03.971908 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:03.971859    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:23:03.971908 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:03.971873    2566 projected.go:196] Error preparing data for projected volume kube-api-access-lfwf2 for pod openshift-network-diagnostics/network-check-target-rvrql: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:23:03.972067 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:03.971941    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a743f0d2-5033-4955-bbca-1e7327255603-kube-api-access-lfwf2 podName:a743f0d2-5033-4955-bbca-1e7327255603 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:35.971921925 +0000 UTC m=+66.282087432 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lfwf2" (UniqueName: "kubernetes.io/projected/a743f0d2-5033-4955-bbca-1e7327255603-kube-api-access-lfwf2") pod "network-check-target-rvrql" (UID: "a743f0d2-5033-4955-bbca-1e7327255603") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:23:05.066660 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.066629    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-233.ec2.internal" event="NodeReady"
Jul 28 17:23:05.067340 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.066787    2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 17:23:05.093066 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.093036    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpv42"
Jul 28 17:23:05.093244 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.093036    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znptx"
Jul 28 17:23:05.093304 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.093037    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvrql"
Jul 28 17:23:05.096350 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.096309    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:23:05.096350 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.096330    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 17:23:05.096600 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.096394    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ztptx\""
Jul 28 17:23:05.096600 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.096334    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:23:05.097482 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.097461    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-27qqv\""
Jul 28 17:23:05.097590 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.097497    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 17:23:05.123946 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.123921    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tpb27"]
Jul 28 17:23:05.129476 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.129455    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b67m9"]
Jul 28 17:23:05.129596 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.129570    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tpb27"
Jul 28 17:23:05.133065 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.133047    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 17:23:05.133256 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.133218    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fqthr\""
Jul 28 17:23:05.133384 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.133271    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 17:23:05.133476 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.133457    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b67m9"
Jul 28 17:23:05.135919 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.135899    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 17:23:05.136020 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.136008    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 17:23:05.136365 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.136315    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 17:23:05.136486 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.136469    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-h4rhp\""
Jul 28 17:23:05.149254 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.149229    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tpb27"]
Jul 28 17:23:05.150159 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.150138    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b67m9"]
Jul 28 17:23:05.281115 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.281083    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpz6q\" (UniqueName: \"kubernetes.io/projected/b244121d-33d8-46d2-aa87-b451301005c8-kube-api-access-kpz6q\") pod \"ingress-canary-b67m9\" (UID: \"b244121d-33d8-46d2-aa87-b451301005c8\") " pod="openshift-ingress-canary/ingress-canary-b67m9"
Jul 28 17:23:05.281287 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.281125    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e17b0fe1-b406-4126-b193-f673a5846bd1-tmp-dir\") pod \"dns-default-tpb27\" (UID: \"e17b0fe1-b406-4126-b193-f673a5846bd1\") " pod="openshift-dns/dns-default-tpb27"
Jul 28 17:23:05.281287 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.281153    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e17b0fe1-b406-4126-b193-f673a5846bd1-config-volume\") pod \"dns-default-tpb27\" (UID: \"e17b0fe1-b406-4126-b193-f673a5846bd1\") " pod="openshift-dns/dns-default-tpb27"
Jul 28 17:23:05.281287 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.281258    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzxzc\" (UniqueName: \"kubernetes.io/projected/e17b0fe1-b406-4126-b193-f673a5846bd1-kube-api-access-hzxzc\") pod \"dns-default-tpb27\" (UID: \"e17b0fe1-b406-4126-b193-f673a5846bd1\") " pod="openshift-dns/dns-default-tpb27"
Jul 28 17:23:05.281461 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.281335    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b244121d-33d8-46d2-aa87-b451301005c8-cert\") pod \"ingress-canary-b67m9\" (UID: \"b244121d-33d8-46d2-aa87-b451301005c8\") " pod="openshift-ingress-canary/ingress-canary-b67m9"
Jul 28 17:23:05.281461 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.281412    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e17b0fe1-b406-4126-b193-f673a5846bd1-metrics-tls\") pod \"dns-default-tpb27\" (UID: \"e17b0fe1-b406-4126-b193-f673a5846bd1\") " pod="openshift-dns/dns-default-tpb27"
Jul 28 17:23:05.382554 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.382460    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e17b0fe1-b406-4126-b193-f673a5846bd1-tmp-dir\") pod \"dns-default-tpb27\" (UID: \"e17b0fe1-b406-4126-b193-f673a5846bd1\") " pod="openshift-dns/dns-default-tpb27"
Jul 28 17:23:05.382554 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.382517    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e17b0fe1-b406-4126-b193-f673a5846bd1-config-volume\") pod \"dns-default-tpb27\" (UID: \"e17b0fe1-b406-4126-b193-f673a5846bd1\") " pod="openshift-dns/dns-default-tpb27"
Jul 28 17:23:05.382784 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.382565    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzxzc\" (UniqueName: \"kubernetes.io/projected/e17b0fe1-b406-4126-b193-f673a5846bd1-kube-api-access-hzxzc\") pod \"dns-default-tpb27\" (UID: \"e17b0fe1-b406-4126-b193-f673a5846bd1\") " pod="openshift-dns/dns-default-tpb27"
Jul 28 17:23:05.382784 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.382614    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b244121d-33d8-46d2-aa87-b451301005c8-cert\") pod \"ingress-canary-b67m9\" (UID: \"b244121d-33d8-46d2-aa87-b451301005c8\") " pod="openshift-ingress-canary/ingress-canary-b67m9"
Jul 28 17:23:05.382784 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.382714    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e17b0fe1-b406-4126-b193-f673a5846bd1-metrics-tls\") pod \"dns-default-tpb27\" (UID: \"e17b0fe1-b406-4126-b193-f673a5846bd1\") " pod="openshift-dns/dns-default-tpb27"
Jul 28 17:23:05.382784 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.382755    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpz6q\" (UniqueName: \"kubernetes.io/projected/b244121d-33d8-46d2-aa87-b451301005c8-kube-api-access-kpz6q\") pod \"ingress-canary-b67m9\" (UID: \"b244121d-33d8-46d2-aa87-b451301005c8\") " pod="openshift-ingress-canary/ingress-canary-b67m9"
Jul 28 17:23:05.383017 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.382888    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e17b0fe1-b406-4126-b193-f673a5846bd1-tmp-dir\") pod \"dns-default-tpb27\" (UID: \"e17b0fe1-b406-4126-b193-f673a5846bd1\") " pod="openshift-dns/dns-default-tpb27"
Jul 28 17:23:05.383017 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:05.382992    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:23:05.383104 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:05.383052    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17b0fe1-b406-4126-b193-f673a5846bd1-metrics-tls podName:e17b0fe1-b406-4126-b193-f673a5846bd1 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:05.883033912 +0000 UTC m=+36.193199403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e17b0fe1-b406-4126-b193-f673a5846bd1-metrics-tls") pod "dns-default-tpb27" (UID: "e17b0fe1-b406-4126-b193-f673a5846bd1") : secret "dns-default-metrics-tls" not found
Jul 28 17:23:05.383193 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.383165    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e17b0fe1-b406-4126-b193-f673a5846bd1-config-volume\") pod \"dns-default-tpb27\" (UID: \"e17b0fe1-b406-4126-b193-f673a5846bd1\") " pod="openshift-dns/dns-default-tpb27"
Jul 28 17:23:05.387001 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.386969    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b244121d-33d8-46d2-aa87-b451301005c8-cert\") pod \"ingress-canary-b67m9\" (UID: \"b244121d-33d8-46d2-aa87-b451301005c8\") " pod="openshift-ingress-canary/ingress-canary-b67m9"
Jul 28 17:23:05.393130 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.393100    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzxzc\" (UniqueName: \"kubernetes.io/projected/e17b0fe1-b406-4126-b193-f673a5846bd1-kube-api-access-hzxzc\") pod \"dns-default-tpb27\" (UID: \"e17b0fe1-b406-4126-b193-f673a5846bd1\") " pod="openshift-dns/dns-default-tpb27"
Jul 28 17:23:05.393369 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.393349    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpz6q\" (UniqueName: \"kubernetes.io/projected/b244121d-33d8-46d2-aa87-b451301005c8-kube-api-access-kpz6q\") pod \"ingress-canary-b67m9\" (UID: \"b244121d-33d8-46d2-aa87-b451301005c8\") " pod="openshift-ingress-canary/ingress-canary-b67m9"
Jul 28 17:23:05.446830 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.446797    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b67m9"
Jul 28 17:23:05.887848 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.887812    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e17b0fe1-b406-4126-b193-f673a5846bd1-metrics-tls\") pod \"dns-default-tpb27\" (UID: \"e17b0fe1-b406-4126-b193-f673a5846bd1\") " pod="openshift-dns/dns-default-tpb27"
Jul 28 17:23:05.888044 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:05.887991    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:23:05.888094 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:05.888070    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17b0fe1-b406-4126-b193-f673a5846bd1-metrics-tls podName:e17b0fe1-b406-4126-b193-f673a5846bd1 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:06.888050417 +0000 UTC m=+37.198215899 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e17b0fe1-b406-4126-b193-f673a5846bd1-metrics-tls") pod "dns-default-tpb27" (UID: "e17b0fe1-b406-4126-b193-f673a5846bd1") : secret "dns-default-metrics-tls" not found
Jul 28 17:23:05.901879 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.901843    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-566c6cdbb8-47rvj"]
Jul 28 17:23:05.905454 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.905428    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-67686b4db5-wtnp8"]
Jul 28 17:23:05.905607 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.905583    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-566c6cdbb8-47rvj"
Jul 28 17:23:05.908712 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.908692    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67686b4db5-wtnp8"
Jul 28 17:23:05.909621 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.909452    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-xqwvk\""
Jul 28 17:23:05.909621 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.909504    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 28 17:23:05.909621 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.909452    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 17:23:05.909621 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.909598    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 17:23:05.909879 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.909643    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 17:23:05.911709 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.911687    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 28 17:23:05.916052 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.916005    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-566c6cdbb8-47rvj"]
Jul 28 17:23:05.918548 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.918526    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-67686b4db5-wtnp8"]
Jul 28 17:23:05.958685 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.958652    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-599946f5cf-f22zd"]
Jul 28 17:23:05.962303 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.962278    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-599946f5cf-f22zd"
Jul 28 17:23:05.965149 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.965125    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 28 17:23:05.965266 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.965161    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 28 17:23:05.965376 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.965282    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 28 17:23:05.965376 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.965334    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 28 17:23:05.978771 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.978743    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-599946f5cf-f22zd"]
Jul 28 17:23:05.988878 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.988844    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/292404fc-32c0-4eed-b545-c82d91dd91c8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-566c6cdbb8-47rvj\" (UID: \"292404fc-32c0-4eed-b545-c82d91dd91c8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-566c6cdbb8-47rvj"
Jul 28 17:23:05.989027 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:05.988907    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj44m\" (UniqueName: \"kubernetes.io/projected/292404fc-32c0-4eed-b545-c82d91dd91c8-kube-api-access-wj44m\") pod \"managed-serviceaccount-addon-agent-566c6cdbb8-47rvj\" (UID: \"292404fc-32c0-4eed-b545-c82d91dd91c8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-566c6cdbb8-47rvj"
Jul 28 17:23:06.090309 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.090269    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d771ff91-6330-45c6-844a-7d5b62ded5f0-ca\") pod \"cluster-proxy-proxy-agent-599946f5cf-f22zd\" (UID: \"d771ff91-6330-45c6-844a-7d5b62ded5f0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-599946f5cf-f22zd"
Jul 28 17:23:06.090309 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.090312    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a31cbaae-07a7-4584-8b0e-a1b4aea43595-tmp\") pod \"klusterlet-addon-workmgr-67686b4db5-wtnp8\" (UID: \"a31cbaae-07a7-4584-8b0e-a1b4aea43595\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67686b4db5-wtnp8"
Jul 28 17:23:06.091036 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.090466    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d771ff91-6330-45c6-844a-7d5b62ded5f0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-599946f5cf-f22zd\" (UID: \"d771ff91-6330-45c6-844a-7d5b62ded5f0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-599946f5cf-f22zd"
Jul 28 17:23:06.091036 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.090494    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a31cbaae-07a7-4584-8b0e-a1b4aea43595-klusterlet-config\") pod \"klusterlet-addon-workmgr-67686b4db5-wtnp8\" (UID: \"a31cbaae-07a7-4584-8b0e-a1b4aea43595\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67686b4db5-wtnp8"
Jul 28 17:23:06.091036 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.090513    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm6xw\" (UniqueName: \"kubernetes.io/projected/a31cbaae-07a7-4584-8b0e-a1b4aea43595-kube-api-access-zm6xw\") pod \"klusterlet-addon-workmgr-67686b4db5-wtnp8\" (UID: \"a31cbaae-07a7-4584-8b0e-a1b4aea43595\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67686b4db5-wtnp8"
Jul 28 17:23:06.091036 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.090542    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/292404fc-32c0-4eed-b545-c82d91dd91c8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-566c6cdbb8-47rvj\" (UID: \"292404fc-32c0-4eed-b545-c82d91dd91c8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-566c6cdbb8-47rvj"
Jul 28 17:23:06.091036 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.090600    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj44m\" (UniqueName: \"kubernetes.io/projected/292404fc-32c0-4eed-b545-c82d91dd91c8-kube-api-access-wj44m\") pod \"managed-serviceaccount-addon-agent-566c6cdbb8-47rvj\" (UID: \"292404fc-32c0-4eed-b545-c82d91dd91c8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-566c6cdbb8-47rvj"
Jul 28 17:23:06.091036 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.090631    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d771ff91-6330-45c6-844a-7d5b62ded5f0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-599946f5cf-f22zd\" (UID: \"d771ff91-6330-45c6-844a-7d5b62ded5f0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-599946f5cf-f22zd"
Jul 28 17:23:06.091036 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.090670    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d771ff91-6330-45c6-844a-7d5b62ded5f0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-599946f5cf-f22zd\" (UID: \"d771ff91-6330-45c6-844a-7d5b62ded5f0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-599946f5cf-f22zd"
Jul 28 17:23:06.091036 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.090696    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8kjw\" (UniqueName: \"kubernetes.io/projected/d771ff91-6330-45c6-844a-7d5b62ded5f0-kube-api-access-d8kjw\") pod \"cluster-proxy-proxy-agent-599946f5cf-f22zd\" (UID: \"d771ff91-6330-45c6-844a-7d5b62ded5f0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-599946f5cf-f22zd"
Jul 28 17:23:06.091036 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.090746    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d771ff91-6330-45c6-844a-7d5b62ded5f0-hub\") pod \"cluster-proxy-proxy-agent-599946f5cf-f22zd\" (UID: \"d771ff91-6330-45c6-844a-7d5b62ded5f0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-599946f5cf-f22zd"
Jul 28 17:23:06.093407 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.093380    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/292404fc-32c0-4eed-b545-c82d91dd91c8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-566c6cdbb8-47rvj\" (UID: \"292404fc-32c0-4eed-b545-c82d91dd91c8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-566c6cdbb8-47rvj"
Jul 28 17:23:06.099797 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.099772    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj44m\" (UniqueName: \"kubernetes.io/projected/292404fc-32c0-4eed-b545-c82d91dd91c8-kube-api-access-wj44m\") pod \"managed-serviceaccount-addon-agent-566c6cdbb8-47rvj\" (UID: \"292404fc-32c0-4eed-b545-c82d91dd91c8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-566c6cdbb8-47rvj"
Jul 28 17:23:06.191831 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.191788    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d771ff91-6330-45c6-844a-7d5b62ded5f0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-599946f5cf-f22zd\" (UID: \"d771ff91-6330-45c6-844a-7d5b62ded5f0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-599946f5cf-f22zd"
Jul 28 17:23:06.191831 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.191836    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a31cbaae-07a7-4584-8b0e-a1b4aea43595-klusterlet-config\") pod \"klusterlet-addon-workmgr-67686b4db5-wtnp8\" (UID: \"a31cbaae-07a7-4584-8b0e-a1b4aea43595\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67686b4db5-wtnp8"
Jul 28 17:23:06.192102 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.191855    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm6xw\" (UniqueName: \"kubernetes.io/projected/a31cbaae-07a7-4584-8b0e-a1b4aea43595-kube-api-access-zm6xw\") pod \"klusterlet-addon-workmgr-67686b4db5-wtnp8\" (UID: \"a31cbaae-07a7-4584-8b0e-a1b4aea43595\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67686b4db5-wtnp8"
Jul 28 17:23:06.192102 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.191928    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d771ff91-6330-45c6-844a-7d5b62ded5f0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-599946f5cf-f22zd\" (UID: \"d771ff91-6330-45c6-844a-7d5b62ded5f0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-599946f5cf-f22zd"
Jul 28 17:23:06.192102 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.191981    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d771ff91-6330-45c6-844a-7d5b62ded5f0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-599946f5cf-f22zd\" (UID: \"d771ff91-6330-45c6-844a-7d5b62ded5f0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-599946f5cf-f22zd"
Jul 28 17:23:06.192102 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.192000    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8kjw\" (UniqueName: \"kubernetes.io/projected/d771ff91-6330-45c6-844a-7d5b62ded5f0-kube-api-access-d8kjw\") pod \"cluster-proxy-proxy-agent-599946f5cf-f22zd\" (UID: \"d771ff91-6330-45c6-844a-7d5b62ded5f0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-599946f5cf-f22zd"
Jul 28 17:23:06.192102 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.192094    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d771ff91-6330-45c6-844a-7d5b62ded5f0-hub\") pod \"cluster-proxy-proxy-agent-599946f5cf-f22zd\" (UID: \"d771ff91-6330-45c6-844a-7d5b62ded5f0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-599946f5cf-f22zd"
Jul 28 17:23:06.192365 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.192141    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d771ff91-6330-45c6-844a-7d5b62ded5f0-ca\") pod \"cluster-proxy-proxy-agent-599946f5cf-f22zd\" (UID: \"d771ff91-6330-45c6-844a-7d5b62ded5f0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-599946f5cf-f22zd"
Jul 28 17:23:06.192365 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.192171    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a31cbaae-07a7-4584-8b0e-a1b4aea43595-tmp\") pod \"klusterlet-addon-workmgr-67686b4db5-wtnp8\" (UID: \"a31cbaae-07a7-4584-8b0e-a1b4aea43595\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67686b4db5-wtnp8"
Jul 28 17:23:06.192563 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.192543    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a31cbaae-07a7-4584-8b0e-a1b4aea43595-tmp\") pod \"klusterlet-addon-workmgr-67686b4db5-wtnp8\" (UID: \"a31cbaae-07a7-4584-8b0e-a1b4aea43595\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67686b4db5-wtnp8"
Jul 28 17:23:06.192912 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.192862    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d771ff91-6330-45c6-844a-7d5b62ded5f0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-599946f5cf-f22zd\" (UID: \"d771ff91-6330-45c6-844a-7d5b62ded5f0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-599946f5cf-f22zd"
Jul 28 17:23:06.194848 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.194826    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a31cbaae-07a7-4584-8b0e-a1b4aea43595-klusterlet-config\") pod \"klusterlet-addon-workmgr-67686b4db5-wtnp8\" (UID: \"a31cbaae-07a7-4584-8b0e-a1b4aea43595\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67686b4db5-wtnp8"
Jul 28 17:23:06.194963 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.194926    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d771ff91-6330-45c6-844a-7d5b62ded5f0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-599946f5cf-f22zd\" (UID: \"d771ff91-6330-45c6-844a-7d5b62ded5f0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-599946f5cf-f22zd"
Jul 28 17:23:06.195091 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.195070    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d771ff91-6330-45c6-844a-7d5b62ded5f0-hub\") pod \"cluster-proxy-proxy-agent-599946f5cf-f22zd\" (UID: \"d771ff91-6330-45c6-844a-7d5b62ded5f0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-599946f5cf-f22zd"
Jul 28 17:23:06.195151 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.195118    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d771ff91-6330-45c6-844a-7d5b62ded5f0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-599946f5cf-f22zd\" (UID: \"d771ff91-6330-45c6-844a-7d5b62ded5f0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-599946f5cf-f22zd"
Jul 28 17:23:06.195206 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.195151    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d771ff91-6330-45c6-844a-7d5b62ded5f0-ca\") pod \"cluster-proxy-proxy-agent-599946f5cf-f22zd\" (UID: \"d771ff91-6330-45c6-844a-7d5b62ded5f0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-599946f5cf-f22zd"
Jul 28 17:23:06.203415 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.203391    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8kjw\" (UniqueName: \"kubernetes.io/projected/d771ff91-6330-45c6-844a-7d5b62ded5f0-kube-api-access-d8kjw\") pod \"cluster-proxy-proxy-agent-599946f5cf-f22zd\" (UID: \"d771ff91-6330-45c6-844a-7d5b62ded5f0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-599946f5cf-f22zd"
Jul 28 17:23:06.204156 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.204134    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm6xw\" (UniqueName: \"kubernetes.io/projected/a31cbaae-07a7-4584-8b0e-a1b4aea43595-kube-api-access-zm6xw\") pod \"klusterlet-addon-workmgr-67686b4db5-wtnp8\" (UID: \"a31cbaae-07a7-4584-8b0e-a1b4aea43595\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67686b4db5-wtnp8"
Jul 28 17:23:06.225155 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.225124    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-566c6cdbb8-47rvj"
Jul 28 17:23:06.231426 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.231402    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67686b4db5-wtnp8"
Jul 28 17:23:06.273268 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.273121    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-599946f5cf-f22zd"
Jul 28 17:23:06.508410 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.508229    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b67m9"]
Jul 28 17:23:06.513313 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.512216    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-566c6cdbb8-47rvj"]
Jul 28 17:23:06.513313 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.513244    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-599946f5cf-f22zd"]
Jul 28 17:23:06.520144 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.520076    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-67686b4db5-wtnp8"]
Jul 28 17:23:06.897723 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:06.897638    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e17b0fe1-b406-4126-b193-f673a5846bd1-metrics-tls\") pod \"dns-default-tpb27\" (UID: \"e17b0fe1-b406-4126-b193-f673a5846bd1\") " pod="openshift-dns/dns-default-tpb27"
Jul 28 17:23:06.897873 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:06.897809    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:23:06.897919 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:06.897879    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17b0fe1-b406-4126-b193-f673a5846bd1-metrics-tls podName:e17b0fe1-b406-4126-b193-f673a5846bd1 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:08.897862336 +0000 UTC m=+39.208027818 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e17b0fe1-b406-4126-b193-f673a5846bd1-metrics-tls") pod "dns-default-tpb27" (UID: "e17b0fe1-b406-4126-b193-f673a5846bd1") : secret "dns-default-metrics-tls" not found
Jul 28 17:23:07.280659 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:07.280614    2566 generic.go:356] "Generic (PLEG): container finished" podID="15fc65f5-e122-489d-b56c-1999d101223f" containerID="f106d7cde5d87b5d7a52fb26b88c1ce4aa5832da7f9b417ab6f39a0d73532e89" exitCode=0
Jul 28 17:23:07.281132 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:07.280682    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-48d52" event={"ID":"15fc65f5-e122-489d-b56c-1999d101223f","Type":"ContainerDied","Data":"f106d7cde5d87b5d7a52fb26b88c1ce4aa5832da7f9b417ab6f39a0d73532e89"}
Jul 28 17:23:07.282097 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:07.281957    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-599946f5cf-f22zd" event={"ID":"d771ff91-6330-45c6-844a-7d5b62ded5f0","Type":"ContainerStarted","Data":"065e909764e082cdad35b4957894734cbd219289390c1d9f59a6a2e7873f6b67"}
Jul 28 17:23:07.283286 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:07.283244    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-566c6cdbb8-47rvj" event={"ID":"292404fc-32c0-4eed-b545-c82d91dd91c8","Type":"ContainerStarted","Data":"dd4217b00d9da26e688a8207886083d1c52219d71cb64031e59a537356bc43f8"}
Jul 28 17:23:07.284505 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:07.284480    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b67m9" event={"ID":"b244121d-33d8-46d2-aa87-b451301005c8","Type":"ContainerStarted","Data":"f8c5b9718986ea3b56dd0457e3a1c0be282aba8d5fe7293a26f9355885a51789"}
Jul 28 17:23:07.285732 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:07.285703    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67686b4db5-wtnp8" event={"ID":"a31cbaae-07a7-4584-8b0e-a1b4aea43595","Type":"ContainerStarted","Data":"efe69056abe16a4c8466ab812431a09e92b62aec821816664034d816d26b3e7c"}
Jul 28 17:23:08.299152 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:08.299093    2566 generic.go:356] "Generic (PLEG): container finished" podID="15fc65f5-e122-489d-b56c-1999d101223f" containerID="ae02267aa115e8bd41ebcf5bc90f9981cef51a7365bdafbafdb978bb9f1dd873" exitCode=0
Jul 28 17:23:08.299152 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:08.299153    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-48d52" event={"ID":"15fc65f5-e122-489d-b56c-1999d101223f","Type":"ContainerDied","Data":"ae02267aa115e8bd41ebcf5bc90f9981cef51a7365bdafbafdb978bb9f1dd873"}
Jul 28 17:23:08.919568 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:08.919311    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e17b0fe1-b406-4126-b193-f673a5846bd1-metrics-tls\") pod \"dns-default-tpb27\" (UID: \"e17b0fe1-b406-4126-b193-f673a5846bd1\") " pod="openshift-dns/dns-default-tpb27"
Jul 28 17:23:08.919736 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:08.919690    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:23:08.919793 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:08.919756    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17b0fe1-b406-4126-b193-f673a5846bd1-metrics-tls podName:e17b0fe1-b406-4126-b193-f673a5846bd1 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:12.919736908 +0000 UTC m=+43.229902394 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e17b0fe1-b406-4126-b193-f673a5846bd1-metrics-tls") pod "dns-default-tpb27" (UID: "e17b0fe1-b406-4126-b193-f673a5846bd1") : secret "dns-default-metrics-tls" not found
Jul 28 17:23:09.308199 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:09.308131    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-48d52" event={"ID":"15fc65f5-e122-489d-b56c-1999d101223f","Type":"ContainerStarted","Data":"f53d9b5541e0a5b6e49b95724af6115801375352c08cdf3bec287227cc6bfb99"}
Jul 28 17:23:09.336676 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:09.336396    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-48d52" podStartSLOduration=9.876972305 podStartE2EDuration="39.336374661s" totalImagesPullingTime="29.459402356s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:22:30 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:09.334035311 +0000 UTC m=+39.644200820" watchObservedRunningTime="2026-07-28 17:23:09.336374661 +0000 UTC m=+39.646540169"
Jul 28 17:23:12.951947 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:12.951906    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e17b0fe1-b406-4126-b193-f673a5846bd1-metrics-tls\") pod \"dns-default-tpb27\" (UID: \"e17b0fe1-b406-4126-b193-f673a5846bd1\") " pod="openshift-dns/dns-default-tpb27"
Jul 28 17:23:12.952285 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:12.952023    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:23:12.952285 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:12.952081    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17b0fe1-b406-4126-b193-f673a5846bd1-metrics-tls podName:e17b0fe1-b406-4126-b193-f673a5846bd1 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:20.952069046 +0000 UTC m=+51.262234528 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e17b0fe1-b406-4126-b193-f673a5846bd1-metrics-tls") pod "dns-default-tpb27" (UID: "e17b0fe1-b406-4126-b193-f673a5846bd1") : secret "dns-default-metrics-tls" not found
Jul 28 17:23:15.323744 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:15.323661    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-566c6cdbb8-47rvj" event={"ID":"292404fc-32c0-4eed-b545-c82d91dd91c8","Type":"ContainerStarted","Data":"8606b91710c925c5be38d62e730e937ff44f51f3dae443b0bc4a90fe232eb29c"}
Jul 28 17:23:15.325077 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:15.325055    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b67m9" event={"ID":"b244121d-33d8-46d2-aa87-b451301005c8","Type":"ContainerStarted","Data":"75bde929d563de8af40ff38332c91c0e4b41712cb2e0211e6cf9d7ab6d411983"}
Jul 28 17:23:15.326478 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:15.326449    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67686b4db5-wtnp8" event={"ID":"a31cbaae-07a7-4584-8b0e-a1b4aea43595","Type":"ContainerStarted","Data":"24e3bd36a2998a4e70655b93b91e1608fcbf34f293ea6a509e4157d5bffde214"}
Jul 28 17:23:15.326560 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:15.326504    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67686b4db5-wtnp8"
Jul 28 17:23:15.328376 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:15.328356    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67686b4db5-wtnp8"
Jul 28 17:23:15.328497 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:15.328479    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-599946f5cf-f22zd" event={"ID":"d771ff91-6330-45c6-844a-7d5b62ded5f0","Type":"ContainerStarted","Data":"cf64a71a34035688e3dc60f041737acbece6640c1b6bc3ff50a9a2240116d032"}
Jul 28 17:23:15.328542 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:15.328508    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-599946f5cf-f22zd" event={"ID":"d771ff91-6330-45c6-844a-7d5b62ded5f0","Type":"ContainerStarted","Data":"99bdbe0dcc9b52ffa18d396331a4e1f70db9730322cc6d4516c6a79efeebafa4"}
Jul 28 17:23:15.328542 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:15.328520    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-599946f5cf-f22zd" event={"ID":"d771ff91-6330-45c6-844a-7d5b62ded5f0","Type":"ContainerStarted","Data":"3190cfb4eed435ea803bd796aafc4e0266b712fac8e612972c2b327bc474e316"}
Jul 28 17:23:15.340735 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:15.340678    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-566c6cdbb8-47rvj" podStartSLOduration=2.507649774 podStartE2EDuration="10.340665136s" totalImagesPullingTime="7.833015362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:15.340233421 +0000 UTC m=+45.650398925" watchObservedRunningTime="2026-07-28 17:23:15.340665136 +0000 UTC m=+45.650830634"
Jul 28 17:23:15.359143 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:15.359099    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b67m9" podStartSLOduration=2.526026157 podStartE2EDuration="10.359086437s" totalImagesPullingTime="7.83306028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:15.358805581 +0000 UTC m=+45.668971085" watchObservedRunningTime="2026-07-28 17:23:15.359086437 +0000 UTC m=+45.669251944"
Jul 28 17:23:15.433842 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:15.433797    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-599946f5cf-f22zd" podStartSLOduration=2.242440386 podStartE2EDuration="10.433780902s" totalImagesPullingTime="8.191340516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:15.379274293 +0000 UTC m=+45.689439797" watchObservedRunningTime="2026-07-28 17:23:15.433780902 +0000 UTC m=+45.743946442"
Jul 28 17:23:15.434134 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:15.434106    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67686b4db5-wtnp8" podStartSLOduration=2.6172912 podStartE2EDuration="10.434100202s" totalImagesPullingTime="7.816809002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:15.433595093 +0000 UTC m=+45.743760597" watchObservedRunningTime="2026-07-28 17:23:15.434100202 +0000 UTC m=+45.744265707"
Jul 28 17:23:15.501920 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:15.501890    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 17:23:16.088787 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:16.088682    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T17:23:15.501908958Z","UUID":"5137714d-176e-4fb9-b465-baadd6d98101","Handler":null,"Name":"","Endpoint":""}
Jul 28 17:23:16.090309 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:16.090287    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 17:23:16.090438 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:16.090347    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 17:23:16.334440 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:16.334393    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jv7m" event={"ID":"22ea97a4-e795-4d89-8257-fe0baf30b83e","Type":"ContainerStarted","Data":"9b09a7d8e82124d244c0fe91933bd943cd367444920c91e2a910ca2ed7e0a414"}
Jul 28 17:23:16.355718 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:16.355634    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jv7m" podStartSLOduration=4.228048726 podStartE2EDuration="46.35561904s" totalImagesPullingTime="42.127570314s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:22:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-28 17:23:16.354374018 +0000 UTC m=+46.664539535" watchObservedRunningTime="2026-07-28 17:23:16.35561904 +0000 UTC m=+46.665784545"
Jul 28 17:23:21.013832 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:21.013793    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e17b0fe1-b406-4126-b193-f673a5846bd1-metrics-tls\") pod \"dns-default-tpb27\" (UID: \"e17b0fe1-b406-4126-b193-f673a5846bd1\") " pod="openshift-dns/dns-default-tpb27"
Jul 28 17:23:21.014207 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:21.013940    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:23:21.014207 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:21.014004    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17b0fe1-b406-4126-b193-f673a5846bd1-metrics-tls podName:e17b0fe1-b406-4126-b193-f673a5846bd1 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:37.013988042 +0000 UTC m=+67.324153529 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e17b0fe1-b406-4126-b193-f673a5846bd1-metrics-tls") pod "dns-default-tpb27" (UID: "e17b0fe1-b406-4126-b193-f673a5846bd1") : secret "dns-default-metrics-tls" not found
Jul 28 17:23:31.274126 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:31.274094    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5sdnx"
Jul 28 17:23:35.926480 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:35.926432    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b881ad9-ece5-40ab-b106-32e6d520c37e-original-pull-secret\") pod \"global-pull-secret-syncer-dpv42\" (UID: \"3b881ad9-ece5-40ab-b106-32e6d520c37e\") " pod="kube-system/global-pull-secret-syncer-dpv42"
Jul 28 17:23:35.926954 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:35.926507    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dedc8b78-7469-4e8d-a7b8-b11c741186d7-metrics-certs\") pod \"network-metrics-daemon-znptx\" (UID: \"dedc8b78-7469-4e8d-a7b8-b11c741186d7\") " pod="openshift-multus/network-metrics-daemon-znptx"
Jul 28 17:23:35.929315 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:35.929289    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:23:35.929414 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:35.929352    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:23:35.936789 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:35.936765    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 28 17:23:35.936888 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:35.936853    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dedc8b78-7469-4e8d-a7b8-b11c741186d7-metrics-certs podName:dedc8b78-7469-4e8d-a7b8-b11c741186d7 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:39.936829152 +0000 UTC m=+130.246994652 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dedc8b78-7469-4e8d-a7b8-b11c741186d7-metrics-certs") pod "network-metrics-daemon-znptx" (UID: "dedc8b78-7469-4e8d-a7b8-b11c741186d7") : secret "metrics-daemon-secret" not found
Jul 28 17:23:35.940553 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:35.940533    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b881ad9-ece5-40ab-b106-32e6d520c37e-original-pull-secret\") pod \"global-pull-secret-syncer-dpv42\" (UID: \"3b881ad9-ece5-40ab-b106-32e6d520c37e\") " pod="kube-system/global-pull-secret-syncer-dpv42"
Jul 28 17:23:36.007099 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:36.007060    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpv42"
Jul 28 17:23:36.027104 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:36.027077    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfwf2\" (UniqueName: \"kubernetes.io/projected/a743f0d2-5033-4955-bbca-1e7327255603-kube-api-access-lfwf2\") pod \"network-check-target-rvrql\" (UID: \"a743f0d2-5033-4955-bbca-1e7327255603\") " pod="openshift-network-diagnostics/network-check-target-rvrql"
Jul 28 17:23:36.029964 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:36.029940    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 17:23:36.040482 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:36.040459    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 17:23:36.050921 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:36.050897    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfwf2\" (UniqueName: \"kubernetes.io/projected/a743f0d2-5033-4955-bbca-1e7327255603-kube-api-access-lfwf2\") pod \"network-check-target-rvrql\" (UID: \"a743f0d2-5033-4955-bbca-1e7327255603\") " pod="openshift-network-diagnostics/network-check-target-rvrql"
Jul 28 17:23:36.129765 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:36.129735    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dpv42"]
Jul 28 17:23:36.132904 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:23:36.132872    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b881ad9_ece5_40ab_b106_32e6d520c37e.slice/crio-b64c06ed63e4b1b45574b8aec143b5f94526dfa48a72e506af61a95a63f81d31 WatchSource:0}: Error finding container b64c06ed63e4b1b45574b8aec143b5f94526dfa48a72e506af61a95a63f81d31: Status 404 returned error can't find the container with id b64c06ed63e4b1b45574b8aec143b5f94526dfa48a72e506af61a95a63f81d31
Jul 28 17:23:36.319799 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:36.319768    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-27qqv\""
Jul 28 17:23:36.327837 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:36.327812    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvrql"
Jul 28 17:23:36.395684 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:36.395656    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dpv42" event={"ID":"3b881ad9-ece5-40ab-b106-32e6d520c37e","Type":"ContainerStarted","Data":"b64c06ed63e4b1b45574b8aec143b5f94526dfa48a72e506af61a95a63f81d31"}
Jul 28 17:23:36.447431 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:36.447396    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rvrql"]
Jul 28 17:23:36.451082 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:23:36.451054    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda743f0d2_5033_4955_bbca_1e7327255603.slice/crio-38ea04a60e570750a4fdd07d5511d2c08896c9092b026854dc49fc8f85a9fd69 WatchSource:0}: Error finding container 38ea04a60e570750a4fdd07d5511d2c08896c9092b026854dc49fc8f85a9fd69: Status 404 returned error can't find the container with id 38ea04a60e570750a4fdd07d5511d2c08896c9092b026854dc49fc8f85a9fd69
Jul 28 17:23:37.033977 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:37.033940    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e17b0fe1-b406-4126-b193-f673a5846bd1-metrics-tls\") pod \"dns-default-tpb27\" (UID: \"e17b0fe1-b406-4126-b193-f673a5846bd1\") " pod="openshift-dns/dns-default-tpb27"
Jul 28 17:23:37.034408 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:37.034104    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:23:37.034408 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:37.034178    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17b0fe1-b406-4126-b193-f673a5846bd1-metrics-tls podName:e17b0fe1-b406-4126-b193-f673a5846bd1 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:09.034158529 +0000 UTC m=+99.344324015 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e17b0fe1-b406-4126-b193-f673a5846bd1-metrics-tls") pod "dns-default-tpb27" (UID: "e17b0fe1-b406-4126-b193-f673a5846bd1") : secret "dns-default-metrics-tls" not found
Jul 28 17:23:37.401031 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:37.400938    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rvrql" event={"ID":"a743f0d2-5033-4955-bbca-1e7327255603","Type":"ContainerStarted","Data":"38ea04a60e570750a4fdd07d5511d2c08896c9092b026854dc49fc8f85a9fd69"}
Jul 28 17:23:42.422109 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:42.422073    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rvrql" event={"ID":"a743f0d2-5033-4955-bbca-1e7327255603","Type":"ContainerStarted","Data":"2532c154a7f1fbd73f41b66853fae89bc8c096d1c26c31cff400d2793a7d6482"}
Jul 28 17:23:42.422626 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:42.422196    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rvrql"
Jul 28 17:23:42.423501 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:42.423481    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dpv42" event={"ID":"3b881ad9-ece5-40ab-b106-32e6d520c37e","Type":"ContainerStarted","Data":"500773bc6ac55c5b1ab7e5422cf163456292b4b69edb7c70dfe72c71baa4cd3f"}
Jul 28 17:23:42.440121 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:42.440081    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rvrql" podStartSLOduration=67.542650725 podStartE2EDuration="1m12.440068346s" totalImagesPullingTime="4.897417621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:22:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:42.438649448 +0000 UTC m=+72.748814977" watchObservedRunningTime="2026-07-28 17:23:42.440068346 +0000 UTC m=+72.750233851"
Jul 28 17:23:42.457250 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:42.454679    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dpv42" podStartSLOduration=67.227991451 podStartE2EDuration="1m12.454660073s" totalImagesPullingTime="5.226668622s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:22:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:42.454209892 +0000 UTC m=+72.764375400" watchObservedRunningTime="2026-07-28 17:23:42.454660073 +0000 UTC m=+72.764825580"
Jul 28 17:23:58.753930 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:58.753784    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-79zvc"]
Jul 28 17:23:58.758375 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:58.758354    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-79zvc"
Jul 28 17:23:58.761191 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:58.761155    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 17:23:58.761299 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:58.761198    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 17:23:58.761299 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:58.761161    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 17:23:58.761438 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:58.761160    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 17:23:58.762480 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:58.762463    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hw8gz\""
Jul 28 17:23:58.767870 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:58.767850    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-79zvc"]
Jul 28 17:23:58.893955 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:58.893918    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfxjc\" (UniqueName: \"kubernetes.io/projected/3a9203ae-f509-43f1-ae1e-2841014548a4-kube-api-access-jfxjc\") pod \"insights-runtime-extractor-79zvc\" (UID: \"3a9203ae-f509-43f1-ae1e-2841014548a4\") " pod="openshift-insights/insights-runtime-extractor-79zvc"
Jul 28 17:23:58.894103 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:58.893978    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3a9203ae-f509-43f1-ae1e-2841014548a4-crio-socket\") pod \"insights-runtime-extractor-79zvc\" (UID: \"3a9203ae-f509-43f1-ae1e-2841014548a4\") " pod="openshift-insights/insights-runtime-extractor-79zvc"
Jul 28 17:23:58.894103 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:58.894030    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a9203ae-f509-43f1-ae1e-2841014548a4-data-volume\") pod \"insights-runtime-extractor-79zvc\" (UID: \"3a9203ae-f509-43f1-ae1e-2841014548a4\") " pod="openshift-insights/insights-runtime-extractor-79zvc"
Jul 28 17:23:58.894103 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:58.894076    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3a9203ae-f509-43f1-ae1e-2841014548a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-79zvc\" (UID: \"3a9203ae-f509-43f1-ae1e-2841014548a4\") " pod="openshift-insights/insights-runtime-extractor-79zvc"
Jul 28 17:23:58.894281 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:58.894108    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3a9203ae-f509-43f1-ae1e-2841014548a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-79zvc\" (UID: \"3a9203ae-f509-43f1-ae1e-2841014548a4\") " pod="openshift-insights/insights-runtime-extractor-79zvc"
Jul 28 17:23:58.994618 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:58.994568    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3a9203ae-f509-43f1-ae1e-2841014548a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-79zvc\" (UID: \"3a9203ae-f509-43f1-ae1e-2841014548a4\") " pod="openshift-insights/insights-runtime-extractor-79zvc"
Jul 28 17:23:58.994806 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:58.994643    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfxjc\" (UniqueName: \"kubernetes.io/projected/3a9203ae-f509-43f1-ae1e-2841014548a4-kube-api-access-jfxjc\") pod \"insights-runtime-extractor-79zvc\" (UID: \"3a9203ae-f509-43f1-ae1e-2841014548a4\") " pod="openshift-insights/insights-runtime-extractor-79zvc"
Jul 28 17:23:58.994806 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:58.994695    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3a9203ae-f509-43f1-ae1e-2841014548a4-crio-socket\") pod \"insights-runtime-extractor-79zvc\" (UID: \"3a9203ae-f509-43f1-ae1e-2841014548a4\") " pod="openshift-insights/insights-runtime-extractor-79zvc"
Jul 28 17:23:58.994806 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:58.994722    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:23:58.994806 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:58.994725    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a9203ae-f509-43f1-ae1e-2841014548a4-data-volume\") pod \"insights-runtime-extractor-79zvc\" (UID: \"3a9203ae-f509-43f1-ae1e-2841014548a4\") " pod="openshift-insights/insights-runtime-extractor-79zvc"
Jul 28 17:23:58.995018 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:58.994814    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a9203ae-f509-43f1-ae1e-2841014548a4-insights-runtime-extractor-tls podName:3a9203ae-f509-43f1-ae1e-2841014548a4 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:59.494794402 +0000 UTC m=+89.804959888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3a9203ae-f509-43f1-ae1e-2841014548a4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-79zvc" (UID: "3a9203ae-f509-43f1-ae1e-2841014548a4") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:23:58.995018 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:58.994843    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3a9203ae-f509-43f1-ae1e-2841014548a4-crio-socket\") pod \"insights-runtime-extractor-79zvc\" (UID: \"3a9203ae-f509-43f1-ae1e-2841014548a4\") " pod="openshift-insights/insights-runtime-extractor-79zvc"
Jul 28 17:23:58.995018 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:58.994858    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3a9203ae-f509-43f1-ae1e-2841014548a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-79zvc\" (UID: \"3a9203ae-f509-43f1-ae1e-2841014548a4\") " pod="openshift-insights/insights-runtime-extractor-79zvc"
Jul 28 17:23:58.995018 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:58.994917    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:23:58.995018 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:58.994953    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a9203ae-f509-43f1-ae1e-2841014548a4-kube-rbac-proxy-cm podName:3a9203ae-f509-43f1-ae1e-2841014548a4 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:59.494944688 +0000 UTC m=+89.805110171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3a9203ae-f509-43f1-ae1e-2841014548a4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-79zvc" (UID: "3a9203ae-f509-43f1-ae1e-2841014548a4") : configmap "kube-rbac-proxy" not found
Jul 28 17:23:58.995206 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:58.995050    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a9203ae-f509-43f1-ae1e-2841014548a4-data-volume\") pod \"insights-runtime-extractor-79zvc\" (UID: \"3a9203ae-f509-43f1-ae1e-2841014548a4\") " pod="openshift-insights/insights-runtime-extractor-79zvc"
Jul 28 17:23:59.009466 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:59.009403    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfxjc\" (UniqueName: \"kubernetes.io/projected/3a9203ae-f509-43f1-ae1e-2841014548a4-kube-api-access-jfxjc\") pod \"insights-runtime-extractor-79zvc\" (UID: \"3a9203ae-f509-43f1-ae1e-2841014548a4\") " pod="openshift-insights/insights-runtime-extractor-79zvc"
Jul 28 17:23:59.497915 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:59.497877    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3a9203ae-f509-43f1-ae1e-2841014548a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-79zvc\" (UID: \"3a9203ae-f509-43f1-ae1e-2841014548a4\") " pod="openshift-insights/insights-runtime-extractor-79zvc"
Jul 28 17:23:59.498126 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:23:59.497990    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3a9203ae-f509-43f1-ae1e-2841014548a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-79zvc\" (UID: \"3a9203ae-f509-43f1-ae1e-2841014548a4\") " pod="openshift-insights/insights-runtime-extractor-79zvc"
Jul 28 17:23:59.498126 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:59.498027    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:23:59.498126 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:59.498066    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:23:59.498126 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:59.498098    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a9203ae-f509-43f1-ae1e-2841014548a4-insights-runtime-extractor-tls podName:3a9203ae-f509-43f1-ae1e-2841014548a4 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:00.498082516 +0000 UTC m=+90.808247998 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3a9203ae-f509-43f1-ae1e-2841014548a4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-79zvc" (UID: "3a9203ae-f509-43f1-ae1e-2841014548a4") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:23:59.498126 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:23:59.498125    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a9203ae-f509-43f1-ae1e-2841014548a4-kube-rbac-proxy-cm podName:3a9203ae-f509-43f1-ae1e-2841014548a4 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:00.498110452 +0000 UTC m=+90.808275940 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3a9203ae-f509-43f1-ae1e-2841014548a4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-79zvc" (UID: "3a9203ae-f509-43f1-ae1e-2841014548a4") : configmap "kube-rbac-proxy" not found
Jul 28 17:24:00.505545 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:00.505512    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3a9203ae-f509-43f1-ae1e-2841014548a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-79zvc\" (UID: \"3a9203ae-f509-43f1-ae1e-2841014548a4\") " pod="openshift-insights/insights-runtime-extractor-79zvc"
Jul 28 17:24:00.505545 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:00.505552    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3a9203ae-f509-43f1-ae1e-2841014548a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-79zvc\" (UID: \"3a9203ae-f509-43f1-ae1e-2841014548a4\") " pod="openshift-insights/insights-runtime-extractor-79zvc"
Jul 28 17:24:00.505958 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:24:00.505637    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:24:00.505958 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:24:00.505665    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:24:00.505958 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:24:00.505706    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a9203ae-f509-43f1-ae1e-2841014548a4-kube-rbac-proxy-cm podName:3a9203ae-f509-43f1-ae1e-2841014548a4 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:02.50568852 +0000 UTC m=+92.815854015 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3a9203ae-f509-43f1-ae1e-2841014548a4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-79zvc" (UID: "3a9203ae-f509-43f1-ae1e-2841014548a4") : configmap "kube-rbac-proxy" not found
Jul 28 17:24:00.505958 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:24:00.505722    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a9203ae-f509-43f1-ae1e-2841014548a4-insights-runtime-extractor-tls podName:3a9203ae-f509-43f1-ae1e-2841014548a4 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:02.505715335 +0000 UTC m=+92.815880823 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3a9203ae-f509-43f1-ae1e-2841014548a4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-79zvc" (UID: "3a9203ae-f509-43f1-ae1e-2841014548a4") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:24:00.744138 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:00.744109    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s7j5k_091666fc-49ce-46bc-9a7d-5815676ea342/dns-node-resolver/0.log"
Jul 28 17:24:01.544146 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:01.544120    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5bhch_630e1b18-a044-454e-9be4-359de1817e2b/node-ca/0.log"
Jul 28 17:24:02.343194 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:02.343162    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b67m9_b244121d-33d8-46d2-aa87-b451301005c8/serve-healthcheck-canary/0.log"
Jul 28 17:24:02.521630 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:02.521586    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3a9203ae-f509-43f1-ae1e-2841014548a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-79zvc\" (UID: \"3a9203ae-f509-43f1-ae1e-2841014548a4\") " pod="openshift-insights/insights-runtime-extractor-79zvc"
Jul 28 17:24:02.521630 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:02.521635    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3a9203ae-f509-43f1-ae1e-2841014548a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-79zvc\" (UID: \"3a9203ae-f509-43f1-ae1e-2841014548a4\") " pod="openshift-insights/insights-runtime-extractor-79zvc"
Jul 28 17:24:02.521884 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:24:02.521713    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:24:02.521884 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:24:02.521741    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:24:02.521884 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:24:02.521784    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a9203ae-f509-43f1-ae1e-2841014548a4-kube-rbac-proxy-cm podName:3a9203ae-f509-43f1-ae1e-2841014548a4 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:06.521767605 +0000 UTC m=+96.831933090 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3a9203ae-f509-43f1-ae1e-2841014548a4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-79zvc" (UID: "3a9203ae-f509-43f1-ae1e-2841014548a4") : configmap "kube-rbac-proxy" not found
Jul 28 17:24:02.521884 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:24:02.521806    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a9203ae-f509-43f1-ae1e-2841014548a4-insights-runtime-extractor-tls podName:3a9203ae-f509-43f1-ae1e-2841014548a4 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:06.521790957 +0000 UTC m=+96.831956442 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3a9203ae-f509-43f1-ae1e-2841014548a4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-79zvc" (UID: "3a9203ae-f509-43f1-ae1e-2841014548a4") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:24:06.549884 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:06.549847    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3a9203ae-f509-43f1-ae1e-2841014548a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-79zvc\" (UID: \"3a9203ae-f509-43f1-ae1e-2841014548a4\") " pod="openshift-insights/insights-runtime-extractor-79zvc"
Jul 28 17:24:06.549884 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:06.549892    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3a9203ae-f509-43f1-ae1e-2841014548a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-79zvc\" (UID: \"3a9203ae-f509-43f1-ae1e-2841014548a4\") " pod="openshift-insights/insights-runtime-extractor-79zvc"
Jul 28 17:24:06.550419 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:24:06.549931    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:24:06.550419 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:24:06.550009    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a9203ae-f509-43f1-ae1e-2841014548a4-kube-rbac-proxy-cm podName:3a9203ae-f509-43f1-ae1e-2841014548a4 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:14.549990102 +0000 UTC m=+104.860155608 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3a9203ae-f509-43f1-ae1e-2841014548a4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-79zvc" (UID: "3a9203ae-f509-43f1-ae1e-2841014548a4") : configmap "kube-rbac-proxy" not found
Jul 28 17:24:06.550419 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:24:06.550029    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:24:06.550419 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:24:06.550068    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a9203ae-f509-43f1-ae1e-2841014548a4-insights-runtime-extractor-tls podName:3a9203ae-f509-43f1-ae1e-2841014548a4 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:14.550056363 +0000 UTC m=+104.860221854 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3a9203ae-f509-43f1-ae1e-2841014548a4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-79zvc" (UID: "3a9203ae-f509-43f1-ae1e-2841014548a4") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:24:09.071622 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:09.071586    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e17b0fe1-b406-4126-b193-f673a5846bd1-metrics-tls\") pod \"dns-default-tpb27\" (UID: \"e17b0fe1-b406-4126-b193-f673a5846bd1\") " pod="openshift-dns/dns-default-tpb27"
Jul 28 17:24:09.073836 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:09.073806    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e17b0fe1-b406-4126-b193-f673a5846bd1-metrics-tls\") pod \"dns-default-tpb27\" (UID: \"e17b0fe1-b406-4126-b193-f673a5846bd1\") " pod="openshift-dns/dns-default-tpb27"
Jul 28 17:24:09.344172 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:09.344143    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fqthr\""
Jul 28 17:24:09.351657 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:09.351634    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tpb27"
Jul 28 17:24:09.480102 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:09.480072    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tpb27"]
Jul 28 17:24:09.485658 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:24:09.485618    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode17b0fe1_b406_4126_b193_f673a5846bd1.slice/crio-2dad6e5ff6437aa309c242f4c8ba9cb442df13a6b1c6c3bc5f3482be30050690 WatchSource:0}: Error finding container 2dad6e5ff6437aa309c242f4c8ba9cb442df13a6b1c6c3bc5f3482be30050690: Status 404 returned error can't find the container with id 2dad6e5ff6437aa309c242f4c8ba9cb442df13a6b1c6c3bc5f3482be30050690
Jul 28 17:24:09.508238 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:09.508205    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tpb27" event={"ID":"e17b0fe1-b406-4126-b193-f673a5846bd1","Type":"ContainerStarted","Data":"2dad6e5ff6437aa309c242f4c8ba9cb442df13a6b1c6c3bc5f3482be30050690"}
Jul 28 17:24:11.516674 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:11.516639    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tpb27" event={"ID":"e17b0fe1-b406-4126-b193-f673a5846bd1","Type":"ContainerStarted","Data":"b60803fc1a587ad9356422a3dd2371377a320ca62ff241e731301f0e1d130056"}
Jul 28 17:24:12.524518 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:12.524486    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tpb27" event={"ID":"e17b0fe1-b406-4126-b193-f673a5846bd1","Type":"ContainerStarted","Data":"9a118b9d3c6be94e1574ec1c59f83613bb17aea873f357d6165b92d79e606283"}
Jul 28 17:24:12.524926 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:12.524642    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tpb27"
Jul 28 17:24:12.545055 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:12.545006    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tpb27" podStartSLOduration=65.618439808 podStartE2EDuration="1m7.544990231s" totalImagesPullingTime="1.926550423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:12.544156529 +0000 UTC m=+102.854322047" watchObservedRunningTime="2026-07-28 17:24:12.544990231 +0000 UTC m=+102.855155737"
Jul 28 17:24:13.428704 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:13.428675    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rvrql"
Jul 28 17:24:14.619836 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:14.619804    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3a9203ae-f509-43f1-ae1e-2841014548a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-79zvc\" (UID: \"3a9203ae-f509-43f1-ae1e-2841014548a4\") " pod="openshift-insights/insights-runtime-extractor-79zvc"
Jul 28 17:24:14.619836 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:14.619846    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3a9203ae-f509-43f1-ae1e-2841014548a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-79zvc\" (UID: \"3a9203ae-f509-43f1-ae1e-2841014548a4\") " pod="openshift-insights/insights-runtime-extractor-79zvc"
Jul 28 17:24:14.620514 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:14.620490    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3a9203ae-f509-43f1-ae1e-2841014548a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-79zvc\" (UID: \"3a9203ae-f509-43f1-ae1e-2841014548a4\") " pod="openshift-insights/insights-runtime-extractor-79zvc"
Jul 28 17:24:14.622097 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:14.622079    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3a9203ae-f509-43f1-ae1e-2841014548a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-79zvc\" (UID: \"3a9203ae-f509-43f1-ae1e-2841014548a4\") " pod="openshift-insights/insights-runtime-extractor-79zvc"
Jul 28 17:24:14.666824 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:14.666791    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-79zvc"
Jul 28 17:24:14.831371 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:14.831336    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-79zvc"]
Jul 28 17:24:14.834731 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:24:14.834698    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a9203ae_f509_43f1_ae1e_2841014548a4.slice/crio-5783d8667174434e6e4393e71b08f42999ffa71c67f81b76f10e235caff1134e WatchSource:0}: Error finding container 5783d8667174434e6e4393e71b08f42999ffa71c67f81b76f10e235caff1134e: Status 404 returned error can't find the container with id 5783d8667174434e6e4393e71b08f42999ffa71c67f81b76f10e235caff1134e
Jul 28 17:24:15.535840 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:15.535780    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-79zvc" event={"ID":"3a9203ae-f509-43f1-ae1e-2841014548a4","Type":"ContainerStarted","Data":"5783d8667174434e6e4393e71b08f42999ffa71c67f81b76f10e235caff1134e"}
Jul 28 17:24:20.554177 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:20.554137    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-79zvc" event={"ID":"3a9203ae-f509-43f1-ae1e-2841014548a4","Type":"ContainerStarted","Data":"d102abd3d8d52b1d34981ed4efb892a11984c23827cc83593f24df2001256073"}
Jul 28 17:24:22.529866 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:22.529828    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tpb27"
Jul 28 17:24:24.570908 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:24.570873    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-79zvc" event={"ID":"3a9203ae-f509-43f1-ae1e-2841014548a4","Type":"ContainerStarted","Data":"ec426e464df317aedd492b26414368ff4ed946315370a21e3b16d34202398964"}
Jul 28 17:24:27.587507 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:27.587467    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-79zvc" event={"ID":"3a9203ae-f509-43f1-ae1e-2841014548a4","Type":"ContainerStarted","Data":"b7686e9e122dcb7b2f74be4f6183c4dfdd20357523c34c39d33b19dc0e5a1f66"}
Jul 28 17:24:27.608775 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:27.608729    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-79zvc" podStartSLOduration=17.520206849 podStartE2EDuration="29.60871199s" totalImagesPullingTime="12.088505141s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:23:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:27.60846276 +0000 UTC m=+117.918628276" watchObservedRunningTime="2026-07-28 17:24:27.60871199 +0000 UTC m=+117.918877496"
Jul 28 17:24:36.274777 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:36.274733    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-599946f5cf-f22zd" podUID="d771ff91-6330-45c6-844a-7d5b62ded5f0" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 28 17:24:37.123053 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.123011    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7fzw8"]
Jul 28 17:24:37.126271 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.126255    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7fzw8"
Jul 28 17:24:37.131105 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.131079    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 17:24:37.131244 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.131138    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 17:24:37.139774 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.139757    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 17:24:37.140044 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.140029    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 17:24:37.140099 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.140083    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dm6j5\""
Jul 28 17:24:37.140255 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.140239    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 17:24:37.149301 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.149284    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 17:24:37.178428 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.178393    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc-sys\") pod \"node-exporter-7fzw8\" (UID: \"fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc\") " pod="openshift-monitoring/node-exporter-7fzw8"
Jul 28 17:24:37.178428 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.178429    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc-node-exporter-tls\") pod \"node-exporter-7fzw8\" (UID: \"fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc\") " pod="openshift-monitoring/node-exporter-7fzw8"
Jul 28 17:24:37.178594 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.178449    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8dh6\" (UniqueName: \"kubernetes.io/projected/fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc-kube-api-access-n8dh6\") pod \"node-exporter-7fzw8\" (UID: \"fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc\") " pod="openshift-monitoring/node-exporter-7fzw8"
Jul 28 17:24:37.178594 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.178465    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc-root\") pod \"node-exporter-7fzw8\" (UID: \"fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc\") " pod="openshift-monitoring/node-exporter-7fzw8"
Jul 28 17:24:37.178594 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.178510    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc-node-exporter-textfile\") pod \"node-exporter-7fzw8\" (UID: \"fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc\") " pod="openshift-monitoring/node-exporter-7fzw8"
Jul 28 17:24:37.178594 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.178525    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc-node-exporter-wtmp\") pod \"node-exporter-7fzw8\" (UID: \"fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc\") " pod="openshift-monitoring/node-exporter-7fzw8"
Jul 28 17:24:37.178594 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.178587    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7fzw8\" (UID: \"fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc\") " pod="openshift-monitoring/node-exporter-7fzw8"
Jul 28 17:24:37.178749 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.178631    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc-metrics-client-ca\") pod \"node-exporter-7fzw8\" (UID: \"fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc\") " pod="openshift-monitoring/node-exporter-7fzw8"
Jul 28 17:24:37.178749 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.178649    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc-node-exporter-accelerators-collector-config\") pod \"node-exporter-7fzw8\" (UID: \"fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc\") " pod="openshift-monitoring/node-exporter-7fzw8"
Jul 28 17:24:37.279363 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.279307    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc-sys\") pod \"node-exporter-7fzw8\" (UID: \"fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc\") " pod="openshift-monitoring/node-exporter-7fzw8"
Jul 28 17:24:37.279363 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.279363    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc-node-exporter-tls\") pod \"node-exporter-7fzw8\" (UID: \"fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc\") " pod="openshift-monitoring/node-exporter-7fzw8"
Jul 28 17:24:37.279852 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.279426    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc-sys\") pod \"node-exporter-7fzw8\" (UID: \"fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc\") " pod="openshift-monitoring/node-exporter-7fzw8"
Jul 28 17:24:37.279852 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:24:37.279455    2566 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 28 17:24:37.279852 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.279476    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8dh6\" (UniqueName: \"kubernetes.io/projected/fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc-kube-api-access-n8dh6\") pod \"node-exporter-7fzw8\" (UID: \"fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc\") " pod="openshift-monitoring/node-exporter-7fzw8"
Jul 28 17:24:37.279852 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:24:37.279510    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc-node-exporter-tls podName:fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc nodeName:}" failed. No retries permitted until 2026-07-28 17:24:37.779494533 +0000 UTC m=+128.089660015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc-node-exporter-tls") pod "node-exporter-7fzw8" (UID: "fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc") : secret "node-exporter-tls" not found
Jul 28 17:24:37.279852 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.279522    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc-root\") pod \"node-exporter-7fzw8\" (UID: \"fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc\") " pod="openshift-monitoring/node-exporter-7fzw8"
Jul 28 17:24:37.279852 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.279558    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc-node-exporter-textfile\") pod \"node-exporter-7fzw8\" (UID: \"fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc\") " pod="openshift-monitoring/node-exporter-7fzw8"
Jul 28 17:24:37.279852 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.279575    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc-node-exporter-wtmp\") pod \"node-exporter-7fzw8\" (UID: \"fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc\") " pod="openshift-monitoring/node-exporter-7fzw8"
Jul 28 17:24:37.279852 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.279613    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7fzw8\" (UID: \"fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc\") " pod="openshift-monitoring/node-exporter-7fzw8"
Jul 28 17:24:37.279852 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.279631    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc-root\") pod \"node-exporter-7fzw8\" (UID: \"fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc\") " pod="openshift-monitoring/node-exporter-7fzw8"
Jul 28 17:24:37.279852 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.279642    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc-metrics-client-ca\") pod \"node-exporter-7fzw8\" (UID: \"fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc\") " pod="openshift-monitoring/node-exporter-7fzw8"
Jul 28 17:24:37.279852 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.279736    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc-node-exporter-wtmp\") pod \"node-exporter-7fzw8\" (UID: \"fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc\") " pod="openshift-monitoring/node-exporter-7fzw8"
Jul 28 17:24:37.279852 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.279756    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc-node-exporter-accelerators-collector-config\") pod \"node-exporter-7fzw8\" (UID: \"fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc\") " pod="openshift-monitoring/node-exporter-7fzw8"
Jul 28 17:24:37.280444 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.279967    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc-node-exporter-textfile\") pod \"node-exporter-7fzw8\" (UID: \"fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc\") " pod="openshift-monitoring/node-exporter-7fzw8"
Jul 28 17:24:37.280444 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.280170    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc-metrics-client-ca\") pod \"node-exporter-7fzw8\" (UID: \"fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc\") " pod="openshift-monitoring/node-exporter-7fzw8"
Jul 28 17:24:37.280444 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.280347    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc-node-exporter-accelerators-collector-config\") pod \"node-exporter-7fzw8\" (UID: \"fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc\") " pod="openshift-monitoring/node-exporter-7fzw8"
Jul 28 17:24:37.281973 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.281950    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7fzw8\" (UID: \"fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc\") " pod="openshift-monitoring/node-exporter-7fzw8"
Jul 28 17:24:37.304449 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.304420    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8dh6\" (UniqueName: \"kubernetes.io/projected/fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc-kube-api-access-n8dh6\") pod \"node-exporter-7fzw8\" (UID: \"fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc\") " pod="openshift-monitoring/node-exporter-7fzw8"
Jul 28 17:24:37.783598 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.783548    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc-node-exporter-tls\") pod \"node-exporter-7fzw8\" (UID: \"fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc\") " pod="openshift-monitoring/node-exporter-7fzw8"
Jul 28 17:24:37.785771 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:37.785740    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc-node-exporter-tls\") pod \"node-exporter-7fzw8\" (UID: \"fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc\") " pod="openshift-monitoring/node-exporter-7fzw8"
Jul 28 17:24:38.034549 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:38.034466    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7fzw8"
Jul 28 17:24:38.045248 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:24:38.045217    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd5bfe5_8b39_4f6d_a5ec_85009c390ebc.slice/crio-1b5aef007f4a49331caaefcf4c5a05d80f63a09b5ccc65de497ee784affc98d0 WatchSource:0}: Error finding container 1b5aef007f4a49331caaefcf4c5a05d80f63a09b5ccc65de497ee784affc98d0: Status 404 returned error can't find the container with id 1b5aef007f4a49331caaefcf4c5a05d80f63a09b5ccc65de497ee784affc98d0
Jul 28 17:24:38.622907 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:38.622866    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7fzw8" event={"ID":"fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc","Type":"ContainerStarted","Data":"1b5aef007f4a49331caaefcf4c5a05d80f63a09b5ccc65de497ee784affc98d0"}
Jul 28 17:24:39.627965 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:39.627934    2566 generic.go:356] "Generic (PLEG): container finished" podID="fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc" containerID="9230612ce7f7c24b8402717cbf0e25c2b8743bc594705b14359bf7a2cd735bc3" exitCode=0
Jul 28 17:24:39.628295 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:39.628009    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7fzw8" event={"ID":"fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc","Type":"ContainerDied","Data":"9230612ce7f7c24b8402717cbf0e25c2b8743bc594705b14359bf7a2cd735bc3"}
Jul 28 17:24:40.003125 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:40.003088    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dedc8b78-7469-4e8d-a7b8-b11c741186d7-metrics-certs\") pod \"network-metrics-daemon-znptx\" (UID: \"dedc8b78-7469-4e8d-a7b8-b11c741186d7\") " pod="openshift-multus/network-metrics-daemon-znptx"
Jul 28 17:24:40.005420 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:40.005399    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dedc8b78-7469-4e8d-a7b8-b11c741186d7-metrics-certs\") pod \"network-metrics-daemon-znptx\" (UID: \"dedc8b78-7469-4e8d-a7b8-b11c741186d7\") " pod="openshift-multus/network-metrics-daemon-znptx"
Jul 28 17:24:40.214598 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:40.214568    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ztptx\""
Jul 28 17:24:40.221720 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:40.221700    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znptx"
Jul 28 17:24:40.353448 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:40.351745    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-znptx"]
Jul 28 17:24:40.633485 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:40.633399    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7fzw8" event={"ID":"fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc","Type":"ContainerStarted","Data":"3dcd557de3c2a85a3313c4325401ca49d00784a14081ba483479d64a29085c7f"}
Jul 28 17:24:40.633485 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:40.633456    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7fzw8" event={"ID":"fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc","Type":"ContainerStarted","Data":"69c9550c637ed64a3874d1511562c4ef5e056c751c8da987f63408e7bed251a1"}
Jul 28 17:24:40.637074 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:40.637044    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-znptx" event={"ID":"dedc8b78-7469-4e8d-a7b8-b11c741186d7","Type":"ContainerStarted","Data":"fe4fdd7c78db1e78011a70fa74db35d3a40f8e2f3a4ea5fb3185407a4e6be72e"}
Jul 28 17:24:40.657919 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:40.657875    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7fzw8" podStartSLOduration=2.670677232 podStartE2EDuration="3.657857658s" totalImagesPullingTime="925.576128ms" totalInitContainerRuntime="61.604298ms" isStatefulPod=true podCreationTimestamp="2026-07-28 17:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:40.65598557 +0000 UTC m=+130.966151076" watchObservedRunningTime="2026-07-28 17:24:40.657857658 +0000 UTC m=+130.968023163"
Jul 28 17:24:41.497649 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:41.497623    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-588f9d6645-ftg7q"]
Jul 28 17:24:41.501756 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:41.501741    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-588f9d6645-ftg7q"
Jul 28 17:24:41.507637 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:41.507599    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-r2x7t\""
Jul 28 17:24:41.507735 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:41.507624    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 17:24:41.508218 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:41.508139    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5n3cg7kb8qbv0\""
Jul 28 17:24:41.508218 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:41.508150    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 28 17:24:41.508218 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:41.508187    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 28 17:24:41.508434 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:41.508362    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 28 17:24:41.514938 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:41.514882    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-588f9d6645-ftg7q"]
Jul 28 17:24:41.613193 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:41.613167    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f9b80343-ffd9-457d-a1d7-e40f1f467a3a-audit-log\") pod \"metrics-server-588f9d6645-ftg7q\" (UID: \"f9b80343-ffd9-457d-a1d7-e40f1f467a3a\") " pod="openshift-monitoring/metrics-server-588f9d6645-ftg7q"
Jul 28 17:24:41.613309 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:41.613205    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9b80343-ffd9-457d-a1d7-e40f1f467a3a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-588f9d6645-ftg7q\" (UID: \"f9b80343-ffd9-457d-a1d7-e40f1f467a3a\") " pod="openshift-monitoring/metrics-server-588f9d6645-ftg7q"
Jul 28 17:24:41.613309 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:41.613229    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f9b80343-ffd9-457d-a1d7-e40f1f467a3a-client-ca-bundle\") pod \"metrics-server-588f9d6645-ftg7q\" (UID: \"f9b80343-ffd9-457d-a1d7-e40f1f467a3a\") " pod="openshift-monitoring/metrics-server-588f9d6645-ftg7q"
Jul 28 17:24:41.613444 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:41.613329    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzrxt\" (UniqueName: \"kubernetes.io/projected/f9b80343-ffd9-457d-a1d7-e40f1f467a3a-kube-api-access-wzrxt\") pod \"metrics-server-588f9d6645-ftg7q\" (UID: \"f9b80343-ffd9-457d-a1d7-e40f1f467a3a\") " pod="openshift-monitoring/metrics-server-588f9d6645-ftg7q"
Jul 28 17:24:41.613444 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:41.613364    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f9b80343-ffd9-457d-a1d7-e40f1f467a3a-metrics-server-audit-profiles\") pod \"metrics-server-588f9d6645-ftg7q\" (UID: \"f9b80343-ffd9-457d-a1d7-e40f1f467a3a\") " pod="openshift-monitoring/metrics-server-588f9d6645-ftg7q"
Jul 28 17:24:41.613444 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:41.613385    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f9b80343-ffd9-457d-a1d7-e40f1f467a3a-secret-metrics-server-tls\") pod \"metrics-server-588f9d6645-ftg7q\" (UID: \"f9b80343-ffd9-457d-a1d7-e40f1f467a3a\") " pod="openshift-monitoring/metrics-server-588f9d6645-ftg7q"
Jul 28 17:24:41.613444 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:41.613419    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f9b80343-ffd9-457d-a1d7-e40f1f467a3a-secret-metrics-server-client-certs\") pod \"metrics-server-588f9d6645-ftg7q\" (UID: \"f9b80343-ffd9-457d-a1d7-e40f1f467a3a\") " pod="openshift-monitoring/metrics-server-588f9d6645-ftg7q"
Jul 28 17:24:41.642463 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:41.642430    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-znptx" event={"ID":"dedc8b78-7469-4e8d-a7b8-b11c741186d7","Type":"ContainerStarted","Data":"30f3012e08ae07d60dde277b2bee8cb2e86af10f8ba2cf4dfab1529f7cba8705"}
Jul 28 17:24:41.642837 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:41.642484    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-znptx" event={"ID":"dedc8b78-7469-4e8d-a7b8-b11c741186d7","Type":"ContainerStarted","Data":"7cf8a2461a535acaa74c08557174c13dee59bd9888d47cdb95eb745ab8e9516f"}
Jul 28 17:24:41.661649 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:41.661606    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-znptx" podStartSLOduration=130.591982326 podStartE2EDuration="2m11.66159041s" totalImagesPullingTime="1.069608084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:22:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:41.660644329 +0000 UTC m=+131.970809834" watchObservedRunningTime="2026-07-28 17:24:41.66159041 +0000 UTC m=+131.971755914"
Jul 28 17:24:41.714234 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:41.714204    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f9b80343-ffd9-457d-a1d7-e40f1f467a3a-client-ca-bundle\") pod \"metrics-server-588f9d6645-ftg7q\" (UID: \"f9b80343-ffd9-457d-a1d7-e40f1f467a3a\") " pod="openshift-monitoring/metrics-server-588f9d6645-ftg7q"
Jul 28 17:24:41.714426 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:41.714273    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzrxt\" (UniqueName: \"kubernetes.io/projected/f9b80343-ffd9-457d-a1d7-e40f1f467a3a-kube-api-access-wzrxt\") pod \"metrics-server-588f9d6645-ftg7q\" (UID: \"f9b80343-ffd9-457d-a1d7-e40f1f467a3a\") " pod="openshift-monitoring/metrics-server-588f9d6645-ftg7q"
Jul 28 17:24:41.714528 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:41.714506    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f9b80343-ffd9-457d-a1d7-e40f1f467a3a-metrics-server-audit-profiles\") pod \"metrics-server-588f9d6645-ftg7q\" (UID: \"f9b80343-ffd9-457d-a1d7-e40f1f467a3a\") " pod="openshift-monitoring/metrics-server-588f9d6645-ftg7q"
Jul 28 17:24:41.714588 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:41.714554    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f9b80343-ffd9-457d-a1d7-e40f1f467a3a-secret-metrics-server-tls\") pod \"metrics-server-588f9d6645-ftg7q\" (UID: \"f9b80343-ffd9-457d-a1d7-e40f1f467a3a\") " pod="openshift-monitoring/metrics-server-588f9d6645-ftg7q"
Jul 28 17:24:41.714662 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:41.714640    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f9b80343-ffd9-457d-a1d7-e40f1f467a3a-secret-metrics-server-client-certs\") pod \"metrics-server-588f9d6645-ftg7q\" (UID: \"f9b80343-ffd9-457d-a1d7-e40f1f467a3a\") " pod="openshift-monitoring/metrics-server-588f9d6645-ftg7q"
Jul 28 17:24:41.714736 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:41.714719    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f9b80343-ffd9-457d-a1d7-e40f1f467a3a-audit-log\") pod \"metrics-server-588f9d6645-ftg7q\" (UID: \"f9b80343-ffd9-457d-a1d7-e40f1f467a3a\") " pod="openshift-monitoring/metrics-server-588f9d6645-ftg7q"
Jul 28 17:24:41.714824 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:41.714805    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9b80343-ffd9-457d-a1d7-e40f1f467a3a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-588f9d6645-ftg7q\" (UID: \"f9b80343-ffd9-457d-a1d7-e40f1f467a3a\") " pod="openshift-monitoring/metrics-server-588f9d6645-ftg7q"
Jul 28 17:24:41.715188 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:41.715168    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f9b80343-ffd9-457d-a1d7-e40f1f467a3a-audit-log\") pod \"metrics-server-588f9d6645-ftg7q\" (UID: \"f9b80343-ffd9-457d-a1d7-e40f1f467a3a\") " pod="openshift-monitoring/metrics-server-588f9d6645-ftg7q"
Jul 28 17:24:41.715492 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:41.715448    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9b80343-ffd9-457d-a1d7-e40f1f467a3a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-588f9d6645-ftg7q\" (UID: \"f9b80343-ffd9-457d-a1d7-e40f1f467a3a\") " pod="openshift-monitoring/metrics-server-588f9d6645-ftg7q"
Jul 28 17:24:41.715597 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:41.715554    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f9b80343-ffd9-457d-a1d7-e40f1f467a3a-metrics-server-audit-profiles\") pod \"metrics-server-588f9d6645-ftg7q\" (UID: \"f9b80343-ffd9-457d-a1d7-e40f1f467a3a\") " pod="openshift-monitoring/metrics-server-588f9d6645-ftg7q"
Jul 28 17:24:41.716941 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:41.716923    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f9b80343-ffd9-457d-a1d7-e40f1f467a3a-secret-metrics-server-tls\") pod \"metrics-server-588f9d6645-ftg7q\" (UID: \"f9b80343-ffd9-457d-a1d7-e40f1f467a3a\") " pod="openshift-monitoring/metrics-server-588f9d6645-ftg7q"
Jul 28 17:24:41.717034 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:41.716980    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f9b80343-ffd9-457d-a1d7-e40f1f467a3a-client-ca-bundle\") pod \"metrics-server-588f9d6645-ftg7q\" (UID: \"f9b80343-ffd9-457d-a1d7-e40f1f467a3a\") " pod="openshift-monitoring/metrics-server-588f9d6645-ftg7q"
Jul 28 17:24:41.717034 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:41.717015    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f9b80343-ffd9-457d-a1d7-e40f1f467a3a-secret-metrics-server-client-certs\") pod \"metrics-server-588f9d6645-ftg7q\" (UID: \"f9b80343-ffd9-457d-a1d7-e40f1f467a3a\") " pod="openshift-monitoring/metrics-server-588f9d6645-ftg7q"
Jul 28 17:24:41.723488 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:41.723461    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzrxt\" (UniqueName: \"kubernetes.io/projected/f9b80343-ffd9-457d-a1d7-e40f1f467a3a-kube-api-access-wzrxt\") pod \"metrics-server-588f9d6645-ftg7q\" (UID: \"f9b80343-ffd9-457d-a1d7-e40f1f467a3a\") " pod="openshift-monitoring/metrics-server-588f9d6645-ftg7q"
Jul 28 17:24:41.812256 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:41.812226    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-588f9d6645-ftg7q"
Jul 28 17:24:41.952021 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:41.951986    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-588f9d6645-ftg7q"]
Jul 28 17:24:41.956197 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:24:41.956162    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b80343_ffd9_457d_a1d7_e40f1f467a3a.slice/crio-b6959bd28e1c7181db215ba76dd0ac44053fdbaca0086a42070665e1b54e9dae WatchSource:0}: Error finding container b6959bd28e1c7181db215ba76dd0ac44053fdbaca0086a42070665e1b54e9dae: Status 404 returned error can't find the container with id b6959bd28e1c7181db215ba76dd0ac44053fdbaca0086a42070665e1b54e9dae
Jul 28 17:24:42.646764 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:42.646728    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-588f9d6645-ftg7q" event={"ID":"f9b80343-ffd9-457d-a1d7-e40f1f467a3a","Type":"ContainerStarted","Data":"b6959bd28e1c7181db215ba76dd0ac44053fdbaca0086a42070665e1b54e9dae"}
Jul 28 17:24:43.653371 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:43.653314    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-588f9d6645-ftg7q" event={"ID":"f9b80343-ffd9-457d-a1d7-e40f1f467a3a","Type":"ContainerStarted","Data":"499ac8a45fe3de9f894f0789eb1d1c30adced5d55a34281686c652805c8c38d3"}
Jul 28 17:24:43.673095 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:43.673039    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-588f9d6645-ftg7q" podStartSLOduration=1.055131469 podStartE2EDuration="2.673024257s" totalImagesPullingTime="1.617892788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:43.67193766 +0000 UTC m=+133.982103242" watchObservedRunningTime="2026-07-28 17:24:43.673024257 +0000 UTC m=+133.983189762"
Jul 28 17:24:46.274805 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:46.274758    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-599946f5cf-f22zd" podUID="d771ff91-6330-45c6-844a-7d5b62ded5f0" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 28 17:24:56.274079 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:56.274039    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-599946f5cf-f22zd" podUID="d771ff91-6330-45c6-844a-7d5b62ded5f0" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 28 17:24:56.274461 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:56.274112    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-599946f5cf-f22zd"
Jul 28 17:24:56.274607 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:56.274576    2566 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"cf64a71a34035688e3dc60f041737acbece6640c1b6bc3ff50a9a2240116d032"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-599946f5cf-f22zd" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 28 17:24:56.274647 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:56.274630    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-599946f5cf-f22zd" podUID="d771ff91-6330-45c6-844a-7d5b62ded5f0" containerName="service-proxy" containerID="cri-o://cf64a71a34035688e3dc60f041737acbece6640c1b6bc3ff50a9a2240116d032" gracePeriod=30
Jul 28 17:24:56.695903 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:56.695871    2566 generic.go:356] "Generic (PLEG): container finished" podID="d771ff91-6330-45c6-844a-7d5b62ded5f0" containerID="cf64a71a34035688e3dc60f041737acbece6640c1b6bc3ff50a9a2240116d032" exitCode=2
Jul 28 17:24:56.696058 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:56.695929    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-599946f5cf-f22zd" event={"ID":"d771ff91-6330-45c6-844a-7d5b62ded5f0","Type":"ContainerDied","Data":"cf64a71a34035688e3dc60f041737acbece6640c1b6bc3ff50a9a2240116d032"}
Jul 28 17:24:57.701451 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:24:57.701417    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-599946f5cf-f22zd" event={"ID":"d771ff91-6330-45c6-844a-7d5b62ded5f0","Type":"ContainerStarted","Data":"4c03b71ad1c522513039074a292a76307be666d184e676e52a60ecabae568807"}
Jul 28 17:25:01.812481 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:25:01.812422    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-588f9d6645-ftg7q"
Jul 28 17:25:01.812849 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:25:01.812526    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-588f9d6645-ftg7q"
Jul 28 17:25:21.818231 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:25:21.818203    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-588f9d6645-ftg7q"
Jul 28 17:25:21.821927 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:25:21.821905    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-588f9d6645-ftg7q"
Jul 28 17:27:30.011751 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:27:30.011723    2566 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 17:29:06.528641 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:29:06.528612    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-9k58d"]
Jul 28 17:29:06.531521 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:29:06.531506    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-9k58d"
Jul 28 17:29:06.535006 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:29:06.534985    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 17:29:06.536839 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:29:06.536816    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 17:29:06.536839 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:29:06.536829    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 28 17:29:06.536985 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:29:06.536917    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-hrntp\""
Jul 28 17:29:06.587586 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:29:06.587559    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-9k58d"]
Jul 28 17:29:06.613494 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:29:06.613467    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bshml\" (UniqueName: \"kubernetes.io/projected/8312b01c-e651-40f5-bf99-d5fbd4359133-kube-api-access-bshml\") pod \"llmisvc-controller-manager-654c557856-9k58d\" (UID: \"8312b01c-e651-40f5-bf99-d5fbd4359133\") " pod="kserve/llmisvc-controller-manager-654c557856-9k58d"
Jul 28 17:29:06.613597 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:29:06.613517    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8312b01c-e651-40f5-bf99-d5fbd4359133-cert\") pod \"llmisvc-controller-manager-654c557856-9k58d\" (UID: \"8312b01c-e651-40f5-bf99-d5fbd4359133\") " pod="kserve/llmisvc-controller-manager-654c557856-9k58d"
Jul 28 17:29:06.714196 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:29:06.714168    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bshml\" (UniqueName: \"kubernetes.io/projected/8312b01c-e651-40f5-bf99-d5fbd4359133-kube-api-access-bshml\") pod \"llmisvc-controller-manager-654c557856-9k58d\" (UID: \"8312b01c-e651-40f5-bf99-d5fbd4359133\") " pod="kserve/llmisvc-controller-manager-654c557856-9k58d"
Jul 28 17:29:06.714351 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:29:06.714226    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8312b01c-e651-40f5-bf99-d5fbd4359133-cert\") pod \"llmisvc-controller-manager-654c557856-9k58d\" (UID: \"8312b01c-e651-40f5-bf99-d5fbd4359133\") " pod="kserve/llmisvc-controller-manager-654c557856-9k58d"
Jul 28 17:29:06.714351 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:29:06.714345    2566 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 28 17:29:06.714468 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:29:06.714409    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8312b01c-e651-40f5-bf99-d5fbd4359133-cert podName:8312b01c-e651-40f5-bf99-d5fbd4359133 nodeName:}" failed. No retries permitted until 2026-07-28 17:29:07.214393438 +0000 UTC m=+397.524558921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8312b01c-e651-40f5-bf99-d5fbd4359133-cert") pod "llmisvc-controller-manager-654c557856-9k58d" (UID: "8312b01c-e651-40f5-bf99-d5fbd4359133") : secret "llmisvc-webhook-server-cert" not found
Jul 28 17:29:06.752586 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:29:06.752562    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bshml\" (UniqueName: \"kubernetes.io/projected/8312b01c-e651-40f5-bf99-d5fbd4359133-kube-api-access-bshml\") pod \"llmisvc-controller-manager-654c557856-9k58d\" (UID: \"8312b01c-e651-40f5-bf99-d5fbd4359133\") " pod="kserve/llmisvc-controller-manager-654c557856-9k58d"
Jul 28 17:29:07.217903 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:29:07.217864    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8312b01c-e651-40f5-bf99-d5fbd4359133-cert\") pod \"llmisvc-controller-manager-654c557856-9k58d\" (UID: \"8312b01c-e651-40f5-bf99-d5fbd4359133\") " pod="kserve/llmisvc-controller-manager-654c557856-9k58d"
Jul 28 17:29:07.220147 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:29:07.220130    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8312b01c-e651-40f5-bf99-d5fbd4359133-cert\") pod \"llmisvc-controller-manager-654c557856-9k58d\" (UID: \"8312b01c-e651-40f5-bf99-d5fbd4359133\") " pod="kserve/llmisvc-controller-manager-654c557856-9k58d"
Jul 28 17:29:07.440340 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:29:07.440284    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-9k58d"
Jul 28 17:29:07.594310 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:29:07.594271    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-9k58d"]
Jul 28 17:29:07.597581 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:29:07.597555    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8312b01c_e651_40f5_bf99_d5fbd4359133.slice/crio-092ec8ad09e7ec77e7963cb1f0d86774cf2f42f57c0362b993017744c1fa3f4a WatchSource:0}: Error finding container 092ec8ad09e7ec77e7963cb1f0d86774cf2f42f57c0362b993017744c1fa3f4a: Status 404 returned error can't find the container with id 092ec8ad09e7ec77e7963cb1f0d86774cf2f42f57c0362b993017744c1fa3f4a
Jul 28 17:29:07.598908 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:29:07.598891    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:29:08.486379 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:29:08.486347    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-9k58d" event={"ID":"8312b01c-e651-40f5-bf99-d5fbd4359133","Type":"ContainerStarted","Data":"092ec8ad09e7ec77e7963cb1f0d86774cf2f42f57c0362b993017744c1fa3f4a"}
Jul 28 17:29:12.500117 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:29:12.500082    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-9k58d" event={"ID":"8312b01c-e651-40f5-bf99-d5fbd4359133","Type":"ContainerStarted","Data":"15bdfbd07042893df12d137568165869816cde25e6367c3391fda90c62b6bfe8"}
Jul 28 17:29:12.500662 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:29:12.500182    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-654c557856-9k58d"
Jul 28 17:29:12.524051 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:29:12.524000    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-654c557856-9k58d" podStartSLOduration=2.5349250740000002 podStartE2EDuration="6.523986499s" totalImagesPullingTime="3.989061425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:29:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:29:12.523735048 +0000 UTC m=+402.833900554" watchObservedRunningTime="2026-07-28 17:29:12.523986499 +0000 UTC m=+402.834152004"
Jul 28 17:29:43.505114 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:29:43.505083    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-654c557856-9k58d"
Jul 28 17:32:39.014937 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:32:39.014901    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn"]
Jul 28 17:32:39.017948 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:32:39.017932    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn"
Jul 28 17:32:39.021464 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:32:39.021443    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f0903-kube-rbac-proxy-sar-config\""
Jul 28 17:32:39.021588 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:32:39.021496    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f0903-predictor-serving-cert\""
Jul 28 17:32:39.021868 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:32:39.021853    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:32:39.021868 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:32:39.021863    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:32:39.022723 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:32:39.022709    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7f5hv\""
Jul 28 17:32:39.030276 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:32:39.030255    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn"]
Jul 28 17:32:39.138556 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:32:39.138525    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00e17ba8-b199-4f61-83e7-9483f2ba0598-proxy-tls\") pod \"success-200-isvc-f0903-predictor-b88cb88c-8g2fn\" (UID: \"00e17ba8-b199-4f61-83e7-9483f2ba0598\") " pod="kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn"
Jul 28 17:32:39.138724 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:32:39.138563    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95ftm\" (UniqueName: \"kubernetes.io/projected/00e17ba8-b199-4f61-83e7-9483f2ba0598-kube-api-access-95ftm\") pod \"success-200-isvc-f0903-predictor-b88cb88c-8g2fn\" (UID: \"00e17ba8-b199-4f61-83e7-9483f2ba0598\") " pod="kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn"
Jul 28 17:32:39.138724 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:32:39.138587    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00e17ba8-b199-4f61-83e7-9483f2ba0598-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f0903-predictor-b88cb88c-8g2fn\" (UID: \"00e17ba8-b199-4f61-83e7-9483f2ba0598\") " pod="kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn"
Jul 28 17:32:39.239801 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:32:39.239775    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00e17ba8-b199-4f61-83e7-9483f2ba0598-proxy-tls\") pod \"success-200-isvc-f0903-predictor-b88cb88c-8g2fn\" (UID: \"00e17ba8-b199-4f61-83e7-9483f2ba0598\") " pod="kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn"
Jul 28 17:32:39.239955 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:32:39.239815    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95ftm\" (UniqueName: \"kubernetes.io/projected/00e17ba8-b199-4f61-83e7-9483f2ba0598-kube-api-access-95ftm\") pod \"success-200-isvc-f0903-predictor-b88cb88c-8g2fn\" (UID: \"00e17ba8-b199-4f61-83e7-9483f2ba0598\") " pod="kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn"
Jul 28 17:32:39.239955 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:32:39.239841    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00e17ba8-b199-4f61-83e7-9483f2ba0598-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f0903-predictor-b88cb88c-8g2fn\" (UID: \"00e17ba8-b199-4f61-83e7-9483f2ba0598\") " pod="kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn"
Jul 28 17:32:39.239955 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:32:39.239937    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-f0903-predictor-serving-cert: secret "success-200-isvc-f0903-predictor-serving-cert" not found
Jul 28 17:32:39.240095 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:32:39.240009    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00e17ba8-b199-4f61-83e7-9483f2ba0598-proxy-tls podName:00e17ba8-b199-4f61-83e7-9483f2ba0598 nodeName:}" failed. No retries permitted until 2026-07-28 17:32:39.739990106 +0000 UTC m=+610.050155592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/00e17ba8-b199-4f61-83e7-9483f2ba0598-proxy-tls") pod "success-200-isvc-f0903-predictor-b88cb88c-8g2fn" (UID: "00e17ba8-b199-4f61-83e7-9483f2ba0598") : secret "success-200-isvc-f0903-predictor-serving-cert" not found
Jul 28 17:32:39.240489 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:32:39.240470    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00e17ba8-b199-4f61-83e7-9483f2ba0598-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f0903-predictor-b88cb88c-8g2fn\" (UID: \"00e17ba8-b199-4f61-83e7-9483f2ba0598\") " pod="kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn"
Jul 28 17:32:39.270215 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:32:39.270150    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95ftm\" (UniqueName: \"kubernetes.io/projected/00e17ba8-b199-4f61-83e7-9483f2ba0598-kube-api-access-95ftm\") pod \"success-200-isvc-f0903-predictor-b88cb88c-8g2fn\" (UID: \"00e17ba8-b199-4f61-83e7-9483f2ba0598\") " pod="kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn"
Jul 28 17:32:39.744011 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:32:39.743979    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00e17ba8-b199-4f61-83e7-9483f2ba0598-proxy-tls\") pod \"success-200-isvc-f0903-predictor-b88cb88c-8g2fn\" (UID: \"00e17ba8-b199-4f61-83e7-9483f2ba0598\") " pod="kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn"
Jul 28 17:32:39.746434 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:32:39.746411    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00e17ba8-b199-4f61-83e7-9483f2ba0598-proxy-tls\") pod \"success-200-isvc-f0903-predictor-b88cb88c-8g2fn\" (UID: \"00e17ba8-b199-4f61-83e7-9483f2ba0598\") " pod="kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn"
Jul 28 17:32:39.927332 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:32:39.927279    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn"
Jul 28 17:32:40.054081 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:32:40.054052    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn"]
Jul 28 17:32:40.057221 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:32:40.057195    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00e17ba8_b199_4f61_83e7_9483f2ba0598.slice/crio-8e078dce17a53f2159172eb77e07dafc2aa373acb6e3a1a8d4e3662d8675a2d7 WatchSource:0}: Error finding container 8e078dce17a53f2159172eb77e07dafc2aa373acb6e3a1a8d4e3662d8675a2d7: Status 404 returned error can't find the container with id 8e078dce17a53f2159172eb77e07dafc2aa373acb6e3a1a8d4e3662d8675a2d7
Jul 28 17:32:40.168115 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:32:40.168079    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn" event={"ID":"00e17ba8-b199-4f61-83e7-9483f2ba0598","Type":"ContainerStarted","Data":"8e078dce17a53f2159172eb77e07dafc2aa373acb6e3a1a8d4e3662d8675a2d7"}
Jul 28 17:32:52.221268 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:32:52.221183    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn" event={"ID":"00e17ba8-b199-4f61-83e7-9483f2ba0598","Type":"ContainerStarted","Data":"b1fb7fd036c5c9d26f235b6cab7666d9e8af81694529bde0a87273ff70e9e3b8"}
Jul 28 17:32:54.230638 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:32:54.230564    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn" event={"ID":"00e17ba8-b199-4f61-83e7-9483f2ba0598","Type":"ContainerStarted","Data":"48e4eaf11d0d7ca68b3c5f92ad395dbe6f004c4a9143df2e9cd4779821369afc"}
Jul 28 17:32:54.231026 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:32:54.230816    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn"
Jul 28 17:32:54.231026 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:32:54.230840    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn"
Jul 28 17:32:54.232133 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:32:54.232110    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn" podUID="00e17ba8-b199-4f61-83e7-9483f2ba0598" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.14:8080: connect: connection refused"
Jul 28 17:32:54.250813 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:32:54.250755    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn" podStartSLOduration=2.420240455 podStartE2EDuration="16.250736047s" totalImagesPullingTime="13.830495592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:32:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:32:54.249791372 +0000 UTC m=+624.559956876" watchObservedRunningTime="2026-07-28 17:32:54.250736047 +0000 UTC m=+624.560901552"
Jul 28 17:32:55.234144 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:32:55.234105    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn" podUID="00e17ba8-b199-4f61-83e7-9483f2ba0598" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.14:8080: connect: connection refused"
Jul 28 17:33:00.238740 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:33:00.238713    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn"
Jul 28 17:33:00.239165 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:33:00.239098    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn" podUID="00e17ba8-b199-4f61-83e7-9483f2ba0598" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.14:8080: connect: connection refused"
Jul 28 17:33:10.239744 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:33:10.239707    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn" podUID="00e17ba8-b199-4f61-83e7-9483f2ba0598" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.14:8080: connect: connection refused"
Jul 28 17:33:20.239132 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:33:20.239099    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn" podUID="00e17ba8-b199-4f61-83e7-9483f2ba0598" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.14:8080: connect: connection refused"
Jul 28 17:33:30.239838 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:33:30.239802    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn" podUID="00e17ba8-b199-4f61-83e7-9483f2ba0598" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.14:8080: connect: connection refused"
Jul 28 17:33:40.240376 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:33:40.240348    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn"
Jul 28 17:33:59.104373 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:33:59.104341    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-f0903-56678cb99b-4nwlp"]
Jul 28 17:33:59.107408 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:33:59.107391    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-f0903-56678cb99b-4nwlp"
Jul 28 17:33:59.110003 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:33:59.109983    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-f0903-kube-rbac-proxy-sar-config\""
Jul 28 17:33:59.110121 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:33:59.109986    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-f0903-serving-cert\""
Jul 28 17:33:59.121907 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:33:59.121886    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-f0903-56678cb99b-4nwlp"]
Jul 28 17:33:59.213053 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:33:59.213026    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/484ebf5f-3805-45f2-97a1-f3cb4caec5dd-proxy-tls\") pod \"switch-graph-f0903-56678cb99b-4nwlp\" (UID: \"484ebf5f-3805-45f2-97a1-f3cb4caec5dd\") " pod="kserve-ci-e2e-test/switch-graph-f0903-56678cb99b-4nwlp"
Jul 28 17:33:59.213232 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:33:59.213065    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/484ebf5f-3805-45f2-97a1-f3cb4caec5dd-openshift-service-ca-bundle\") pod \"switch-graph-f0903-56678cb99b-4nwlp\" (UID: \"484ebf5f-3805-45f2-97a1-f3cb4caec5dd\") " pod="kserve-ci-e2e-test/switch-graph-f0903-56678cb99b-4nwlp"
Jul 28 17:33:59.313633 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:33:59.313602    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/484ebf5f-3805-45f2-97a1-f3cb4caec5dd-openshift-service-ca-bundle\") pod \"switch-graph-f0903-56678cb99b-4nwlp\" (UID: \"484ebf5f-3805-45f2-97a1-f3cb4caec5dd\") " pod="kserve-ci-e2e-test/switch-graph-f0903-56678cb99b-4nwlp"
Jul 28 17:33:59.313814 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:33:59.313666    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/484ebf5f-3805-45f2-97a1-f3cb4caec5dd-proxy-tls\") pod \"switch-graph-f0903-56678cb99b-4nwlp\" (UID: \"484ebf5f-3805-45f2-97a1-f3cb4caec5dd\") " pod="kserve-ci-e2e-test/switch-graph-f0903-56678cb99b-4nwlp"
Jul 28 17:33:59.313814 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:33:59.313759    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-f0903-serving-cert: secret "switch-graph-f0903-serving-cert" not found
Jul 28 17:33:59.313928 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:33:59.313819    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/484ebf5f-3805-45f2-97a1-f3cb4caec5dd-proxy-tls podName:484ebf5f-3805-45f2-97a1-f3cb4caec5dd nodeName:}" failed. No retries permitted until 2026-07-28 17:33:59.813804015 +0000 UTC m=+690.123969498 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/484ebf5f-3805-45f2-97a1-f3cb4caec5dd-proxy-tls") pod "switch-graph-f0903-56678cb99b-4nwlp" (UID: "484ebf5f-3805-45f2-97a1-f3cb4caec5dd") : secret "switch-graph-f0903-serving-cert" not found
Jul 28 17:33:59.314377 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:33:59.314352    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/484ebf5f-3805-45f2-97a1-f3cb4caec5dd-openshift-service-ca-bundle\") pod \"switch-graph-f0903-56678cb99b-4nwlp\" (UID: \"484ebf5f-3805-45f2-97a1-f3cb4caec5dd\") " pod="kserve-ci-e2e-test/switch-graph-f0903-56678cb99b-4nwlp"
Jul 28 17:33:59.816963 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:33:59.816932    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/484ebf5f-3805-45f2-97a1-f3cb4caec5dd-proxy-tls\") pod \"switch-graph-f0903-56678cb99b-4nwlp\" (UID: \"484ebf5f-3805-45f2-97a1-f3cb4caec5dd\") " pod="kserve-ci-e2e-test/switch-graph-f0903-56678cb99b-4nwlp"
Jul 28 17:33:59.819168 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:33:59.819141    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/484ebf5f-3805-45f2-97a1-f3cb4caec5dd-proxy-tls\") pod \"switch-graph-f0903-56678cb99b-4nwlp\" (UID: \"484ebf5f-3805-45f2-97a1-f3cb4caec5dd\") " pod="kserve-ci-e2e-test/switch-graph-f0903-56678cb99b-4nwlp"
Jul 28 17:34:00.017350 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:00.017313    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-f0903-56678cb99b-4nwlp"
Jul 28 17:34:00.138954 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:00.138923    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-f0903-56678cb99b-4nwlp"]
Jul 28 17:34:00.142123 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:34:00.142086    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod484ebf5f_3805_45f2_97a1_f3cb4caec5dd.slice/crio-11faedbe2b96e6645453b7aa970c34bf834c078a2ae9a98f5e856a31ea5c9097 WatchSource:0}: Error finding container 11faedbe2b96e6645453b7aa970c34bf834c078a2ae9a98f5e856a31ea5c9097: Status 404 returned error can't find the container with id 11faedbe2b96e6645453b7aa970c34bf834c078a2ae9a98f5e856a31ea5c9097
Jul 28 17:34:00.446251 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:00.446211    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-f0903-56678cb99b-4nwlp" event={"ID":"484ebf5f-3805-45f2-97a1-f3cb4caec5dd","Type":"ContainerStarted","Data":"11faedbe2b96e6645453b7aa970c34bf834c078a2ae9a98f5e856a31ea5c9097"}
Jul 28 17:34:03.461425 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:03.461389    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-f0903-56678cb99b-4nwlp" event={"ID":"484ebf5f-3805-45f2-97a1-f3cb4caec5dd","Type":"ContainerStarted","Data":"d17c668547600d5610736857b2711ea5cb3127fe206a97b0196b7a4a49c9e357"}
Jul 28 17:34:03.461794 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:03.461497    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-f0903-56678cb99b-4nwlp"
Jul 28 17:34:03.478380 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:03.478311    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-f0903-56678cb99b-4nwlp" podStartSLOduration=2.13440341 podStartE2EDuration="4.478294907s" totalImagesPullingTime="2.343891497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:34:03.478264321 +0000 UTC m=+693.788429830" watchObservedRunningTime="2026-07-28 17:34:03.478294907 +0000 UTC m=+693.788460393"
Jul 28 17:34:09.471588 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:09.471560    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-f0903-56678cb99b-4nwlp"
Jul 28 17:34:13.275738 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:13.275699    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-f0903-56678cb99b-4nwlp"]
Jul 28 17:34:13.276207 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:13.275971    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-f0903-56678cb99b-4nwlp" podUID="484ebf5f-3805-45f2-97a1-f3cb4caec5dd" containerName="switch-graph-f0903" containerID="cri-o://d17c668547600d5610736857b2711ea5cb3127fe206a97b0196b7a4a49c9e357" gracePeriod=30
Jul 28 17:34:13.456651 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:13.456620    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn"]
Jul 28 17:34:13.456892 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:13.456866    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn" podUID="00e17ba8-b199-4f61-83e7-9483f2ba0598" containerName="kube-rbac-proxy" containerID="cri-o://48e4eaf11d0d7ca68b3c5f92ad395dbe6f004c4a9143df2e9cd4779821369afc" gracePeriod=30
Jul 28 17:34:13.456981 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:13.456857    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn" podUID="00e17ba8-b199-4f61-83e7-9483f2ba0598" containerName="kserve-container" containerID="cri-o://b1fb7fd036c5c9d26f235b6cab7666d9e8af81694529bde0a87273ff70e9e3b8" gracePeriod=30
Jul 28 17:34:13.502971 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:13.502945    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4"]
Jul 28 17:34:13.506364 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:13.506347    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4"
Jul 28 17:34:13.508972 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:13.508954    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-22e0e-predictor-serving-cert\""
Jul 28 17:34:13.509312 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:13.509294    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-22e0e-kube-rbac-proxy-sar-config\""
Jul 28 17:34:13.517275 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:13.517255    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4"]
Jul 28 17:34:13.629168 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:13.629136    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxwsz\" (UniqueName: \"kubernetes.io/projected/134b6f2e-e57e-44e6-b74c-e33cf4af9166-kube-api-access-fxwsz\") pod \"success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4\" (UID: \"134b6f2e-e57e-44e6-b74c-e33cf4af9166\") " pod="kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4"
Jul 28 17:34:13.629355 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:13.629187    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/134b6f2e-e57e-44e6-b74c-e33cf4af9166-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4\" (UID: \"134b6f2e-e57e-44e6-b74c-e33cf4af9166\") " pod="kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4"
Jul 28 17:34:13.629355 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:13.629248    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/134b6f2e-e57e-44e6-b74c-e33cf4af9166-proxy-tls\") pod \"success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4\" (UID: \"134b6f2e-e57e-44e6-b74c-e33cf4af9166\") " pod="kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4"
Jul 28 17:34:13.729972 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:13.729941    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxwsz\" (UniqueName: \"kubernetes.io/projected/134b6f2e-e57e-44e6-b74c-e33cf4af9166-kube-api-access-fxwsz\") pod \"success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4\" (UID: \"134b6f2e-e57e-44e6-b74c-e33cf4af9166\") " pod="kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4"
Jul 28 17:34:13.730142 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:13.729980    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/134b6f2e-e57e-44e6-b74c-e33cf4af9166-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4\" (UID: \"134b6f2e-e57e-44e6-b74c-e33cf4af9166\") " pod="kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4"
Jul 28 17:34:13.730142 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:13.730007    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/134b6f2e-e57e-44e6-b74c-e33cf4af9166-proxy-tls\") pod \"success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4\" (UID: \"134b6f2e-e57e-44e6-b74c-e33cf4af9166\") " pod="kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4"
Jul 28 17:34:13.730756 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:13.730734    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/134b6f2e-e57e-44e6-b74c-e33cf4af9166-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4\" (UID: \"134b6f2e-e57e-44e6-b74c-e33cf4af9166\") " pod="kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4"
Jul 28 17:34:13.732552 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:13.732531    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/134b6f2e-e57e-44e6-b74c-e33cf4af9166-proxy-tls\") pod \"success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4\" (UID: \"134b6f2e-e57e-44e6-b74c-e33cf4af9166\") " pod="kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4"
Jul 28 17:34:13.738719 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:13.738700    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxwsz\" (UniqueName: \"kubernetes.io/projected/134b6f2e-e57e-44e6-b74c-e33cf4af9166-kube-api-access-fxwsz\") pod \"success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4\" (UID: \"134b6f2e-e57e-44e6-b74c-e33cf4af9166\") " pod="kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4"
Jul 28 17:34:13.816216 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:13.816188    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4"
Jul 28 17:34:13.938412 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:13.938388    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4"]
Jul 28 17:34:13.940206 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:34:13.940182    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod134b6f2e_e57e_44e6_b74c_e33cf4af9166.slice/crio-7a6139efa993527f368b108e0c5103d33c21641bc7b98f955be586d903cf65a8 WatchSource:0}: Error finding container 7a6139efa993527f368b108e0c5103d33c21641bc7b98f955be586d903cf65a8: Status 404 returned error can't find the container with id 7a6139efa993527f368b108e0c5103d33c21641bc7b98f955be586d903cf65a8
Jul 28 17:34:14.469274 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:14.469234    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-f0903-56678cb99b-4nwlp" podUID="484ebf5f-3805-45f2-97a1-f3cb4caec5dd" containerName="switch-graph-f0903" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:34:14.499814 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:14.499787    2566 generic.go:356] "Generic (PLEG): container finished" podID="00e17ba8-b199-4f61-83e7-9483f2ba0598" containerID="48e4eaf11d0d7ca68b3c5f92ad395dbe6f004c4a9143df2e9cd4779821369afc" exitCode=2
Jul 28 17:34:14.499971 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:14.499864    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn" event={"ID":"00e17ba8-b199-4f61-83e7-9483f2ba0598","Type":"ContainerDied","Data":"48e4eaf11d0d7ca68b3c5f92ad395dbe6f004c4a9143df2e9cd4779821369afc"}
Jul 28 17:34:14.501538 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:14.501512    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4" event={"ID":"134b6f2e-e57e-44e6-b74c-e33cf4af9166","Type":"ContainerStarted","Data":"66b749214276a9424521ccd7fcf96c0b78cfe00339b7235d636bcfad7ad6d520"}
Jul 28 17:34:14.501641 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:14.501566    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4" event={"ID":"134b6f2e-e57e-44e6-b74c-e33cf4af9166","Type":"ContainerStarted","Data":"a29589b1b74488da4aea215b38c53cfccae8c08b71fb43d9dc669fa1c2210d33"}
Jul 28 17:34:14.501641 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:14.501589    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4" event={"ID":"134b6f2e-e57e-44e6-b74c-e33cf4af9166","Type":"ContainerStarted","Data":"7a6139efa993527f368b108e0c5103d33c21641bc7b98f955be586d903cf65a8"}
Jul 28 17:34:14.501641 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:14.501621    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4"
Jul 28 17:34:14.501804 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:14.501692    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4"
Jul 28 17:34:14.502953 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:14.502925    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4" podUID="134b6f2e-e57e-44e6-b74c-e33cf4af9166" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 28 17:34:14.521261 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:14.521223    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4" podStartSLOduration=1.521208327 podStartE2EDuration="1.521208327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:34:14.520165917 +0000 UTC m=+704.830331456" watchObservedRunningTime="2026-07-28 17:34:14.521208327 +0000 UTC m=+704.831373832"
Jul 28 17:34:15.234941 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:15.234903    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn" podUID="00e17ba8-b199-4f61-83e7-9483f2ba0598" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.14:8643/healthz\": dial tcp 10.133.0.14:8643: connect: connection refused"
Jul 28 17:34:15.506354 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:15.506254    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4" podUID="134b6f2e-e57e-44e6-b74c-e33cf4af9166" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 28 17:34:16.402887 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:16.402866    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn"
Jul 28 17:34:16.511191 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:16.511113    2566 generic.go:356] "Generic (PLEG): container finished" podID="00e17ba8-b199-4f61-83e7-9483f2ba0598" containerID="b1fb7fd036c5c9d26f235b6cab7666d9e8af81694529bde0a87273ff70e9e3b8" exitCode=0
Jul 28 17:34:16.511191 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:16.511177    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn" event={"ID":"00e17ba8-b199-4f61-83e7-9483f2ba0598","Type":"ContainerDied","Data":"b1fb7fd036c5c9d26f235b6cab7666d9e8af81694529bde0a87273ff70e9e3b8"}
Jul 28 17:34:16.511617 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:16.511210    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn"
Jul 28 17:34:16.511617 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:16.511246    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn" event={"ID":"00e17ba8-b199-4f61-83e7-9483f2ba0598","Type":"ContainerDied","Data":"8e078dce17a53f2159172eb77e07dafc2aa373acb6e3a1a8d4e3662d8675a2d7"}
Jul 28 17:34:16.511617 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:16.511285    2566 scope.go:117] "RemoveContainer" containerID="48e4eaf11d0d7ca68b3c5f92ad395dbe6f004c4a9143df2e9cd4779821369afc"
Jul 28 17:34:16.518832 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:16.518813    2566 scope.go:117] "RemoveContainer" containerID="b1fb7fd036c5c9d26f235b6cab7666d9e8af81694529bde0a87273ff70e9e3b8"
Jul 28 17:34:16.525672 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:16.525654    2566 scope.go:117] "RemoveContainer" containerID="48e4eaf11d0d7ca68b3c5f92ad395dbe6f004c4a9143df2e9cd4779821369afc"
Jul 28 17:34:16.525936 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:34:16.525916    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48e4eaf11d0d7ca68b3c5f92ad395dbe6f004c4a9143df2e9cd4779821369afc\": container with ID starting with 48e4eaf11d0d7ca68b3c5f92ad395dbe6f004c4a9143df2e9cd4779821369afc not found: ID does not exist" containerID="48e4eaf11d0d7ca68b3c5f92ad395dbe6f004c4a9143df2e9cd4779821369afc"
Jul 28 17:34:16.526005 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:16.525949    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48e4eaf11d0d7ca68b3c5f92ad395dbe6f004c4a9143df2e9cd4779821369afc"} err="failed to get container status \"48e4eaf11d0d7ca68b3c5f92ad395dbe6f004c4a9143df2e9cd4779821369afc\": rpc error: code = NotFound desc = could not find container \"48e4eaf11d0d7ca68b3c5f92ad395dbe6f004c4a9143df2e9cd4779821369afc\": container with ID starting with 48e4eaf11d0d7ca68b3c5f92ad395dbe6f004c4a9143df2e9cd4779821369afc not found: ID does not exist"
Jul 28 17:34:16.526005 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:16.525974    2566 scope.go:117] "RemoveContainer" containerID="b1fb7fd036c5c9d26f235b6cab7666d9e8af81694529bde0a87273ff70e9e3b8"
Jul 28 17:34:16.526251 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:34:16.526225    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1fb7fd036c5c9d26f235b6cab7666d9e8af81694529bde0a87273ff70e9e3b8\": container with ID starting with b1fb7fd036c5c9d26f235b6cab7666d9e8af81694529bde0a87273ff70e9e3b8 not found: ID does not exist" containerID="b1fb7fd036c5c9d26f235b6cab7666d9e8af81694529bde0a87273ff70e9e3b8"
Jul 28 17:34:16.526287 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:16.526258    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1fb7fd036c5c9d26f235b6cab7666d9e8af81694529bde0a87273ff70e9e3b8"} err="failed to get container status \"b1fb7fd036c5c9d26f235b6cab7666d9e8af81694529bde0a87273ff70e9e3b8\": rpc error: code = NotFound desc = could not find container \"b1fb7fd036c5c9d26f235b6cab7666d9e8af81694529bde0a87273ff70e9e3b8\": container with ID starting with b1fb7fd036c5c9d26f235b6cab7666d9e8af81694529bde0a87273ff70e9e3b8 not found: ID does not exist"
Jul 28 17:34:16.554005 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:16.553985    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00e17ba8-b199-4f61-83e7-9483f2ba0598-kube-rbac-proxy-sar-config\") pod \"00e17ba8-b199-4f61-83e7-9483f2ba0598\" (UID: \"00e17ba8-b199-4f61-83e7-9483f2ba0598\") "
Jul 28 17:34:16.554082 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:16.554018    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95ftm\" (UniqueName: \"kubernetes.io/projected/00e17ba8-b199-4f61-83e7-9483f2ba0598-kube-api-access-95ftm\") pod \"00e17ba8-b199-4f61-83e7-9483f2ba0598\" (UID: \"00e17ba8-b199-4f61-83e7-9483f2ba0598\") "
Jul 28 17:34:16.554122 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:16.554090    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00e17ba8-b199-4f61-83e7-9483f2ba0598-proxy-tls\") pod \"00e17ba8-b199-4f61-83e7-9483f2ba0598\" (UID: \"00e17ba8-b199-4f61-83e7-9483f2ba0598\") "
Jul 28 17:34:16.554348 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:16.554310    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00e17ba8-b199-4f61-83e7-9483f2ba0598-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "00e17ba8-b199-4f61-83e7-9483f2ba0598" (UID: "00e17ba8-b199-4f61-83e7-9483f2ba0598"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:34:16.556078 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:16.556058    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00e17ba8-b199-4f61-83e7-9483f2ba0598-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "00e17ba8-b199-4f61-83e7-9483f2ba0598" (UID: "00e17ba8-b199-4f61-83e7-9483f2ba0598"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:34:16.556191 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:16.556171    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00e17ba8-b199-4f61-83e7-9483f2ba0598-kube-api-access-95ftm" (OuterVolumeSpecName: "kube-api-access-95ftm") pod "00e17ba8-b199-4f61-83e7-9483f2ba0598" (UID: "00e17ba8-b199-4f61-83e7-9483f2ba0598"). InnerVolumeSpecName "kube-api-access-95ftm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:34:16.655308 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:16.655281    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00e17ba8-b199-4f61-83e7-9483f2ba0598-proxy-tls\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:34:16.655308 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:16.655303    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00e17ba8-b199-4f61-83e7-9483f2ba0598-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:34:16.655468 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:16.655313    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95ftm\" (UniqueName: \"kubernetes.io/projected/00e17ba8-b199-4f61-83e7-9483f2ba0598-kube-api-access-95ftm\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:34:16.834187 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:16.834160    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn"]
Jul 28 17:34:16.837719 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:16.837696    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn"]
Jul 28 17:34:18.097879 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:18.097845    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00e17ba8-b199-4f61-83e7-9483f2ba0598" path="/var/lib/kubelet/pods/00e17ba8-b199-4f61-83e7-9483f2ba0598/volumes"
Jul 28 17:34:19.469338 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:19.469296    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-f0903-56678cb99b-4nwlp" podUID="484ebf5f-3805-45f2-97a1-f3cb4caec5dd" containerName="switch-graph-f0903" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:34:20.511190 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:20.511156    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4"
Jul 28 17:34:20.511685 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:20.511606    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4" podUID="134b6f2e-e57e-44e6-b74c-e33cf4af9166" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 28 17:34:24.469348 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:24.469251    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-f0903-56678cb99b-4nwlp" podUID="484ebf5f-3805-45f2-97a1-f3cb4caec5dd" containerName="switch-graph-f0903" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:34:24.469915 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:24.469357    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-f0903-56678cb99b-4nwlp"
Jul 28 17:34:29.469175 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:29.469137    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-f0903-56678cb99b-4nwlp" podUID="484ebf5f-3805-45f2-97a1-f3cb4caec5dd" containerName="switch-graph-f0903" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:34:30.511741 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:30.511706    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4" podUID="134b6f2e-e57e-44e6-b74c-e33cf4af9166" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 28 17:34:34.469266 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:34.469225    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-f0903-56678cb99b-4nwlp" podUID="484ebf5f-3805-45f2-97a1-f3cb4caec5dd" containerName="switch-graph-f0903" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:34:39.469689 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:39.469647    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-f0903-56678cb99b-4nwlp" podUID="484ebf5f-3805-45f2-97a1-f3cb4caec5dd" containerName="switch-graph-f0903" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:34:40.512076 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:40.512034    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4" podUID="134b6f2e-e57e-44e6-b74c-e33cf4af9166" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 28 17:34:43.433161 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:43.433134    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-f0903-56678cb99b-4nwlp"
Jul 28 17:34:43.540239 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:43.540169    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/484ebf5f-3805-45f2-97a1-f3cb4caec5dd-openshift-service-ca-bundle\") pod \"484ebf5f-3805-45f2-97a1-f3cb4caec5dd\" (UID: \"484ebf5f-3805-45f2-97a1-f3cb4caec5dd\") "
Jul 28 17:34:43.540239 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:43.540214    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/484ebf5f-3805-45f2-97a1-f3cb4caec5dd-proxy-tls\") pod \"484ebf5f-3805-45f2-97a1-f3cb4caec5dd\" (UID: \"484ebf5f-3805-45f2-97a1-f3cb4caec5dd\") "
Jul 28 17:34:43.540535 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:43.540514    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/484ebf5f-3805-45f2-97a1-f3cb4caec5dd-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "484ebf5f-3805-45f2-97a1-f3cb4caec5dd" (UID: "484ebf5f-3805-45f2-97a1-f3cb4caec5dd"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:34:43.542160 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:43.542137    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/484ebf5f-3805-45f2-97a1-f3cb4caec5dd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "484ebf5f-3805-45f2-97a1-f3cb4caec5dd" (UID: "484ebf5f-3805-45f2-97a1-f3cb4caec5dd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:34:43.600703 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:43.600671    2566 generic.go:356] "Generic (PLEG): container finished" podID="484ebf5f-3805-45f2-97a1-f3cb4caec5dd" containerID="d17c668547600d5610736857b2711ea5cb3127fe206a97b0196b7a4a49c9e357" exitCode=137
Jul 28 17:34:43.600865 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:43.600713    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-f0903-56678cb99b-4nwlp" event={"ID":"484ebf5f-3805-45f2-97a1-f3cb4caec5dd","Type":"ContainerDied","Data":"d17c668547600d5610736857b2711ea5cb3127fe206a97b0196b7a4a49c9e357"}
Jul 28 17:34:43.600865 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:43.600720    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-f0903-56678cb99b-4nwlp"
Jul 28 17:34:43.600865 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:43.600764    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-f0903-56678cb99b-4nwlp" event={"ID":"484ebf5f-3805-45f2-97a1-f3cb4caec5dd","Type":"ContainerDied","Data":"11faedbe2b96e6645453b7aa970c34bf834c078a2ae9a98f5e856a31ea5c9097"}
Jul 28 17:34:43.600865 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:43.600792    2566 scope.go:117] "RemoveContainer" containerID="d17c668547600d5610736857b2711ea5cb3127fe206a97b0196b7a4a49c9e357"
Jul 28 17:34:43.609887 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:43.609869    2566 scope.go:117] "RemoveContainer" containerID="d17c668547600d5610736857b2711ea5cb3127fe206a97b0196b7a4a49c9e357"
Jul 28 17:34:43.610173 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:34:43.610143    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d17c668547600d5610736857b2711ea5cb3127fe206a97b0196b7a4a49c9e357\": container with ID starting with d17c668547600d5610736857b2711ea5cb3127fe206a97b0196b7a4a49c9e357 not found: ID does not exist" containerID="d17c668547600d5610736857b2711ea5cb3127fe206a97b0196b7a4a49c9e357"
Jul 28 17:34:43.610265 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:43.610183    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d17c668547600d5610736857b2711ea5cb3127fe206a97b0196b7a4a49c9e357"} err="failed to get container status \"d17c668547600d5610736857b2711ea5cb3127fe206a97b0196b7a4a49c9e357\": rpc error: code = NotFound desc = could not find container \"d17c668547600d5610736857b2711ea5cb3127fe206a97b0196b7a4a49c9e357\": container with ID starting with d17c668547600d5610736857b2711ea5cb3127fe206a97b0196b7a4a49c9e357 not found: ID does not exist"
Jul 28 17:34:43.623704 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:43.623681    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-f0903-56678cb99b-4nwlp"]
Jul 28 17:34:43.630231 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:43.630209    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-f0903-56678cb99b-4nwlp"]
Jul 28 17:34:43.641484 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:43.641458    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/484ebf5f-3805-45f2-97a1-f3cb4caec5dd-openshift-service-ca-bundle\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:34:43.641484 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:43.641479    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/484ebf5f-3805-45f2-97a1-f3cb4caec5dd-proxy-tls\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:34:44.097246 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:44.097216    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="484ebf5f-3805-45f2-97a1-f3cb4caec5dd" path="/var/lib/kubelet/pods/484ebf5f-3805-45f2-97a1-f3cb4caec5dd/volumes"
Jul 28 17:34:50.512472 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:34:50.512436    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4" podUID="134b6f2e-e57e-44e6-b74c-e33cf4af9166" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 28 17:35:00.512181 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:00.512152    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4"
Jul 28 17:35:09.015381 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:09.015346    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-64698579b9-v2bps"]
Jul 28 17:35:09.015745 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:09.015608    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00e17ba8-b199-4f61-83e7-9483f2ba0598" containerName="kube-rbac-proxy"
Jul 28 17:35:09.015745 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:09.015618    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="00e17ba8-b199-4f61-83e7-9483f2ba0598" containerName="kube-rbac-proxy"
Jul 28 17:35:09.015745 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:09.015636    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00e17ba8-b199-4f61-83e7-9483f2ba0598" containerName="kserve-container"
Jul 28 17:35:09.015745 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:09.015641    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="00e17ba8-b199-4f61-83e7-9483f2ba0598" containerName="kserve-container"
Jul 28 17:35:09.015745 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:09.015648    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="484ebf5f-3805-45f2-97a1-f3cb4caec5dd" containerName="switch-graph-f0903"
Jul 28 17:35:09.015745 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:09.015654    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="484ebf5f-3805-45f2-97a1-f3cb4caec5dd" containerName="switch-graph-f0903"
Jul 28 17:35:09.015745 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:09.015692    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="00e17ba8-b199-4f61-83e7-9483f2ba0598" containerName="kserve-container"
Jul 28 17:35:09.015745 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:09.015701    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="00e17ba8-b199-4f61-83e7-9483f2ba0598" containerName="kube-rbac-proxy"
Jul 28 17:35:09.015745 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:09.015707    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="484ebf5f-3805-45f2-97a1-f3cb4caec5dd" containerName="switch-graph-f0903"
Jul 28 17:35:09.018505 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:09.018488    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-64698579b9-v2bps"
Jul 28 17:35:09.022611 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:09.022590    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
Jul 28 17:35:09.022705 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:09.022610    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
Jul 28 17:35:09.039085 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:09.039057    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-64698579b9-v2bps"]
Jul 28 17:35:09.117579 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:09.117544    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8c677af-aff7-468b-a3fa-c9718b98b0b6-openshift-service-ca-bundle\") pod \"model-chainer-64698579b9-v2bps\" (UID: \"b8c677af-aff7-468b-a3fa-c9718b98b0b6\") " pod="kserve-ci-e2e-test/model-chainer-64698579b9-v2bps"
Jul 28 17:35:09.117726 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:09.117600    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8c677af-aff7-468b-a3fa-c9718b98b0b6-proxy-tls\") pod \"model-chainer-64698579b9-v2bps\" (UID: \"b8c677af-aff7-468b-a3fa-c9718b98b0b6\") " pod="kserve-ci-e2e-test/model-chainer-64698579b9-v2bps"
Jul 28 17:35:09.218432 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:09.218395    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8c677af-aff7-468b-a3fa-c9718b98b0b6-proxy-tls\") pod \"model-chainer-64698579b9-v2bps\" (UID: \"b8c677af-aff7-468b-a3fa-c9718b98b0b6\") " pod="kserve-ci-e2e-test/model-chainer-64698579b9-v2bps"
Jul 28 17:35:09.218591 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:09.218444    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8c677af-aff7-468b-a3fa-c9718b98b0b6-openshift-service-ca-bundle\") pod \"model-chainer-64698579b9-v2bps\" (UID: \"b8c677af-aff7-468b-a3fa-c9718b98b0b6\") " pod="kserve-ci-e2e-test/model-chainer-64698579b9-v2bps"
Jul 28 17:35:09.218591 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:35:09.218571    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-serving-cert: secret "model-chainer-serving-cert" not found
Jul 28 17:35:09.218707 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:35:09.218660    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8c677af-aff7-468b-a3fa-c9718b98b0b6-proxy-tls podName:b8c677af-aff7-468b-a3fa-c9718b98b0b6 nodeName:}" failed. No retries permitted until 2026-07-28 17:35:09.718638333 +0000 UTC m=+760.028803830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b8c677af-aff7-468b-a3fa-c9718b98b0b6-proxy-tls") pod "model-chainer-64698579b9-v2bps" (UID: "b8c677af-aff7-468b-a3fa-c9718b98b0b6") : secret "model-chainer-serving-cert" not found
Jul 28 17:35:09.219078 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:09.219059    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8c677af-aff7-468b-a3fa-c9718b98b0b6-openshift-service-ca-bundle\") pod \"model-chainer-64698579b9-v2bps\" (UID: \"b8c677af-aff7-468b-a3fa-c9718b98b0b6\") " pod="kserve-ci-e2e-test/model-chainer-64698579b9-v2bps"
Jul 28 17:35:09.723306 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:09.723248    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8c677af-aff7-468b-a3fa-c9718b98b0b6-proxy-tls\") pod \"model-chainer-64698579b9-v2bps\" (UID: \"b8c677af-aff7-468b-a3fa-c9718b98b0b6\") " pod="kserve-ci-e2e-test/model-chainer-64698579b9-v2bps"
Jul 28 17:35:09.725715 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:09.725690    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8c677af-aff7-468b-a3fa-c9718b98b0b6-proxy-tls\") pod \"model-chainer-64698579b9-v2bps\" (UID: \"b8c677af-aff7-468b-a3fa-c9718b98b0b6\") " pod="kserve-ci-e2e-test/model-chainer-64698579b9-v2bps"
Jul 28 17:35:09.927752 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:09.927701    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-64698579b9-v2bps"
Jul 28 17:35:10.052749 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:10.052722    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-64698579b9-v2bps"]
Jul 28 17:35:10.054993 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:35:10.054968    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c677af_aff7_468b_a3fa_c9718b98b0b6.slice/crio-8f0c89bd781ac631489db4be0b9225a6923de552f21adae6d78b55120443aacc WatchSource:0}: Error finding container 8f0c89bd781ac631489db4be0b9225a6923de552f21adae6d78b55120443aacc: Status 404 returned error can't find the container with id 8f0c89bd781ac631489db4be0b9225a6923de552f21adae6d78b55120443aacc
Jul 28 17:35:10.688122 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:10.688080    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-64698579b9-v2bps" event={"ID":"b8c677af-aff7-468b-a3fa-c9718b98b0b6","Type":"ContainerStarted","Data":"712139ce7ae1f58571eafc5d3140414eb0c8c19a6585877aecd4777022324ac9"}
Jul 28 17:35:10.688287 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:10.688141    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-64698579b9-v2bps" event={"ID":"b8c677af-aff7-468b-a3fa-c9718b98b0b6","Type":"ContainerStarted","Data":"8f0c89bd781ac631489db4be0b9225a6923de552f21adae6d78b55120443aacc"}
Jul 28 17:35:10.688287 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:10.688222    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-64698579b9-v2bps"
Jul 28 17:35:10.706048 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:10.705999    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-64698579b9-v2bps" podStartSLOduration=2.705985701 podStartE2EDuration="2.705985701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:35:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:35:10.705479576 +0000 UTC m=+761.015645093" watchObservedRunningTime="2026-07-28 17:35:10.705985701 +0000 UTC m=+761.016151206"
Jul 28 17:35:13.521909 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:13.521875    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-22e0e-5d85974d85-2j87g"]
Jul 28 17:35:13.525021 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:13.525005    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-22e0e-5d85974d85-2j87g"
Jul 28 17:35:13.527928 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:13.527904    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-22e0e-serving-cert\""
Jul 28 17:35:13.528047 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:13.527951    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-22e0e-kube-rbac-proxy-sar-config\""
Jul 28 17:35:13.535240 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:13.535216    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-22e0e-5d85974d85-2j87g"]
Jul 28 17:35:13.550525 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:13.550500    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d84d88cc-b604-4ff6-8534-40919f22c7b5-proxy-tls\") pod \"switch-graph-22e0e-5d85974d85-2j87g\" (UID: \"d84d88cc-b604-4ff6-8534-40919f22c7b5\") " pod="kserve-ci-e2e-test/switch-graph-22e0e-5d85974d85-2j87g"
Jul 28 17:35:13.550653 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:13.550548    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d84d88cc-b604-4ff6-8534-40919f22c7b5-openshift-service-ca-bundle\") pod \"switch-graph-22e0e-5d85974d85-2j87g\" (UID: \"d84d88cc-b604-4ff6-8534-40919f22c7b5\") " pod="kserve-ci-e2e-test/switch-graph-22e0e-5d85974d85-2j87g"
Jul 28 17:35:13.651367 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:13.651311    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d84d88cc-b604-4ff6-8534-40919f22c7b5-openshift-service-ca-bundle\") pod \"switch-graph-22e0e-5d85974d85-2j87g\" (UID: \"d84d88cc-b604-4ff6-8534-40919f22c7b5\") " pod="kserve-ci-e2e-test/switch-graph-22e0e-5d85974d85-2j87g"
Jul 28 17:35:13.651541 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:13.651388    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d84d88cc-b604-4ff6-8534-40919f22c7b5-proxy-tls\") pod \"switch-graph-22e0e-5d85974d85-2j87g\" (UID: \"d84d88cc-b604-4ff6-8534-40919f22c7b5\") " pod="kserve-ci-e2e-test/switch-graph-22e0e-5d85974d85-2j87g"
Jul 28 17:35:13.651541 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:35:13.651488    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-22e0e-serving-cert: secret "switch-graph-22e0e-serving-cert" not found
Jul 28 17:35:13.651631 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:35:13.651549    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d84d88cc-b604-4ff6-8534-40919f22c7b5-proxy-tls podName:d84d88cc-b604-4ff6-8534-40919f22c7b5 nodeName:}" failed. No retries permitted until 2026-07-28 17:35:14.151533433 +0000 UTC m=+764.461698917 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d84d88cc-b604-4ff6-8534-40919f22c7b5-proxy-tls") pod "switch-graph-22e0e-5d85974d85-2j87g" (UID: "d84d88cc-b604-4ff6-8534-40919f22c7b5") : secret "switch-graph-22e0e-serving-cert" not found
Jul 28 17:35:13.651981 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:13.651961    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d84d88cc-b604-4ff6-8534-40919f22c7b5-openshift-service-ca-bundle\") pod \"switch-graph-22e0e-5d85974d85-2j87g\" (UID: \"d84d88cc-b604-4ff6-8534-40919f22c7b5\") " pod="kserve-ci-e2e-test/switch-graph-22e0e-5d85974d85-2j87g"
Jul 28 17:35:14.155244 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:14.155209    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d84d88cc-b604-4ff6-8534-40919f22c7b5-proxy-tls\") pod \"switch-graph-22e0e-5d85974d85-2j87g\" (UID: \"d84d88cc-b604-4ff6-8534-40919f22c7b5\") " pod="kserve-ci-e2e-test/switch-graph-22e0e-5d85974d85-2j87g"
Jul 28 17:35:14.157787 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:14.157762    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d84d88cc-b604-4ff6-8534-40919f22c7b5-proxy-tls\") pod \"switch-graph-22e0e-5d85974d85-2j87g\" (UID: \"d84d88cc-b604-4ff6-8534-40919f22c7b5\") " pod="kserve-ci-e2e-test/switch-graph-22e0e-5d85974d85-2j87g"
Jul 28 17:35:14.434815 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:14.434784    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-22e0e-5d85974d85-2j87g"
Jul 28 17:35:14.560442 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:14.560297    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-22e0e-5d85974d85-2j87g"]
Jul 28 17:35:14.563385 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:35:14.563352    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd84d88cc_b604_4ff6_8534_40919f22c7b5.slice/crio-54627b9b85c6c561f6bae4373ecdef9c6cb0d4cac5682177229d869847110305 WatchSource:0}: Error finding container 54627b9b85c6c561f6bae4373ecdef9c6cb0d4cac5682177229d869847110305: Status 404 returned error can't find the container with id 54627b9b85c6c561f6bae4373ecdef9c6cb0d4cac5682177229d869847110305
Jul 28 17:35:14.704240 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:14.704151    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-22e0e-5d85974d85-2j87g" event={"ID":"d84d88cc-b604-4ff6-8534-40919f22c7b5","Type":"ContainerStarted","Data":"1c42caf0d37171d1ae25349926c9e37216d8c2f3722bcceb5ddd787d45f6ce4d"}
Jul 28 17:35:14.704240 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:14.704203    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-22e0e-5d85974d85-2j87g" event={"ID":"d84d88cc-b604-4ff6-8534-40919f22c7b5","Type":"ContainerStarted","Data":"54627b9b85c6c561f6bae4373ecdef9c6cb0d4cac5682177229d869847110305"}
Jul 28 17:35:14.704481 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:14.704282    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-22e0e-5d85974d85-2j87g"
Jul 28 17:35:14.722940 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:14.722899    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-22e0e-5d85974d85-2j87g" podStartSLOduration=1.7228857180000001 podStartE2EDuration="1.722885718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:35:14.722285798 +0000 UTC m=+765.032451325" watchObservedRunningTime="2026-07-28 17:35:14.722885718 +0000 UTC m=+765.033051223"
Jul 28 17:35:16.698438 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:16.698407    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-64698579b9-v2bps"
Jul 28 17:35:20.713643 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:20.713612    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-22e0e-5d85974d85-2j87g"
Jul 28 17:35:23.104848 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:23.104818    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-64698579b9-v2bps"]
Jul 28 17:35:23.105212 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:23.105012    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-64698579b9-v2bps" podUID="b8c677af-aff7-468b-a3fa-c9718b98b0b6" containerName="model-chainer" containerID="cri-o://712139ce7ae1f58571eafc5d3140414eb0c8c19a6585877aecd4777022324ac9" gracePeriod=30
Jul 28 17:35:23.332311 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:23.332278    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj"]
Jul 28 17:35:23.335736 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:23.335713    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj"
Jul 28 17:35:23.338229 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:23.338210    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b1dfe-predictor-serving-cert\""
Jul 28 17:35:23.338362 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:23.338256    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b1dfe-kube-rbac-proxy-sar-config\""
Jul 28 17:35:23.345959 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:23.345933    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj"]
Jul 28 17:35:23.423126 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:23.423042    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4b3a598-cb9d-46a5-a17c-481a7e45fddb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj\" (UID: \"d4b3a598-cb9d-46a5-a17c-481a7e45fddb\") " pod="kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj"
Jul 28 17:35:23.423126 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:23.423083    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5gb5\" (UniqueName: \"kubernetes.io/projected/d4b3a598-cb9d-46a5-a17c-481a7e45fddb-kube-api-access-q5gb5\") pod \"success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj\" (UID: \"d4b3a598-cb9d-46a5-a17c-481a7e45fddb\") " pod="kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj"
Jul 28 17:35:23.423126 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:23.423106    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4b3a598-cb9d-46a5-a17c-481a7e45fddb-proxy-tls\") pod \"success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj\" (UID: \"d4b3a598-cb9d-46a5-a17c-481a7e45fddb\") " pod="kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj"
Jul 28 17:35:23.524002 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:23.523972    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4b3a598-cb9d-46a5-a17c-481a7e45fddb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj\" (UID: \"d4b3a598-cb9d-46a5-a17c-481a7e45fddb\") " pod="kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj"
Jul 28 17:35:23.524002 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:23.524007    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5gb5\" (UniqueName: \"kubernetes.io/projected/d4b3a598-cb9d-46a5-a17c-481a7e45fddb-kube-api-access-q5gb5\") pod \"success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj\" (UID: \"d4b3a598-cb9d-46a5-a17c-481a7e45fddb\") " pod="kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj"
Jul 28 17:35:23.524244 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:23.524026    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4b3a598-cb9d-46a5-a17c-481a7e45fddb-proxy-tls\") pod \"success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj\" (UID: \"d4b3a598-cb9d-46a5-a17c-481a7e45fddb\") " pod="kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj"
Jul 28 17:35:23.524684 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:23.524660    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4b3a598-cb9d-46a5-a17c-481a7e45fddb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj\" (UID: \"d4b3a598-cb9d-46a5-a17c-481a7e45fddb\") " pod="kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj"
Jul 28 17:35:23.526360 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:23.526340    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4b3a598-cb9d-46a5-a17c-481a7e45fddb-proxy-tls\") pod \"success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj\" (UID: \"d4b3a598-cb9d-46a5-a17c-481a7e45fddb\") " pod="kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj"
Jul 28 17:35:23.532810 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:23.532788    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5gb5\" (UniqueName: \"kubernetes.io/projected/d4b3a598-cb9d-46a5-a17c-481a7e45fddb-kube-api-access-q5gb5\") pod \"success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj\" (UID: \"d4b3a598-cb9d-46a5-a17c-481a7e45fddb\") " pod="kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj"
Jul 28 17:35:23.646112 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:23.646075    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj"
Jul 28 17:35:23.770431 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:23.770408    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj"]
Jul 28 17:35:23.772950 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:35:23.772922    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b3a598_cb9d_46a5_a17c_481a7e45fddb.slice/crio-56c64348190e9fd53173e3fa05a1cf28ec47dc5df75071fc7fd63d79ef7e12c3 WatchSource:0}: Error finding container 56c64348190e9fd53173e3fa05a1cf28ec47dc5df75071fc7fd63d79ef7e12c3: Status 404 returned error can't find the container with id 56c64348190e9fd53173e3fa05a1cf28ec47dc5df75071fc7fd63d79ef7e12c3
Jul 28 17:35:24.740773 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:24.740740    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj" event={"ID":"d4b3a598-cb9d-46a5-a17c-481a7e45fddb","Type":"ContainerStarted","Data":"0a7fdda66282e72e2f883123bca7d6e87808764d4a815045562845f5e4a46780"}
Jul 28 17:35:24.741163 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:24.740798    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj" event={"ID":"d4b3a598-cb9d-46a5-a17c-481a7e45fddb","Type":"ContainerStarted","Data":"b07f10d87eb275a6bdac47ac153af194e57ac4d8d7659f6daa5dc22de4e08402"}
Jul 28 17:35:24.741163 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:24.740816    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj" event={"ID":"d4b3a598-cb9d-46a5-a17c-481a7e45fddb","Type":"ContainerStarted","Data":"56c64348190e9fd53173e3fa05a1cf28ec47dc5df75071fc7fd63d79ef7e12c3"}
Jul 28 17:35:24.741163 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:24.740929    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj"
Jul 28 17:35:24.741163 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:24.740970    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj"
Jul 28 17:35:24.742360 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:24.742315    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj" podUID="d4b3a598-cb9d-46a5-a17c-481a7e45fddb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 28 17:35:24.761783 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:24.761720    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj" podStartSLOduration=1.761686612 podStartE2EDuration="1.761686612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:35:24.759936167 +0000 UTC m=+775.070101690" watchObservedRunningTime="2026-07-28 17:35:24.761686612 +0000 UTC m=+775.071852116"
Jul 28 17:35:25.744962 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:25.744928    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj" podUID="d4b3a598-cb9d-46a5-a17c-481a7e45fddb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 28 17:35:26.695972 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:26.695928    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-64698579b9-v2bps" podUID="b8c677af-aff7-468b-a3fa-c9718b98b0b6" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:35:30.749951 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:30.749920    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj"
Jul 28 17:35:30.750450 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:30.750425    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj" podUID="d4b3a598-cb9d-46a5-a17c-481a7e45fddb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 28 17:35:31.696499 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:31.696456    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-64698579b9-v2bps" podUID="b8c677af-aff7-468b-a3fa-c9718b98b0b6" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:35:36.696173 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:36.696127    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-64698579b9-v2bps" podUID="b8c677af-aff7-468b-a3fa-c9718b98b0b6" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:35:36.696650 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:36.696226    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-64698579b9-v2bps"
Jul 28 17:35:40.750414 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:40.750368    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj" podUID="d4b3a598-cb9d-46a5-a17c-481a7e45fddb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 28 17:35:41.695752 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:41.695712    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-64698579b9-v2bps" podUID="b8c677af-aff7-468b-a3fa-c9718b98b0b6" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:35:46.695933 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:46.695892    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-64698579b9-v2bps" podUID="b8c677af-aff7-468b-a3fa-c9718b98b0b6" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:35:50.751161 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:50.751123    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj" podUID="d4b3a598-cb9d-46a5-a17c-481a7e45fddb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 28 17:35:51.696252 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:51.696212    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-64698579b9-v2bps" podUID="b8c677af-aff7-468b-a3fa-c9718b98b0b6" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:35:53.241731 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:53.241704    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-64698579b9-v2bps"
Jul 28 17:35:53.353593 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:53.353556    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8c677af-aff7-468b-a3fa-c9718b98b0b6-openshift-service-ca-bundle\") pod \"b8c677af-aff7-468b-a3fa-c9718b98b0b6\" (UID: \"b8c677af-aff7-468b-a3fa-c9718b98b0b6\") "
Jul 28 17:35:53.353758 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:53.353632    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8c677af-aff7-468b-a3fa-c9718b98b0b6-proxy-tls\") pod \"b8c677af-aff7-468b-a3fa-c9718b98b0b6\" (UID: \"b8c677af-aff7-468b-a3fa-c9718b98b0b6\") "
Jul 28 17:35:53.353919 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:53.353898    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8c677af-aff7-468b-a3fa-c9718b98b0b6-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "b8c677af-aff7-468b-a3fa-c9718b98b0b6" (UID: "b8c677af-aff7-468b-a3fa-c9718b98b0b6"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:35:53.355685 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:53.355661    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8c677af-aff7-468b-a3fa-c9718b98b0b6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b8c677af-aff7-468b-a3fa-c9718b98b0b6" (UID: "b8c677af-aff7-468b-a3fa-c9718b98b0b6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:35:53.454198 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:53.454167    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8c677af-aff7-468b-a3fa-c9718b98b0b6-proxy-tls\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:35:53.454198 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:53.454191    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8c677af-aff7-468b-a3fa-c9718b98b0b6-openshift-service-ca-bundle\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:35:53.840442 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:53.840406    2566 generic.go:356] "Generic (PLEG): container finished" podID="b8c677af-aff7-468b-a3fa-c9718b98b0b6" containerID="712139ce7ae1f58571eafc5d3140414eb0c8c19a6585877aecd4777022324ac9" exitCode=0
Jul 28 17:35:53.840633 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:53.840450    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-64698579b9-v2bps"
Jul 28 17:35:53.840633 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:53.840492    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-64698579b9-v2bps" event={"ID":"b8c677af-aff7-468b-a3fa-c9718b98b0b6","Type":"ContainerDied","Data":"712139ce7ae1f58571eafc5d3140414eb0c8c19a6585877aecd4777022324ac9"}
Jul 28 17:35:53.840633 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:53.840547    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-64698579b9-v2bps" event={"ID":"b8c677af-aff7-468b-a3fa-c9718b98b0b6","Type":"ContainerDied","Data":"8f0c89bd781ac631489db4be0b9225a6923de552f21adae6d78b55120443aacc"}
Jul 28 17:35:53.840633 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:53.840577    2566 scope.go:117] "RemoveContainer" containerID="712139ce7ae1f58571eafc5d3140414eb0c8c19a6585877aecd4777022324ac9"
Jul 28 17:35:53.848607 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:53.848591    2566 scope.go:117] "RemoveContainer" containerID="712139ce7ae1f58571eafc5d3140414eb0c8c19a6585877aecd4777022324ac9"
Jul 28 17:35:53.848859 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:35:53.848841    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"712139ce7ae1f58571eafc5d3140414eb0c8c19a6585877aecd4777022324ac9\": container with ID starting with 712139ce7ae1f58571eafc5d3140414eb0c8c19a6585877aecd4777022324ac9 not found: ID does not exist" containerID="712139ce7ae1f58571eafc5d3140414eb0c8c19a6585877aecd4777022324ac9"
Jul 28 17:35:53.848934 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:53.848866    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"712139ce7ae1f58571eafc5d3140414eb0c8c19a6585877aecd4777022324ac9"} err="failed to get container status \"712139ce7ae1f58571eafc5d3140414eb0c8c19a6585877aecd4777022324ac9\": rpc error: code = NotFound desc = could not find container \"712139ce7ae1f58571eafc5d3140414eb0c8c19a6585877aecd4777022324ac9\": container with ID starting with 712139ce7ae1f58571eafc5d3140414eb0c8c19a6585877aecd4777022324ac9 not found: ID does not exist"
Jul 28 17:35:53.862542 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:53.862519    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-64698579b9-v2bps"]
Jul 28 17:35:53.867038 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:53.867019    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-64698579b9-v2bps"]
Jul 28 17:35:54.097747 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:35:54.097674    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8c677af-aff7-468b-a3fa-c9718b98b0b6" path="/var/lib/kubelet/pods/b8c677af-aff7-468b-a3fa-c9718b98b0b6/volumes"
Jul 28 17:36:00.750553 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:36:00.750519    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj" podUID="d4b3a598-cb9d-46a5-a17c-481a7e45fddb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 28 17:36:10.751051 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:36:10.751023    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj"
Jul 28 17:36:23.353206 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:36:23.353170    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-b1dfe-bf8f99978-2vj8v"]
Jul 28 17:36:23.353669 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:36:23.353463    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8c677af-aff7-468b-a3fa-c9718b98b0b6" containerName="model-chainer"
Jul 28 17:36:23.353669 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:36:23.353474    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8c677af-aff7-468b-a3fa-c9718b98b0b6" containerName="model-chainer"
Jul 28 17:36:23.353669 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:36:23.353517    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8c677af-aff7-468b-a3fa-c9718b98b0b6" containerName="model-chainer"
Jul 28 17:36:23.356425 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:36:23.356407    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-b1dfe-bf8f99978-2vj8v"
Jul 28 17:36:23.359371 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:36:23.359354    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-b1dfe-serving-cert\""
Jul 28 17:36:23.359457 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:36:23.359356    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-b1dfe-kube-rbac-proxy-sar-config\""
Jul 28 17:36:23.366808 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:36:23.366785    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-b1dfe-bf8f99978-2vj8v"]
Jul 28 17:36:23.372660 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:36:23.372640    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd5242f7-86f5-4cde-9584-8c75ce5dfe4b-proxy-tls\") pod \"sequence-graph-b1dfe-bf8f99978-2vj8v\" (UID: \"bd5242f7-86f5-4cde-9584-8c75ce5dfe4b\") " pod="kserve-ci-e2e-test/sequence-graph-b1dfe-bf8f99978-2vj8v"
Jul 28 17:36:23.372754 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:36:23.372673    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd5242f7-86f5-4cde-9584-8c75ce5dfe4b-openshift-service-ca-bundle\") pod \"sequence-graph-b1dfe-bf8f99978-2vj8v\" (UID: \"bd5242f7-86f5-4cde-9584-8c75ce5dfe4b\") " pod="kserve-ci-e2e-test/sequence-graph-b1dfe-bf8f99978-2vj8v"
Jul 28 17:36:23.473960 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:36:23.473935    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd5242f7-86f5-4cde-9584-8c75ce5dfe4b-proxy-tls\") pod \"sequence-graph-b1dfe-bf8f99978-2vj8v\" (UID: \"bd5242f7-86f5-4cde-9584-8c75ce5dfe4b\") " pod="kserve-ci-e2e-test/sequence-graph-b1dfe-bf8f99978-2vj8v"
Jul 28 17:36:23.474097 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:36:23.473971    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd5242f7-86f5-4cde-9584-8c75ce5dfe4b-openshift-service-ca-bundle\") pod \"sequence-graph-b1dfe-bf8f99978-2vj8v\" (UID: \"bd5242f7-86f5-4cde-9584-8c75ce5dfe4b\") " pod="kserve-ci-e2e-test/sequence-graph-b1dfe-bf8f99978-2vj8v"
Jul 28 17:36:23.474611 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:36:23.474590    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd5242f7-86f5-4cde-9584-8c75ce5dfe4b-openshift-service-ca-bundle\") pod \"sequence-graph-b1dfe-bf8f99978-2vj8v\" (UID: \"bd5242f7-86f5-4cde-9584-8c75ce5dfe4b\") " pod="kserve-ci-e2e-test/sequence-graph-b1dfe-bf8f99978-2vj8v"
Jul 28 17:36:23.476469 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:36:23.476444    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd5242f7-86f5-4cde-9584-8c75ce5dfe4b-proxy-tls\") pod \"sequence-graph-b1dfe-bf8f99978-2vj8v\" (UID: \"bd5242f7-86f5-4cde-9584-8c75ce5dfe4b\") " pod="kserve-ci-e2e-test/sequence-graph-b1dfe-bf8f99978-2vj8v"
Jul 28 17:36:23.666555 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:36:23.666471    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-b1dfe-bf8f99978-2vj8v"
Jul 28 17:36:23.787617 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:36:23.787590    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-b1dfe-bf8f99978-2vj8v"]
Jul 28 17:36:23.789990 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:36:23.789959    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd5242f7_86f5_4cde_9584_8c75ce5dfe4b.slice/crio-3933d5fd7b23a31d291f3375dc0023775c455ef717297fa43d8bc5bc968a677e WatchSource:0}: Error finding container 3933d5fd7b23a31d291f3375dc0023775c455ef717297fa43d8bc5bc968a677e: Status 404 returned error can't find the container with id 3933d5fd7b23a31d291f3375dc0023775c455ef717297fa43d8bc5bc968a677e
Jul 28 17:36:23.946187 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:36:23.946153    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-b1dfe-bf8f99978-2vj8v" event={"ID":"bd5242f7-86f5-4cde-9584-8c75ce5dfe4b","Type":"ContainerStarted","Data":"4373602408a35eea4fc6c2e587c242e952ddbebda4a79c3d27ef0cb0a5856d31"}
Jul 28 17:36:23.946365 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:36:23.946209    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-b1dfe-bf8f99978-2vj8v" event={"ID":"bd5242f7-86f5-4cde-9584-8c75ce5dfe4b","Type":"ContainerStarted","Data":"3933d5fd7b23a31d291f3375dc0023775c455ef717297fa43d8bc5bc968a677e"}
Jul 28 17:36:23.946365 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:36:23.946251    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-b1dfe-bf8f99978-2vj8v"
Jul 28 17:36:23.964127 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:36:23.964081    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-b1dfe-bf8f99978-2vj8v" podStartSLOduration=0.964066896 podStartE2EDuration="964.066896ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:36:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:36:23.962642767 +0000 UTC m=+834.272808273" watchObservedRunningTime="2026-07-28 17:36:23.964066896 +0000 UTC m=+834.274232401"
Jul 28 17:36:29.955111 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:36:29.955085    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-b1dfe-bf8f99978-2vj8v"
Jul 28 17:43:28.058502 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:28.058427    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-22e0e-5d85974d85-2j87g"]
Jul 28 17:43:28.059012 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:28.058625    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-22e0e-5d85974d85-2j87g" podUID="d84d88cc-b604-4ff6-8534-40919f22c7b5" containerName="switch-graph-22e0e" containerID="cri-o://1c42caf0d37171d1ae25349926c9e37216d8c2f3722bcceb5ddd787d45f6ce4d" gracePeriod=30
Jul 28 17:43:28.320418 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:28.320344    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk"]
Jul 28 17:43:28.323698 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:28.323676    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk"
Jul 28 17:43:28.328338 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:28.328298    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8b8b5-kube-rbac-proxy-sar-config\""
Jul 28 17:43:28.328482 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:28.328338    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8b8b5-predictor-serving-cert\""
Jul 28 17:43:28.329626 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:28.329600    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4"]
Jul 28 17:43:28.329856 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:28.329811    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4" podUID="134b6f2e-e57e-44e6-b74c-e33cf4af9166" containerName="kserve-container" containerID="cri-o://a29589b1b74488da4aea215b38c53cfccae8c08b71fb43d9dc669fa1c2210d33" gracePeriod=30
Jul 28 17:43:28.329972 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:28.329886    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4" podUID="134b6f2e-e57e-44e6-b74c-e33cf4af9166" containerName="kube-rbac-proxy" containerID="cri-o://66b749214276a9424521ccd7fcf96c0b78cfe00339b7235d636bcfad7ad6d520" gracePeriod=30
Jul 28 17:43:28.340822 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:28.340798    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk"]
Jul 28 17:43:28.409598 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:28.409563    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c561db9f-a9b5-4c4f-87e5-4d5d5df07782-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk\" (UID: \"c561db9f-a9b5-4c4f-87e5-4d5d5df07782\") " pod="kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk"
Jul 28 17:43:28.409770 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:28.409610    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdn5j\" (UniqueName: \"kubernetes.io/projected/c561db9f-a9b5-4c4f-87e5-4d5d5df07782-kube-api-access-tdn5j\") pod \"success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk\" (UID: \"c561db9f-a9b5-4c4f-87e5-4d5d5df07782\") " pod="kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk"
Jul 28 17:43:28.409770 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:28.409699    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c561db9f-a9b5-4c4f-87e5-4d5d5df07782-proxy-tls\") pod \"success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk\" (UID: \"c561db9f-a9b5-4c4f-87e5-4d5d5df07782\") " pod="kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk"
Jul 28 17:43:28.510553 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:28.510518    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c561db9f-a9b5-4c4f-87e5-4d5d5df07782-proxy-tls\") pod \"success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk\" (UID: \"c561db9f-a9b5-4c4f-87e5-4d5d5df07782\") " pod="kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk"
Jul 28 17:43:28.510553 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:28.510554    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c561db9f-a9b5-4c4f-87e5-4d5d5df07782-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk\" (UID: \"c561db9f-a9b5-4c4f-87e5-4d5d5df07782\") " pod="kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk"
Jul 28 17:43:28.510792 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:28.510580    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdn5j\" (UniqueName: \"kubernetes.io/projected/c561db9f-a9b5-4c4f-87e5-4d5d5df07782-kube-api-access-tdn5j\") pod \"success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk\" (UID: \"c561db9f-a9b5-4c4f-87e5-4d5d5df07782\") " pod="kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk"
Jul 28 17:43:28.510792 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:43:28.510723    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-serving-cert: secret "success-200-isvc-8b8b5-predictor-serving-cert" not found
Jul 28 17:43:28.510792 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:43:28.510786    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c561db9f-a9b5-4c4f-87e5-4d5d5df07782-proxy-tls podName:c561db9f-a9b5-4c4f-87e5-4d5d5df07782 nodeName:}" failed. No retries permitted until 2026-07-28 17:43:29.010767121 +0000 UTC m=+1259.320932604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c561db9f-a9b5-4c4f-87e5-4d5d5df07782-proxy-tls") pod "success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk" (UID: "c561db9f-a9b5-4c4f-87e5-4d5d5df07782") : secret "success-200-isvc-8b8b5-predictor-serving-cert" not found
Jul 28 17:43:28.511229 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:28.511205    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c561db9f-a9b5-4c4f-87e5-4d5d5df07782-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk\" (UID: \"c561db9f-a9b5-4c4f-87e5-4d5d5df07782\") " pod="kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk"
Jul 28 17:43:28.522232 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:28.522203    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdn5j\" (UniqueName: \"kubernetes.io/projected/c561db9f-a9b5-4c4f-87e5-4d5d5df07782-kube-api-access-tdn5j\") pod \"success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk\" (UID: \"c561db9f-a9b5-4c4f-87e5-4d5d5df07782\") " pod="kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk"
Jul 28 17:43:29.015337 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:29.015287    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c561db9f-a9b5-4c4f-87e5-4d5d5df07782-proxy-tls\") pod \"success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk\" (UID: \"c561db9f-a9b5-4c4f-87e5-4d5d5df07782\") " pod="kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk"
Jul 28 17:43:29.017686 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:29.017657    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c561db9f-a9b5-4c4f-87e5-4d5d5df07782-proxy-tls\") pod \"success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk\" (UID: \"c561db9f-a9b5-4c4f-87e5-4d5d5df07782\") " pod="kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk"
Jul 28 17:43:29.236383 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:29.236342    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk"
Jul 28 17:43:29.369140 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:29.368965    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk"]
Jul 28 17:43:29.371671 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:43:29.371647    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc561db9f_a9b5_4c4f_87e5_4d5d5df07782.slice/crio-45619d3b57036d64bbc8b94eeb7747d667a4279b0e4626cc1f9217e1bc190645 WatchSource:0}: Error finding container 45619d3b57036d64bbc8b94eeb7747d667a4279b0e4626cc1f9217e1bc190645: Status 404 returned error can't find the container with id 45619d3b57036d64bbc8b94eeb7747d667a4279b0e4626cc1f9217e1bc190645
Jul 28 17:43:29.399683 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:29.399660    2566 generic.go:356] "Generic (PLEG): container finished" podID="134b6f2e-e57e-44e6-b74c-e33cf4af9166" containerID="66b749214276a9424521ccd7fcf96c0b78cfe00339b7235d636bcfad7ad6d520" exitCode=2
Jul 28 17:43:29.399814 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:29.399727    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4" event={"ID":"134b6f2e-e57e-44e6-b74c-e33cf4af9166","Type":"ContainerDied","Data":"66b749214276a9424521ccd7fcf96c0b78cfe00339b7235d636bcfad7ad6d520"}
Jul 28 17:43:29.400799 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:29.400778    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk" event={"ID":"c561db9f-a9b5-4c4f-87e5-4d5d5df07782","Type":"ContainerStarted","Data":"45619d3b57036d64bbc8b94eeb7747d667a4279b0e4626cc1f9217e1bc190645"}
Jul 28 17:43:30.406945 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:30.406895    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk" event={"ID":"c561db9f-a9b5-4c4f-87e5-4d5d5df07782","Type":"ContainerStarted","Data":"4ecfe79b54afb4742771064c51f0913c53e46b70870f1fe5a6955e07f5b19119"}
Jul 28 17:43:30.407374 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:30.406968    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk" event={"ID":"c561db9f-a9b5-4c4f-87e5-4d5d5df07782","Type":"ContainerStarted","Data":"3ba38287289cca3e127aea72a45a131e83a2b1e1c1d47d02077f174b77fd6232"}
Jul 28 17:43:30.407374 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:30.407053    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk"
Jul 28 17:43:30.438708 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:30.438660    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk" podStartSLOduration=2.438646803 podStartE2EDuration="2.438646803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:43:30.437903009 +0000 UTC m=+1260.748068513" watchObservedRunningTime="2026-07-28 17:43:30.438646803 +0000 UTC m=+1260.748812308"
Jul 28 17:43:30.506723 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:30.506681    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4" podUID="134b6f2e-e57e-44e6-b74c-e33cf4af9166" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.16:8643/healthz\": dial tcp 10.133.0.16:8643: connect: connection refused"
Jul 28 17:43:30.512357 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:30.512309    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4" podUID="134b6f2e-e57e-44e6-b74c-e33cf4af9166" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 28 17:43:30.711833 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:30.711793    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-22e0e-5d85974d85-2j87g" podUID="d84d88cc-b604-4ff6-8534-40919f22c7b5" containerName="switch-graph-22e0e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:43:31.413530 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:31.413500    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk"
Jul 28 17:43:31.414841 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:31.414815    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk" podUID="c561db9f-a9b5-4c4f-87e5-4d5d5df07782" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 28 17:43:31.575139 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:31.575110    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4"
Jul 28 17:43:31.634753 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:31.634719    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxwsz\" (UniqueName: \"kubernetes.io/projected/134b6f2e-e57e-44e6-b74c-e33cf4af9166-kube-api-access-fxwsz\") pod \"134b6f2e-e57e-44e6-b74c-e33cf4af9166\" (UID: \"134b6f2e-e57e-44e6-b74c-e33cf4af9166\") "
Jul 28 17:43:31.634753 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:31.634754    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/134b6f2e-e57e-44e6-b74c-e33cf4af9166-proxy-tls\") pod \"134b6f2e-e57e-44e6-b74c-e33cf4af9166\" (UID: \"134b6f2e-e57e-44e6-b74c-e33cf4af9166\") "
Jul 28 17:43:31.635008 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:31.634796    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/134b6f2e-e57e-44e6-b74c-e33cf4af9166-kube-rbac-proxy-sar-config\") pod \"134b6f2e-e57e-44e6-b74c-e33cf4af9166\" (UID: \"134b6f2e-e57e-44e6-b74c-e33cf4af9166\") "
Jul 28 17:43:31.635177 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:31.635152    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/134b6f2e-e57e-44e6-b74c-e33cf4af9166-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "134b6f2e-e57e-44e6-b74c-e33cf4af9166" (UID: "134b6f2e-e57e-44e6-b74c-e33cf4af9166"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:43:31.636826 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:31.636803    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/134b6f2e-e57e-44e6-b74c-e33cf4af9166-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "134b6f2e-e57e-44e6-b74c-e33cf4af9166" (UID: "134b6f2e-e57e-44e6-b74c-e33cf4af9166"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:43:31.636920 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:31.636825    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/134b6f2e-e57e-44e6-b74c-e33cf4af9166-kube-api-access-fxwsz" (OuterVolumeSpecName: "kube-api-access-fxwsz") pod "134b6f2e-e57e-44e6-b74c-e33cf4af9166" (UID: "134b6f2e-e57e-44e6-b74c-e33cf4af9166"). InnerVolumeSpecName "kube-api-access-fxwsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:43:31.735381 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:31.735342    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxwsz\" (UniqueName: \"kubernetes.io/projected/134b6f2e-e57e-44e6-b74c-e33cf4af9166-kube-api-access-fxwsz\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:43:31.735381 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:31.735376    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/134b6f2e-e57e-44e6-b74c-e33cf4af9166-proxy-tls\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:43:31.735381 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:31.735386    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/134b6f2e-e57e-44e6-b74c-e33cf4af9166-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:43:32.418850 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:32.418768    2566 generic.go:356] "Generic (PLEG): container finished" podID="134b6f2e-e57e-44e6-b74c-e33cf4af9166" containerID="a29589b1b74488da4aea215b38c53cfccae8c08b71fb43d9dc669fa1c2210d33" exitCode=0
Jul 28 17:43:32.418850 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:32.418817    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4"
Jul 28 17:43:32.419347 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:32.418853    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4" event={"ID":"134b6f2e-e57e-44e6-b74c-e33cf4af9166","Type":"ContainerDied","Data":"a29589b1b74488da4aea215b38c53cfccae8c08b71fb43d9dc669fa1c2210d33"}
Jul 28 17:43:32.419347 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:32.418918    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4" event={"ID":"134b6f2e-e57e-44e6-b74c-e33cf4af9166","Type":"ContainerDied","Data":"7a6139efa993527f368b108e0c5103d33c21641bc7b98f955be586d903cf65a8"}
Jul 28 17:43:32.419347 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:32.418962    2566 scope.go:117] "RemoveContainer" containerID="66b749214276a9424521ccd7fcf96c0b78cfe00339b7235d636bcfad7ad6d520"
Jul 28 17:43:32.419560 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:32.419512    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk" podUID="c561db9f-a9b5-4c4f-87e5-4d5d5df07782" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 28 17:43:32.426984 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:32.426965    2566 scope.go:117] "RemoveContainer" containerID="a29589b1b74488da4aea215b38c53cfccae8c08b71fb43d9dc669fa1c2210d33"
Jul 28 17:43:32.433657 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:32.433642    2566 scope.go:117] "RemoveContainer" containerID="66b749214276a9424521ccd7fcf96c0b78cfe00339b7235d636bcfad7ad6d520"
Jul 28 17:43:32.433892 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:43:32.433875    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66b749214276a9424521ccd7fcf96c0b78cfe00339b7235d636bcfad7ad6d520\": container with ID starting with 66b749214276a9424521ccd7fcf96c0b78cfe00339b7235d636bcfad7ad6d520 not found: ID does not exist" containerID="66b749214276a9424521ccd7fcf96c0b78cfe00339b7235d636bcfad7ad6d520"
Jul 28 17:43:32.433941 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:32.433900    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66b749214276a9424521ccd7fcf96c0b78cfe00339b7235d636bcfad7ad6d520"} err="failed to get container status \"66b749214276a9424521ccd7fcf96c0b78cfe00339b7235d636bcfad7ad6d520\": rpc error: code = NotFound desc = could not find container \"66b749214276a9424521ccd7fcf96c0b78cfe00339b7235d636bcfad7ad6d520\": container with ID starting with 66b749214276a9424521ccd7fcf96c0b78cfe00339b7235d636bcfad7ad6d520 not found: ID does not exist"
Jul 28 17:43:32.433941 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:32.433915    2566 scope.go:117] "RemoveContainer" containerID="a29589b1b74488da4aea215b38c53cfccae8c08b71fb43d9dc669fa1c2210d33"
Jul 28 17:43:32.434171 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:43:32.434153    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a29589b1b74488da4aea215b38c53cfccae8c08b71fb43d9dc669fa1c2210d33\": container with ID starting with a29589b1b74488da4aea215b38c53cfccae8c08b71fb43d9dc669fa1c2210d33 not found: ID does not exist" containerID="a29589b1b74488da4aea215b38c53cfccae8c08b71fb43d9dc669fa1c2210d33"
Jul 28 17:43:32.434219 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:32.434178    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a29589b1b74488da4aea215b38c53cfccae8c08b71fb43d9dc669fa1c2210d33"} err="failed to get container status \"a29589b1b74488da4aea215b38c53cfccae8c08b71fb43d9dc669fa1c2210d33\": rpc error: code = NotFound desc = could not find container \"a29589b1b74488da4aea215b38c53cfccae8c08b71fb43d9dc669fa1c2210d33\": container with ID starting with a29589b1b74488da4aea215b38c53cfccae8c08b71fb43d9dc669fa1c2210d33 not found: ID does not exist"
Jul 28 17:43:32.438307 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:32.438279    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4"]
Jul 28 17:43:32.447412 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:32.447393    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4"]
Jul 28 17:43:34.097524 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:34.097489    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="134b6f2e-e57e-44e6-b74c-e33cf4af9166" path="/var/lib/kubelet/pods/134b6f2e-e57e-44e6-b74c-e33cf4af9166/volumes"
Jul 28 17:43:35.712917 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:35.712879    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-22e0e-5d85974d85-2j87g" podUID="d84d88cc-b604-4ff6-8534-40919f22c7b5" containerName="switch-graph-22e0e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:43:37.423937 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:37.423909    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk"
Jul 28 17:43:37.424372 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:37.424348    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk" podUID="c561db9f-a9b5-4c4f-87e5-4d5d5df07782" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 28 17:43:40.712832 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:40.712796    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-22e0e-5d85974d85-2j87g" podUID="d84d88cc-b604-4ff6-8534-40919f22c7b5" containerName="switch-graph-22e0e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:43:40.713198 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:40.712889    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-22e0e-5d85974d85-2j87g"
Jul 28 17:43:45.712665 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:45.712619    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-22e0e-5d85974d85-2j87g" podUID="d84d88cc-b604-4ff6-8534-40919f22c7b5" containerName="switch-graph-22e0e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:43:47.424740 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:47.424702    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk" podUID="c561db9f-a9b5-4c4f-87e5-4d5d5df07782" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 28 17:43:50.712651 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:50.712613    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-22e0e-5d85974d85-2j87g" podUID="d84d88cc-b604-4ff6-8534-40919f22c7b5" containerName="switch-graph-22e0e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:43:55.712283 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:55.712234    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-22e0e-5d85974d85-2j87g" podUID="d84d88cc-b604-4ff6-8534-40919f22c7b5" containerName="switch-graph-22e0e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:43:57.424877 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:57.424838    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk" podUID="c561db9f-a9b5-4c4f-87e5-4d5d5df07782" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 28 17:43:58.191215 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:58.191191    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-22e0e-5d85974d85-2j87g"
Jul 28 17:43:58.231118 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:58.231088    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d84d88cc-b604-4ff6-8534-40919f22c7b5-proxy-tls\") pod \"d84d88cc-b604-4ff6-8534-40919f22c7b5\" (UID: \"d84d88cc-b604-4ff6-8534-40919f22c7b5\") "
Jul 28 17:43:58.231505 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:58.231134    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d84d88cc-b604-4ff6-8534-40919f22c7b5-openshift-service-ca-bundle\") pod \"d84d88cc-b604-4ff6-8534-40919f22c7b5\" (UID: \"d84d88cc-b604-4ff6-8534-40919f22c7b5\") "
Jul 28 17:43:58.231973 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:58.231944    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d84d88cc-b604-4ff6-8534-40919f22c7b5-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "d84d88cc-b604-4ff6-8534-40919f22c7b5" (UID: "d84d88cc-b604-4ff6-8534-40919f22c7b5"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:43:58.233525 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:58.233501    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d84d88cc-b604-4ff6-8534-40919f22c7b5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d84d88cc-b604-4ff6-8534-40919f22c7b5" (UID: "d84d88cc-b604-4ff6-8534-40919f22c7b5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:43:58.332511 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:58.332424    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d84d88cc-b604-4ff6-8534-40919f22c7b5-proxy-tls\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:43:58.332511 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:58.332455    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d84d88cc-b604-4ff6-8534-40919f22c7b5-openshift-service-ca-bundle\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:43:58.508956 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:58.508921    2566 generic.go:356] "Generic (PLEG): container finished" podID="d84d88cc-b604-4ff6-8534-40919f22c7b5" containerID="1c42caf0d37171d1ae25349926c9e37216d8c2f3722bcceb5ddd787d45f6ce4d" exitCode=0
Jul 28 17:43:58.508956 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:58.508959    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-22e0e-5d85974d85-2j87g"
Jul 28 17:43:58.509472 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:58.509008    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-22e0e-5d85974d85-2j87g" event={"ID":"d84d88cc-b604-4ff6-8534-40919f22c7b5","Type":"ContainerDied","Data":"1c42caf0d37171d1ae25349926c9e37216d8c2f3722bcceb5ddd787d45f6ce4d"}
Jul 28 17:43:58.509472 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:58.509069    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-22e0e-5d85974d85-2j87g" event={"ID":"d84d88cc-b604-4ff6-8534-40919f22c7b5","Type":"ContainerDied","Data":"54627b9b85c6c561f6bae4373ecdef9c6cb0d4cac5682177229d869847110305"}
Jul 28 17:43:58.509472 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:58.509105    2566 scope.go:117] "RemoveContainer" containerID="1c42caf0d37171d1ae25349926c9e37216d8c2f3722bcceb5ddd787d45f6ce4d"
Jul 28 17:43:58.518470 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:58.518454    2566 scope.go:117] "RemoveContainer" containerID="1c42caf0d37171d1ae25349926c9e37216d8c2f3722bcceb5ddd787d45f6ce4d"
Jul 28 17:43:58.518754 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:43:58.518728    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c42caf0d37171d1ae25349926c9e37216d8c2f3722bcceb5ddd787d45f6ce4d\": container with ID starting with 1c42caf0d37171d1ae25349926c9e37216d8c2f3722bcceb5ddd787d45f6ce4d not found: ID does not exist" containerID="1c42caf0d37171d1ae25349926c9e37216d8c2f3722bcceb5ddd787d45f6ce4d"
Jul 28 17:43:58.518844 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:58.518761    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c42caf0d37171d1ae25349926c9e37216d8c2f3722bcceb5ddd787d45f6ce4d"} err="failed to get container status \"1c42caf0d37171d1ae25349926c9e37216d8c2f3722bcceb5ddd787d45f6ce4d\": rpc error: code = NotFound desc = could not find container \"1c42caf0d37171d1ae25349926c9e37216d8c2f3722bcceb5ddd787d45f6ce4d\": container with ID starting with 1c42caf0d37171d1ae25349926c9e37216d8c2f3722bcceb5ddd787d45f6ce4d not found: ID does not exist"
Jul 28 17:43:58.531295 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:58.531265    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-22e0e-5d85974d85-2j87g"]
Jul 28 17:43:58.537133 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:43:58.537110    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-22e0e-5d85974d85-2j87g"]
Jul 28 17:44:00.101549 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:00.101503    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d84d88cc-b604-4ff6-8534-40919f22c7b5" path="/var/lib/kubelet/pods/d84d88cc-b604-4ff6-8534-40919f22c7b5/volumes"
Jul 28 17:44:07.424507 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:07.424467    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk" podUID="c561db9f-a9b5-4c4f-87e5-4d5d5df07782" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 28 17:44:17.425121 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:17.425092    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk"
Jul 28 17:44:28.364104 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:28.364064    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-8b8b5-846dd8bd57-vd26j"]
Jul 28 17:44:28.364590 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:28.364369    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d84d88cc-b604-4ff6-8534-40919f22c7b5" containerName="switch-graph-22e0e"
Jul 28 17:44:28.364590 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:28.364383    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d84d88cc-b604-4ff6-8534-40919f22c7b5" containerName="switch-graph-22e0e"
Jul 28 17:44:28.364590 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:28.364401    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="134b6f2e-e57e-44e6-b74c-e33cf4af9166" containerName="kube-rbac-proxy"
Jul 28 17:44:28.364590 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:28.364407    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="134b6f2e-e57e-44e6-b74c-e33cf4af9166" containerName="kube-rbac-proxy"
Jul 28 17:44:28.364590 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:28.364415    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="134b6f2e-e57e-44e6-b74c-e33cf4af9166" containerName="kserve-container"
Jul 28 17:44:28.364590 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:28.364422    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="134b6f2e-e57e-44e6-b74c-e33cf4af9166" containerName="kserve-container"
Jul 28 17:44:28.364590 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:28.364470    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d84d88cc-b604-4ff6-8534-40919f22c7b5" containerName="switch-graph-22e0e"
Jul 28 17:44:28.364590 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:28.364478    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="134b6f2e-e57e-44e6-b74c-e33cf4af9166" containerName="kserve-container"
Jul 28 17:44:28.364590 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:28.364485    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="134b6f2e-e57e-44e6-b74c-e33cf4af9166" containerName="kube-rbac-proxy"
Jul 28 17:44:28.368751 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:28.368731    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-8b8b5-846dd8bd57-vd26j"
Jul 28 17:44:28.372513 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:28.372495    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-8b8b5-serving-cert\""
Jul 28 17:44:28.372592 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:28.372500    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-8b8b5-kube-rbac-proxy-sar-config\""
Jul 28 17:44:28.381664 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:28.381639    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-8b8b5-846dd8bd57-vd26j"]
Jul 28 17:44:28.469481 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:28.469441    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f51a692-a6d4-40f1-9588-2944c833a3e5-openshift-service-ca-bundle\") pod \"ensemble-graph-8b8b5-846dd8bd57-vd26j\" (UID: \"5f51a692-a6d4-40f1-9588-2944c833a3e5\") " pod="kserve-ci-e2e-test/ensemble-graph-8b8b5-846dd8bd57-vd26j"
Jul 28 17:44:28.469647 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:28.469494    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f51a692-a6d4-40f1-9588-2944c833a3e5-proxy-tls\") pod \"ensemble-graph-8b8b5-846dd8bd57-vd26j\" (UID: \"5f51a692-a6d4-40f1-9588-2944c833a3e5\") " pod="kserve-ci-e2e-test/ensemble-graph-8b8b5-846dd8bd57-vd26j"
Jul 28 17:44:28.570219 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:28.570182    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f51a692-a6d4-40f1-9588-2944c833a3e5-openshift-service-ca-bundle\") pod \"ensemble-graph-8b8b5-846dd8bd57-vd26j\" (UID: \"5f51a692-a6d4-40f1-9588-2944c833a3e5\") " pod="kserve-ci-e2e-test/ensemble-graph-8b8b5-846dd8bd57-vd26j"
Jul 28 17:44:28.570442 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:28.570235    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f51a692-a6d4-40f1-9588-2944c833a3e5-proxy-tls\") pod \"ensemble-graph-8b8b5-846dd8bd57-vd26j\" (UID: \"5f51a692-a6d4-40f1-9588-2944c833a3e5\") " pod="kserve-ci-e2e-test/ensemble-graph-8b8b5-846dd8bd57-vd26j"
Jul 28 17:44:28.570852 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:28.570829    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f51a692-a6d4-40f1-9588-2944c833a3e5-openshift-service-ca-bundle\") pod \"ensemble-graph-8b8b5-846dd8bd57-vd26j\" (UID: \"5f51a692-a6d4-40f1-9588-2944c833a3e5\") " pod="kserve-ci-e2e-test/ensemble-graph-8b8b5-846dd8bd57-vd26j"
Jul 28 17:44:28.572624 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:28.572605    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f51a692-a6d4-40f1-9588-2944c833a3e5-proxy-tls\") pod \"ensemble-graph-8b8b5-846dd8bd57-vd26j\" (UID: \"5f51a692-a6d4-40f1-9588-2944c833a3e5\") " pod="kserve-ci-e2e-test/ensemble-graph-8b8b5-846dd8bd57-vd26j"
Jul 28 17:44:28.678488 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:28.678397    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-8b8b5-846dd8bd57-vd26j"
Jul 28 17:44:28.815120 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:28.815088    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-8b8b5-846dd8bd57-vd26j"]
Jul 28 17:44:28.818192 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:44:28.818164    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f51a692_a6d4_40f1_9588_2944c833a3e5.slice/crio-7bbb02fd94adbc9b9bd194fd2500387946bcfa977193389c8c14ae11d4500900 WatchSource:0}: Error finding container 7bbb02fd94adbc9b9bd194fd2500387946bcfa977193389c8c14ae11d4500900: Status 404 returned error can't find the container with id 7bbb02fd94adbc9b9bd194fd2500387946bcfa977193389c8c14ae11d4500900
Jul 28 17:44:29.619851 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:29.619815    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-8b8b5-846dd8bd57-vd26j" event={"ID":"5f51a692-a6d4-40f1-9588-2944c833a3e5","Type":"ContainerStarted","Data":"8d599028f70c2d8535914d8b711ebe0a08679aa3ad336a4d07bb9031dc5f789d"}
Jul 28 17:44:29.620248 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:29.619892    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-8b8b5-846dd8bd57-vd26j" event={"ID":"5f51a692-a6d4-40f1-9588-2944c833a3e5","Type":"ContainerStarted","Data":"7bbb02fd94adbc9b9bd194fd2500387946bcfa977193389c8c14ae11d4500900"}
Jul 28 17:44:29.620248 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:29.619918    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-8b8b5-846dd8bd57-vd26j"
Jul 28 17:44:29.637262 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:29.637213    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-8b8b5-846dd8bd57-vd26j" podStartSLOduration=1.637198502 podStartE2EDuration="1.637198502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:44:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:44:29.635922788 +0000 UTC m=+1319.946088294" watchObservedRunningTime="2026-07-28 17:44:29.637198502 +0000 UTC m=+1319.947364007"
Jul 28 17:44:35.629971 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:35.629936    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-8b8b5-846dd8bd57-vd26j"
Jul 28 17:44:37.974706 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:37.974673    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-b1dfe-bf8f99978-2vj8v"]
Jul 28 17:44:37.975111 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:37.974861    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-b1dfe-bf8f99978-2vj8v" podUID="bd5242f7-86f5-4cde-9584-8c75ce5dfe4b" containerName="sequence-graph-b1dfe" containerID="cri-o://4373602408a35eea4fc6c2e587c242e952ddbebda4a79c3d27ef0cb0a5856d31" gracePeriod=30
Jul 28 17:44:38.286840 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:38.286758    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k"]
Jul 28 17:44:38.291517 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:38.291495    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k"
Jul 28 17:44:38.296583 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:38.296561    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-61a23-predictor-serving-cert\""
Jul 28 17:44:38.296703 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:38.296655    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-61a23-kube-rbac-proxy-sar-config\""
Jul 28 17:44:38.316903 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:38.316875    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k"]
Jul 28 17:44:38.343800 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:38.343773    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca-proxy-tls\") pod \"success-200-isvc-61a23-predictor-644f77f65c-7gb8k\" (UID: \"52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca\") " pod="kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k"
Jul 28 17:44:38.343800 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:38.343803    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpzpt\" (UniqueName: \"kubernetes.io/projected/52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca-kube-api-access-vpzpt\") pod \"success-200-isvc-61a23-predictor-644f77f65c-7gb8k\" (UID: \"52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca\") " pod="kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k"
Jul 28 17:44:38.343977 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:38.343827    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-61a23-predictor-644f77f65c-7gb8k\" (UID: \"52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca\") " pod="kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k"
Jul 28 17:44:38.386242 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:38.386210    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj"]
Jul 28 17:44:38.386539 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:38.386507    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj" podUID="d4b3a598-cb9d-46a5-a17c-481a7e45fddb" containerName="kserve-container" containerID="cri-o://b07f10d87eb275a6bdac47ac153af194e57ac4d8d7659f6daa5dc22de4e08402" gracePeriod=30
Jul 28 17:44:38.386657 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:38.386537    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj" podUID="d4b3a598-cb9d-46a5-a17c-481a7e45fddb" containerName="kube-rbac-proxy" containerID="cri-o://0a7fdda66282e72e2f883123bca7d6e87808764d4a815045562845f5e4a46780" gracePeriod=30
Jul 28 17:44:38.443973 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:38.443937    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-8b8b5-846dd8bd57-vd26j"]
Jul 28 17:44:38.444195 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:38.444160    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-8b8b5-846dd8bd57-vd26j" podUID="5f51a692-a6d4-40f1-9588-2944c833a3e5" containerName="ensemble-graph-8b8b5" containerID="cri-o://8d599028f70c2d8535914d8b711ebe0a08679aa3ad336a4d07bb9031dc5f789d" gracePeriod=30
Jul 28 17:44:38.444347 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:38.444206    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca-proxy-tls\") pod \"success-200-isvc-61a23-predictor-644f77f65c-7gb8k\" (UID: \"52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca\") " pod="kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k"
Jul 28 17:44:38.444347 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:38.444242    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpzpt\" (UniqueName: \"kubernetes.io/projected/52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca-kube-api-access-vpzpt\") pod \"success-200-isvc-61a23-predictor-644f77f65c-7gb8k\" (UID: \"52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca\") " pod="kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k"
Jul 28 17:44:38.444347 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:38.444279    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-61a23-predictor-644f77f65c-7gb8k\" (UID: \"52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca\") " pod="kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k"
Jul 28 17:44:38.444513 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:44:38.444352    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-61a23-predictor-serving-cert: secret "success-200-isvc-61a23-predictor-serving-cert" not found
Jul 28 17:44:38.444513 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:44:38.444403    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca-proxy-tls podName:52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca nodeName:}" failed. No retries permitted until 2026-07-28 17:44:38.944384053 +0000 UTC m=+1329.254549536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca-proxy-tls") pod "success-200-isvc-61a23-predictor-644f77f65c-7gb8k" (UID: "52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca") : secret "success-200-isvc-61a23-predictor-serving-cert" not found
Jul 28 17:44:38.445003 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:38.444986    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-61a23-predictor-644f77f65c-7gb8k\" (UID: \"52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca\") " pod="kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k"
Jul 28 17:44:38.459439 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:38.459412    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpzpt\" (UniqueName: \"kubernetes.io/projected/52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca-kube-api-access-vpzpt\") pod \"success-200-isvc-61a23-predictor-644f77f65c-7gb8k\" (UID: \"52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca\") " pod="kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k"
Jul 28 17:44:38.652764 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:38.652681    2566 generic.go:356] "Generic (PLEG): container finished" podID="d4b3a598-cb9d-46a5-a17c-481a7e45fddb" containerID="0a7fdda66282e72e2f883123bca7d6e87808764d4a815045562845f5e4a46780" exitCode=2
Jul 28 17:44:38.652764 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:38.652727    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj" event={"ID":"d4b3a598-cb9d-46a5-a17c-481a7e45fddb","Type":"ContainerDied","Data":"0a7fdda66282e72e2f883123bca7d6e87808764d4a815045562845f5e4a46780"}
Jul 28 17:44:38.948746 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:38.948717    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca-proxy-tls\") pod \"success-200-isvc-61a23-predictor-644f77f65c-7gb8k\" (UID: \"52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca\") " pod="kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k"
Jul 28 17:44:38.948900 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:44:38.948832    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-61a23-predictor-serving-cert: secret "success-200-isvc-61a23-predictor-serving-cert" not found
Jul 28 17:44:38.948900 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:44:38.948886    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca-proxy-tls podName:52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca nodeName:}" failed. No retries permitted until 2026-07-28 17:44:39.948872786 +0000 UTC m=+1330.259038269 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca-proxy-tls") pod "success-200-isvc-61a23-predictor-644f77f65c-7gb8k" (UID: "52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca") : secret "success-200-isvc-61a23-predictor-serving-cert" not found
Jul 28 17:44:39.059794 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:39.059759    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8"]
Jul 28 17:44:39.063083 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:39.063064    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8"
Jul 28 17:44:39.068868 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:39.068829    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-614a5-predictor-serving-cert\""
Jul 28 17:44:39.078582 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:39.078560    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-614a5-kube-rbac-proxy-sar-config\""
Jul 28 17:44:39.097243 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:39.097207    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8"]
Jul 28 17:44:39.150544 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:39.150507    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt8rt\" (UniqueName: \"kubernetes.io/projected/0da81b68-4834-4e53-b9b1-1305fcec6bc6-kube-api-access-kt8rt\") pod \"success-200-isvc-614a5-predictor-776c647d45-4xwq8\" (UID: \"0da81b68-4834-4e53-b9b1-1305fcec6bc6\") " pod="kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8"
Jul 28 17:44:39.150703 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:39.150566    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0da81b68-4834-4e53-b9b1-1305fcec6bc6-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-614a5-predictor-776c647d45-4xwq8\" (UID: \"0da81b68-4834-4e53-b9b1-1305fcec6bc6\") " pod="kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8"
Jul 28 17:44:39.150703 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:39.150667    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0da81b68-4834-4e53-b9b1-1305fcec6bc6-proxy-tls\") pod \"success-200-isvc-614a5-predictor-776c647d45-4xwq8\" (UID: \"0da81b68-4834-4e53-b9b1-1305fcec6bc6\") " pod="kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8"
Jul 28 17:44:39.205254 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:39.205169    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk"]
Jul 28 17:44:39.205465 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:39.205415    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk" podUID="c561db9f-a9b5-4c4f-87e5-4d5d5df07782" containerName="kserve-container" containerID="cri-o://3ba38287289cca3e127aea72a45a131e83a2b1e1c1d47d02077f174b77fd6232" gracePeriod=30
Jul 28 17:44:39.205552 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:39.205446    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk" podUID="c561db9f-a9b5-4c4f-87e5-4d5d5df07782" containerName="kube-rbac-proxy" containerID="cri-o://4ecfe79b54afb4742771064c51f0913c53e46b70870f1fe5a6955e07f5b19119" gracePeriod=30
Jul 28 17:44:39.252009 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:39.251982    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0da81b68-4834-4e53-b9b1-1305fcec6bc6-proxy-tls\") pod \"success-200-isvc-614a5-predictor-776c647d45-4xwq8\" (UID: \"0da81b68-4834-4e53-b9b1-1305fcec6bc6\") " pod="kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8"
Jul 28 17:44:39.252170 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:39.252045    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt8rt\" (UniqueName: \"kubernetes.io/projected/0da81b68-4834-4e53-b9b1-1305fcec6bc6-kube-api-access-kt8rt\") pod \"success-200-isvc-614a5-predictor-776c647d45-4xwq8\" (UID: \"0da81b68-4834-4e53-b9b1-1305fcec6bc6\") " pod="kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8"
Jul 28 17:44:39.252170 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:39.252065    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0da81b68-4834-4e53-b9b1-1305fcec6bc6-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-614a5-predictor-776c647d45-4xwq8\" (UID: \"0da81b68-4834-4e53-b9b1-1305fcec6bc6\") " pod="kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8"
Jul 28 17:44:39.252682 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:39.252663    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0da81b68-4834-4e53-b9b1-1305fcec6bc6-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-614a5-predictor-776c647d45-4xwq8\" (UID: \"0da81b68-4834-4e53-b9b1-1305fcec6bc6\") " pod="kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8"
Jul 28 17:44:39.254538 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:39.254502    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0da81b68-4834-4e53-b9b1-1305fcec6bc6-proxy-tls\") pod \"success-200-isvc-614a5-predictor-776c647d45-4xwq8\" (UID: \"0da81b68-4834-4e53-b9b1-1305fcec6bc6\") " pod="kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8"
Jul 28 17:44:39.286560 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:39.286526    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt8rt\" (UniqueName: \"kubernetes.io/projected/0da81b68-4834-4e53-b9b1-1305fcec6bc6-kube-api-access-kt8rt\") pod \"success-200-isvc-614a5-predictor-776c647d45-4xwq8\" (UID: \"0da81b68-4834-4e53-b9b1-1305fcec6bc6\") " pod="kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8"
Jul 28 17:44:39.373642 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:39.373612    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8"
Jul 28 17:44:39.519775 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:39.519746    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8"]
Jul 28 17:44:39.522523 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:44:39.522492    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da81b68_4834_4e53_b9b1_1305fcec6bc6.slice/crio-207d9810f6cf426c7b44b27fe70577e37aa45f651e0c2085ff92780da949e146 WatchSource:0}: Error finding container 207d9810f6cf426c7b44b27fe70577e37aa45f651e0c2085ff92780da949e146: Status 404 returned error can't find the container with id 207d9810f6cf426c7b44b27fe70577e37aa45f651e0c2085ff92780da949e146
Jul 28 17:44:39.657716 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:39.657689    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8" event={"ID":"0da81b68-4834-4e53-b9b1-1305fcec6bc6","Type":"ContainerStarted","Data":"4b7fbb09d5a17d97b9de73c7657355a7b6b9d49dbae07786d8b9080dab8abaa5"}
Jul 28 17:44:39.657832 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:39.657788    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8" event={"ID":"0da81b68-4834-4e53-b9b1-1305fcec6bc6","Type":"ContainerStarted","Data":"d95b7574486f54196383609ef9ba293dbbbbcd8a0bec34681aa006d6145d1662"}
Jul 28 17:44:39.657892 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:39.657831    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8"
Jul 28 17:44:39.657892 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:39.657861    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8" event={"ID":"0da81b68-4834-4e53-b9b1-1305fcec6bc6","Type":"ContainerStarted","Data":"207d9810f6cf426c7b44b27fe70577e37aa45f651e0c2085ff92780da949e146"}
Jul 28 17:44:39.657984 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:39.657893    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8"
Jul 28 17:44:39.659597 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:39.659571    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8" podUID="0da81b68-4834-4e53-b9b1-1305fcec6bc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 17:44:39.659847 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:39.659825    2566 generic.go:356] "Generic (PLEG): container finished" podID="c561db9f-a9b5-4c4f-87e5-4d5d5df07782" containerID="4ecfe79b54afb4742771064c51f0913c53e46b70870f1fe5a6955e07f5b19119" exitCode=2
Jul 28 17:44:39.659892 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:39.659859    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk" event={"ID":"c561db9f-a9b5-4c4f-87e5-4d5d5df07782","Type":"ContainerDied","Data":"4ecfe79b54afb4742771064c51f0913c53e46b70870f1fe5a6955e07f5b19119"}
Jul 28 17:44:39.677435 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:39.677371    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8" podStartSLOduration=1.677359231 podStartE2EDuration="1.677359231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:44:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:44:39.67648485 +0000 UTC m=+1329.986650356" watchObservedRunningTime="2026-07-28 17:44:39.677359231 +0000 UTC m=+1329.987524818"
Jul 28 17:44:39.953992 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:39.953955    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-b1dfe-bf8f99978-2vj8v" podUID="bd5242f7-86f5-4cde-9584-8c75ce5dfe4b" containerName="sequence-graph-b1dfe" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:44:39.957427 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:39.957400    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca-proxy-tls\") pod \"success-200-isvc-61a23-predictor-644f77f65c-7gb8k\" (UID: \"52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca\") " pod="kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k"
Jul 28 17:44:39.959805 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:39.959783    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca-proxy-tls\") pod \"success-200-isvc-61a23-predictor-644f77f65c-7gb8k\" (UID: \"52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca\") " pod="kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k"
Jul 28 17:44:40.101856 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:40.101828    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k"
Jul 28 17:44:40.232681 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:40.232330    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k"]
Jul 28 17:44:40.234926 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:44:40.234888    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52699f3f_9cb0_45a0_a3ba_f1a1a1c5a1ca.slice/crio-4452e550f1deb489e7c4c31de51270b8905e55d7b506086f9c20afd9a12a5bb2 WatchSource:0}: Error finding container 4452e550f1deb489e7c4c31de51270b8905e55d7b506086f9c20afd9a12a5bb2: Status 404 returned error can't find the container with id 4452e550f1deb489e7c4c31de51270b8905e55d7b506086f9c20afd9a12a5bb2
Jul 28 17:44:40.628106 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:40.628010    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-8b8b5-846dd8bd57-vd26j" podUID="5f51a692-a6d4-40f1-9588-2944c833a3e5" containerName="ensemble-graph-8b8b5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:44:40.665692 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:40.665653    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k" event={"ID":"52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca","Type":"ContainerStarted","Data":"1e54f0e42d53ebb4c9b23c268a7ca7f44d3c7db61ec5b66782476ecc19231b0e"}
Jul 28 17:44:40.665842 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:40.665721    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k" event={"ID":"52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca","Type":"ContainerStarted","Data":"04f2d894d059dbb678aac08dec360146b674bc2dfc8c0aa69edf39920d644056"}
Jul 28 17:44:40.665842 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:40.665747    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k" event={"ID":"52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca","Type":"ContainerStarted","Data":"4452e550f1deb489e7c4c31de51270b8905e55d7b506086f9c20afd9a12a5bb2"}
Jul 28 17:44:40.665997 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:40.665964    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8" podUID="0da81b68-4834-4e53-b9b1-1305fcec6bc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 17:44:40.686274 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:40.686222    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k" podStartSLOduration=2.686201699 podStartE2EDuration="2.686201699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:44:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:44:40.685118552 +0000 UTC m=+1330.995284069" watchObservedRunningTime="2026-07-28 17:44:40.686201699 +0000 UTC m=+1330.996367204"
Jul 28 17:44:40.745648 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:40.745600    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj" podUID="d4b3a598-cb9d-46a5-a17c-481a7e45fddb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.19:8643/healthz\": dial tcp 10.133.0.19:8643: connect: connection refused"
Jul 28 17:44:40.750960 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:40.750926    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj" podUID="d4b3a598-cb9d-46a5-a17c-481a7e45fddb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 28 17:44:41.669675 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:41.669636    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k"
Jul 28 17:44:41.670040 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:41.669703    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k"
Jul 28 17:44:41.671035 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:41.671005    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k" podUID="52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 28 17:44:42.132044 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.132019    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj"
Jul 28 17:44:42.173674 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.173642    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4b3a598-cb9d-46a5-a17c-481a7e45fddb-proxy-tls\") pod \"d4b3a598-cb9d-46a5-a17c-481a7e45fddb\" (UID: \"d4b3a598-cb9d-46a5-a17c-481a7e45fddb\") "
Jul 28 17:44:42.173821 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.173693    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4b3a598-cb9d-46a5-a17c-481a7e45fddb-kube-rbac-proxy-sar-config\") pod \"d4b3a598-cb9d-46a5-a17c-481a7e45fddb\" (UID: \"d4b3a598-cb9d-46a5-a17c-481a7e45fddb\") "
Jul 28 17:44:42.173896 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.173874    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5gb5\" (UniqueName: \"kubernetes.io/projected/d4b3a598-cb9d-46a5-a17c-481a7e45fddb-kube-api-access-q5gb5\") pod \"d4b3a598-cb9d-46a5-a17c-481a7e45fddb\" (UID: \"d4b3a598-cb9d-46a5-a17c-481a7e45fddb\") "
Jul 28 17:44:42.174053 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.174031    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4b3a598-cb9d-46a5-a17c-481a7e45fddb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d4b3a598-cb9d-46a5-a17c-481a7e45fddb" (UID: "d4b3a598-cb9d-46a5-a17c-481a7e45fddb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:44:42.174118 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.174104    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4b3a598-cb9d-46a5-a17c-481a7e45fddb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:44:42.175794 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.175768    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4b3a598-cb9d-46a5-a17c-481a7e45fddb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d4b3a598-cb9d-46a5-a17c-481a7e45fddb" (UID: "d4b3a598-cb9d-46a5-a17c-481a7e45fddb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:44:42.175907 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.175846    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b3a598-cb9d-46a5-a17c-481a7e45fddb-kube-api-access-q5gb5" (OuterVolumeSpecName: "kube-api-access-q5gb5") pod "d4b3a598-cb9d-46a5-a17c-481a7e45fddb" (UID: "d4b3a598-cb9d-46a5-a17c-481a7e45fddb"). InnerVolumeSpecName "kube-api-access-q5gb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:44:42.274915 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.274833    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4b3a598-cb9d-46a5-a17c-481a7e45fddb-proxy-tls\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:44:42.274915 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.274859    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5gb5\" (UniqueName: \"kubernetes.io/projected/d4b3a598-cb9d-46a5-a17c-481a7e45fddb-kube-api-access-q5gb5\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:44:42.419999 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.419960    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk" podUID="c561db9f-a9b5-4c4f-87e5-4d5d5df07782" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Jul 28 17:44:42.543251 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.543227    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk"
Jul 28 17:44:42.576145 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.576116    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c561db9f-a9b5-4c4f-87e5-4d5d5df07782-kube-rbac-proxy-sar-config\") pod \"c561db9f-a9b5-4c4f-87e5-4d5d5df07782\" (UID: \"c561db9f-a9b5-4c4f-87e5-4d5d5df07782\") "
Jul 28 17:44:42.576333 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.576158    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdn5j\" (UniqueName: \"kubernetes.io/projected/c561db9f-a9b5-4c4f-87e5-4d5d5df07782-kube-api-access-tdn5j\") pod \"c561db9f-a9b5-4c4f-87e5-4d5d5df07782\" (UID: \"c561db9f-a9b5-4c4f-87e5-4d5d5df07782\") "
Jul 28 17:44:42.576333 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.576204    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c561db9f-a9b5-4c4f-87e5-4d5d5df07782-proxy-tls\") pod \"c561db9f-a9b5-4c4f-87e5-4d5d5df07782\" (UID: \"c561db9f-a9b5-4c4f-87e5-4d5d5df07782\") "
Jul 28 17:44:42.576542 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.576503    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c561db9f-a9b5-4c4f-87e5-4d5d5df07782-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c561db9f-a9b5-4c4f-87e5-4d5d5df07782" (UID: "c561db9f-a9b5-4c4f-87e5-4d5d5df07782"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:44:42.578245 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.578220    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c561db9f-a9b5-4c4f-87e5-4d5d5df07782-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c561db9f-a9b5-4c4f-87e5-4d5d5df07782" (UID: "c561db9f-a9b5-4c4f-87e5-4d5d5df07782"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:44:42.578245 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.578223    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c561db9f-a9b5-4c4f-87e5-4d5d5df07782-kube-api-access-tdn5j" (OuterVolumeSpecName: "kube-api-access-tdn5j") pod "c561db9f-a9b5-4c4f-87e5-4d5d5df07782" (UID: "c561db9f-a9b5-4c4f-87e5-4d5d5df07782"). InnerVolumeSpecName "kube-api-access-tdn5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:44:42.675471 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.675440    2566 generic.go:356] "Generic (PLEG): container finished" podID="c561db9f-a9b5-4c4f-87e5-4d5d5df07782" containerID="3ba38287289cca3e127aea72a45a131e83a2b1e1c1d47d02077f174b77fd6232" exitCode=0
Jul 28 17:44:42.675930 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.675487    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk"
Jul 28 17:44:42.675930 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.675513    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk" event={"ID":"c561db9f-a9b5-4c4f-87e5-4d5d5df07782","Type":"ContainerDied","Data":"3ba38287289cca3e127aea72a45a131e83a2b1e1c1d47d02077f174b77fd6232"}
Jul 28 17:44:42.675930 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.675571    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk" event={"ID":"c561db9f-a9b5-4c4f-87e5-4d5d5df07782","Type":"ContainerDied","Data":"45619d3b57036d64bbc8b94eeb7747d667a4279b0e4626cc1f9217e1bc190645"}
Jul 28 17:44:42.675930 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.675607    2566 scope.go:117] "RemoveContainer" containerID="4ecfe79b54afb4742771064c51f0913c53e46b70870f1fe5a6955e07f5b19119"
Jul 28 17:44:42.676788 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.676760    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c561db9f-a9b5-4c4f-87e5-4d5d5df07782-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:44:42.676788 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.676783    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdn5j\" (UniqueName: \"kubernetes.io/projected/c561db9f-a9b5-4c4f-87e5-4d5d5df07782-kube-api-access-tdn5j\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:44:42.676788 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.676794    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c561db9f-a9b5-4c4f-87e5-4d5d5df07782-proxy-tls\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:44:42.677195 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.677177    2566 generic.go:356] "Generic (PLEG): container finished" podID="d4b3a598-cb9d-46a5-a17c-481a7e45fddb" containerID="b07f10d87eb275a6bdac47ac153af194e57ac4d8d7659f6daa5dc22de4e08402" exitCode=0
Jul 28 17:44:42.677272 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.677228    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj"
Jul 28 17:44:42.677272 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.677265    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj" event={"ID":"d4b3a598-cb9d-46a5-a17c-481a7e45fddb","Type":"ContainerDied","Data":"b07f10d87eb275a6bdac47ac153af194e57ac4d8d7659f6daa5dc22de4e08402"}
Jul 28 17:44:42.677404 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.677299    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj" event={"ID":"d4b3a598-cb9d-46a5-a17c-481a7e45fddb","Type":"ContainerDied","Data":"56c64348190e9fd53173e3fa05a1cf28ec47dc5df75071fc7fd63d79ef7e12c3"}
Jul 28 17:44:42.677757 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.677726    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k" podUID="52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 28 17:44:42.689607 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.689583    2566 scope.go:117] "RemoveContainer" containerID="3ba38287289cca3e127aea72a45a131e83a2b1e1c1d47d02077f174b77fd6232"
Jul 28 17:44:42.696931 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.696915    2566 scope.go:117] "RemoveContainer" containerID="4ecfe79b54afb4742771064c51f0913c53e46b70870f1fe5a6955e07f5b19119"
Jul 28 17:44:42.697187 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:44:42.697168    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ecfe79b54afb4742771064c51f0913c53e46b70870f1fe5a6955e07f5b19119\": container with ID starting with 4ecfe79b54afb4742771064c51f0913c53e46b70870f1fe5a6955e07f5b19119 not found: ID does not exist" containerID="4ecfe79b54afb4742771064c51f0913c53e46b70870f1fe5a6955e07f5b19119"
Jul 28 17:44:42.697232 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.697195    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ecfe79b54afb4742771064c51f0913c53e46b70870f1fe5a6955e07f5b19119"} err="failed to get container status \"4ecfe79b54afb4742771064c51f0913c53e46b70870f1fe5a6955e07f5b19119\": rpc error: code = NotFound desc = could not find container \"4ecfe79b54afb4742771064c51f0913c53e46b70870f1fe5a6955e07f5b19119\": container with ID starting with 4ecfe79b54afb4742771064c51f0913c53e46b70870f1fe5a6955e07f5b19119 not found: ID does not exist"
Jul 28 17:44:42.697232 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.697212    2566 scope.go:117] "RemoveContainer" containerID="3ba38287289cca3e127aea72a45a131e83a2b1e1c1d47d02077f174b77fd6232"
Jul 28 17:44:42.697546 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:44:42.697518    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ba38287289cca3e127aea72a45a131e83a2b1e1c1d47d02077f174b77fd6232\": container with ID starting with 3ba38287289cca3e127aea72a45a131e83a2b1e1c1d47d02077f174b77fd6232 not found: ID does not exist" containerID="3ba38287289cca3e127aea72a45a131e83a2b1e1c1d47d02077f174b77fd6232"
Jul 28 17:44:42.697643 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.697552    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ba38287289cca3e127aea72a45a131e83a2b1e1c1d47d02077f174b77fd6232"} err="failed to get container status \"3ba38287289cca3e127aea72a45a131e83a2b1e1c1d47d02077f174b77fd6232\": rpc error: code = NotFound desc = could not find container \"3ba38287289cca3e127aea72a45a131e83a2b1e1c1d47d02077f174b77fd6232\": container with ID starting with 3ba38287289cca3e127aea72a45a131e83a2b1e1c1d47d02077f174b77fd6232 not found: ID does not exist"
Jul 28 17:44:42.697643 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.697572    2566 scope.go:117] "RemoveContainer" containerID="0a7fdda66282e72e2f883123bca7d6e87808764d4a815045562845f5e4a46780"
Jul 28 17:44:42.707480 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.707431    2566 scope.go:117] "RemoveContainer" containerID="b07f10d87eb275a6bdac47ac153af194e57ac4d8d7659f6daa5dc22de4e08402"
Jul 28 17:44:42.708751 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.708063    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj"]
Jul 28 17:44:42.709796 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.709775    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj"]
Jul 28 17:44:42.715550 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.715533    2566 scope.go:117] "RemoveContainer" containerID="0a7fdda66282e72e2f883123bca7d6e87808764d4a815045562845f5e4a46780"
Jul 28 17:44:42.715817 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:44:42.715801    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a7fdda66282e72e2f883123bca7d6e87808764d4a815045562845f5e4a46780\": container with ID starting with 0a7fdda66282e72e2f883123bca7d6e87808764d4a815045562845f5e4a46780 not found: ID does not exist" containerID="0a7fdda66282e72e2f883123bca7d6e87808764d4a815045562845f5e4a46780"
Jul 28 17:44:42.715877 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.715825    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a7fdda66282e72e2f883123bca7d6e87808764d4a815045562845f5e4a46780"} err="failed to get container status \"0a7fdda66282e72e2f883123bca7d6e87808764d4a815045562845f5e4a46780\": rpc error: code = NotFound desc = could not find container \"0a7fdda66282e72e2f883123bca7d6e87808764d4a815045562845f5e4a46780\": container with ID starting with 0a7fdda66282e72e2f883123bca7d6e87808764d4a815045562845f5e4a46780 not found: ID does not exist"
Jul 28 17:44:42.715877 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.715842    2566 scope.go:117] "RemoveContainer" containerID="b07f10d87eb275a6bdac47ac153af194e57ac4d8d7659f6daa5dc22de4e08402"
Jul 28 17:44:42.716080 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:44:42.716049    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b07f10d87eb275a6bdac47ac153af194e57ac4d8d7659f6daa5dc22de4e08402\": container with ID starting with b07f10d87eb275a6bdac47ac153af194e57ac4d8d7659f6daa5dc22de4e08402 not found: ID does not exist" containerID="b07f10d87eb275a6bdac47ac153af194e57ac4d8d7659f6daa5dc22de4e08402"
Jul 28 17:44:42.716163 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.716082    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b07f10d87eb275a6bdac47ac153af194e57ac4d8d7659f6daa5dc22de4e08402"} err="failed to get container status \"b07f10d87eb275a6bdac47ac153af194e57ac4d8d7659f6daa5dc22de4e08402\": rpc error: code = NotFound desc = could not find container \"b07f10d87eb275a6bdac47ac153af194e57ac4d8d7659f6daa5dc22de4e08402\": container with ID starting with b07f10d87eb275a6bdac47ac153af194e57ac4d8d7659f6daa5dc22de4e08402 not found: ID does not exist"
Jul 28 17:44:42.724898 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.724877    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk"]
Jul 28 17:44:42.727465 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:42.727447    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk"]
Jul 28 17:44:44.097937 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:44.097905    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c561db9f-a9b5-4c4f-87e5-4d5d5df07782" path="/var/lib/kubelet/pods/c561db9f-a9b5-4c4f-87e5-4d5d5df07782/volumes"
Jul 28 17:44:44.098310 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:44.098296    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4b3a598-cb9d-46a5-a17c-481a7e45fddb" path="/var/lib/kubelet/pods/d4b3a598-cb9d-46a5-a17c-481a7e45fddb/volumes"
Jul 28 17:44:44.953743 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:44.953709    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-b1dfe-bf8f99978-2vj8v" podUID="bd5242f7-86f5-4cde-9584-8c75ce5dfe4b" containerName="sequence-graph-b1dfe" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:44:45.627930 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:45.627893    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-8b8b5-846dd8bd57-vd26j" podUID="5f51a692-a6d4-40f1-9588-2944c833a3e5" containerName="ensemble-graph-8b8b5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:44:45.671044 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:45.671019    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8"
Jul 28 17:44:45.671512 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:45.671483    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8" podUID="0da81b68-4834-4e53-b9b1-1305fcec6bc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 17:44:47.682250 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:47.682220    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k"
Jul 28 17:44:47.682699 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:47.682672    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k" podUID="52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 28 17:44:49.953726 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:49.953681    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-b1dfe-bf8f99978-2vj8v" podUID="bd5242f7-86f5-4cde-9584-8c75ce5dfe4b" containerName="sequence-graph-b1dfe" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:44:49.954163 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:49.953805    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-b1dfe-bf8f99978-2vj8v"
Jul 28 17:44:50.628117 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:50.628078    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-8b8b5-846dd8bd57-vd26j" podUID="5f51a692-a6d4-40f1-9588-2944c833a3e5" containerName="ensemble-graph-8b8b5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:44:50.628284 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:50.628173    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-8b8b5-846dd8bd57-vd26j"
Jul 28 17:44:54.953843 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:54.953760    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-b1dfe-bf8f99978-2vj8v" podUID="bd5242f7-86f5-4cde-9584-8c75ce5dfe4b" containerName="sequence-graph-b1dfe" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:44:55.628065 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:55.628027    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-8b8b5-846dd8bd57-vd26j" podUID="5f51a692-a6d4-40f1-9588-2944c833a3e5" containerName="ensemble-graph-8b8b5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:44:55.672002 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:55.671963    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8" podUID="0da81b68-4834-4e53-b9b1-1305fcec6bc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 17:44:57.683496 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:57.683461    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k" podUID="52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 28 17:44:59.954168 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:44:59.954127    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-b1dfe-bf8f99978-2vj8v" podUID="bd5242f7-86f5-4cde-9584-8c75ce5dfe4b" containerName="sequence-graph-b1dfe" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:45:00.627268 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:00.627229    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-8b8b5-846dd8bd57-vd26j" podUID="5f51a692-a6d4-40f1-9588-2944c833a3e5" containerName="ensemble-graph-8b8b5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:45:04.954055 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:04.954016    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-b1dfe-bf8f99978-2vj8v" podUID="bd5242f7-86f5-4cde-9584-8c75ce5dfe4b" containerName="sequence-graph-b1dfe" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:45:05.627747 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:05.627710    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-8b8b5-846dd8bd57-vd26j" podUID="5f51a692-a6d4-40f1-9588-2944c833a3e5" containerName="ensemble-graph-8b8b5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:45:05.672489 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:05.672449    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8" podUID="0da81b68-4834-4e53-b9b1-1305fcec6bc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 17:45:07.683218 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:07.683182    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k" podUID="52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 28 17:45:08.122092 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:08.122064    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-b1dfe-bf8f99978-2vj8v"
Jul 28 17:45:08.173638 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:08.173604    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd5242f7-86f5-4cde-9584-8c75ce5dfe4b-openshift-service-ca-bundle\") pod \"bd5242f7-86f5-4cde-9584-8c75ce5dfe4b\" (UID: \"bd5242f7-86f5-4cde-9584-8c75ce5dfe4b\") "
Jul 28 17:45:08.173791 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:08.173647    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd5242f7-86f5-4cde-9584-8c75ce5dfe4b-proxy-tls\") pod \"bd5242f7-86f5-4cde-9584-8c75ce5dfe4b\" (UID: \"bd5242f7-86f5-4cde-9584-8c75ce5dfe4b\") "
Jul 28 17:45:08.174048 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:08.174021    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd5242f7-86f5-4cde-9584-8c75ce5dfe4b-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "bd5242f7-86f5-4cde-9584-8c75ce5dfe4b" (UID: "bd5242f7-86f5-4cde-9584-8c75ce5dfe4b"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:45:08.175789 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:08.175765    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd5242f7-86f5-4cde-9584-8c75ce5dfe4b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bd5242f7-86f5-4cde-9584-8c75ce5dfe4b" (UID: "bd5242f7-86f5-4cde-9584-8c75ce5dfe4b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:45:08.274822 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:08.274741    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd5242f7-86f5-4cde-9584-8c75ce5dfe4b-openshift-service-ca-bundle\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:45:08.274822 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:08.274768    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd5242f7-86f5-4cde-9584-8c75ce5dfe4b-proxy-tls\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:45:08.773646 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:08.773611    2566 generic.go:356] "Generic (PLEG): container finished" podID="5f51a692-a6d4-40f1-9588-2944c833a3e5" containerID="8d599028f70c2d8535914d8b711ebe0a08679aa3ad336a4d07bb9031dc5f789d" exitCode=0
Jul 28 17:45:08.774034 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:08.773683    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-8b8b5-846dd8bd57-vd26j" event={"ID":"5f51a692-a6d4-40f1-9588-2944c833a3e5","Type":"ContainerDied","Data":"8d599028f70c2d8535914d8b711ebe0a08679aa3ad336a4d07bb9031dc5f789d"}
Jul 28 17:45:08.774823 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:08.774806    2566 generic.go:356] "Generic (PLEG): container finished" podID="bd5242f7-86f5-4cde-9584-8c75ce5dfe4b" containerID="4373602408a35eea4fc6c2e587c242e952ddbebda4a79c3d27ef0cb0a5856d31" exitCode=0
Jul 28 17:45:08.774877 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:08.774839    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-b1dfe-bf8f99978-2vj8v" event={"ID":"bd5242f7-86f5-4cde-9584-8c75ce5dfe4b","Type":"ContainerDied","Data":"4373602408a35eea4fc6c2e587c242e952ddbebda4a79c3d27ef0cb0a5856d31"}
Jul 28 17:45:08.774877 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:08.774866    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-b1dfe-bf8f99978-2vj8v"
Jul 28 17:45:08.774953 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:08.774890    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-b1dfe-bf8f99978-2vj8v" event={"ID":"bd5242f7-86f5-4cde-9584-8c75ce5dfe4b","Type":"ContainerDied","Data":"3933d5fd7b23a31d291f3375dc0023775c455ef717297fa43d8bc5bc968a677e"}
Jul 28 17:45:08.774953 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:08.774912    2566 scope.go:117] "RemoveContainer" containerID="4373602408a35eea4fc6c2e587c242e952ddbebda4a79c3d27ef0cb0a5856d31"
Jul 28 17:45:08.783397 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:08.783380    2566 scope.go:117] "RemoveContainer" containerID="4373602408a35eea4fc6c2e587c242e952ddbebda4a79c3d27ef0cb0a5856d31"
Jul 28 17:45:08.783627 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:45:08.783609    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4373602408a35eea4fc6c2e587c242e952ddbebda4a79c3d27ef0cb0a5856d31\": container with ID starting with 4373602408a35eea4fc6c2e587c242e952ddbebda4a79c3d27ef0cb0a5856d31 not found: ID does not exist" containerID="4373602408a35eea4fc6c2e587c242e952ddbebda4a79c3d27ef0cb0a5856d31"
Jul 28 17:45:08.783674 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:08.783634    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4373602408a35eea4fc6c2e587c242e952ddbebda4a79c3d27ef0cb0a5856d31"} err="failed to get container status \"4373602408a35eea4fc6c2e587c242e952ddbebda4a79c3d27ef0cb0a5856d31\": rpc error: code = NotFound desc = could not find container \"4373602408a35eea4fc6c2e587c242e952ddbebda4a79c3d27ef0cb0a5856d31\": container with ID starting with 4373602408a35eea4fc6c2e587c242e952ddbebda4a79c3d27ef0cb0a5856d31 not found: ID does not exist"
Jul 28 17:45:08.797161 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:08.797134    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-b1dfe-bf8f99978-2vj8v"]
Jul 28 17:45:08.801817 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:08.801797    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-b1dfe-bf8f99978-2vj8v"]
Jul 28 17:45:09.092832 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:09.092805    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-8b8b5-846dd8bd57-vd26j"
Jul 28 17:45:09.179995 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:09.179963    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f51a692-a6d4-40f1-9588-2944c833a3e5-openshift-service-ca-bundle\") pod \"5f51a692-a6d4-40f1-9588-2944c833a3e5\" (UID: \"5f51a692-a6d4-40f1-9588-2944c833a3e5\") "
Jul 28 17:45:09.180241 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:09.180008    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f51a692-a6d4-40f1-9588-2944c833a3e5-proxy-tls\") pod \"5f51a692-a6d4-40f1-9588-2944c833a3e5\" (UID: \"5f51a692-a6d4-40f1-9588-2944c833a3e5\") "
Jul 28 17:45:09.180342 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:09.180300    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f51a692-a6d4-40f1-9588-2944c833a3e5-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "5f51a692-a6d4-40f1-9588-2944c833a3e5" (UID: "5f51a692-a6d4-40f1-9588-2944c833a3e5"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:45:09.182053 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:09.182029    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f51a692-a6d4-40f1-9588-2944c833a3e5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5f51a692-a6d4-40f1-9588-2944c833a3e5" (UID: "5f51a692-a6d4-40f1-9588-2944c833a3e5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:45:09.280946 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:09.280917    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f51a692-a6d4-40f1-9588-2944c833a3e5-openshift-service-ca-bundle\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:45:09.280946 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:09.280944    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f51a692-a6d4-40f1-9588-2944c833a3e5-proxy-tls\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:45:09.779914 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:09.779888    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-8b8b5-846dd8bd57-vd26j"
Jul 28 17:45:09.779914 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:09.779895    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-8b8b5-846dd8bd57-vd26j" event={"ID":"5f51a692-a6d4-40f1-9588-2944c833a3e5","Type":"ContainerDied","Data":"7bbb02fd94adbc9b9bd194fd2500387946bcfa977193389c8c14ae11d4500900"}
Jul 28 17:45:09.780466 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:09.779947    2566 scope.go:117] "RemoveContainer" containerID="8d599028f70c2d8535914d8b711ebe0a08679aa3ad336a4d07bb9031dc5f789d"
Jul 28 17:45:09.800939 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:09.800915    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-8b8b5-846dd8bd57-vd26j"]
Jul 28 17:45:09.807743 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:09.807717    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-8b8b5-846dd8bd57-vd26j"]
Jul 28 17:45:10.098474 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:10.098395    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f51a692-a6d4-40f1-9588-2944c833a3e5" path="/var/lib/kubelet/pods/5f51a692-a6d4-40f1-9588-2944c833a3e5/volumes"
Jul 28 17:45:10.098741 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:10.098727    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd5242f7-86f5-4cde-9584-8c75ce5dfe4b" path="/var/lib/kubelet/pods/bd5242f7-86f5-4cde-9584-8c75ce5dfe4b/volumes"
Jul 28 17:45:15.672357 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:15.672296    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8" podUID="0da81b68-4834-4e53-b9b1-1305fcec6bc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 17:45:17.682855 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:17.682813    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k" podUID="52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 28 17:45:25.672470 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:25.672440    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8"
Jul 28 17:45:27.684130 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:27.684104    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k"
Jul 28 17:45:38.392119 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.392032    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-61a23-7dfbd4b7cf-ll569"]
Jul 28 17:45:38.392642 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.392406    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd5242f7-86f5-4cde-9584-8c75ce5dfe4b" containerName="sequence-graph-b1dfe"
Jul 28 17:45:38.392642 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.392427    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd5242f7-86f5-4cde-9584-8c75ce5dfe4b" containerName="sequence-graph-b1dfe"
Jul 28 17:45:38.392642 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.392441    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c561db9f-a9b5-4c4f-87e5-4d5d5df07782" containerName="kserve-container"
Jul 28 17:45:38.392642 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.392450    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c561db9f-a9b5-4c4f-87e5-4d5d5df07782" containerName="kserve-container"
Jul 28 17:45:38.392642 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.392463    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f51a692-a6d4-40f1-9588-2944c833a3e5" containerName="ensemble-graph-8b8b5"
Jul 28 17:45:38.392642 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.392472    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f51a692-a6d4-40f1-9588-2944c833a3e5" containerName="ensemble-graph-8b8b5"
Jul 28 17:45:38.392642 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.392481    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4b3a598-cb9d-46a5-a17c-481a7e45fddb" containerName="kube-rbac-proxy"
Jul 28 17:45:38.392642 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.392489    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4b3a598-cb9d-46a5-a17c-481a7e45fddb" containerName="kube-rbac-proxy"
Jul 28 17:45:38.392642 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.392507    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4b3a598-cb9d-46a5-a17c-481a7e45fddb" containerName="kserve-container"
Jul 28 17:45:38.392642 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.392515    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4b3a598-cb9d-46a5-a17c-481a7e45fddb" containerName="kserve-container"
Jul 28 17:45:38.392642 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.392527    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c561db9f-a9b5-4c4f-87e5-4d5d5df07782" containerName="kube-rbac-proxy"
Jul 28 17:45:38.392642 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.392534    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c561db9f-a9b5-4c4f-87e5-4d5d5df07782" containerName="kube-rbac-proxy"
Jul 28 17:45:38.392642 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.392607    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd5242f7-86f5-4cde-9584-8c75ce5dfe4b" containerName="sequence-graph-b1dfe"
Jul 28 17:45:38.392642 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.392616    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c561db9f-a9b5-4c4f-87e5-4d5d5df07782" containerName="kserve-container"
Jul 28 17:45:38.392642 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.392627    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f51a692-a6d4-40f1-9588-2944c833a3e5" containerName="ensemble-graph-8b8b5"
Jul 28 17:45:38.392642 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.392638    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4b3a598-cb9d-46a5-a17c-481a7e45fddb" containerName="kserve-container"
Jul 28 17:45:38.392642 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.392647    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4b3a598-cb9d-46a5-a17c-481a7e45fddb" containerName="kube-rbac-proxy"
Jul 28 17:45:38.393462 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.392655    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c561db9f-a9b5-4c4f-87e5-4d5d5df07782" containerName="kube-rbac-proxy"
Jul 28 17:45:38.397137 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.397117    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-61a23-7dfbd4b7cf-ll569"
Jul 28 17:45:38.400043 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.400024    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-61a23-kube-rbac-proxy-sar-config\""
Jul 28 17:45:38.400119 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.400032    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-61a23-serving-cert\""
Jul 28 17:45:38.413120 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.413098    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-61a23-7dfbd4b7cf-ll569"]
Jul 28 17:45:38.506090 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.506059    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/989befef-f869-4ed5-ac37-9e447c15b756-openshift-service-ca-bundle\") pod \"sequence-graph-61a23-7dfbd4b7cf-ll569\" (UID: \"989befef-f869-4ed5-ac37-9e447c15b756\") " pod="kserve-ci-e2e-test/sequence-graph-61a23-7dfbd4b7cf-ll569"
Jul 28 17:45:38.506257 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.506099    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/989befef-f869-4ed5-ac37-9e447c15b756-proxy-tls\") pod \"sequence-graph-61a23-7dfbd4b7cf-ll569\" (UID: \"989befef-f869-4ed5-ac37-9e447c15b756\") " pod="kserve-ci-e2e-test/sequence-graph-61a23-7dfbd4b7cf-ll569"
Jul 28 17:45:38.607019 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.606973    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/989befef-f869-4ed5-ac37-9e447c15b756-openshift-service-ca-bundle\") pod \"sequence-graph-61a23-7dfbd4b7cf-ll569\" (UID: \"989befef-f869-4ed5-ac37-9e447c15b756\") " pod="kserve-ci-e2e-test/sequence-graph-61a23-7dfbd4b7cf-ll569"
Jul 28 17:45:38.607190 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.607030    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/989befef-f869-4ed5-ac37-9e447c15b756-proxy-tls\") pod \"sequence-graph-61a23-7dfbd4b7cf-ll569\" (UID: \"989befef-f869-4ed5-ac37-9e447c15b756\") " pod="kserve-ci-e2e-test/sequence-graph-61a23-7dfbd4b7cf-ll569"
Jul 28 17:45:38.607668 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.607646    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/989befef-f869-4ed5-ac37-9e447c15b756-openshift-service-ca-bundle\") pod \"sequence-graph-61a23-7dfbd4b7cf-ll569\" (UID: \"989befef-f869-4ed5-ac37-9e447c15b756\") " pod="kserve-ci-e2e-test/sequence-graph-61a23-7dfbd4b7cf-ll569"
Jul 28 17:45:38.609490 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.609470    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/989befef-f869-4ed5-ac37-9e447c15b756-proxy-tls\") pod \"sequence-graph-61a23-7dfbd4b7cf-ll569\" (UID: \"989befef-f869-4ed5-ac37-9e447c15b756\") " pod="kserve-ci-e2e-test/sequence-graph-61a23-7dfbd4b7cf-ll569"
Jul 28 17:45:38.706475 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.706441    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-61a23-7dfbd4b7cf-ll569"
Jul 28 17:45:38.726483 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.726457    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-614a5-7dcdd75c-qs5jf"]
Jul 28 17:45:38.730610 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.730590    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-614a5-7dcdd75c-qs5jf"
Jul 28 17:45:38.736449 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.736423    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-614a5-serving-cert\""
Jul 28 17:45:38.736562 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.736456    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-614a5-kube-rbac-proxy-sar-config\""
Jul 28 17:45:38.745391 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.745359    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-614a5-7dcdd75c-qs5jf"]
Jul 28 17:45:38.839792 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.839769    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-61a23-7dfbd4b7cf-ll569"]
Jul 28 17:45:38.842266 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:45:38.842229    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod989befef_f869_4ed5_ac37_9e447c15b756.slice/crio-68ec2413b392a3931348860a18011d4b1c6a31669088bba95b1577aa31b2c760 WatchSource:0}: Error finding container 68ec2413b392a3931348860a18011d4b1c6a31669088bba95b1577aa31b2c760: Status 404 returned error can't find the container with id 68ec2413b392a3931348860a18011d4b1c6a31669088bba95b1577aa31b2c760
Jul 28 17:45:38.890913 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.890879    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-61a23-7dfbd4b7cf-ll569" event={"ID":"989befef-f869-4ed5-ac37-9e447c15b756","Type":"ContainerStarted","Data":"68ec2413b392a3931348860a18011d4b1c6a31669088bba95b1577aa31b2c760"}
Jul 28 17:45:38.909830 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.909805    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf0370bc-f2ce-4d77-bab2-057cf8adefd5-openshift-service-ca-bundle\") pod \"ensemble-graph-614a5-7dcdd75c-qs5jf\" (UID: \"bf0370bc-f2ce-4d77-bab2-057cf8adefd5\") " pod="kserve-ci-e2e-test/ensemble-graph-614a5-7dcdd75c-qs5jf"
Jul 28 17:45:38.909967 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:38.909864    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf0370bc-f2ce-4d77-bab2-057cf8adefd5-proxy-tls\") pod \"ensemble-graph-614a5-7dcdd75c-qs5jf\" (UID: \"bf0370bc-f2ce-4d77-bab2-057cf8adefd5\") " pod="kserve-ci-e2e-test/ensemble-graph-614a5-7dcdd75c-qs5jf"
Jul 28 17:45:39.010652 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:39.010560    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf0370bc-f2ce-4d77-bab2-057cf8adefd5-openshift-service-ca-bundle\") pod \"ensemble-graph-614a5-7dcdd75c-qs5jf\" (UID: \"bf0370bc-f2ce-4d77-bab2-057cf8adefd5\") " pod="kserve-ci-e2e-test/ensemble-graph-614a5-7dcdd75c-qs5jf"
Jul 28 17:45:39.010652 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:39.010612    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf0370bc-f2ce-4d77-bab2-057cf8adefd5-proxy-tls\") pod \"ensemble-graph-614a5-7dcdd75c-qs5jf\" (UID: \"bf0370bc-f2ce-4d77-bab2-057cf8adefd5\") " pod="kserve-ci-e2e-test/ensemble-graph-614a5-7dcdd75c-qs5jf"
Jul 28 17:45:39.010847 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:45:39.010728    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-614a5-serving-cert: secret "ensemble-graph-614a5-serving-cert" not found
Jul 28 17:45:39.010847 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:45:39.010781    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf0370bc-f2ce-4d77-bab2-057cf8adefd5-proxy-tls podName:bf0370bc-f2ce-4d77-bab2-057cf8adefd5 nodeName:}" failed. No retries permitted until 2026-07-28 17:45:39.510764564 +0000 UTC m=+1389.820930047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bf0370bc-f2ce-4d77-bab2-057cf8adefd5-proxy-tls") pod "ensemble-graph-614a5-7dcdd75c-qs5jf" (UID: "bf0370bc-f2ce-4d77-bab2-057cf8adefd5") : secret "ensemble-graph-614a5-serving-cert" not found
Jul 28 17:45:39.011234 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:39.011215    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf0370bc-f2ce-4d77-bab2-057cf8adefd5-openshift-service-ca-bundle\") pod \"ensemble-graph-614a5-7dcdd75c-qs5jf\" (UID: \"bf0370bc-f2ce-4d77-bab2-057cf8adefd5\") " pod="kserve-ci-e2e-test/ensemble-graph-614a5-7dcdd75c-qs5jf"
Jul 28 17:45:39.515525 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:39.515490    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf0370bc-f2ce-4d77-bab2-057cf8adefd5-proxy-tls\") pod \"ensemble-graph-614a5-7dcdd75c-qs5jf\" (UID: \"bf0370bc-f2ce-4d77-bab2-057cf8adefd5\") " pod="kserve-ci-e2e-test/ensemble-graph-614a5-7dcdd75c-qs5jf"
Jul 28 17:45:39.517847 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:39.517816    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf0370bc-f2ce-4d77-bab2-057cf8adefd5-proxy-tls\") pod \"ensemble-graph-614a5-7dcdd75c-qs5jf\" (UID: \"bf0370bc-f2ce-4d77-bab2-057cf8adefd5\") " pod="kserve-ci-e2e-test/ensemble-graph-614a5-7dcdd75c-qs5jf"
Jul 28 17:45:39.640408 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:39.640372    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-614a5-7dcdd75c-qs5jf"
Jul 28 17:45:39.764594 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:39.764409    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-614a5-7dcdd75c-qs5jf"]
Jul 28 17:45:39.766782 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:45:39.766734    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf0370bc_f2ce_4d77_bab2_057cf8adefd5.slice/crio-d8e34ff0f40acfc7a04bbb2d7ad9ae6ca99f3d50f641b5fe4d633657b9633e51 WatchSource:0}: Error finding container d8e34ff0f40acfc7a04bbb2d7ad9ae6ca99f3d50f641b5fe4d633657b9633e51: Status 404 returned error can't find the container with id d8e34ff0f40acfc7a04bbb2d7ad9ae6ca99f3d50f641b5fe4d633657b9633e51
Jul 28 17:45:39.896212 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:39.896173    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-61a23-7dfbd4b7cf-ll569" event={"ID":"989befef-f869-4ed5-ac37-9e447c15b756","Type":"ContainerStarted","Data":"0d993d901b8b1017f9076202332642d07f38eea7b3454c44e12c1a7afad58ec7"}
Jul 28 17:45:39.896417 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:39.896244    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-61a23-7dfbd4b7cf-ll569"
Jul 28 17:45:39.897527 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:39.897502    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-614a5-7dcdd75c-qs5jf" event={"ID":"bf0370bc-f2ce-4d77-bab2-057cf8adefd5","Type":"ContainerStarted","Data":"cdec6481805f1ee801afadd1a740699b15359414ae0f5880dce6c5a8a830a49d"}
Jul 28 17:45:39.897642 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:39.897540    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-614a5-7dcdd75c-qs5jf" event={"ID":"bf0370bc-f2ce-4d77-bab2-057cf8adefd5","Type":"ContainerStarted","Data":"d8e34ff0f40acfc7a04bbb2d7ad9ae6ca99f3d50f641b5fe4d633657b9633e51"}
Jul 28 17:45:39.897642 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:39.897634    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-614a5-7dcdd75c-qs5jf"
Jul 28 17:45:39.914132 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:39.914085    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-61a23-7dfbd4b7cf-ll569" podStartSLOduration=1.914069552 podStartE2EDuration="1.914069552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:45:39.912475772 +0000 UTC m=+1390.222641277" watchObservedRunningTime="2026-07-28 17:45:39.914069552 +0000 UTC m=+1390.224235057"
Jul 28 17:45:39.930918 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:39.930877    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-614a5-7dcdd75c-qs5jf" podStartSLOduration=1.930861777 podStartE2EDuration="1.930861777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:45:39.929726941 +0000 UTC m=+1390.239892467" watchObservedRunningTime="2026-07-28 17:45:39.930861777 +0000 UTC m=+1390.241027310"
Jul 28 17:45:45.907016 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:45.906986    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-61a23-7dfbd4b7cf-ll569"
Jul 28 17:45:45.907503 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:45.907410    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-614a5-7dcdd75c-qs5jf"
Jul 28 17:45:52.323145 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:52.323104    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-61a23-7dfbd4b7cf-ll569"]
Jul 28 17:45:52.323619 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:52.323519    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-61a23-7dfbd4b7cf-ll569" podUID="989befef-f869-4ed5-ac37-9e447c15b756" containerName="sequence-graph-61a23" containerID="cri-o://0d993d901b8b1017f9076202332642d07f38eea7b3454c44e12c1a7afad58ec7" gracePeriod=30
Jul 28 17:45:52.466629 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:52.466595    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k"]
Jul 28 17:45:52.466851 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:52.466815    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k" podUID="52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca" containerName="kserve-container" containerID="cri-o://04f2d894d059dbb678aac08dec360146b674bc2dfc8c0aa69edf39920d644056" gracePeriod=30
Jul 28 17:45:52.466981 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:52.466861    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k" podUID="52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca" containerName="kube-rbac-proxy" containerID="cri-o://1e54f0e42d53ebb4c9b23c268a7ca7f44d3c7db61ec5b66782476ecc19231b0e" gracePeriod=30
Jul 28 17:45:52.512158 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:52.512120    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk"]
Jul 28 17:45:52.515700 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:52.515674    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk"
Jul 28 17:45:52.518144 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:52.518101    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39137e71-bcda-4ba3-bdca-60c6e8a91b1a-proxy-tls\") pod \"success-200-isvc-787a1-predictor-869dbdb759-qnbjk\" (UID: \"39137e71-bcda-4ba3-bdca-60c6e8a91b1a\") " pod="kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk"
Jul 28 17:45:52.518305 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:52.518170    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39137e71-bcda-4ba3-bdca-60c6e8a91b1a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-787a1-predictor-869dbdb759-qnbjk\" (UID: \"39137e71-bcda-4ba3-bdca-60c6e8a91b1a\") " pod="kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk"
Jul 28 17:45:52.518305 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:52.518201    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5msq\" (UniqueName: \"kubernetes.io/projected/39137e71-bcda-4ba3-bdca-60c6e8a91b1a-kube-api-access-r5msq\") pod \"success-200-isvc-787a1-predictor-869dbdb759-qnbjk\" (UID: \"39137e71-bcda-4ba3-bdca-60c6e8a91b1a\") " pod="kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk"
Jul 28 17:45:52.518759 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:52.518732    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-787a1-predictor-serving-cert\""
Jul 28 17:45:52.520750 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:52.520726    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-787a1-kube-rbac-proxy-sar-config\""
Jul 28 17:45:52.528092 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:52.528058    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk"]
Jul 28 17:45:52.619423 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:52.619379    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39137e71-bcda-4ba3-bdca-60c6e8a91b1a-proxy-tls\") pod \"success-200-isvc-787a1-predictor-869dbdb759-qnbjk\" (UID: \"39137e71-bcda-4ba3-bdca-60c6e8a91b1a\") " pod="kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk"
Jul 28 17:45:52.619623 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:52.619438    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39137e71-bcda-4ba3-bdca-60c6e8a91b1a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-787a1-predictor-869dbdb759-qnbjk\" (UID: \"39137e71-bcda-4ba3-bdca-60c6e8a91b1a\") " pod="kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk"
Jul 28 17:45:52.619623 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:45:52.619533    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-787a1-predictor-serving-cert: secret "success-200-isvc-787a1-predictor-serving-cert" not found
Jul 28 17:45:52.619623 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:52.619562    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5msq\" (UniqueName: \"kubernetes.io/projected/39137e71-bcda-4ba3-bdca-60c6e8a91b1a-kube-api-access-r5msq\") pod \"success-200-isvc-787a1-predictor-869dbdb759-qnbjk\" (UID: \"39137e71-bcda-4ba3-bdca-60c6e8a91b1a\") " pod="kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk"
Jul 28 17:45:52.619623 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:45:52.619607    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39137e71-bcda-4ba3-bdca-60c6e8a91b1a-proxy-tls podName:39137e71-bcda-4ba3-bdca-60c6e8a91b1a nodeName:}" failed. No retries permitted until 2026-07-28 17:45:53.119586548 +0000 UTC m=+1403.429752032 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/39137e71-bcda-4ba3-bdca-60c6e8a91b1a-proxy-tls") pod "success-200-isvc-787a1-predictor-869dbdb759-qnbjk" (UID: "39137e71-bcda-4ba3-bdca-60c6e8a91b1a") : secret "success-200-isvc-787a1-predictor-serving-cert" not found
Jul 28 17:45:52.620026 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:52.620008    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39137e71-bcda-4ba3-bdca-60c6e8a91b1a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-787a1-predictor-869dbdb759-qnbjk\" (UID: \"39137e71-bcda-4ba3-bdca-60c6e8a91b1a\") " pod="kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk"
Jul 28 17:45:52.636450 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:52.636410    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5msq\" (UniqueName: \"kubernetes.io/projected/39137e71-bcda-4ba3-bdca-60c6e8a91b1a-kube-api-access-r5msq\") pod \"success-200-isvc-787a1-predictor-869dbdb759-qnbjk\" (UID: \"39137e71-bcda-4ba3-bdca-60c6e8a91b1a\") " pod="kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk"
Jul 28 17:45:52.678707 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:52.678664    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k" podUID="52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 28 17:45:52.944417 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:52.944384    2566 generic.go:356] "Generic (PLEG): container finished" podID="52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca" containerID="1e54f0e42d53ebb4c9b23c268a7ca7f44d3c7db61ec5b66782476ecc19231b0e" exitCode=2
Jul 28 17:45:52.944587 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:52.944451    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k" event={"ID":"52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca","Type":"ContainerDied","Data":"1e54f0e42d53ebb4c9b23c268a7ca7f44d3c7db61ec5b66782476ecc19231b0e"}
Jul 28 17:45:53.123383 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:53.123352    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39137e71-bcda-4ba3-bdca-60c6e8a91b1a-proxy-tls\") pod \"success-200-isvc-787a1-predictor-869dbdb759-qnbjk\" (UID: \"39137e71-bcda-4ba3-bdca-60c6e8a91b1a\") " pod="kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk"
Jul 28 17:45:53.125760 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:53.125731    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39137e71-bcda-4ba3-bdca-60c6e8a91b1a-proxy-tls\") pod \"success-200-isvc-787a1-predictor-869dbdb759-qnbjk\" (UID: \"39137e71-bcda-4ba3-bdca-60c6e8a91b1a\") " pod="kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk"
Jul 28 17:45:53.127904 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:53.127884    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk"
Jul 28 17:45:53.257248 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:53.257098    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk"]
Jul 28 17:45:53.259884 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:45:53.259855    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39137e71_bcda_4ba3_bdca_60c6e8a91b1a.slice/crio-ba8f726308968499fddef055639570086b5649daeea6a9c33ffe27834e7d7d6d WatchSource:0}: Error finding container ba8f726308968499fddef055639570086b5649daeea6a9c33ffe27834e7d7d6d: Status 404 returned error can't find the container with id ba8f726308968499fddef055639570086b5649daeea6a9c33ffe27834e7d7d6d
Jul 28 17:45:53.950210 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:53.950171    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk" event={"ID":"39137e71-bcda-4ba3-bdca-60c6e8a91b1a","Type":"ContainerStarted","Data":"9c7c719a3358b4b7a05dc1947c3d8c8e0975486281b618ed0894bbace4825db9"}
Jul 28 17:45:53.950688 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:53.950244    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk" event={"ID":"39137e71-bcda-4ba3-bdca-60c6e8a91b1a","Type":"ContainerStarted","Data":"8a35008083e2cdad97726f99755029a0164a064e904ad1a460592bb46d0256a5"}
Jul 28 17:45:53.950688 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:53.950271    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk" event={"ID":"39137e71-bcda-4ba3-bdca-60c6e8a91b1a","Type":"ContainerStarted","Data":"ba8f726308968499fddef055639570086b5649daeea6a9c33ffe27834e7d7d6d"}
Jul 28 17:45:53.950688 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:53.950302    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk"
Jul 28 17:45:53.950688 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:53.950356    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk"
Jul 28 17:45:53.951634 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:53.951607    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk" podUID="39137e71-bcda-4ba3-bdca-60c6e8a91b1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 17:45:54.954303 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:54.954257    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk" podUID="39137e71-bcda-4ba3-bdca-60c6e8a91b1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 17:45:55.609866 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:55.609841    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k"
Jul 28 17:45:55.640207 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:55.640178    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpzpt\" (UniqueName: \"kubernetes.io/projected/52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca-kube-api-access-vpzpt\") pod \"52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca\" (UID: \"52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca\") "
Jul 28 17:45:55.640349 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:55.640215    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca-kube-rbac-proxy-sar-config\") pod \"52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca\" (UID: \"52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca\") "
Jul 28 17:45:55.640349 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:55.640256    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca-proxy-tls\") pod \"52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca\" (UID: \"52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca\") "
Jul 28 17:45:55.640757 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:55.640723    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca" (UID: "52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:45:55.642529 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:55.642502    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca-kube-api-access-vpzpt" (OuterVolumeSpecName: "kube-api-access-vpzpt") pod "52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca" (UID: "52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca"). InnerVolumeSpecName "kube-api-access-vpzpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:45:55.642630 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:55.642539    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca" (UID: "52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:45:55.646540 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:55.646375    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk" podStartSLOduration=3.646358691 podStartE2EDuration="3.646358691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:45:53.97265762 +0000 UTC m=+1404.282823151" watchObservedRunningTime="2026-07-28 17:45:55.646358691 +0000 UTC m=+1405.956524196"
Jul 28 17:45:55.741237 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:55.741200    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpzpt\" (UniqueName: \"kubernetes.io/projected/52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca-kube-api-access-vpzpt\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:45:55.741237 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:55.741229    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:45:55.741237 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:55.741238    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca-proxy-tls\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:45:55.905926 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:55.905886    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-61a23-7dfbd4b7cf-ll569" podUID="989befef-f869-4ed5-ac37-9e447c15b756" containerName="sequence-graph-61a23" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:45:55.959138 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:55.959105    2566 generic.go:356] "Generic (PLEG): container finished" podID="52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca" containerID="04f2d894d059dbb678aac08dec360146b674bc2dfc8c0aa69edf39920d644056" exitCode=0
Jul 28 17:45:55.959556 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:55.959171    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k"
Jul 28 17:45:55.959556 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:55.959189    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k" event={"ID":"52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca","Type":"ContainerDied","Data":"04f2d894d059dbb678aac08dec360146b674bc2dfc8c0aa69edf39920d644056"}
Jul 28 17:45:55.959556 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:55.959255    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k" event={"ID":"52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca","Type":"ContainerDied","Data":"4452e550f1deb489e7c4c31de51270b8905e55d7b506086f9c20afd9a12a5bb2"}
Jul 28 17:45:55.959556 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:55.959293    2566 scope.go:117] "RemoveContainer" containerID="1e54f0e42d53ebb4c9b23c268a7ca7f44d3c7db61ec5b66782476ecc19231b0e"
Jul 28 17:45:55.967926 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:55.967905    2566 scope.go:117] "RemoveContainer" containerID="04f2d894d059dbb678aac08dec360146b674bc2dfc8c0aa69edf39920d644056"
Jul 28 17:45:55.974749 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:55.974733    2566 scope.go:117] "RemoveContainer" containerID="1e54f0e42d53ebb4c9b23c268a7ca7f44d3c7db61ec5b66782476ecc19231b0e"
Jul 28 17:45:55.975023 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:45:55.975005    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e54f0e42d53ebb4c9b23c268a7ca7f44d3c7db61ec5b66782476ecc19231b0e\": container with ID starting with 1e54f0e42d53ebb4c9b23c268a7ca7f44d3c7db61ec5b66782476ecc19231b0e not found: ID does not exist" containerID="1e54f0e42d53ebb4c9b23c268a7ca7f44d3c7db61ec5b66782476ecc19231b0e"
Jul 28 17:45:55.975069 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:55.975031    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e54f0e42d53ebb4c9b23c268a7ca7f44d3c7db61ec5b66782476ecc19231b0e"} err="failed to get container status \"1e54f0e42d53ebb4c9b23c268a7ca7f44d3c7db61ec5b66782476ecc19231b0e\": rpc error: code = NotFound desc = could not find container \"1e54f0e42d53ebb4c9b23c268a7ca7f44d3c7db61ec5b66782476ecc19231b0e\": container with ID starting with 1e54f0e42d53ebb4c9b23c268a7ca7f44d3c7db61ec5b66782476ecc19231b0e not found: ID does not exist"
Jul 28 17:45:55.975069 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:55.975048    2566 scope.go:117] "RemoveContainer" containerID="04f2d894d059dbb678aac08dec360146b674bc2dfc8c0aa69edf39920d644056"
Jul 28 17:45:55.975263 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:45:55.975243    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04f2d894d059dbb678aac08dec360146b674bc2dfc8c0aa69edf39920d644056\": container with ID starting with 04f2d894d059dbb678aac08dec360146b674bc2dfc8c0aa69edf39920d644056 not found: ID does not exist" containerID="04f2d894d059dbb678aac08dec360146b674bc2dfc8c0aa69edf39920d644056"
Jul 28 17:45:55.975309 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:55.975270    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04f2d894d059dbb678aac08dec360146b674bc2dfc8c0aa69edf39920d644056"} err="failed to get container status \"04f2d894d059dbb678aac08dec360146b674bc2dfc8c0aa69edf39920d644056\": rpc error: code = NotFound desc = could not find container \"04f2d894d059dbb678aac08dec360146b674bc2dfc8c0aa69edf39920d644056\": container with ID starting with 04f2d894d059dbb678aac08dec360146b674bc2dfc8c0aa69edf39920d644056 not found: ID does not exist"
Jul 28 17:45:55.983986 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:55.983963    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k"]
Jul 28 17:45:55.990825 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:55.990802    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k"]
Jul 28 17:45:56.097361 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:56.097262    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca" path="/var/lib/kubelet/pods/52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca/volumes"
Jul 28 17:45:59.959293 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:59.959265    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk"
Jul 28 17:45:59.959802 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:45:59.959771    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk" podUID="39137e71-bcda-4ba3-bdca-60c6e8a91b1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 17:46:00.905886 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:00.905845    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-61a23-7dfbd4b7cf-ll569" podUID="989befef-f869-4ed5-ac37-9e447c15b756" containerName="sequence-graph-61a23" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:46:05.905359 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:05.905296    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-61a23-7dfbd4b7cf-ll569" podUID="989befef-f869-4ed5-ac37-9e447c15b756" containerName="sequence-graph-61a23" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:46:05.905822 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:05.905432    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-61a23-7dfbd4b7cf-ll569"
Jul 28 17:46:09.960010 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:09.959963    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk" podUID="39137e71-bcda-4ba3-bdca-60c6e8a91b1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 17:46:10.906776 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:10.906727    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-61a23-7dfbd4b7cf-ll569" podUID="989befef-f869-4ed5-ac37-9e447c15b756" containerName="sequence-graph-61a23" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:46:15.905922 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:15.905880    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-61a23-7dfbd4b7cf-ll569" podUID="989befef-f869-4ed5-ac37-9e447c15b756" containerName="sequence-graph-61a23" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:46:19.960669 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:19.960624    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk" podUID="39137e71-bcda-4ba3-bdca-60c6e8a91b1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 17:46:20.906704 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:20.906652    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-61a23-7dfbd4b7cf-ll569" podUID="989befef-f869-4ed5-ac37-9e447c15b756" containerName="sequence-graph-61a23" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:46:22.479866 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:22.479832    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-61a23-7dfbd4b7cf-ll569"
Jul 28 17:46:22.549191 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:22.549150    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/989befef-f869-4ed5-ac37-9e447c15b756-openshift-service-ca-bundle\") pod \"989befef-f869-4ed5-ac37-9e447c15b756\" (UID: \"989befef-f869-4ed5-ac37-9e447c15b756\") "
Jul 28 17:46:22.549375 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:22.549248    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/989befef-f869-4ed5-ac37-9e447c15b756-proxy-tls\") pod \"989befef-f869-4ed5-ac37-9e447c15b756\" (UID: \"989befef-f869-4ed5-ac37-9e447c15b756\") "
Jul 28 17:46:22.549612 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:22.549592    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/989befef-f869-4ed5-ac37-9e447c15b756-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "989befef-f869-4ed5-ac37-9e447c15b756" (UID: "989befef-f869-4ed5-ac37-9e447c15b756"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:46:22.551360 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:22.551312    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/989befef-f869-4ed5-ac37-9e447c15b756-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "989befef-f869-4ed5-ac37-9e447c15b756" (UID: "989befef-f869-4ed5-ac37-9e447c15b756"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:46:22.650137 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:22.650036    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/989befef-f869-4ed5-ac37-9e447c15b756-openshift-service-ca-bundle\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:46:22.650137 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:22.650071    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/989befef-f869-4ed5-ac37-9e447c15b756-proxy-tls\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:46:23.060228 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:23.060193    2566 generic.go:356] "Generic (PLEG): container finished" podID="989befef-f869-4ed5-ac37-9e447c15b756" containerID="0d993d901b8b1017f9076202332642d07f38eea7b3454c44e12c1a7afad58ec7" exitCode=0
Jul 28 17:46:23.060437 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:23.060250    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-61a23-7dfbd4b7cf-ll569"
Jul 28 17:46:23.060437 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:23.060251    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-61a23-7dfbd4b7cf-ll569" event={"ID":"989befef-f869-4ed5-ac37-9e447c15b756","Type":"ContainerDied","Data":"0d993d901b8b1017f9076202332642d07f38eea7b3454c44e12c1a7afad58ec7"}
Jul 28 17:46:23.060437 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:23.060412    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-61a23-7dfbd4b7cf-ll569" event={"ID":"989befef-f869-4ed5-ac37-9e447c15b756","Type":"ContainerDied","Data":"68ec2413b392a3931348860a18011d4b1c6a31669088bba95b1577aa31b2c760"}
Jul 28 17:46:23.060437 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:23.060431    2566 scope.go:117] "RemoveContainer" containerID="0d993d901b8b1017f9076202332642d07f38eea7b3454c44e12c1a7afad58ec7"
Jul 28 17:46:23.071480 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:23.071459    2566 scope.go:117] "RemoveContainer" containerID="0d993d901b8b1017f9076202332642d07f38eea7b3454c44e12c1a7afad58ec7"
Jul 28 17:46:23.071872 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:46:23.071845    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d993d901b8b1017f9076202332642d07f38eea7b3454c44e12c1a7afad58ec7\": container with ID starting with 0d993d901b8b1017f9076202332642d07f38eea7b3454c44e12c1a7afad58ec7 not found: ID does not exist" containerID="0d993d901b8b1017f9076202332642d07f38eea7b3454c44e12c1a7afad58ec7"
Jul 28 17:46:23.071931 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:23.071884    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d993d901b8b1017f9076202332642d07f38eea7b3454c44e12c1a7afad58ec7"} err="failed to get container status \"0d993d901b8b1017f9076202332642d07f38eea7b3454c44e12c1a7afad58ec7\": rpc error: code = NotFound desc = could not find container \"0d993d901b8b1017f9076202332642d07f38eea7b3454c44e12c1a7afad58ec7\": container with ID starting with 0d993d901b8b1017f9076202332642d07f38eea7b3454c44e12c1a7afad58ec7 not found: ID does not exist"
Jul 28 17:46:23.084641 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:23.084599    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-61a23-7dfbd4b7cf-ll569"]
Jul 28 17:46:23.091048 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:23.091009    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-61a23-7dfbd4b7cf-ll569"]
Jul 28 17:46:24.098063 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:24.098022    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="989befef-f869-4ed5-ac37-9e447c15b756" path="/var/lib/kubelet/pods/989befef-f869-4ed5-ac37-9e447c15b756/volumes"
Jul 28 17:46:29.960663 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:29.960620    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk" podUID="39137e71-bcda-4ba3-bdca-60c6e8a91b1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 17:46:39.960535 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:39.960502    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk"
Jul 28 17:46:52.575565 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:52.575516    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-787a1-77c7b956dc-cpngn"]
Jul 28 17:46:52.576197 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:52.576176    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="989befef-f869-4ed5-ac37-9e447c15b756" containerName="sequence-graph-61a23"
Jul 28 17:46:52.576291 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:52.576200    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="989befef-f869-4ed5-ac37-9e447c15b756" containerName="sequence-graph-61a23"
Jul 28 17:46:52.576291 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:52.576219    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca" containerName="kserve-container"
Jul 28 17:46:52.576291 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:52.576229    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca" containerName="kserve-container"
Jul 28 17:46:52.576291 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:52.576242    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca" containerName="kube-rbac-proxy"
Jul 28 17:46:52.576291 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:52.576251    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca" containerName="kube-rbac-proxy"
Jul 28 17:46:52.576577 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:52.576311    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca" containerName="kube-rbac-proxy"
Jul 28 17:46:52.576577 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:52.576340    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="989befef-f869-4ed5-ac37-9e447c15b756" containerName="sequence-graph-61a23"
Jul 28 17:46:52.576577 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:52.576349    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="52699f3f-9cb0-45a0-a3ba-f1a1a1c5a1ca" containerName="kserve-container"
Jul 28 17:46:52.579460 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:52.579432    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-787a1-77c7b956dc-cpngn"
Jul 28 17:46:52.582345 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:52.582296    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-787a1-kube-rbac-proxy-sar-config\""
Jul 28 17:46:52.582481 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:52.582301    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-787a1-serving-cert\""
Jul 28 17:46:52.590823 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:52.590792    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-787a1-77c7b956dc-cpngn"]
Jul 28 17:46:52.680549 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:52.680488    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/811f7ddb-91b4-4102-9ee5-ecb2a04ab80f-proxy-tls\") pod \"sequence-graph-787a1-77c7b956dc-cpngn\" (UID: \"811f7ddb-91b4-4102-9ee5-ecb2a04ab80f\") " pod="kserve-ci-e2e-test/sequence-graph-787a1-77c7b956dc-cpngn"
Jul 28 17:46:52.680549 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:52.680555    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/811f7ddb-91b4-4102-9ee5-ecb2a04ab80f-openshift-service-ca-bundle\") pod \"sequence-graph-787a1-77c7b956dc-cpngn\" (UID: \"811f7ddb-91b4-4102-9ee5-ecb2a04ab80f\") " pod="kserve-ci-e2e-test/sequence-graph-787a1-77c7b956dc-cpngn"
Jul 28 17:46:52.781258 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:52.781215    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/811f7ddb-91b4-4102-9ee5-ecb2a04ab80f-openshift-service-ca-bundle\") pod \"sequence-graph-787a1-77c7b956dc-cpngn\" (UID: \"811f7ddb-91b4-4102-9ee5-ecb2a04ab80f\") " pod="kserve-ci-e2e-test/sequence-graph-787a1-77c7b956dc-cpngn"
Jul 28 17:46:52.781460 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:52.781292    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/811f7ddb-91b4-4102-9ee5-ecb2a04ab80f-proxy-tls\") pod \"sequence-graph-787a1-77c7b956dc-cpngn\" (UID: \"811f7ddb-91b4-4102-9ee5-ecb2a04ab80f\") " pod="kserve-ci-e2e-test/sequence-graph-787a1-77c7b956dc-cpngn"
Jul 28 17:46:52.781936 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:52.781906    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/811f7ddb-91b4-4102-9ee5-ecb2a04ab80f-openshift-service-ca-bundle\") pod \"sequence-graph-787a1-77c7b956dc-cpngn\" (UID: \"811f7ddb-91b4-4102-9ee5-ecb2a04ab80f\") " pod="kserve-ci-e2e-test/sequence-graph-787a1-77c7b956dc-cpngn"
Jul 28 17:46:52.783754 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:52.783732    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/811f7ddb-91b4-4102-9ee5-ecb2a04ab80f-proxy-tls\") pod \"sequence-graph-787a1-77c7b956dc-cpngn\" (UID: \"811f7ddb-91b4-4102-9ee5-ecb2a04ab80f\") " pod="kserve-ci-e2e-test/sequence-graph-787a1-77c7b956dc-cpngn"
Jul 28 17:46:52.889169 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:52.889049    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-787a1-77c7b956dc-cpngn"
Jul 28 17:46:53.026994 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:53.026910    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-787a1-77c7b956dc-cpngn"]
Jul 28 17:46:53.029704 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:46:53.029676    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod811f7ddb_91b4_4102_9ee5_ecb2a04ab80f.slice/crio-a9684a7765d7709e7e17292cf02c3ac63a974565f66004d78a9688ef2ea6ecf6 WatchSource:0}: Error finding container a9684a7765d7709e7e17292cf02c3ac63a974565f66004d78a9688ef2ea6ecf6: Status 404 returned error can't find the container with id a9684a7765d7709e7e17292cf02c3ac63a974565f66004d78a9688ef2ea6ecf6
Jul 28 17:46:53.171695 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:53.171605    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-787a1-77c7b956dc-cpngn" event={"ID":"811f7ddb-91b4-4102-9ee5-ecb2a04ab80f","Type":"ContainerStarted","Data":"899d9270fd1b1a2a969fb589eae5ceffef74ec534219e99cd5b7de466030bde6"}
Jul 28 17:46:53.171695 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:53.171655    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-787a1-77c7b956dc-cpngn" event={"ID":"811f7ddb-91b4-4102-9ee5-ecb2a04ab80f","Type":"ContainerStarted","Data":"a9684a7765d7709e7e17292cf02c3ac63a974565f66004d78a9688ef2ea6ecf6"}
Jul 28 17:46:53.171695 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:53.171687    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-787a1-77c7b956dc-cpngn"
Jul 28 17:46:53.191252 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:53.191194    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-787a1-77c7b956dc-cpngn" podStartSLOduration=1.191171951 podStartE2EDuration="1.191171951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:46:53.189870332 +0000 UTC m=+1463.500035837" watchObservedRunningTime="2026-07-28 17:46:53.191171951 +0000 UTC m=+1463.501337455"
Jul 28 17:46:59.181765 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:46:59.181733    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-787a1-77c7b956dc-cpngn"
Jul 28 17:53:53.432190 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:53.432104    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-614a5-7dcdd75c-qs5jf"]
Jul 28 17:53:53.432787 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:53.432417    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-614a5-7dcdd75c-qs5jf" podUID="bf0370bc-f2ce-4d77-bab2-057cf8adefd5" containerName="ensemble-graph-614a5" containerID="cri-o://cdec6481805f1ee801afadd1a740699b15359414ae0f5880dce6c5a8a830a49d" gracePeriod=30
Jul 28 17:53:53.591587 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:53.591550    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8"]
Jul 28 17:53:53.591796 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:53.591773    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8" podUID="0da81b68-4834-4e53-b9b1-1305fcec6bc6" containerName="kserve-container" containerID="cri-o://d95b7574486f54196383609ef9ba293dbbbbcd8a0bec34681aa006d6145d1662" gracePeriod=30
Jul 28 17:53:53.591849 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:53.591801    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8" podUID="0da81b68-4834-4e53-b9b1-1305fcec6bc6" containerName="kube-rbac-proxy" containerID="cri-o://4b7fbb09d5a17d97b9de73c7657355a7b6b9d49dbae07786d8b9080dab8abaa5" gracePeriod=30
Jul 28 17:53:53.662644 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:53.662610    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft"]
Jul 28 17:53:53.666013 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:53.665982    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft"
Jul 28 17:53:53.668632 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:53.668604    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f7704-predictor-serving-cert\""
Jul 28 17:53:53.668778 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:53.668707    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f7704-kube-rbac-proxy-sar-config\""
Jul 28 17:53:53.678739 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:53.678704    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft"]
Jul 28 17:53:53.809640 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:53.809606    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b7f1dcc-7061-4a60-ae5e-be54d0b5171e-proxy-tls\") pod \"success-200-isvc-f7704-predictor-865dcf7dff-bjhft\" (UID: \"0b7f1dcc-7061-4a60-ae5e-be54d0b5171e\") " pod="kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft"
Jul 28 17:53:53.809640 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:53.809642    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szzfl\" (UniqueName: \"kubernetes.io/projected/0b7f1dcc-7061-4a60-ae5e-be54d0b5171e-kube-api-access-szzfl\") pod \"success-200-isvc-f7704-predictor-865dcf7dff-bjhft\" (UID: \"0b7f1dcc-7061-4a60-ae5e-be54d0b5171e\") " pod="kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft"
Jul 28 17:53:53.809845 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:53.809756    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b7f1dcc-7061-4a60-ae5e-be54d0b5171e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f7704-predictor-865dcf7dff-bjhft\" (UID: \"0b7f1dcc-7061-4a60-ae5e-be54d0b5171e\") " pod="kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft"
Jul 28 17:53:53.910312 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:53.910266    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b7f1dcc-7061-4a60-ae5e-be54d0b5171e-proxy-tls\") pod \"success-200-isvc-f7704-predictor-865dcf7dff-bjhft\" (UID: \"0b7f1dcc-7061-4a60-ae5e-be54d0b5171e\") " pod="kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft"
Jul 28 17:53:53.910312 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:53.910309    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szzfl\" (UniqueName: \"kubernetes.io/projected/0b7f1dcc-7061-4a60-ae5e-be54d0b5171e-kube-api-access-szzfl\") pod \"success-200-isvc-f7704-predictor-865dcf7dff-bjhft\" (UID: \"0b7f1dcc-7061-4a60-ae5e-be54d0b5171e\") " pod="kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft"
Jul 28 17:53:53.910584 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:53.910392    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b7f1dcc-7061-4a60-ae5e-be54d0b5171e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f7704-predictor-865dcf7dff-bjhft\" (UID: \"0b7f1dcc-7061-4a60-ae5e-be54d0b5171e\") " pod="kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft"
Jul 28 17:53:53.911031 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:53.911008    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b7f1dcc-7061-4a60-ae5e-be54d0b5171e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f7704-predictor-865dcf7dff-bjhft\" (UID: \"0b7f1dcc-7061-4a60-ae5e-be54d0b5171e\") " pod="kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft"
Jul 28 17:53:53.912841 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:53.912818    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b7f1dcc-7061-4a60-ae5e-be54d0b5171e-proxy-tls\") pod \"success-200-isvc-f7704-predictor-865dcf7dff-bjhft\" (UID: \"0b7f1dcc-7061-4a60-ae5e-be54d0b5171e\") " pod="kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft"
Jul 28 17:53:53.919462 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:53.919434    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szzfl\" (UniqueName: \"kubernetes.io/projected/0b7f1dcc-7061-4a60-ae5e-be54d0b5171e-kube-api-access-szzfl\") pod \"success-200-isvc-f7704-predictor-865dcf7dff-bjhft\" (UID: \"0b7f1dcc-7061-4a60-ae5e-be54d0b5171e\") " pod="kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft"
Jul 28 17:53:53.977914 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:53.977813    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft"
Jul 28 17:53:54.114880 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:54.114845    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft"]
Jul 28 17:53:54.118055 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:53:54.118026    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b7f1dcc_7061_4a60_ae5e_be54d0b5171e.slice/crio-7f19c81fd9163addfce99db9fb11d32dee9680d06d14d140698bcbc118946115 WatchSource:0}: Error finding container 7f19c81fd9163addfce99db9fb11d32dee9680d06d14d140698bcbc118946115: Status 404 returned error can't find the container with id 7f19c81fd9163addfce99db9fb11d32dee9680d06d14d140698bcbc118946115
Jul 28 17:53:54.693645 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:54.693603    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft" event={"ID":"0b7f1dcc-7061-4a60-ae5e-be54d0b5171e","Type":"ContainerStarted","Data":"2672b1a332e61e66faaab98c610fa5633e852b5f276b939060ae0b28b3c628b8"}
Jul 28 17:53:54.694127 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:54.693692    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft"
Jul 28 17:53:54.694127 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:54.693727    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft" event={"ID":"0b7f1dcc-7061-4a60-ae5e-be54d0b5171e","Type":"ContainerStarted","Data":"ce9ae889ff149ab56f76e278b8474b0f509be9dea2a0152feb0fd9d50ef9fa46"}
Jul 28 17:53:54.694127 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:54.693763    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft" event={"ID":"0b7f1dcc-7061-4a60-ae5e-be54d0b5171e","Type":"ContainerStarted","Data":"7f19c81fd9163addfce99db9fb11d32dee9680d06d14d140698bcbc118946115"}
Jul 28 17:53:54.695669 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:54.695639    2566 generic.go:356] "Generic (PLEG): container finished" podID="0da81b68-4834-4e53-b9b1-1305fcec6bc6" containerID="4b7fbb09d5a17d97b9de73c7657355a7b6b9d49dbae07786d8b9080dab8abaa5" exitCode=2
Jul 28 17:53:54.695810 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:54.695705    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8" event={"ID":"0da81b68-4834-4e53-b9b1-1305fcec6bc6","Type":"ContainerDied","Data":"4b7fbb09d5a17d97b9de73c7657355a7b6b9d49dbae07786d8b9080dab8abaa5"}
Jul 28 17:53:55.666893 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:55.666848    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8" podUID="0da81b68-4834-4e53-b9b1-1305fcec6bc6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Jul 28 17:53:55.672331 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:55.672293    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8" podUID="0da81b68-4834-4e53-b9b1-1305fcec6bc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 17:53:55.709558 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:55.709520    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft"
Jul 28 17:53:55.711217 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:55.711175    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft" podUID="0b7f1dcc-7061-4a60-ae5e-be54d0b5171e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 17:53:55.907051 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:55.906999    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-614a5-7dcdd75c-qs5jf" podUID="bf0370bc-f2ce-4d77-bab2-057cf8adefd5" containerName="ensemble-graph-614a5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:53:56.714351 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:56.714296    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft" podUID="0b7f1dcc-7061-4a60-ae5e-be54d0b5171e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 17:53:57.338115 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:57.338073    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8"
Jul 28 17:53:57.362561 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:57.362445    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft" podStartSLOduration=4.36242219 podStartE2EDuration="4.36242219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:53:54.727881349 +0000 UTC m=+1885.038046878" watchObservedRunningTime="2026-07-28 17:53:57.36242219 +0000 UTC m=+1887.672587695"
Jul 28 17:53:57.440201 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:57.440162    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0da81b68-4834-4e53-b9b1-1305fcec6bc6-proxy-tls\") pod \"0da81b68-4834-4e53-b9b1-1305fcec6bc6\" (UID: \"0da81b68-4834-4e53-b9b1-1305fcec6bc6\") "
Jul 28 17:53:57.440424 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:57.440241    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kt8rt\" (UniqueName: \"kubernetes.io/projected/0da81b68-4834-4e53-b9b1-1305fcec6bc6-kube-api-access-kt8rt\") pod \"0da81b68-4834-4e53-b9b1-1305fcec6bc6\" (UID: \"0da81b68-4834-4e53-b9b1-1305fcec6bc6\") "
Jul 28 17:53:57.440424 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:57.440266    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0da81b68-4834-4e53-b9b1-1305fcec6bc6-kube-rbac-proxy-sar-config\") pod \"0da81b68-4834-4e53-b9b1-1305fcec6bc6\" (UID: \"0da81b68-4834-4e53-b9b1-1305fcec6bc6\") "
Jul 28 17:53:57.440668 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:57.440638    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da81b68-4834-4e53-b9b1-1305fcec6bc6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0da81b68-4834-4e53-b9b1-1305fcec6bc6" (UID: "0da81b68-4834-4e53-b9b1-1305fcec6bc6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:53:57.442491 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:57.442461    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da81b68-4834-4e53-b9b1-1305fcec6bc6-kube-api-access-kt8rt" (OuterVolumeSpecName: "kube-api-access-kt8rt") pod "0da81b68-4834-4e53-b9b1-1305fcec6bc6" (UID: "0da81b68-4834-4e53-b9b1-1305fcec6bc6"). InnerVolumeSpecName "kube-api-access-kt8rt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:53:57.442594 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:57.442505    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da81b68-4834-4e53-b9b1-1305fcec6bc6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0da81b68-4834-4e53-b9b1-1305fcec6bc6" (UID: "0da81b68-4834-4e53-b9b1-1305fcec6bc6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:53:57.541290 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:57.541246    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0da81b68-4834-4e53-b9b1-1305fcec6bc6-proxy-tls\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:53:57.541290 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:57.541283    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt8rt\" (UniqueName: \"kubernetes.io/projected/0da81b68-4834-4e53-b9b1-1305fcec6bc6-kube-api-access-kt8rt\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:53:57.541290 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:57.541294    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0da81b68-4834-4e53-b9b1-1305fcec6bc6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:53:57.719127 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:57.719088    2566 generic.go:356] "Generic (PLEG): container finished" podID="0da81b68-4834-4e53-b9b1-1305fcec6bc6" containerID="d95b7574486f54196383609ef9ba293dbbbbcd8a0bec34681aa006d6145d1662" exitCode=0
Jul 28 17:53:57.719639 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:57.719156    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8"
Jul 28 17:53:57.719639 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:57.719170    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8" event={"ID":"0da81b68-4834-4e53-b9b1-1305fcec6bc6","Type":"ContainerDied","Data":"d95b7574486f54196383609ef9ba293dbbbbcd8a0bec34681aa006d6145d1662"}
Jul 28 17:53:57.719639 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:57.719235    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8" event={"ID":"0da81b68-4834-4e53-b9b1-1305fcec6bc6","Type":"ContainerDied","Data":"207d9810f6cf426c7b44b27fe70577e37aa45f651e0c2085ff92780da949e146"}
Jul 28 17:53:57.719639 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:57.719265    2566 scope.go:117] "RemoveContainer" containerID="4b7fbb09d5a17d97b9de73c7657355a7b6b9d49dbae07786d8b9080dab8abaa5"
Jul 28 17:53:57.728201 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:57.728179    2566 scope.go:117] "RemoveContainer" containerID="d95b7574486f54196383609ef9ba293dbbbbcd8a0bec34681aa006d6145d1662"
Jul 28 17:53:57.735959 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:57.735935    2566 scope.go:117] "RemoveContainer" containerID="4b7fbb09d5a17d97b9de73c7657355a7b6b9d49dbae07786d8b9080dab8abaa5"
Jul 28 17:53:57.736384 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:53:57.736362    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b7fbb09d5a17d97b9de73c7657355a7b6b9d49dbae07786d8b9080dab8abaa5\": container with ID starting with 4b7fbb09d5a17d97b9de73c7657355a7b6b9d49dbae07786d8b9080dab8abaa5 not found: ID does not exist" containerID="4b7fbb09d5a17d97b9de73c7657355a7b6b9d49dbae07786d8b9080dab8abaa5"
Jul 28 17:53:57.736461 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:57.736395    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b7fbb09d5a17d97b9de73c7657355a7b6b9d49dbae07786d8b9080dab8abaa5"} err="failed to get container status \"4b7fbb09d5a17d97b9de73c7657355a7b6b9d49dbae07786d8b9080dab8abaa5\": rpc error: code = NotFound desc = could not find container \"4b7fbb09d5a17d97b9de73c7657355a7b6b9d49dbae07786d8b9080dab8abaa5\": container with ID starting with 4b7fbb09d5a17d97b9de73c7657355a7b6b9d49dbae07786d8b9080dab8abaa5 not found: ID does not exist"
Jul 28 17:53:57.736461 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:57.736415    2566 scope.go:117] "RemoveContainer" containerID="d95b7574486f54196383609ef9ba293dbbbbcd8a0bec34681aa006d6145d1662"
Jul 28 17:53:57.736673 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:53:57.736658    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d95b7574486f54196383609ef9ba293dbbbbcd8a0bec34681aa006d6145d1662\": container with ID starting with d95b7574486f54196383609ef9ba293dbbbbcd8a0bec34681aa006d6145d1662 not found: ID does not exist" containerID="d95b7574486f54196383609ef9ba293dbbbbcd8a0bec34681aa006d6145d1662"
Jul 28 17:53:57.736714 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:57.736681    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d95b7574486f54196383609ef9ba293dbbbbcd8a0bec34681aa006d6145d1662"} err="failed to get container status \"d95b7574486f54196383609ef9ba293dbbbbcd8a0bec34681aa006d6145d1662\": rpc error: code = NotFound desc = could not find container \"d95b7574486f54196383609ef9ba293dbbbbcd8a0bec34681aa006d6145d1662\": container with ID starting with d95b7574486f54196383609ef9ba293dbbbbcd8a0bec34681aa006d6145d1662 not found: ID does not exist"
Jul 28 17:53:57.750533 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:57.750507    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8"]
Jul 28 17:53:57.755880 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:57.755849    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8"]
Jul 28 17:53:58.097898 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:53:58.097812    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0da81b68-4834-4e53-b9b1-1305fcec6bc6" path="/var/lib/kubelet/pods/0da81b68-4834-4e53-b9b1-1305fcec6bc6/volumes"
Jul 28 17:54:00.905892 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:00.905851    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-614a5-7dcdd75c-qs5jf" podUID="bf0370bc-f2ce-4d77-bab2-057cf8adefd5" containerName="ensemble-graph-614a5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:54:01.719474 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:01.719442    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft"
Jul 28 17:54:01.719999 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:01.719965    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft" podUID="0b7f1dcc-7061-4a60-ae5e-be54d0b5171e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 17:54:05.904924 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:05.904878    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-614a5-7dcdd75c-qs5jf" podUID="bf0370bc-f2ce-4d77-bab2-057cf8adefd5" containerName="ensemble-graph-614a5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:54:05.905411 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:05.905019    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-614a5-7dcdd75c-qs5jf"
Jul 28 17:54:10.904947 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:10.904899    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-614a5-7dcdd75c-qs5jf" podUID="bf0370bc-f2ce-4d77-bab2-057cf8adefd5" containerName="ensemble-graph-614a5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:54:11.719966 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:11.719923    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft" podUID="0b7f1dcc-7061-4a60-ae5e-be54d0b5171e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 17:54:15.905784 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:15.905743    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-614a5-7dcdd75c-qs5jf" podUID="bf0370bc-f2ce-4d77-bab2-057cf8adefd5" containerName="ensemble-graph-614a5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:54:20.905984 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:20.905946    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-614a5-7dcdd75c-qs5jf" podUID="bf0370bc-f2ce-4d77-bab2-057cf8adefd5" containerName="ensemble-graph-614a5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:54:21.720511 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:21.720471    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft" podUID="0b7f1dcc-7061-4a60-ae5e-be54d0b5171e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 17:54:23.588019 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:23.587991    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-614a5-7dcdd75c-qs5jf"
Jul 28 17:54:23.634568 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:23.634533    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf0370bc-f2ce-4d77-bab2-057cf8adefd5-openshift-service-ca-bundle\") pod \"bf0370bc-f2ce-4d77-bab2-057cf8adefd5\" (UID: \"bf0370bc-f2ce-4d77-bab2-057cf8adefd5\") "
Jul 28 17:54:23.634770 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:23.634581    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf0370bc-f2ce-4d77-bab2-057cf8adefd5-proxy-tls\") pod \"bf0370bc-f2ce-4d77-bab2-057cf8adefd5\" (UID: \"bf0370bc-f2ce-4d77-bab2-057cf8adefd5\") "
Jul 28 17:54:23.634943 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:23.634909    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf0370bc-f2ce-4d77-bab2-057cf8adefd5-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "bf0370bc-f2ce-4d77-bab2-057cf8adefd5" (UID: "bf0370bc-f2ce-4d77-bab2-057cf8adefd5"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:54:23.636812 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:23.636786    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf0370bc-f2ce-4d77-bab2-057cf8adefd5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bf0370bc-f2ce-4d77-bab2-057cf8adefd5" (UID: "bf0370bc-f2ce-4d77-bab2-057cf8adefd5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:54:23.735846 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:23.735765    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf0370bc-f2ce-4d77-bab2-057cf8adefd5-openshift-service-ca-bundle\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:54:23.735846 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:23.735795    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf0370bc-f2ce-4d77-bab2-057cf8adefd5-proxy-tls\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:54:23.812667 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:23.812635    2566 generic.go:356] "Generic (PLEG): container finished" podID="bf0370bc-f2ce-4d77-bab2-057cf8adefd5" containerID="cdec6481805f1ee801afadd1a740699b15359414ae0f5880dce6c5a8a830a49d" exitCode=137
Jul 28 17:54:23.812824 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:23.812679    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-614a5-7dcdd75c-qs5jf"
Jul 28 17:54:23.812824 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:23.812723    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-614a5-7dcdd75c-qs5jf" event={"ID":"bf0370bc-f2ce-4d77-bab2-057cf8adefd5","Type":"ContainerDied","Data":"cdec6481805f1ee801afadd1a740699b15359414ae0f5880dce6c5a8a830a49d"}
Jul 28 17:54:23.812824 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:23.812775    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-614a5-7dcdd75c-qs5jf" event={"ID":"bf0370bc-f2ce-4d77-bab2-057cf8adefd5","Type":"ContainerDied","Data":"d8e34ff0f40acfc7a04bbb2d7ad9ae6ca99f3d50f641b5fe4d633657b9633e51"}
Jul 28 17:54:23.812824 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:23.812799    2566 scope.go:117] "RemoveContainer" containerID="cdec6481805f1ee801afadd1a740699b15359414ae0f5880dce6c5a8a830a49d"
Jul 28 17:54:23.822071 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:23.822046    2566 scope.go:117] "RemoveContainer" containerID="cdec6481805f1ee801afadd1a740699b15359414ae0f5880dce6c5a8a830a49d"
Jul 28 17:54:23.822454 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:54:23.822429    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdec6481805f1ee801afadd1a740699b15359414ae0f5880dce6c5a8a830a49d\": container with ID starting with cdec6481805f1ee801afadd1a740699b15359414ae0f5880dce6c5a8a830a49d not found: ID does not exist" containerID="cdec6481805f1ee801afadd1a740699b15359414ae0f5880dce6c5a8a830a49d"
Jul 28 17:54:23.822538 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:23.822468    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdec6481805f1ee801afadd1a740699b15359414ae0f5880dce6c5a8a830a49d"} err="failed to get container status \"cdec6481805f1ee801afadd1a740699b15359414ae0f5880dce6c5a8a830a49d\": rpc error: code = NotFound desc = could not find container \"cdec6481805f1ee801afadd1a740699b15359414ae0f5880dce6c5a8a830a49d\": container with ID starting with cdec6481805f1ee801afadd1a740699b15359414ae0f5880dce6c5a8a830a49d not found: ID does not exist"
Jul 28 17:54:23.838366 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:23.838314    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-614a5-7dcdd75c-qs5jf"]
Jul 28 17:54:23.844053 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:23.844020    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-614a5-7dcdd75c-qs5jf"]
Jul 28 17:54:24.098156 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:24.098076    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf0370bc-f2ce-4d77-bab2-057cf8adefd5" path="/var/lib/kubelet/pods/bf0370bc-f2ce-4d77-bab2-057cf8adefd5/volumes"
Jul 28 17:54:31.720334 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:31.720276    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft" podUID="0b7f1dcc-7061-4a60-ae5e-be54d0b5171e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 17:54:41.720474 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:41.720439    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft"
Jul 28 17:54:53.713083 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:53.713041    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-f7704-758c568bf-l924l"]
Jul 28 17:54:53.713781 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:53.713445    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf0370bc-f2ce-4d77-bab2-057cf8adefd5" containerName="ensemble-graph-614a5"
Jul 28 17:54:53.713781 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:53.713464    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf0370bc-f2ce-4d77-bab2-057cf8adefd5" containerName="ensemble-graph-614a5"
Jul 28 17:54:53.713781 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:53.713488    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0da81b68-4834-4e53-b9b1-1305fcec6bc6" containerName="kserve-container"
Jul 28 17:54:53.713781 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:53.713497    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0da81b68-4834-4e53-b9b1-1305fcec6bc6" containerName="kserve-container"
Jul 28 17:54:53.713781 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:53.713521    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0da81b68-4834-4e53-b9b1-1305fcec6bc6" containerName="kube-rbac-proxy"
Jul 28 17:54:53.713781 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:53.713529    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0da81b68-4834-4e53-b9b1-1305fcec6bc6" containerName="kube-rbac-proxy"
Jul 28 17:54:53.713781 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:53.713587    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0da81b68-4834-4e53-b9b1-1305fcec6bc6" containerName="kserve-container"
Jul 28 17:54:53.713781 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:53.713598    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf0370bc-f2ce-4d77-bab2-057cf8adefd5" containerName="ensemble-graph-614a5"
Jul 28 17:54:53.713781 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:53.713610    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0da81b68-4834-4e53-b9b1-1305fcec6bc6" containerName="kube-rbac-proxy"
Jul 28 17:54:53.716462 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:53.716444    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-f7704-758c568bf-l924l"
Jul 28 17:54:53.719161 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:53.719132    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-f7704-serving-cert\""
Jul 28 17:54:53.719161 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:53.719151    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-f7704-kube-rbac-proxy-sar-config\""
Jul 28 17:54:53.725204 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:53.725179    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-f7704-758c568bf-l924l"]
Jul 28 17:54:53.770676 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:53.770637    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcb68714-b150-4245-8506-40ff87015604-proxy-tls\") pod \"splitter-graph-f7704-758c568bf-l924l\" (UID: \"fcb68714-b150-4245-8506-40ff87015604\") " pod="kserve-ci-e2e-test/splitter-graph-f7704-758c568bf-l924l"
Jul 28 17:54:53.770676 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:53.770683    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcb68714-b150-4245-8506-40ff87015604-openshift-service-ca-bundle\") pod \"splitter-graph-f7704-758c568bf-l924l\" (UID: \"fcb68714-b150-4245-8506-40ff87015604\") " pod="kserve-ci-e2e-test/splitter-graph-f7704-758c568bf-l924l"
Jul 28 17:54:53.871346 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:53.871281    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcb68714-b150-4245-8506-40ff87015604-proxy-tls\") pod \"splitter-graph-f7704-758c568bf-l924l\" (UID: \"fcb68714-b150-4245-8506-40ff87015604\") " pod="kserve-ci-e2e-test/splitter-graph-f7704-758c568bf-l924l"
Jul 28 17:54:53.871346 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:53.871352    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcb68714-b150-4245-8506-40ff87015604-openshift-service-ca-bundle\") pod \"splitter-graph-f7704-758c568bf-l924l\" (UID: \"fcb68714-b150-4245-8506-40ff87015604\") " pod="kserve-ci-e2e-test/splitter-graph-f7704-758c568bf-l924l"
Jul 28 17:54:53.871554 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:54:53.871437    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-f7704-serving-cert: secret "splitter-graph-f7704-serving-cert" not found
Jul 28 17:54:53.871554 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:54:53.871512    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcb68714-b150-4245-8506-40ff87015604-proxy-tls podName:fcb68714-b150-4245-8506-40ff87015604 nodeName:}" failed. No retries permitted until 2026-07-28 17:54:54.371496391 +0000 UTC m=+1944.681661873 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fcb68714-b150-4245-8506-40ff87015604-proxy-tls") pod "splitter-graph-f7704-758c568bf-l924l" (UID: "fcb68714-b150-4245-8506-40ff87015604") : secret "splitter-graph-f7704-serving-cert" not found
Jul 28 17:54:53.871982 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:53.871963    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcb68714-b150-4245-8506-40ff87015604-openshift-service-ca-bundle\") pod \"splitter-graph-f7704-758c568bf-l924l\" (UID: \"fcb68714-b150-4245-8506-40ff87015604\") " pod="kserve-ci-e2e-test/splitter-graph-f7704-758c568bf-l924l"
Jul 28 17:54:54.374625 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:54.374574    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcb68714-b150-4245-8506-40ff87015604-proxy-tls\") pod \"splitter-graph-f7704-758c568bf-l924l\" (UID: \"fcb68714-b150-4245-8506-40ff87015604\") " pod="kserve-ci-e2e-test/splitter-graph-f7704-758c568bf-l924l"
Jul 28 17:54:54.377136 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:54.377101    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcb68714-b150-4245-8506-40ff87015604-proxy-tls\") pod \"splitter-graph-f7704-758c568bf-l924l\" (UID: \"fcb68714-b150-4245-8506-40ff87015604\") " pod="kserve-ci-e2e-test/splitter-graph-f7704-758c568bf-l924l"
Jul 28 17:54:54.627227 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:54.627123    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-f7704-758c568bf-l924l"
Jul 28 17:54:54.772048 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:54.772020    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-f7704-758c568bf-l924l"]
Jul 28 17:54:54.774361 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:54:54.774310    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb68714_b150_4245_8506_40ff87015604.slice/crio-5183b2c47a1db7176ef59b673b05ad0aa7adb6f8ddab3fc8dec587d4d40dd8f2 WatchSource:0}: Error finding container 5183b2c47a1db7176ef59b673b05ad0aa7adb6f8ddab3fc8dec587d4d40dd8f2: Status 404 returned error can't find the container with id 5183b2c47a1db7176ef59b673b05ad0aa7adb6f8ddab3fc8dec587d4d40dd8f2
Jul 28 17:54:54.933117 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:54.933068    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-f7704-758c568bf-l924l" event={"ID":"fcb68714-b150-4245-8506-40ff87015604","Type":"ContainerStarted","Data":"173ab4903d91c05dd98020f6a9719c8438c649548bd7ec15fa6377bc94c517eb"}
Jul 28 17:54:54.933298 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:54.933135    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-f7704-758c568bf-l924l" event={"ID":"fcb68714-b150-4245-8506-40ff87015604","Type":"ContainerStarted","Data":"5183b2c47a1db7176ef59b673b05ad0aa7adb6f8ddab3fc8dec587d4d40dd8f2"}
Jul 28 17:54:54.933298 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:54.933174    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-f7704-758c568bf-l924l"
Jul 28 17:54:54.953523 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:54:54.953466    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-f7704-758c568bf-l924l" podStartSLOduration=1.953419739 podStartE2EDuration="1.953419739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:54:54.952622014 +0000 UTC m=+1945.262787518" watchObservedRunningTime="2026-07-28 17:54:54.953419739 +0000 UTC m=+1945.263585246"
Jul 28 17:55:00.942376 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:00.942343    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-f7704-758c568bf-l924l"
Jul 28 17:55:03.819359 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:03.819307    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-f7704-758c568bf-l924l"]
Jul 28 17:55:03.819767 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:03.819521    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-f7704-758c568bf-l924l" podUID="fcb68714-b150-4245-8506-40ff87015604" containerName="splitter-graph-f7704" containerID="cri-o://173ab4903d91c05dd98020f6a9719c8438c649548bd7ec15fa6377bc94c517eb" gracePeriod=30
Jul 28 17:55:04.004446 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:04.004413    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft"]
Jul 28 17:55:04.004663 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:04.004636    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft" podUID="0b7f1dcc-7061-4a60-ae5e-be54d0b5171e" containerName="kserve-container" containerID="cri-o://ce9ae889ff149ab56f76e278b8474b0f509be9dea2a0152feb0fd9d50ef9fa46" gracePeriod=30
Jul 28 17:55:04.004730 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:04.004674    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft" podUID="0b7f1dcc-7061-4a60-ae5e-be54d0b5171e" containerName="kube-rbac-proxy" containerID="cri-o://2672b1a332e61e66faaab98c610fa5633e852b5f276b939060ae0b28b3c628b8" gracePeriod=30
Jul 28 17:55:04.040901 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:04.040861    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl"]
Jul 28 17:55:04.044033 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:04.044013    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl"
Jul 28 17:55:04.047602 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:04.047577    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7d999-predictor-serving-cert\""
Jul 28 17:55:04.047755 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:04.047577    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7d999-kube-rbac-proxy-sar-config\""
Jul 28 17:55:04.071536 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:04.071456    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl"]
Jul 28 17:55:04.152354 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:04.152301    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8155544f-2dae-4598-962f-97a2c83dcdf2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7d999-predictor-7f48846ddc-d5stl\" (UID: \"8155544f-2dae-4598-962f-97a2c83dcdf2\") " pod="kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl"
Jul 28 17:55:04.152550 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:04.152369    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8155544f-2dae-4598-962f-97a2c83dcdf2-proxy-tls\") pod \"success-200-isvc-7d999-predictor-7f48846ddc-d5stl\" (UID: \"8155544f-2dae-4598-962f-97a2c83dcdf2\") " pod="kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl"
Jul 28 17:55:04.152550 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:04.152397    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4rwg\" (UniqueName: \"kubernetes.io/projected/8155544f-2dae-4598-962f-97a2c83dcdf2-kube-api-access-n4rwg\") pod \"success-200-isvc-7d999-predictor-7f48846ddc-d5stl\" (UID: \"8155544f-2dae-4598-962f-97a2c83dcdf2\") " pod="kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl"
Jul 28 17:55:04.252914 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:04.252868    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8155544f-2dae-4598-962f-97a2c83dcdf2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7d999-predictor-7f48846ddc-d5stl\" (UID: \"8155544f-2dae-4598-962f-97a2c83dcdf2\") " pod="kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl"
Jul 28 17:55:04.252914 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:04.252915    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8155544f-2dae-4598-962f-97a2c83dcdf2-proxy-tls\") pod \"success-200-isvc-7d999-predictor-7f48846ddc-d5stl\" (UID: \"8155544f-2dae-4598-962f-97a2c83dcdf2\") " pod="kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl"
Jul 28 17:55:04.253214 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:04.252944    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4rwg\" (UniqueName: \"kubernetes.io/projected/8155544f-2dae-4598-962f-97a2c83dcdf2-kube-api-access-n4rwg\") pod \"success-200-isvc-7d999-predictor-7f48846ddc-d5stl\" (UID: \"8155544f-2dae-4598-962f-97a2c83dcdf2\") " pod="kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl"
Jul 28 17:55:04.253214 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:55:04.253063    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-7d999-predictor-serving-cert: secret "success-200-isvc-7d999-predictor-serving-cert" not found
Jul 28 17:55:04.253214 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:55:04.253144    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8155544f-2dae-4598-962f-97a2c83dcdf2-proxy-tls podName:8155544f-2dae-4598-962f-97a2c83dcdf2 nodeName:}" failed. No retries permitted until 2026-07-28 17:55:04.753123948 +0000 UTC m=+1955.063289431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8155544f-2dae-4598-962f-97a2c83dcdf2-proxy-tls") pod "success-200-isvc-7d999-predictor-7f48846ddc-d5stl" (UID: "8155544f-2dae-4598-962f-97a2c83dcdf2") : secret "success-200-isvc-7d999-predictor-serving-cert" not found
Jul 28 17:55:04.253580 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:04.253561    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8155544f-2dae-4598-962f-97a2c83dcdf2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7d999-predictor-7f48846ddc-d5stl\" (UID: \"8155544f-2dae-4598-962f-97a2c83dcdf2\") " pod="kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl"
Jul 28 17:55:04.265145 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:04.265111    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4rwg\" (UniqueName: \"kubernetes.io/projected/8155544f-2dae-4598-962f-97a2c83dcdf2-kube-api-access-n4rwg\") pod \"success-200-isvc-7d999-predictor-7f48846ddc-d5stl\" (UID: \"8155544f-2dae-4598-962f-97a2c83dcdf2\") " pod="kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl"
Jul 28 17:55:04.755684 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:04.755635    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8155544f-2dae-4598-962f-97a2c83dcdf2-proxy-tls\") pod \"success-200-isvc-7d999-predictor-7f48846ddc-d5stl\" (UID: \"8155544f-2dae-4598-962f-97a2c83dcdf2\") " pod="kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl"
Jul 28 17:55:04.758124 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:04.758100    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8155544f-2dae-4598-962f-97a2c83dcdf2-proxy-tls\") pod \"success-200-isvc-7d999-predictor-7f48846ddc-d5stl\" (UID: \"8155544f-2dae-4598-962f-97a2c83dcdf2\") " pod="kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl"
Jul 28 17:55:04.954350 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:04.954267    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl"
Jul 28 17:55:04.972126 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:04.972091    2566 generic.go:356] "Generic (PLEG): container finished" podID="0b7f1dcc-7061-4a60-ae5e-be54d0b5171e" containerID="2672b1a332e61e66faaab98c610fa5633e852b5f276b939060ae0b28b3c628b8" exitCode=2
Jul 28 17:55:04.972276 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:04.972162    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft" event={"ID":"0b7f1dcc-7061-4a60-ae5e-be54d0b5171e","Type":"ContainerDied","Data":"2672b1a332e61e66faaab98c610fa5633e852b5f276b939060ae0b28b3c628b8"}
Jul 28 17:55:05.097347 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:05.096954    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl"]
Jul 28 17:55:05.103213 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:55:05.103176    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8155544f_2dae_4598_962f_97a2c83dcdf2.slice/crio-b0bdf3f14da07882d747b5d7f2a0062044b83ed591a45faf32a8025e5c14bfb4 WatchSource:0}: Error finding container b0bdf3f14da07882d747b5d7f2a0062044b83ed591a45faf32a8025e5c14bfb4: Status 404 returned error can't find the container with id b0bdf3f14da07882d747b5d7f2a0062044b83ed591a45faf32a8025e5c14bfb4
Jul 28 17:55:05.940981 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:05.940937    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-f7704-758c568bf-l924l" podUID="fcb68714-b150-4245-8506-40ff87015604" containerName="splitter-graph-f7704" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:55:05.978245 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:05.978206    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl" event={"ID":"8155544f-2dae-4598-962f-97a2c83dcdf2","Type":"ContainerStarted","Data":"3073bf7f879c3556e5256111838ef6487dba80b955037c6f8c83f97cc5fe7bbf"}
Jul 28 17:55:05.978715 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:05.978275    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl"
Jul 28 17:55:05.978715 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:05.978314    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl" event={"ID":"8155544f-2dae-4598-962f-97a2c83dcdf2","Type":"ContainerStarted","Data":"23a7d05bcd67244d714ecda3294ee22e83421bd4d3e648b9907d14fc1077a527"}
Jul 28 17:55:05.978715 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:05.978368    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl" event={"ID":"8155544f-2dae-4598-962f-97a2c83dcdf2","Type":"ContainerStarted","Data":"b0bdf3f14da07882d747b5d7f2a0062044b83ed591a45faf32a8025e5c14bfb4"}
Jul 28 17:55:05.978715 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:05.978400    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl"
Jul 28 17:55:05.979920 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:05.979892    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl" podUID="8155544f-2dae-4598-962f-97a2c83dcdf2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:55:05.998986 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:05.998934    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl" podStartSLOduration=1.998919049 podStartE2EDuration="1.998919049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:55:05.998542443 +0000 UTC m=+1956.308707960" watchObservedRunningTime="2026-07-28 17:55:05.998919049 +0000 UTC m=+1956.309084553"
Jul 28 17:55:06.714702 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:06.714653    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft" podUID="0b7f1dcc-7061-4a60-ae5e-be54d0b5171e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 28 17:55:06.982793 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:06.982687    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl" podUID="8155544f-2dae-4598-962f-97a2c83dcdf2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:55:07.179681 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:07.179643    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-787a1-77c7b956dc-cpngn"]
Jul 28 17:55:07.179865 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:07.179843    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-787a1-77c7b956dc-cpngn" podUID="811f7ddb-91b4-4102-9ee5-ecb2a04ab80f" containerName="sequence-graph-787a1" containerID="cri-o://899d9270fd1b1a2a969fb589eae5ceffef74ec534219e99cd5b7de466030bde6" gracePeriod=30
Jul 28 17:55:07.363507 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:07.363421    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc"]
Jul 28 17:55:07.366786 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:07.366766    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc"
Jul 28 17:55:07.369997 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:07.369966    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-cb1ee-predictor-serving-cert\""
Jul 28 17:55:07.370732 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:07.370710    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-cb1ee-kube-rbac-proxy-sar-config\""
Jul 28 17:55:07.387398 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:07.387361    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc"]
Jul 28 17:55:07.456291 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:07.456252    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk"]
Jul 28 17:55:07.456542 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:07.456513    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk" podUID="39137e71-bcda-4ba3-bdca-60c6e8a91b1a" containerName="kserve-container" containerID="cri-o://8a35008083e2cdad97726f99755029a0164a064e904ad1a460592bb46d0256a5" gracePeriod=30
Jul 28 17:55:07.456628 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:07.456541    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk" podUID="39137e71-bcda-4ba3-bdca-60c6e8a91b1a" containerName="kube-rbac-proxy" containerID="cri-o://9c7c719a3358b4b7a05dc1947c3d8c8e0975486281b618ed0894bbace4825db9" gracePeriod=30
Jul 28 17:55:07.480653 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:07.480597    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5dqp\" (UniqueName: \"kubernetes.io/projected/b1027091-27cb-46ce-ae3d-ecdcfe582ca3-kube-api-access-k5dqp\") pod \"success-200-isvc-cb1ee-predictor-7b757b875f-46rjc\" (UID: \"b1027091-27cb-46ce-ae3d-ecdcfe582ca3\") " pod="kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc"
Jul 28 17:55:07.480963 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:07.480786    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1027091-27cb-46ce-ae3d-ecdcfe582ca3-proxy-tls\") pod \"success-200-isvc-cb1ee-predictor-7b757b875f-46rjc\" (UID: \"b1027091-27cb-46ce-ae3d-ecdcfe582ca3\") " pod="kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc"
Jul 28 17:55:07.480963 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:07.480826    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1027091-27cb-46ce-ae3d-ecdcfe582ca3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-cb1ee-predictor-7b757b875f-46rjc\" (UID: \"b1027091-27cb-46ce-ae3d-ecdcfe582ca3\") " pod="kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc"
Jul 28 17:55:07.581972 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:07.581937    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1027091-27cb-46ce-ae3d-ecdcfe582ca3-proxy-tls\") pod \"success-200-isvc-cb1ee-predictor-7b757b875f-46rjc\" (UID: \"b1027091-27cb-46ce-ae3d-ecdcfe582ca3\") " pod="kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc"
Jul 28 17:55:07.582140 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:07.582069    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1027091-27cb-46ce-ae3d-ecdcfe582ca3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-cb1ee-predictor-7b757b875f-46rjc\" (UID: \"b1027091-27cb-46ce-ae3d-ecdcfe582ca3\") " pod="kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc"
Jul 28 17:55:07.582140 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:55:07.582092    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-serving-cert: secret "success-200-isvc-cb1ee-predictor-serving-cert" not found
Jul 28 17:55:07.582140 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:07.582121    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5dqp\" (UniqueName: \"kubernetes.io/projected/b1027091-27cb-46ce-ae3d-ecdcfe582ca3-kube-api-access-k5dqp\") pod \"success-200-isvc-cb1ee-predictor-7b757b875f-46rjc\" (UID: \"b1027091-27cb-46ce-ae3d-ecdcfe582ca3\") " pod="kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc"
Jul 28 17:55:07.582303 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:55:07.582183    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1027091-27cb-46ce-ae3d-ecdcfe582ca3-proxy-tls podName:b1027091-27cb-46ce-ae3d-ecdcfe582ca3 nodeName:}" failed. No retries permitted until 2026-07-28 17:55:08.082160008 +0000 UTC m=+1958.392325494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b1027091-27cb-46ce-ae3d-ecdcfe582ca3-proxy-tls") pod "success-200-isvc-cb1ee-predictor-7b757b875f-46rjc" (UID: "b1027091-27cb-46ce-ae3d-ecdcfe582ca3") : secret "success-200-isvc-cb1ee-predictor-serving-cert" not found
Jul 28 17:55:07.582912 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:07.582891    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1027091-27cb-46ce-ae3d-ecdcfe582ca3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-cb1ee-predictor-7b757b875f-46rjc\" (UID: \"b1027091-27cb-46ce-ae3d-ecdcfe582ca3\") " pod="kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc"
Jul 28 17:55:07.596417 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:07.596382    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5dqp\" (UniqueName: \"kubernetes.io/projected/b1027091-27cb-46ce-ae3d-ecdcfe582ca3-kube-api-access-k5dqp\") pod \"success-200-isvc-cb1ee-predictor-7b757b875f-46rjc\" (UID: \"b1027091-27cb-46ce-ae3d-ecdcfe582ca3\") " pod="kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc"
Jul 28 17:55:07.722576 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:07.722545    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft"
Jul 28 17:55:07.885181 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:07.885081    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b7f1dcc-7061-4a60-ae5e-be54d0b5171e-kube-rbac-proxy-sar-config\") pod \"0b7f1dcc-7061-4a60-ae5e-be54d0b5171e\" (UID: \"0b7f1dcc-7061-4a60-ae5e-be54d0b5171e\") "
Jul 28 17:55:07.885181 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:07.885136    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szzfl\" (UniqueName: \"kubernetes.io/projected/0b7f1dcc-7061-4a60-ae5e-be54d0b5171e-kube-api-access-szzfl\") pod \"0b7f1dcc-7061-4a60-ae5e-be54d0b5171e\" (UID: \"0b7f1dcc-7061-4a60-ae5e-be54d0b5171e\") "
Jul 28 17:55:07.885181 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:07.885172    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b7f1dcc-7061-4a60-ae5e-be54d0b5171e-proxy-tls\") pod \"0b7f1dcc-7061-4a60-ae5e-be54d0b5171e\" (UID: \"0b7f1dcc-7061-4a60-ae5e-be54d0b5171e\") "
Jul 28 17:55:07.885495 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:07.885472    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b7f1dcc-7061-4a60-ae5e-be54d0b5171e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0b7f1dcc-7061-4a60-ae5e-be54d0b5171e" (UID: "0b7f1dcc-7061-4a60-ae5e-be54d0b5171e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:55:07.887216 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:07.887195    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b7f1dcc-7061-4a60-ae5e-be54d0b5171e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0b7f1dcc-7061-4a60-ae5e-be54d0b5171e" (UID: "0b7f1dcc-7061-4a60-ae5e-be54d0b5171e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:55:07.887276 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:07.887244    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b7f1dcc-7061-4a60-ae5e-be54d0b5171e-kube-api-access-szzfl" (OuterVolumeSpecName: "kube-api-access-szzfl") pod "0b7f1dcc-7061-4a60-ae5e-be54d0b5171e" (UID: "0b7f1dcc-7061-4a60-ae5e-be54d0b5171e"). InnerVolumeSpecName "kube-api-access-szzfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:55:07.985858 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:07.985830    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b7f1dcc-7061-4a60-ae5e-be54d0b5171e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:55:07.985858 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:07.985854    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szzfl\" (UniqueName: \"kubernetes.io/projected/0b7f1dcc-7061-4a60-ae5e-be54d0b5171e-kube-api-access-szzfl\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:55:07.985858 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:07.985864    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b7f1dcc-7061-4a60-ae5e-be54d0b5171e-proxy-tls\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:55:07.988629 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:07.988600    2566 generic.go:356] "Generic (PLEG): container finished" podID="39137e71-bcda-4ba3-bdca-60c6e8a91b1a" containerID="9c7c719a3358b4b7a05dc1947c3d8c8e0975486281b618ed0894bbace4825db9" exitCode=2
Jul 28 17:55:07.988752 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:07.988670    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk" event={"ID":"39137e71-bcda-4ba3-bdca-60c6e8a91b1a","Type":"ContainerDied","Data":"9c7c719a3358b4b7a05dc1947c3d8c8e0975486281b618ed0894bbace4825db9"}
Jul 28 17:55:07.990102 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:07.990081    2566 generic.go:356] "Generic (PLEG): container finished" podID="0b7f1dcc-7061-4a60-ae5e-be54d0b5171e" containerID="ce9ae889ff149ab56f76e278b8474b0f509be9dea2a0152feb0fd9d50ef9fa46" exitCode=0
Jul 28 17:55:07.990214 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:07.990133    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft"
Jul 28 17:55:07.990214 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:07.990151    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft" event={"ID":"0b7f1dcc-7061-4a60-ae5e-be54d0b5171e","Type":"ContainerDied","Data":"ce9ae889ff149ab56f76e278b8474b0f509be9dea2a0152feb0fd9d50ef9fa46"}
Jul 28 17:55:07.990345 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:07.990212    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft" event={"ID":"0b7f1dcc-7061-4a60-ae5e-be54d0b5171e","Type":"ContainerDied","Data":"7f19c81fd9163addfce99db9fb11d32dee9680d06d14d140698bcbc118946115"}
Jul 28 17:55:07.990345 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:07.990252    2566 scope.go:117] "RemoveContainer" containerID="2672b1a332e61e66faaab98c610fa5633e852b5f276b939060ae0b28b3c628b8"
Jul 28 17:55:07.999493 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:07.999469    2566 scope.go:117] "RemoveContainer" containerID="ce9ae889ff149ab56f76e278b8474b0f509be9dea2a0152feb0fd9d50ef9fa46"
Jul 28 17:55:08.009191 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:08.009169    2566 scope.go:117] "RemoveContainer" containerID="2672b1a332e61e66faaab98c610fa5633e852b5f276b939060ae0b28b3c628b8"
Jul 28 17:55:08.009599 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:55:08.009576    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2672b1a332e61e66faaab98c610fa5633e852b5f276b939060ae0b28b3c628b8\": container with ID starting with 2672b1a332e61e66faaab98c610fa5633e852b5f276b939060ae0b28b3c628b8 not found: ID does not exist" containerID="2672b1a332e61e66faaab98c610fa5633e852b5f276b939060ae0b28b3c628b8"
Jul 28 17:55:08.009704 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:08.009612    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2672b1a332e61e66faaab98c610fa5633e852b5f276b939060ae0b28b3c628b8"} err="failed to get container status \"2672b1a332e61e66faaab98c610fa5633e852b5f276b939060ae0b28b3c628b8\": rpc error: code = NotFound desc = could not find container \"2672b1a332e61e66faaab98c610fa5633e852b5f276b939060ae0b28b3c628b8\": container with ID starting with 2672b1a332e61e66faaab98c610fa5633e852b5f276b939060ae0b28b3c628b8 not found: ID does not exist"
Jul 28 17:55:08.009704 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:08.009641    2566 scope.go:117] "RemoveContainer" containerID="ce9ae889ff149ab56f76e278b8474b0f509be9dea2a0152feb0fd9d50ef9fa46"
Jul 28 17:55:08.009985 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:55:08.009968    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce9ae889ff149ab56f76e278b8474b0f509be9dea2a0152feb0fd9d50ef9fa46\": container with ID starting with ce9ae889ff149ab56f76e278b8474b0f509be9dea2a0152feb0fd9d50ef9fa46 not found: ID does not exist" containerID="ce9ae889ff149ab56f76e278b8474b0f509be9dea2a0152feb0fd9d50ef9fa46"
Jul 28 17:55:08.010030 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:08.009997    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce9ae889ff149ab56f76e278b8474b0f509be9dea2a0152feb0fd9d50ef9fa46"} err="failed to get container status \"ce9ae889ff149ab56f76e278b8474b0f509be9dea2a0152feb0fd9d50ef9fa46\": rpc error: code = NotFound desc = could not find container \"ce9ae889ff149ab56f76e278b8474b0f509be9dea2a0152feb0fd9d50ef9fa46\": container with ID starting with ce9ae889ff149ab56f76e278b8474b0f509be9dea2a0152feb0fd9d50ef9fa46 not found: ID does not exist"
Jul 28 17:55:08.028777 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:08.028745    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft"]
Jul 28 17:55:08.035666 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:08.035622    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft"]
Jul 28 17:55:08.086705 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:08.086665    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1027091-27cb-46ce-ae3d-ecdcfe582ca3-proxy-tls\") pod \"success-200-isvc-cb1ee-predictor-7b757b875f-46rjc\" (UID: \"b1027091-27cb-46ce-ae3d-ecdcfe582ca3\") " pod="kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc"
Jul 28 17:55:08.089104 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:08.089075    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1027091-27cb-46ce-ae3d-ecdcfe582ca3-proxy-tls\") pod \"success-200-isvc-cb1ee-predictor-7b757b875f-46rjc\" (UID: \"b1027091-27cb-46ce-ae3d-ecdcfe582ca3\") " pod="kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc"
Jul 28 17:55:08.097722 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:08.097692    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b7f1dcc-7061-4a60-ae5e-be54d0b5171e" path="/var/lib/kubelet/pods/0b7f1dcc-7061-4a60-ae5e-be54d0b5171e/volumes"
Jul 28 17:55:08.277617 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:08.277582    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc"
Jul 28 17:55:08.427054 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:08.426977    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc"]
Jul 28 17:55:08.429110 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:55:08.429083    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1027091_27cb_46ce_ae3d_ecdcfe582ca3.slice/crio-eb6da2439e95eadf2a1cde84566a85086c4accfb5b49520597ae37fd959ba496 WatchSource:0}: Error finding container eb6da2439e95eadf2a1cde84566a85086c4accfb5b49520597ae37fd959ba496: Status 404 returned error can't find the container with id eb6da2439e95eadf2a1cde84566a85086c4accfb5b49520597ae37fd959ba496
Jul 28 17:55:08.999775 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:08.999736    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc" event={"ID":"b1027091-27cb-46ce-ae3d-ecdcfe582ca3","Type":"ContainerStarted","Data":"c1e0c807a406c27b60a164773d557f160fe502ce290e9bedf37923f26b7ab262"}
Jul 28 17:55:09.000217 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:08.999836    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc" event={"ID":"b1027091-27cb-46ce-ae3d-ecdcfe582ca3","Type":"ContainerStarted","Data":"165dae0eb2700f6233120e6db91e9fd0c2e39c405aa467d3bf5683ad1f5126d2"}
Jul 28 17:55:09.000217 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:08.999867    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc" event={"ID":"b1027091-27cb-46ce-ae3d-ecdcfe582ca3","Type":"ContainerStarted","Data":"eb6da2439e95eadf2a1cde84566a85086c4accfb5b49520597ae37fd959ba496"}
Jul 28 17:55:09.000217 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:08.999901    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc"
Jul 28 17:55:09.000217 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:08.999929    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc"
Jul 28 17:55:09.001525 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:09.001496    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc" podUID="b1027091-27cb-46ce-ae3d-ecdcfe582ca3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 17:55:09.025427 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:09.025366    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc" podStartSLOduration=2.025348593 podStartE2EDuration="2.025348593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:55:09.024461866 +0000 UTC m=+1959.334627371" watchObservedRunningTime="2026-07-28 17:55:09.025348593 +0000 UTC m=+1959.335514098"
Jul 28 17:55:09.180066 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:09.180024    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-787a1-77c7b956dc-cpngn" podUID="811f7ddb-91b4-4102-9ee5-ecb2a04ab80f" containerName="sequence-graph-787a1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:55:09.954795 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:09.954745    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk" podUID="39137e71-bcda-4ba3-bdca-60c6e8a91b1a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 28 17:55:09.960627 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:09.960587    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk" podUID="39137e71-bcda-4ba3-bdca-60c6e8a91b1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 17:55:10.005310 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:10.005262    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc" podUID="b1027091-27cb-46ce-ae3d-ecdcfe582ca3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 17:55:10.941483 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:10.941441    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-f7704-758c568bf-l924l" podUID="fcb68714-b150-4245-8506-40ff87015604" containerName="splitter-graph-f7704" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:55:11.314306 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:11.314271    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk"
Jul 28 17:55:11.415170 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:11.415132    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5msq\" (UniqueName: \"kubernetes.io/projected/39137e71-bcda-4ba3-bdca-60c6e8a91b1a-kube-api-access-r5msq\") pod \"39137e71-bcda-4ba3-bdca-60c6e8a91b1a\" (UID: \"39137e71-bcda-4ba3-bdca-60c6e8a91b1a\") "
Jul 28 17:55:11.415170 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:11.415175    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39137e71-bcda-4ba3-bdca-60c6e8a91b1a-kube-rbac-proxy-sar-config\") pod \"39137e71-bcda-4ba3-bdca-60c6e8a91b1a\" (UID: \"39137e71-bcda-4ba3-bdca-60c6e8a91b1a\") "
Jul 28 17:55:11.415427 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:11.415235    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39137e71-bcda-4ba3-bdca-60c6e8a91b1a-proxy-tls\") pod \"39137e71-bcda-4ba3-bdca-60c6e8a91b1a\" (UID: \"39137e71-bcda-4ba3-bdca-60c6e8a91b1a\") "
Jul 28 17:55:11.415588 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:11.415567    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39137e71-bcda-4ba3-bdca-60c6e8a91b1a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "39137e71-bcda-4ba3-bdca-60c6e8a91b1a" (UID: "39137e71-bcda-4ba3-bdca-60c6e8a91b1a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:55:11.417364 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:11.417310    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39137e71-bcda-4ba3-bdca-60c6e8a91b1a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "39137e71-bcda-4ba3-bdca-60c6e8a91b1a" (UID: "39137e71-bcda-4ba3-bdca-60c6e8a91b1a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:55:11.417492 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:11.417377    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39137e71-bcda-4ba3-bdca-60c6e8a91b1a-kube-api-access-r5msq" (OuterVolumeSpecName: "kube-api-access-r5msq") pod "39137e71-bcda-4ba3-bdca-60c6e8a91b1a" (UID: "39137e71-bcda-4ba3-bdca-60c6e8a91b1a"). InnerVolumeSpecName "kube-api-access-r5msq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:55:11.516715 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:11.516620    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5msq\" (UniqueName: \"kubernetes.io/projected/39137e71-bcda-4ba3-bdca-60c6e8a91b1a-kube-api-access-r5msq\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:55:11.516715 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:11.516653    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39137e71-bcda-4ba3-bdca-60c6e8a91b1a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:55:11.516715 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:11.516663    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39137e71-bcda-4ba3-bdca-60c6e8a91b1a-proxy-tls\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:55:11.987908 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:11.987879    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl"
Jul 28 17:55:11.988506 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:11.988477    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl" podUID="8155544f-2dae-4598-962f-97a2c83dcdf2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:55:12.015241 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:12.015208    2566 generic.go:356] "Generic (PLEG): container finished" podID="39137e71-bcda-4ba3-bdca-60c6e8a91b1a" containerID="8a35008083e2cdad97726f99755029a0164a064e904ad1a460592bb46d0256a5" exitCode=0
Jul 28 17:55:12.015439 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:12.015263    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk"
Jul 28 17:55:12.015439 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:12.015280    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk" event={"ID":"39137e71-bcda-4ba3-bdca-60c6e8a91b1a","Type":"ContainerDied","Data":"8a35008083e2cdad97726f99755029a0164a064e904ad1a460592bb46d0256a5"}
Jul 28 17:55:12.015439 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:12.015371    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk" event={"ID":"39137e71-bcda-4ba3-bdca-60c6e8a91b1a","Type":"ContainerDied","Data":"ba8f726308968499fddef055639570086b5649daeea6a9c33ffe27834e7d7d6d"}
Jul 28 17:55:12.015439 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:12.015412    2566 scope.go:117] "RemoveContainer" containerID="9c7c719a3358b4b7a05dc1947c3d8c8e0975486281b618ed0894bbace4825db9"
Jul 28 17:55:12.024221 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:12.024194    2566 scope.go:117] "RemoveContainer" containerID="8a35008083e2cdad97726f99755029a0164a064e904ad1a460592bb46d0256a5"
Jul 28 17:55:12.031814 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:12.031793    2566 scope.go:117] "RemoveContainer" containerID="9c7c719a3358b4b7a05dc1947c3d8c8e0975486281b618ed0894bbace4825db9"
Jul 28 17:55:12.032174 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:55:12.032150    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c7c719a3358b4b7a05dc1947c3d8c8e0975486281b618ed0894bbace4825db9\": container with ID starting with 9c7c719a3358b4b7a05dc1947c3d8c8e0975486281b618ed0894bbace4825db9 not found: ID does not exist" containerID="9c7c719a3358b4b7a05dc1947c3d8c8e0975486281b618ed0894bbace4825db9"
Jul 28 17:55:12.032215 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:12.032186    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c7c719a3358b4b7a05dc1947c3d8c8e0975486281b618ed0894bbace4825db9"} err="failed to get container status \"9c7c719a3358b4b7a05dc1947c3d8c8e0975486281b618ed0894bbace4825db9\": rpc error: code = NotFound desc = could not find container \"9c7c719a3358b4b7a05dc1947c3d8c8e0975486281b618ed0894bbace4825db9\": container with ID starting with 9c7c719a3358b4b7a05dc1947c3d8c8e0975486281b618ed0894bbace4825db9 not found: ID does not exist"
Jul 28 17:55:12.032215 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:12.032206    2566 scope.go:117] "RemoveContainer" containerID="8a35008083e2cdad97726f99755029a0164a064e904ad1a460592bb46d0256a5"
Jul 28 17:55:12.032535 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:55:12.032518    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a35008083e2cdad97726f99755029a0164a064e904ad1a460592bb46d0256a5\": container with ID starting with 8a35008083e2cdad97726f99755029a0164a064e904ad1a460592bb46d0256a5 not found: ID does not exist" containerID="8a35008083e2cdad97726f99755029a0164a064e904ad1a460592bb46d0256a5"
Jul 28 17:55:12.032586 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:12.032543    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a35008083e2cdad97726f99755029a0164a064e904ad1a460592bb46d0256a5"} err="failed to get container status \"8a35008083e2cdad97726f99755029a0164a064e904ad1a460592bb46d0256a5\": rpc error: code = NotFound desc = could not find container \"8a35008083e2cdad97726f99755029a0164a064e904ad1a460592bb46d0256a5\": container with ID starting with 8a35008083e2cdad97726f99755029a0164a064e904ad1a460592bb46d0256a5 not found: ID does not exist"
Jul 28 17:55:12.038594 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:12.038565    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk"]
Jul 28 17:55:12.040589 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:12.040564    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk"]
Jul 28 17:55:12.098062 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:12.098031    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39137e71-bcda-4ba3-bdca-60c6e8a91b1a" path="/var/lib/kubelet/pods/39137e71-bcda-4ba3-bdca-60c6e8a91b1a/volumes"
Jul 28 17:55:14.179682 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:14.179633    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-787a1-77c7b956dc-cpngn" podUID="811f7ddb-91b4-4102-9ee5-ecb2a04ab80f" containerName="sequence-graph-787a1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:55:15.010034 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:15.010006    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc"
Jul 28 17:55:15.010572 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:15.010539    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc" podUID="b1027091-27cb-46ce-ae3d-ecdcfe582ca3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 17:55:15.941174 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:15.941132    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-f7704-758c568bf-l924l" podUID="fcb68714-b150-4245-8506-40ff87015604" containerName="splitter-graph-f7704" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:55:15.941581 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:15.941237    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-f7704-758c568bf-l924l"
Jul 28 17:55:19.180431 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:19.180393    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-787a1-77c7b956dc-cpngn" podUID="811f7ddb-91b4-4102-9ee5-ecb2a04ab80f" containerName="sequence-graph-787a1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:55:19.180918 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:19.180498    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-787a1-77c7b956dc-cpngn"
Jul 28 17:55:20.940755 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:20.940715    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-f7704-758c568bf-l924l" podUID="fcb68714-b150-4245-8506-40ff87015604" containerName="splitter-graph-f7704" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:55:21.988761 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:21.988721    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl" podUID="8155544f-2dae-4598-962f-97a2c83dcdf2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:55:24.180300 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:24.180255    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-787a1-77c7b956dc-cpngn" podUID="811f7ddb-91b4-4102-9ee5-ecb2a04ab80f" containerName="sequence-graph-787a1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:55:25.011203 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:25.011157    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc" podUID="b1027091-27cb-46ce-ae3d-ecdcfe582ca3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 17:55:25.940848 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:25.940807    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-f7704-758c568bf-l924l" podUID="fcb68714-b150-4245-8506-40ff87015604" containerName="splitter-graph-f7704" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:55:29.179795 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:29.179756    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-787a1-77c7b956dc-cpngn" podUID="811f7ddb-91b4-4102-9ee5-ecb2a04ab80f" containerName="sequence-graph-787a1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:55:30.940566 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:30.940523    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-f7704-758c568bf-l924l" podUID="fcb68714-b150-4245-8506-40ff87015604" containerName="splitter-graph-f7704" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:55:31.988945 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:31.988900    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl" podUID="8155544f-2dae-4598-962f-97a2c83dcdf2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:55:34.013640 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:34.013604    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-f7704-758c568bf-l924l"
Jul 28 17:55:34.080517 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:34.080420    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcb68714-b150-4245-8506-40ff87015604-proxy-tls\") pod \"fcb68714-b150-4245-8506-40ff87015604\" (UID: \"fcb68714-b150-4245-8506-40ff87015604\") "
Jul 28 17:55:34.080517 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:34.080503    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcb68714-b150-4245-8506-40ff87015604-openshift-service-ca-bundle\") pod \"fcb68714-b150-4245-8506-40ff87015604\" (UID: \"fcb68714-b150-4245-8506-40ff87015604\") "
Jul 28 17:55:34.080875 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:34.080845    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcb68714-b150-4245-8506-40ff87015604-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "fcb68714-b150-4245-8506-40ff87015604" (UID: "fcb68714-b150-4245-8506-40ff87015604"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:55:34.082524 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:34.082499    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcb68714-b150-4245-8506-40ff87015604-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fcb68714-b150-4245-8506-40ff87015604" (UID: "fcb68714-b150-4245-8506-40ff87015604"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:55:34.106260 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:34.106215    2566 generic.go:356] "Generic (PLEG): container finished" podID="fcb68714-b150-4245-8506-40ff87015604" containerID="173ab4903d91c05dd98020f6a9719c8438c649548bd7ec15fa6377bc94c517eb" exitCode=0
Jul 28 17:55:34.106260 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:34.106253    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-f7704-758c568bf-l924l"
Jul 28 17:55:34.106550 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:34.106279    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-f7704-758c568bf-l924l" event={"ID":"fcb68714-b150-4245-8506-40ff87015604","Type":"ContainerDied","Data":"173ab4903d91c05dd98020f6a9719c8438c649548bd7ec15fa6377bc94c517eb"}
Jul 28 17:55:34.106550 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:34.106374    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-f7704-758c568bf-l924l" event={"ID":"fcb68714-b150-4245-8506-40ff87015604","Type":"ContainerDied","Data":"5183b2c47a1db7176ef59b673b05ad0aa7adb6f8ddab3fc8dec587d4d40dd8f2"}
Jul 28 17:55:34.106550 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:34.106429    2566 scope.go:117] "RemoveContainer" containerID="173ab4903d91c05dd98020f6a9719c8438c649548bd7ec15fa6377bc94c517eb"
Jul 28 17:55:34.109068 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:55:34.108908    2566 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb68714_b150_4245_8506_40ff87015604.slice/crio-5183b2c47a1db7176ef59b673b05ad0aa7adb6f8ddab3fc8dec587d4d40dd8f2\": RecentStats: unable to find data in memory cache]"
Jul 28 17:55:34.116111 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:34.116089    2566 scope.go:117] "RemoveContainer" containerID="173ab4903d91c05dd98020f6a9719c8438c649548bd7ec15fa6377bc94c517eb"
Jul 28 17:55:34.116427 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:55:34.116405    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"173ab4903d91c05dd98020f6a9719c8438c649548bd7ec15fa6377bc94c517eb\": container with ID starting with 173ab4903d91c05dd98020f6a9719c8438c649548bd7ec15fa6377bc94c517eb not found: ID does not exist" containerID="173ab4903d91c05dd98020f6a9719c8438c649548bd7ec15fa6377bc94c517eb"
Jul 28 17:55:34.116482 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:34.116436    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"173ab4903d91c05dd98020f6a9719c8438c649548bd7ec15fa6377bc94c517eb"} err="failed to get container status \"173ab4903d91c05dd98020f6a9719c8438c649548bd7ec15fa6377bc94c517eb\": rpc error: code = NotFound desc = could not find container \"173ab4903d91c05dd98020f6a9719c8438c649548bd7ec15fa6377bc94c517eb\": container with ID starting with 173ab4903d91c05dd98020f6a9719c8438c649548bd7ec15fa6377bc94c517eb not found: ID does not exist"
Jul 28 17:55:34.129911 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:34.129878    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-f7704-758c568bf-l924l"]
Jul 28 17:55:34.133203 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:34.133177    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-f7704-758c568bf-l924l"]
Jul 28 17:55:34.180027 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:34.179986    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-787a1-77c7b956dc-cpngn" podUID="811f7ddb-91b4-4102-9ee5-ecb2a04ab80f" containerName="sequence-graph-787a1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:55:34.181292 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:34.181264    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcb68714-b150-4245-8506-40ff87015604-openshift-service-ca-bundle\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:55:34.181292 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:34.181290    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcb68714-b150-4245-8506-40ff87015604-proxy-tls\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:55:35.010560 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:35.010510    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc" podUID="b1027091-27cb-46ce-ae3d-ecdcfe582ca3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 17:55:36.101039 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:36.100994    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcb68714-b150-4245-8506-40ff87015604" path="/var/lib/kubelet/pods/fcb68714-b150-4245-8506-40ff87015604/volumes"
Jul 28 17:55:37.333967 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:37.333939    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-787a1-77c7b956dc-cpngn"
Jul 28 17:55:37.404056 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:37.404012    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/811f7ddb-91b4-4102-9ee5-ecb2a04ab80f-openshift-service-ca-bundle\") pod \"811f7ddb-91b4-4102-9ee5-ecb2a04ab80f\" (UID: \"811f7ddb-91b4-4102-9ee5-ecb2a04ab80f\") "
Jul 28 17:55:37.404255 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:37.404091    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/811f7ddb-91b4-4102-9ee5-ecb2a04ab80f-proxy-tls\") pod \"811f7ddb-91b4-4102-9ee5-ecb2a04ab80f\" (UID: \"811f7ddb-91b4-4102-9ee5-ecb2a04ab80f\") "
Jul 28 17:55:37.404392 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:37.404367    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/811f7ddb-91b4-4102-9ee5-ecb2a04ab80f-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "811f7ddb-91b4-4102-9ee5-ecb2a04ab80f" (UID: "811f7ddb-91b4-4102-9ee5-ecb2a04ab80f"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:55:37.406246 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:37.406210    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/811f7ddb-91b4-4102-9ee5-ecb2a04ab80f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "811f7ddb-91b4-4102-9ee5-ecb2a04ab80f" (UID: "811f7ddb-91b4-4102-9ee5-ecb2a04ab80f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:55:37.505144 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:37.505051    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/811f7ddb-91b4-4102-9ee5-ecb2a04ab80f-openshift-service-ca-bundle\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:55:37.505144 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:37.505086    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/811f7ddb-91b4-4102-9ee5-ecb2a04ab80f-proxy-tls\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 17:55:38.129290 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:38.129254    2566 generic.go:356] "Generic (PLEG): container finished" podID="811f7ddb-91b4-4102-9ee5-ecb2a04ab80f" containerID="899d9270fd1b1a2a969fb589eae5ceffef74ec534219e99cd5b7de466030bde6" exitCode=0
Jul 28 17:55:38.129504 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:38.129303    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-787a1-77c7b956dc-cpngn"
Jul 28 17:55:38.129504 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:38.129349    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-787a1-77c7b956dc-cpngn" event={"ID":"811f7ddb-91b4-4102-9ee5-ecb2a04ab80f","Type":"ContainerDied","Data":"899d9270fd1b1a2a969fb589eae5ceffef74ec534219e99cd5b7de466030bde6"}
Jul 28 17:55:38.129504 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:38.129403    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-787a1-77c7b956dc-cpngn" event={"ID":"811f7ddb-91b4-4102-9ee5-ecb2a04ab80f","Type":"ContainerDied","Data":"a9684a7765d7709e7e17292cf02c3ac63a974565f66004d78a9688ef2ea6ecf6"}
Jul 28 17:55:38.129504 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:38.129425    2566 scope.go:117] "RemoveContainer" containerID="899d9270fd1b1a2a969fb589eae5ceffef74ec534219e99cd5b7de466030bde6"
Jul 28 17:55:38.137507 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:38.137476    2566 scope.go:117] "RemoveContainer" containerID="899d9270fd1b1a2a969fb589eae5ceffef74ec534219e99cd5b7de466030bde6"
Jul 28 17:55:38.137825 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:55:38.137800    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"899d9270fd1b1a2a969fb589eae5ceffef74ec534219e99cd5b7de466030bde6\": container with ID starting with 899d9270fd1b1a2a969fb589eae5ceffef74ec534219e99cd5b7de466030bde6 not found: ID does not exist" containerID="899d9270fd1b1a2a969fb589eae5ceffef74ec534219e99cd5b7de466030bde6"
Jul 28 17:55:38.137907 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:38.137838    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"899d9270fd1b1a2a969fb589eae5ceffef74ec534219e99cd5b7de466030bde6"} err="failed to get container status \"899d9270fd1b1a2a969fb589eae5ceffef74ec534219e99cd5b7de466030bde6\": rpc error: code = NotFound desc = could not find container \"899d9270fd1b1a2a969fb589eae5ceffef74ec534219e99cd5b7de466030bde6\": container with ID starting with 899d9270fd1b1a2a969fb589eae5ceffef74ec534219e99cd5b7de466030bde6 not found: ID does not exist"
Jul 28 17:55:38.154654 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:38.154619    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-787a1-77c7b956dc-cpngn"]
Jul 28 17:55:38.159830 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:38.159803    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-787a1-77c7b956dc-cpngn"]
Jul 28 17:55:40.098237 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:40.098205    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="811f7ddb-91b4-4102-9ee5-ecb2a04ab80f" path="/var/lib/kubelet/pods/811f7ddb-91b4-4102-9ee5-ecb2a04ab80f/volumes"
Jul 28 17:55:41.988979 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:41.988936    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl" podUID="8155544f-2dae-4598-962f-97a2c83dcdf2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:55:45.010988 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:45.010942    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc" podUID="b1027091-27cb-46ce-ae3d-ecdcfe582ca3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 17:55:51.989405 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:51.989366    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl"
Jul 28 17:55:55.010975 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:55:55.010935    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc" podUID="b1027091-27cb-46ce-ae3d-ecdcfe582ca3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 17:56:04.065727 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:04.065689    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-7d999-849c5688bd-rrgg7"]
Jul 28 17:56:04.066165 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:04.065989    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="811f7ddb-91b4-4102-9ee5-ecb2a04ab80f" containerName="sequence-graph-787a1"
Jul 28 17:56:04.066165 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:04.066002    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="811f7ddb-91b4-4102-9ee5-ecb2a04ab80f" containerName="sequence-graph-787a1"
Jul 28 17:56:04.066165 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:04.066019    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39137e71-bcda-4ba3-bdca-60c6e8a91b1a" containerName="kserve-container"
Jul 28 17:56:04.066165 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:04.066024    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="39137e71-bcda-4ba3-bdca-60c6e8a91b1a" containerName="kserve-container"
Jul 28 17:56:04.066165 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:04.066032    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b7f1dcc-7061-4a60-ae5e-be54d0b5171e" containerName="kube-rbac-proxy"
Jul 28 17:56:04.066165 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:04.066038    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b7f1dcc-7061-4a60-ae5e-be54d0b5171e" containerName="kube-rbac-proxy"
Jul 28 17:56:04.066165 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:04.066046    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b7f1dcc-7061-4a60-ae5e-be54d0b5171e" containerName="kserve-container"
Jul 28 17:56:04.066165 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:04.066051    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b7f1dcc-7061-4a60-ae5e-be54d0b5171e" containerName="kserve-container"
Jul 28 17:56:04.066165 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:04.066058    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39137e71-bcda-4ba3-bdca-60c6e8a91b1a" containerName="kube-rbac-proxy"
Jul 28 17:56:04.066165 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:04.066063    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="39137e71-bcda-4ba3-bdca-60c6e8a91b1a" containerName="kube-rbac-proxy"
Jul 28 17:56:04.066165 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:04.066069    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcb68714-b150-4245-8506-40ff87015604" containerName="splitter-graph-f7704"
Jul 28 17:56:04.066165 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:04.066074    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcb68714-b150-4245-8506-40ff87015604" containerName="splitter-graph-f7704"
Jul 28 17:56:04.066165 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:04.066115    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="39137e71-bcda-4ba3-bdca-60c6e8a91b1a" containerName="kserve-container"
Jul 28 17:56:04.066165 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:04.066122    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcb68714-b150-4245-8506-40ff87015604" containerName="splitter-graph-f7704"
Jul 28 17:56:04.066165 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:04.066128    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="811f7ddb-91b4-4102-9ee5-ecb2a04ab80f" containerName="sequence-graph-787a1"
Jul 28 17:56:04.066165 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:04.066134    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b7f1dcc-7061-4a60-ae5e-be54d0b5171e" containerName="kserve-container"
Jul 28 17:56:04.066165 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:04.066139    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b7f1dcc-7061-4a60-ae5e-be54d0b5171e" containerName="kube-rbac-proxy"
Jul 28 17:56:04.066165 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:04.066145    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="39137e71-bcda-4ba3-bdca-60c6e8a91b1a" containerName="kube-rbac-proxy"
Jul 28 17:56:04.071257 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:04.071229    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-7d999-849c5688bd-rrgg7"
Jul 28 17:56:04.073786 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:04.073754    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-7d999-kube-rbac-proxy-sar-config\""
Jul 28 17:56:04.073786 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:04.073767    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-7d999-serving-cert\""
Jul 28 17:56:04.081523 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:04.081495    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-7d999-849c5688bd-rrgg7"]
Jul 28 17:56:04.195895 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:04.195846    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5cba631-1a98-41a8-b3c8-c9675bb989df-proxy-tls\") pod \"splitter-graph-7d999-849c5688bd-rrgg7\" (UID: \"a5cba631-1a98-41a8-b3c8-c9675bb989df\") " pod="kserve-ci-e2e-test/splitter-graph-7d999-849c5688bd-rrgg7"
Jul 28 17:56:04.196093 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:04.195924    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5cba631-1a98-41a8-b3c8-c9675bb989df-openshift-service-ca-bundle\") pod \"splitter-graph-7d999-849c5688bd-rrgg7\" (UID: \"a5cba631-1a98-41a8-b3c8-c9675bb989df\") " pod="kserve-ci-e2e-test/splitter-graph-7d999-849c5688bd-rrgg7"
Jul 28 17:56:04.296608 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:04.296566    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5cba631-1a98-41a8-b3c8-c9675bb989df-openshift-service-ca-bundle\") pod \"splitter-graph-7d999-849c5688bd-rrgg7\" (UID: \"a5cba631-1a98-41a8-b3c8-c9675bb989df\") " pod="kserve-ci-e2e-test/splitter-graph-7d999-849c5688bd-rrgg7"
Jul 28 17:56:04.296827 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:04.296639    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5cba631-1a98-41a8-b3c8-c9675bb989df-proxy-tls\") pod \"splitter-graph-7d999-849c5688bd-rrgg7\" (UID: \"a5cba631-1a98-41a8-b3c8-c9675bb989df\") " pod="kserve-ci-e2e-test/splitter-graph-7d999-849c5688bd-rrgg7"
Jul 28 17:56:04.297296 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:04.297270    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5cba631-1a98-41a8-b3c8-c9675bb989df-openshift-service-ca-bundle\") pod \"splitter-graph-7d999-849c5688bd-rrgg7\" (UID: \"a5cba631-1a98-41a8-b3c8-c9675bb989df\") " pod="kserve-ci-e2e-test/splitter-graph-7d999-849c5688bd-rrgg7"
Jul 28 17:56:04.299198 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:04.299178    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5cba631-1a98-41a8-b3c8-c9675bb989df-proxy-tls\") pod \"splitter-graph-7d999-849c5688bd-rrgg7\" (UID: \"a5cba631-1a98-41a8-b3c8-c9675bb989df\") " pod="kserve-ci-e2e-test/splitter-graph-7d999-849c5688bd-rrgg7"
Jul 28 17:56:04.381997 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:04.381895    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-7d999-849c5688bd-rrgg7"
Jul 28 17:56:04.522142 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:04.522109    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-7d999-849c5688bd-rrgg7"]
Jul 28 17:56:04.525447 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:56:04.525418    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5cba631_1a98_41a8_b3c8_c9675bb989df.slice/crio-0ed44048285c524df04f56ff4df9074eed33fa7ac1644625b11fc63c1cc02c5e WatchSource:0}: Error finding container 0ed44048285c524df04f56ff4df9074eed33fa7ac1644625b11fc63c1cc02c5e: Status 404 returned error can't find the container with id 0ed44048285c524df04f56ff4df9074eed33fa7ac1644625b11fc63c1cc02c5e
Jul 28 17:56:05.011113 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:05.011083    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc"
Jul 28 17:56:05.228741 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:05.228700    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-7d999-849c5688bd-rrgg7" event={"ID":"a5cba631-1a98-41a8-b3c8-c9675bb989df","Type":"ContainerStarted","Data":"6ace5a6503ca18c07511c8452cbd25eadf31bfa804210ef4bc1333711d3ad705"}
Jul 28 17:56:05.229125 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:05.228758    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-7d999-849c5688bd-rrgg7" event={"ID":"a5cba631-1a98-41a8-b3c8-c9675bb989df","Type":"ContainerStarted","Data":"0ed44048285c524df04f56ff4df9074eed33fa7ac1644625b11fc63c1cc02c5e"}
Jul 28 17:56:05.229125 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:05.228787    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-7d999-849c5688bd-rrgg7"
Jul 28 17:56:05.258200 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:05.258148    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-7d999-849c5688bd-rrgg7" podStartSLOduration=1.258130348 podStartE2EDuration="1.258130348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:56:05.257747153 +0000 UTC m=+2015.567912659" watchObservedRunningTime="2026-07-28 17:56:05.258130348 +0000 UTC m=+2015.568295913"
Jul 28 17:56:11.238744 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:11.238712    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-7d999-849c5688bd-rrgg7"
Jul 28 17:56:17.542021 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:17.541984    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-cb1ee-7ddbb59689-cbvqc"]
Jul 28 17:56:17.545294 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:17.545276    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-cb1ee-7ddbb59689-cbvqc"
Jul 28 17:56:17.551808 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:17.551632    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-cb1ee-serving-cert\""
Jul 28 17:56:17.553551 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:17.553529    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-cb1ee-kube-rbac-proxy-sar-config\""
Jul 28 17:56:17.578185 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:17.578150    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-cb1ee-7ddbb59689-cbvqc"]
Jul 28 17:56:17.600015 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:17.599970    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0792112d-477d-469c-a9bf-10b4d52ec4ab-proxy-tls\") pod \"switch-graph-cb1ee-7ddbb59689-cbvqc\" (UID: \"0792112d-477d-469c-a9bf-10b4d52ec4ab\") " pod="kserve-ci-e2e-test/switch-graph-cb1ee-7ddbb59689-cbvqc"
Jul 28 17:56:17.600193 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:17.600029    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0792112d-477d-469c-a9bf-10b4d52ec4ab-openshift-service-ca-bundle\") pod \"switch-graph-cb1ee-7ddbb59689-cbvqc\" (UID: \"0792112d-477d-469c-a9bf-10b4d52ec4ab\") " pod="kserve-ci-e2e-test/switch-graph-cb1ee-7ddbb59689-cbvqc"
Jul 28 17:56:17.701042 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:17.701005    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0792112d-477d-469c-a9bf-10b4d52ec4ab-openshift-service-ca-bundle\") pod \"switch-graph-cb1ee-7ddbb59689-cbvqc\" (UID: \"0792112d-477d-469c-a9bf-10b4d52ec4ab\") " pod="kserve-ci-e2e-test/switch-graph-cb1ee-7ddbb59689-cbvqc"
Jul 28 17:56:17.701234 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:17.701081    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0792112d-477d-469c-a9bf-10b4d52ec4ab-proxy-tls\") pod \"switch-graph-cb1ee-7ddbb59689-cbvqc\" (UID: \"0792112d-477d-469c-a9bf-10b4d52ec4ab\") " pod="kserve-ci-e2e-test/switch-graph-cb1ee-7ddbb59689-cbvqc"
Jul 28 17:56:17.701234 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:56:17.701177    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-cb1ee-serving-cert: secret "switch-graph-cb1ee-serving-cert" not found
Jul 28 17:56:17.701234 ip-10-0-131-233 kubenswrapper[2566]: E0728 17:56:17.701234    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0792112d-477d-469c-a9bf-10b4d52ec4ab-proxy-tls podName:0792112d-477d-469c-a9bf-10b4d52ec4ab nodeName:}" failed. No retries permitted until 2026-07-28 17:56:18.201216361 +0000 UTC m=+2028.511381844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0792112d-477d-469c-a9bf-10b4d52ec4ab-proxy-tls") pod "switch-graph-cb1ee-7ddbb59689-cbvqc" (UID: "0792112d-477d-469c-a9bf-10b4d52ec4ab") : secret "switch-graph-cb1ee-serving-cert" not found
Jul 28 17:56:17.701724 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:17.701703    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0792112d-477d-469c-a9bf-10b4d52ec4ab-openshift-service-ca-bundle\") pod \"switch-graph-cb1ee-7ddbb59689-cbvqc\" (UID: \"0792112d-477d-469c-a9bf-10b4d52ec4ab\") " pod="kserve-ci-e2e-test/switch-graph-cb1ee-7ddbb59689-cbvqc"
Jul 28 17:56:18.206260 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:18.206202    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0792112d-477d-469c-a9bf-10b4d52ec4ab-proxy-tls\") pod \"switch-graph-cb1ee-7ddbb59689-cbvqc\" (UID: \"0792112d-477d-469c-a9bf-10b4d52ec4ab\") " pod="kserve-ci-e2e-test/switch-graph-cb1ee-7ddbb59689-cbvqc"
Jul 28 17:56:18.208846 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:18.208820    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0792112d-477d-469c-a9bf-10b4d52ec4ab-proxy-tls\") pod \"switch-graph-cb1ee-7ddbb59689-cbvqc\" (UID: \"0792112d-477d-469c-a9bf-10b4d52ec4ab\") " pod="kserve-ci-e2e-test/switch-graph-cb1ee-7ddbb59689-cbvqc"
Jul 28 17:56:18.455666 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:18.455615    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-cb1ee-7ddbb59689-cbvqc"
Jul 28 17:56:18.599138 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:18.599102    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-cb1ee-7ddbb59689-cbvqc"]
Jul 28 17:56:18.602232 ip-10-0-131-233 kubenswrapper[2566]: W0728 17:56:18.602198    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0792112d_477d_469c_a9bf_10b4d52ec4ab.slice/crio-ac0e6c7f48c0e2ab56cbd905660892de453202f06a2f0ac1ab969514e0f1991e WatchSource:0}: Error finding container ac0e6c7f48c0e2ab56cbd905660892de453202f06a2f0ac1ab969514e0f1991e: Status 404 returned error can't find the container with id ac0e6c7f48c0e2ab56cbd905660892de453202f06a2f0ac1ab969514e0f1991e
Jul 28 17:56:19.279995 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:19.279962    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-cb1ee-7ddbb59689-cbvqc" event={"ID":"0792112d-477d-469c-a9bf-10b4d52ec4ab","Type":"ContainerStarted","Data":"1743b7a858852e09a90aa04a31c4712ca36501c2b807a7d8d3f5e966c84b0adb"}
Jul 28 17:56:19.280178 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:19.280018    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-cb1ee-7ddbb59689-cbvqc" event={"ID":"0792112d-477d-469c-a9bf-10b4d52ec4ab","Type":"ContainerStarted","Data":"ac0e6c7f48c0e2ab56cbd905660892de453202f06a2f0ac1ab969514e0f1991e"}
Jul 28 17:56:19.280178 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:19.280052    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-cb1ee-7ddbb59689-cbvqc"
Jul 28 17:56:19.301875 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:19.301822    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-cb1ee-7ddbb59689-cbvqc" podStartSLOduration=2.301805203 podStartE2EDuration="2.301805203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:56:19.300734127 +0000 UTC m=+2029.610899653" watchObservedRunningTime="2026-07-28 17:56:19.301805203 +0000 UTC m=+2029.611970710"
Jul 28 17:56:25.289520 ip-10-0-131-233 kubenswrapper[2566]: I0728 17:56:25.289491    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-cb1ee-7ddbb59689-cbvqc"
Jul 28 18:04:18.679614 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:18.679580    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-7d999-849c5688bd-rrgg7"]
Jul 28 18:04:18.680095 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:18.679813    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-7d999-849c5688bd-rrgg7" podUID="a5cba631-1a98-41a8-b3c8-c9675bb989df" containerName="splitter-graph-7d999" containerID="cri-o://6ace5a6503ca18c07511c8452cbd25eadf31bfa804210ef4bc1333711d3ad705" gracePeriod=30
Jul 28 18:04:18.868884 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:18.868852    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl"]
Jul 28 18:04:18.869157 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:18.869118    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl" podUID="8155544f-2dae-4598-962f-97a2c83dcdf2" containerName="kserve-container" containerID="cri-o://23a7d05bcd67244d714ecda3294ee22e83421bd4d3e648b9907d14fc1077a527" gracePeriod=30
Jul 28 18:04:18.869277 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:18.869242    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl" podUID="8155544f-2dae-4598-962f-97a2c83dcdf2" containerName="kube-rbac-proxy" containerID="cri-o://3073bf7f879c3556e5256111838ef6487dba80b955037c6f8c83f97cc5fe7bbf" gracePeriod=30
Jul 28 18:04:19.978132 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:19.978101    2566 generic.go:356] "Generic (PLEG): container finished" podID="8155544f-2dae-4598-962f-97a2c83dcdf2" containerID="3073bf7f879c3556e5256111838ef6487dba80b955037c6f8c83f97cc5fe7bbf" exitCode=2
Jul 28 18:04:19.978529 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:19.978175    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl" event={"ID":"8155544f-2dae-4598-962f-97a2c83dcdf2","Type":"ContainerDied","Data":"3073bf7f879c3556e5256111838ef6487dba80b955037c6f8c83f97cc5fe7bbf"}
Jul 28 18:04:21.236911 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:21.236874    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-7d999-849c5688bd-rrgg7" podUID="a5cba631-1a98-41a8-b3c8-c9675bb989df" containerName="splitter-graph-7d999" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 18:04:21.513314 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:21.513291    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl"
Jul 28 18:04:21.603015 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:21.602986    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4rwg\" (UniqueName: \"kubernetes.io/projected/8155544f-2dae-4598-962f-97a2c83dcdf2-kube-api-access-n4rwg\") pod \"8155544f-2dae-4598-962f-97a2c83dcdf2\" (UID: \"8155544f-2dae-4598-962f-97a2c83dcdf2\") "
Jul 28 18:04:21.603138 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:21.603035    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8155544f-2dae-4598-962f-97a2c83dcdf2-proxy-tls\") pod \"8155544f-2dae-4598-962f-97a2c83dcdf2\" (UID: \"8155544f-2dae-4598-962f-97a2c83dcdf2\") "
Jul 28 18:04:21.603138 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:21.603065    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8155544f-2dae-4598-962f-97a2c83dcdf2-kube-rbac-proxy-sar-config\") pod \"8155544f-2dae-4598-962f-97a2c83dcdf2\" (UID: \"8155544f-2dae-4598-962f-97a2c83dcdf2\") "
Jul 28 18:04:21.603446 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:21.603426    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8155544f-2dae-4598-962f-97a2c83dcdf2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8155544f-2dae-4598-962f-97a2c83dcdf2" (UID: "8155544f-2dae-4598-962f-97a2c83dcdf2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:04:21.605006 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:21.604982    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8155544f-2dae-4598-962f-97a2c83dcdf2-kube-api-access-n4rwg" (OuterVolumeSpecName: "kube-api-access-n4rwg") pod "8155544f-2dae-4598-962f-97a2c83dcdf2" (UID: "8155544f-2dae-4598-962f-97a2c83dcdf2"). InnerVolumeSpecName "kube-api-access-n4rwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:04:21.605006 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:21.604990    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8155544f-2dae-4598-962f-97a2c83dcdf2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8155544f-2dae-4598-962f-97a2c83dcdf2" (UID: "8155544f-2dae-4598-962f-97a2c83dcdf2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:04:21.704492 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:21.704461    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8155544f-2dae-4598-962f-97a2c83dcdf2-proxy-tls\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 18:04:21.704492 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:21.704487    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8155544f-2dae-4598-962f-97a2c83dcdf2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 18:04:21.704492 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:21.704496    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4rwg\" (UniqueName: \"kubernetes.io/projected/8155544f-2dae-4598-962f-97a2c83dcdf2-kube-api-access-n4rwg\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 18:04:21.986212 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:21.986184    2566 generic.go:356] "Generic (PLEG): container finished" podID="8155544f-2dae-4598-962f-97a2c83dcdf2" containerID="23a7d05bcd67244d714ecda3294ee22e83421bd4d3e648b9907d14fc1077a527" exitCode=0
Jul 28 18:04:21.986367 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:21.986233    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl"
Jul 28 18:04:21.986367 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:21.986245    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl" event={"ID":"8155544f-2dae-4598-962f-97a2c83dcdf2","Type":"ContainerDied","Data":"23a7d05bcd67244d714ecda3294ee22e83421bd4d3e648b9907d14fc1077a527"}
Jul 28 18:04:21.986367 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:21.986300    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl" event={"ID":"8155544f-2dae-4598-962f-97a2c83dcdf2","Type":"ContainerDied","Data":"b0bdf3f14da07882d747b5d7f2a0062044b83ed591a45faf32a8025e5c14bfb4"}
Jul 28 18:04:21.986367 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:21.986337    2566 scope.go:117] "RemoveContainer" containerID="3073bf7f879c3556e5256111838ef6487dba80b955037c6f8c83f97cc5fe7bbf"
Jul 28 18:04:21.994304 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:21.994279    2566 scope.go:117] "RemoveContainer" containerID="23a7d05bcd67244d714ecda3294ee22e83421bd4d3e648b9907d14fc1077a527"
Jul 28 18:04:22.001150 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:22.001135    2566 scope.go:117] "RemoveContainer" containerID="3073bf7f879c3556e5256111838ef6487dba80b955037c6f8c83f97cc5fe7bbf"
Jul 28 18:04:22.001407 ip-10-0-131-233 kubenswrapper[2566]: E0728 18:04:22.001387    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3073bf7f879c3556e5256111838ef6487dba80b955037c6f8c83f97cc5fe7bbf\": container with ID starting with 3073bf7f879c3556e5256111838ef6487dba80b955037c6f8c83f97cc5fe7bbf not found: ID does not exist" containerID="3073bf7f879c3556e5256111838ef6487dba80b955037c6f8c83f97cc5fe7bbf"
Jul 28 18:04:22.001476 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:22.001416    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3073bf7f879c3556e5256111838ef6487dba80b955037c6f8c83f97cc5fe7bbf"} err="failed to get container status \"3073bf7f879c3556e5256111838ef6487dba80b955037c6f8c83f97cc5fe7bbf\": rpc error: code = NotFound desc = could not find container \"3073bf7f879c3556e5256111838ef6487dba80b955037c6f8c83f97cc5fe7bbf\": container with ID starting with 3073bf7f879c3556e5256111838ef6487dba80b955037c6f8c83f97cc5fe7bbf not found: ID does not exist"
Jul 28 18:04:22.001476 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:22.001434    2566 scope.go:117] "RemoveContainer" containerID="23a7d05bcd67244d714ecda3294ee22e83421bd4d3e648b9907d14fc1077a527"
Jul 28 18:04:22.001678 ip-10-0-131-233 kubenswrapper[2566]: E0728 18:04:22.001659    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23a7d05bcd67244d714ecda3294ee22e83421bd4d3e648b9907d14fc1077a527\": container with ID starting with 23a7d05bcd67244d714ecda3294ee22e83421bd4d3e648b9907d14fc1077a527 not found: ID does not exist" containerID="23a7d05bcd67244d714ecda3294ee22e83421bd4d3e648b9907d14fc1077a527"
Jul 28 18:04:22.001724 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:22.001684    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23a7d05bcd67244d714ecda3294ee22e83421bd4d3e648b9907d14fc1077a527"} err="failed to get container status \"23a7d05bcd67244d714ecda3294ee22e83421bd4d3e648b9907d14fc1077a527\": rpc error: code = NotFound desc = could not find container \"23a7d05bcd67244d714ecda3294ee22e83421bd4d3e648b9907d14fc1077a527\": container with ID starting with 23a7d05bcd67244d714ecda3294ee22e83421bd4d3e648b9907d14fc1077a527 not found: ID does not exist"
Jul 28 18:04:22.010343 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:22.010305    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl"]
Jul 28 18:04:22.023067 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:22.023046    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl"]
Jul 28 18:04:22.097846 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:22.097799    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8155544f-2dae-4598-962f-97a2c83dcdf2" path="/var/lib/kubelet/pods/8155544f-2dae-4598-962f-97a2c83dcdf2/volumes"
Jul 28 18:04:26.236922 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:26.236886    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-7d999-849c5688bd-rrgg7" podUID="a5cba631-1a98-41a8-b3c8-c9675bb989df" containerName="splitter-graph-7d999" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 18:04:31.236530 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:31.236485    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-7d999-849c5688bd-rrgg7" podUID="a5cba631-1a98-41a8-b3c8-c9675bb989df" containerName="splitter-graph-7d999" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 18:04:31.236977 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:31.236628    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-7d999-849c5688bd-rrgg7"
Jul 28 18:04:36.236869 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:36.236831    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-7d999-849c5688bd-rrgg7" podUID="a5cba631-1a98-41a8-b3c8-c9675bb989df" containerName="splitter-graph-7d999" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 18:04:41.236603 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:41.236562    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-7d999-849c5688bd-rrgg7" podUID="a5cba631-1a98-41a8-b3c8-c9675bb989df" containerName="splitter-graph-7d999" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 18:04:46.237001 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:46.236966    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-7d999-849c5688bd-rrgg7" podUID="a5cba631-1a98-41a8-b3c8-c9675bb989df" containerName="splitter-graph-7d999" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 18:04:48.828916 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:48.828887    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-7d999-849c5688bd-rrgg7"
Jul 28 18:04:48.901972 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:48.901941    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5cba631-1a98-41a8-b3c8-c9675bb989df-openshift-service-ca-bundle\") pod \"a5cba631-1a98-41a8-b3c8-c9675bb989df\" (UID: \"a5cba631-1a98-41a8-b3c8-c9675bb989df\") "
Jul 28 18:04:48.902105 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:48.902015    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5cba631-1a98-41a8-b3c8-c9675bb989df-proxy-tls\") pod \"a5cba631-1a98-41a8-b3c8-c9675bb989df\" (UID: \"a5cba631-1a98-41a8-b3c8-c9675bb989df\") "
Jul 28 18:04:48.902261 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:48.902238    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5cba631-1a98-41a8-b3c8-c9675bb989df-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "a5cba631-1a98-41a8-b3c8-c9675bb989df" (UID: "a5cba631-1a98-41a8-b3c8-c9675bb989df"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:04:48.903975 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:48.903954    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5cba631-1a98-41a8-b3c8-c9675bb989df-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a5cba631-1a98-41a8-b3c8-c9675bb989df" (UID: "a5cba631-1a98-41a8-b3c8-c9675bb989df"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:04:49.002884 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:49.002801    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5cba631-1a98-41a8-b3c8-c9675bb989df-proxy-tls\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 18:04:49.002884 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:49.002838    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5cba631-1a98-41a8-b3c8-c9675bb989df-openshift-service-ca-bundle\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 18:04:49.079666 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:49.079636    2566 generic.go:356] "Generic (PLEG): container finished" podID="a5cba631-1a98-41a8-b3c8-c9675bb989df" containerID="6ace5a6503ca18c07511c8452cbd25eadf31bfa804210ef4bc1333711d3ad705" exitCode=0
Jul 28 18:04:49.079816 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:49.079685    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-7d999-849c5688bd-rrgg7"
Jul 28 18:04:49.079816 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:49.079719    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-7d999-849c5688bd-rrgg7" event={"ID":"a5cba631-1a98-41a8-b3c8-c9675bb989df","Type":"ContainerDied","Data":"6ace5a6503ca18c07511c8452cbd25eadf31bfa804210ef4bc1333711d3ad705"}
Jul 28 18:04:49.079816 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:49.079770    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-7d999-849c5688bd-rrgg7" event={"ID":"a5cba631-1a98-41a8-b3c8-c9675bb989df","Type":"ContainerDied","Data":"0ed44048285c524df04f56ff4df9074eed33fa7ac1644625b11fc63c1cc02c5e"}
Jul 28 18:04:49.080536 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:49.079791    2566 scope.go:117] "RemoveContainer" containerID="6ace5a6503ca18c07511c8452cbd25eadf31bfa804210ef4bc1333711d3ad705"
Jul 28 18:04:49.088389 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:49.088375    2566 scope.go:117] "RemoveContainer" containerID="6ace5a6503ca18c07511c8452cbd25eadf31bfa804210ef4bc1333711d3ad705"
Jul 28 18:04:49.088638 ip-10-0-131-233 kubenswrapper[2566]: E0728 18:04:49.088618    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ace5a6503ca18c07511c8452cbd25eadf31bfa804210ef4bc1333711d3ad705\": container with ID starting with 6ace5a6503ca18c07511c8452cbd25eadf31bfa804210ef4bc1333711d3ad705 not found: ID does not exist" containerID="6ace5a6503ca18c07511c8452cbd25eadf31bfa804210ef4bc1333711d3ad705"
Jul 28 18:04:49.088678 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:49.088647    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ace5a6503ca18c07511c8452cbd25eadf31bfa804210ef4bc1333711d3ad705"} err="failed to get container status \"6ace5a6503ca18c07511c8452cbd25eadf31bfa804210ef4bc1333711d3ad705\": rpc error: code = NotFound desc = could not find container \"6ace5a6503ca18c07511c8452cbd25eadf31bfa804210ef4bc1333711d3ad705\": container with ID starting with 6ace5a6503ca18c07511c8452cbd25eadf31bfa804210ef4bc1333711d3ad705 not found: ID does not exist"
Jul 28 18:04:49.102888 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:49.102865    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-7d999-849c5688bd-rrgg7"]
Jul 28 18:04:49.106428 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:49.106407    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-7d999-849c5688bd-rrgg7"]
Jul 28 18:04:50.097107 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:04:50.097075    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5cba631-1a98-41a8-b3c8-c9675bb989df" path="/var/lib/kubelet/pods/a5cba631-1a98-41a8-b3c8-c9675bb989df/volumes"
Jul 28 18:12:36.623233 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:36.623202    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-cb1ee-7ddbb59689-cbvqc"]
Jul 28 18:12:36.625570 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:36.623458    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-cb1ee-7ddbb59689-cbvqc" podUID="0792112d-477d-469c-a9bf-10b4d52ec4ab" containerName="switch-graph-cb1ee" containerID="cri-o://1743b7a858852e09a90aa04a31c4712ca36501c2b807a7d8d3f5e966c84b0adb" gracePeriod=30
Jul 28 18:12:36.778989 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:36.778956    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc"]
Jul 28 18:12:36.779197 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:36.779173    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc" podUID="b1027091-27cb-46ce-ae3d-ecdcfe582ca3" containerName="kserve-container" containerID="cri-o://165dae0eb2700f6233120e6db91e9fd0c2e39c405aa467d3bf5683ad1f5126d2" gracePeriod=30
Jul 28 18:12:36.779281 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:36.779228    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc" podUID="b1027091-27cb-46ce-ae3d-ecdcfe582ca3" containerName="kube-rbac-proxy" containerID="cri-o://c1e0c807a406c27b60a164773d557f160fe502ce290e9bedf37923f26b7ab262" gracePeriod=30
Jul 28 18:12:37.623003 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:37.622972    2566 generic.go:356] "Generic (PLEG): container finished" podID="b1027091-27cb-46ce-ae3d-ecdcfe582ca3" containerID="c1e0c807a406c27b60a164773d557f160fe502ce290e9bedf37923f26b7ab262" exitCode=2
Jul 28 18:12:37.623179 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:37.623022    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc" event={"ID":"b1027091-27cb-46ce-ae3d-ecdcfe582ca3","Type":"ContainerDied","Data":"c1e0c807a406c27b60a164773d557f160fe502ce290e9bedf37923f26b7ab262"}
Jul 28 18:12:39.421064 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:39.421042    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc"
Jul 28 18:12:39.554003 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:39.553929    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1027091-27cb-46ce-ae3d-ecdcfe582ca3-proxy-tls\") pod \"b1027091-27cb-46ce-ae3d-ecdcfe582ca3\" (UID: \"b1027091-27cb-46ce-ae3d-ecdcfe582ca3\") "
Jul 28 18:12:39.554003 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:39.553973    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1027091-27cb-46ce-ae3d-ecdcfe582ca3-kube-rbac-proxy-sar-config\") pod \"b1027091-27cb-46ce-ae3d-ecdcfe582ca3\" (UID: \"b1027091-27cb-46ce-ae3d-ecdcfe582ca3\") "
Jul 28 18:12:39.554158 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:39.554025    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5dqp\" (UniqueName: \"kubernetes.io/projected/b1027091-27cb-46ce-ae3d-ecdcfe582ca3-kube-api-access-k5dqp\") pod \"b1027091-27cb-46ce-ae3d-ecdcfe582ca3\" (UID: \"b1027091-27cb-46ce-ae3d-ecdcfe582ca3\") "
Jul 28 18:12:39.554338 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:39.554294    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1027091-27cb-46ce-ae3d-ecdcfe582ca3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b1027091-27cb-46ce-ae3d-ecdcfe582ca3" (UID: "b1027091-27cb-46ce-ae3d-ecdcfe582ca3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:12:39.555892 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:39.555866    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1027091-27cb-46ce-ae3d-ecdcfe582ca3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b1027091-27cb-46ce-ae3d-ecdcfe582ca3" (UID: "b1027091-27cb-46ce-ae3d-ecdcfe582ca3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:12:39.556003 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:39.555979    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1027091-27cb-46ce-ae3d-ecdcfe582ca3-kube-api-access-k5dqp" (OuterVolumeSpecName: "kube-api-access-k5dqp") pod "b1027091-27cb-46ce-ae3d-ecdcfe582ca3" (UID: "b1027091-27cb-46ce-ae3d-ecdcfe582ca3"). InnerVolumeSpecName "kube-api-access-k5dqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:12:39.631805 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:39.631777    2566 generic.go:356] "Generic (PLEG): container finished" podID="b1027091-27cb-46ce-ae3d-ecdcfe582ca3" containerID="165dae0eb2700f6233120e6db91e9fd0c2e39c405aa467d3bf5683ad1f5126d2" exitCode=0
Jul 28 18:12:39.631919 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:39.631826    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc"
Jul 28 18:12:39.631919 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:39.631841    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc" event={"ID":"b1027091-27cb-46ce-ae3d-ecdcfe582ca3","Type":"ContainerDied","Data":"165dae0eb2700f6233120e6db91e9fd0c2e39c405aa467d3bf5683ad1f5126d2"}
Jul 28 18:12:39.631919 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:39.631888    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc" event={"ID":"b1027091-27cb-46ce-ae3d-ecdcfe582ca3","Type":"ContainerDied","Data":"eb6da2439e95eadf2a1cde84566a85086c4accfb5b49520597ae37fd959ba496"}
Jul 28 18:12:39.631919 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:39.631913    2566 scope.go:117] "RemoveContainer" containerID="c1e0c807a406c27b60a164773d557f160fe502ce290e9bedf37923f26b7ab262"
Jul 28 18:12:39.640336 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:39.640305    2566 scope.go:117] "RemoveContainer" containerID="165dae0eb2700f6233120e6db91e9fd0c2e39c405aa467d3bf5683ad1f5126d2"
Jul 28 18:12:39.646948 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:39.646932    2566 scope.go:117] "RemoveContainer" containerID="c1e0c807a406c27b60a164773d557f160fe502ce290e9bedf37923f26b7ab262"
Jul 28 18:12:39.647198 ip-10-0-131-233 kubenswrapper[2566]: E0728 18:12:39.647181    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1e0c807a406c27b60a164773d557f160fe502ce290e9bedf37923f26b7ab262\": container with ID starting with c1e0c807a406c27b60a164773d557f160fe502ce290e9bedf37923f26b7ab262 not found: ID does not exist" containerID="c1e0c807a406c27b60a164773d557f160fe502ce290e9bedf37923f26b7ab262"
Jul 28 18:12:39.647243 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:39.647204    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1e0c807a406c27b60a164773d557f160fe502ce290e9bedf37923f26b7ab262"} err="failed to get container status \"c1e0c807a406c27b60a164773d557f160fe502ce290e9bedf37923f26b7ab262\": rpc error: code = NotFound desc = could not find container \"c1e0c807a406c27b60a164773d557f160fe502ce290e9bedf37923f26b7ab262\": container with ID starting with c1e0c807a406c27b60a164773d557f160fe502ce290e9bedf37923f26b7ab262 not found: ID does not exist"
Jul 28 18:12:39.647243 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:39.647220    2566 scope.go:117] "RemoveContainer" containerID="165dae0eb2700f6233120e6db91e9fd0c2e39c405aa467d3bf5683ad1f5126d2"
Jul 28 18:12:39.647433 ip-10-0-131-233 kubenswrapper[2566]: E0728 18:12:39.647416    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"165dae0eb2700f6233120e6db91e9fd0c2e39c405aa467d3bf5683ad1f5126d2\": container with ID starting with 165dae0eb2700f6233120e6db91e9fd0c2e39c405aa467d3bf5683ad1f5126d2 not found: ID does not exist" containerID="165dae0eb2700f6233120e6db91e9fd0c2e39c405aa467d3bf5683ad1f5126d2"
Jul 28 18:12:39.647478 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:39.647440    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"165dae0eb2700f6233120e6db91e9fd0c2e39c405aa467d3bf5683ad1f5126d2"} err="failed to get container status \"165dae0eb2700f6233120e6db91e9fd0c2e39c405aa467d3bf5683ad1f5126d2\": rpc error: code = NotFound desc = could not find container \"165dae0eb2700f6233120e6db91e9fd0c2e39c405aa467d3bf5683ad1f5126d2\": container with ID starting with 165dae0eb2700f6233120e6db91e9fd0c2e39c405aa467d3bf5683ad1f5126d2 not found: ID does not exist"
Jul 28 18:12:39.654728 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:39.654706    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc"]
Jul 28 18:12:39.654823 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:39.654728    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1027091-27cb-46ce-ae3d-ecdcfe582ca3-proxy-tls\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 18:12:39.654823 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:39.654744    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1027091-27cb-46ce-ae3d-ecdcfe582ca3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 18:12:39.654823 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:39.654753    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5dqp\" (UniqueName: \"kubernetes.io/projected/b1027091-27cb-46ce-ae3d-ecdcfe582ca3-kube-api-access-k5dqp\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 18:12:39.659500 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:39.659481    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc"]
Jul 28 18:12:40.097147 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:40.097112    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1027091-27cb-46ce-ae3d-ecdcfe582ca3" path="/var/lib/kubelet/pods/b1027091-27cb-46ce-ae3d-ecdcfe582ca3/volumes"
Jul 28 18:12:40.287995 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:40.287963    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-cb1ee-7ddbb59689-cbvqc" podUID="0792112d-477d-469c-a9bf-10b4d52ec4ab" containerName="switch-graph-cb1ee" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 18:12:45.288469 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:45.288428    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-cb1ee-7ddbb59689-cbvqc" podUID="0792112d-477d-469c-a9bf-10b4d52ec4ab" containerName="switch-graph-cb1ee" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 18:12:50.287540 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:50.287499    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-cb1ee-7ddbb59689-cbvqc" podUID="0792112d-477d-469c-a9bf-10b4d52ec4ab" containerName="switch-graph-cb1ee" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 18:12:50.287903 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:50.287596    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-cb1ee-7ddbb59689-cbvqc"
Jul 28 18:12:54.624281 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:54.624253    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-cb1ee-7ddbb59689-cbvqc_0792112d-477d-469c-a9bf-10b4d52ec4ab/switch-graph-cb1ee/0.log"
Jul 28 18:12:55.287993 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:55.287957    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-cb1ee-7ddbb59689-cbvqc" podUID="0792112d-477d-469c-a9bf-10b4d52ec4ab" containerName="switch-graph-cb1ee" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 18:12:55.587205 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:55.587131    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-cb1ee-7ddbb59689-cbvqc_0792112d-477d-469c-a9bf-10b4d52ec4ab/switch-graph-cb1ee/0.log"
Jul 28 18:12:56.501016 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:56.500990    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-cb1ee-7ddbb59689-cbvqc_0792112d-477d-469c-a9bf-10b4d52ec4ab/switch-graph-cb1ee/0.log"
Jul 28 18:12:57.414665 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:57.414636    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-cb1ee-7ddbb59689-cbvqc_0792112d-477d-469c-a9bf-10b4d52ec4ab/switch-graph-cb1ee/0.log"
Jul 28 18:12:58.300634 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:58.300607    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-cb1ee-7ddbb59689-cbvqc_0792112d-477d-469c-a9bf-10b4d52ec4ab/switch-graph-cb1ee/0.log"
Jul 28 18:12:59.231092 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:12:59.231058    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-cb1ee-7ddbb59689-cbvqc_0792112d-477d-469c-a9bf-10b4d52ec4ab/switch-graph-cb1ee/0.log"
Jul 28 18:13:00.115530 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:00.115503    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-cb1ee-7ddbb59689-cbvqc_0792112d-477d-469c-a9bf-10b4d52ec4ab/switch-graph-cb1ee/0.log"
Jul 28 18:13:00.288218 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:00.288181    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-cb1ee-7ddbb59689-cbvqc" podUID="0792112d-477d-469c-a9bf-10b4d52ec4ab" containerName="switch-graph-cb1ee" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 18:13:01.014706 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:01.014676    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-cb1ee-7ddbb59689-cbvqc_0792112d-477d-469c-a9bf-10b4d52ec4ab/switch-graph-cb1ee/0.log"
Jul 28 18:13:01.923405 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:01.923379    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-cb1ee-7ddbb59689-cbvqc_0792112d-477d-469c-a9bf-10b4d52ec4ab/switch-graph-cb1ee/0.log"
Jul 28 18:13:02.908555 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:02.908524    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-cb1ee-7ddbb59689-cbvqc_0792112d-477d-469c-a9bf-10b4d52ec4ab/switch-graph-cb1ee/0.log"
Jul 28 18:13:03.792623 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:03.792600    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-cb1ee-7ddbb59689-cbvqc_0792112d-477d-469c-a9bf-10b4d52ec4ab/switch-graph-cb1ee/0.log"
Jul 28 18:13:04.670086 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:04.670050    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-cb1ee-7ddbb59689-cbvqc_0792112d-477d-469c-a9bf-10b4d52ec4ab/switch-graph-cb1ee/0.log"
Jul 28 18:13:05.287853 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:05.287817    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-cb1ee-7ddbb59689-cbvqc" podUID="0792112d-477d-469c-a9bf-10b4d52ec4ab" containerName="switch-graph-cb1ee" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 18:13:05.605861 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:05.605772    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-cb1ee-7ddbb59689-cbvqc_0792112d-477d-469c-a9bf-10b4d52ec4ab/switch-graph-cb1ee/0.log"
Jul 28 18:13:06.554578 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:06.554552    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-cb1ee-7ddbb59689-cbvqc_0792112d-477d-469c-a9bf-10b4d52ec4ab/switch-graph-cb1ee/0.log"
Jul 28 18:13:06.724340 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:06.724296    2566 generic.go:356] "Generic (PLEG): container finished" podID="0792112d-477d-469c-a9bf-10b4d52ec4ab" containerID="1743b7a858852e09a90aa04a31c4712ca36501c2b807a7d8d3f5e966c84b0adb" exitCode=0
Jul 28 18:13:06.724487 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:06.724357    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-cb1ee-7ddbb59689-cbvqc" event={"ID":"0792112d-477d-469c-a9bf-10b4d52ec4ab","Type":"ContainerDied","Data":"1743b7a858852e09a90aa04a31c4712ca36501c2b807a7d8d3f5e966c84b0adb"}
Jul 28 18:13:06.777026 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:06.777001    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-cb1ee-7ddbb59689-cbvqc"
Jul 28 18:13:06.844880 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:06.844810    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0792112d-477d-469c-a9bf-10b4d52ec4ab-openshift-service-ca-bundle\") pod \"0792112d-477d-469c-a9bf-10b4d52ec4ab\" (UID: \"0792112d-477d-469c-a9bf-10b4d52ec4ab\") "
Jul 28 18:13:06.845025 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:06.844886    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0792112d-477d-469c-a9bf-10b4d52ec4ab-proxy-tls\") pod \"0792112d-477d-469c-a9bf-10b4d52ec4ab\" (UID: \"0792112d-477d-469c-a9bf-10b4d52ec4ab\") "
Jul 28 18:13:06.845195 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:06.845174    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0792112d-477d-469c-a9bf-10b4d52ec4ab-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "0792112d-477d-469c-a9bf-10b4d52ec4ab" (UID: "0792112d-477d-469c-a9bf-10b4d52ec4ab"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:13:06.846812 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:06.846791    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0792112d-477d-469c-a9bf-10b4d52ec4ab-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0792112d-477d-469c-a9bf-10b4d52ec4ab" (UID: "0792112d-477d-469c-a9bf-10b4d52ec4ab"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:13:06.945926 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:06.945893    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0792112d-477d-469c-a9bf-10b4d52ec4ab-proxy-tls\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 18:13:06.945926 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:06.945921    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0792112d-477d-469c-a9bf-10b4d52ec4ab-openshift-service-ca-bundle\") on node \"ip-10-0-131-233.ec2.internal\" DevicePath \"\""
Jul 28 18:13:07.728351 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:07.728309    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-cb1ee-7ddbb59689-cbvqc"
Jul 28 18:13:07.728351 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:07.728306    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-cb1ee-7ddbb59689-cbvqc" event={"ID":"0792112d-477d-469c-a9bf-10b4d52ec4ab","Type":"ContainerDied","Data":"ac0e6c7f48c0e2ab56cbd905660892de453202f06a2f0ac1ab969514e0f1991e"}
Jul 28 18:13:07.728985 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:07.728389    2566 scope.go:117] "RemoveContainer" containerID="1743b7a858852e09a90aa04a31c4712ca36501c2b807a7d8d3f5e966c84b0adb"
Jul 28 18:13:07.761313 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:07.761286    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-cb1ee-7ddbb59689-cbvqc"]
Jul 28 18:13:07.770823 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:07.770803    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-cb1ee-7ddbb59689-cbvqc"]
Jul 28 18:13:08.097929 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:08.097852    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0792112d-477d-469c-a9bf-10b4d52ec4ab" path="/var/lib/kubelet/pods/0792112d-477d-469c-a9bf-10b4d52ec4ab/volumes"
Jul 28 18:13:10.162725 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:10.162693    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vd4v4/must-gather-l68qt"]
Jul 28 18:13:10.163075 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:10.162905    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8155544f-2dae-4598-962f-97a2c83dcdf2" containerName="kube-rbac-proxy"
Jul 28 18:13:10.163075 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:10.162915    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8155544f-2dae-4598-962f-97a2c83dcdf2" containerName="kube-rbac-proxy"
Jul 28 18:13:10.163075 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:10.162927    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5cba631-1a98-41a8-b3c8-c9675bb989df" containerName="splitter-graph-7d999"
Jul 28 18:13:10.163075 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:10.162933    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5cba631-1a98-41a8-b3c8-c9675bb989df" containerName="splitter-graph-7d999"
Jul 28 18:13:10.163075 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:10.162941    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1027091-27cb-46ce-ae3d-ecdcfe582ca3" containerName="kube-rbac-proxy"
Jul 28 18:13:10.163075 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:10.162948    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1027091-27cb-46ce-ae3d-ecdcfe582ca3" containerName="kube-rbac-proxy"
Jul 28 18:13:10.163075 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:10.162961    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8155544f-2dae-4598-962f-97a2c83dcdf2" containerName="kserve-container"
Jul 28 18:13:10.163075 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:10.162966    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8155544f-2dae-4598-962f-97a2c83dcdf2" containerName="kserve-container"
Jul 28 18:13:10.163075 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:10.162972    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1027091-27cb-46ce-ae3d-ecdcfe582ca3" containerName="kserve-container"
Jul 28 18:13:10.163075 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:10.162977    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1027091-27cb-46ce-ae3d-ecdcfe582ca3" containerName="kserve-container"
Jul 28 18:13:10.163075 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:10.162983    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0792112d-477d-469c-a9bf-10b4d52ec4ab" containerName="switch-graph-cb1ee"
Jul 28 18:13:10.163075 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:10.162988    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0792112d-477d-469c-a9bf-10b4d52ec4ab" containerName="switch-graph-cb1ee"
Jul 28 18:13:10.163075 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:10.163021    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1027091-27cb-46ce-ae3d-ecdcfe582ca3" containerName="kube-rbac-proxy"
Jul 28 18:13:10.163075 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:10.163028    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5cba631-1a98-41a8-b3c8-c9675bb989df" containerName="splitter-graph-7d999"
Jul 28 18:13:10.163075 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:10.163035    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8155544f-2dae-4598-962f-97a2c83dcdf2" containerName="kserve-container"
Jul 28 18:13:10.163075 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:10.163040    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1027091-27cb-46ce-ae3d-ecdcfe582ca3" containerName="kserve-container"
Jul 28 18:13:10.163075 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:10.163045    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0792112d-477d-469c-a9bf-10b4d52ec4ab" containerName="switch-graph-cb1ee"
Jul 28 18:13:10.163075 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:10.163051    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8155544f-2dae-4598-962f-97a2c83dcdf2" containerName="kube-rbac-proxy"
Jul 28 18:13:10.165870 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:10.165851    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vd4v4/must-gather-l68qt"
Jul 28 18:13:10.170083 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:10.170064    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vd4v4\"/\"default-dockercfg-6vjwt\""
Jul 28 18:13:10.170191 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:10.170104    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vd4v4\"/\"kube-root-ca.crt\""
Jul 28 18:13:10.171410 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:10.171389    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vd4v4\"/\"openshift-service-ca.crt\""
Jul 28 18:13:10.179305 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:10.179282    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vd4v4/must-gather-l68qt"]
Jul 28 18:13:10.272071 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:10.272044    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6f49\" (UniqueName: \"kubernetes.io/projected/182e2255-3dab-4354-9234-b4bef505e3b9-kube-api-access-z6f49\") pod \"must-gather-l68qt\" (UID: \"182e2255-3dab-4354-9234-b4bef505e3b9\") " pod="openshift-must-gather-vd4v4/must-gather-l68qt"
Jul 28 18:13:10.272208 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:10.272087    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/182e2255-3dab-4354-9234-b4bef505e3b9-must-gather-output\") pod \"must-gather-l68qt\" (UID: \"182e2255-3dab-4354-9234-b4bef505e3b9\") " pod="openshift-must-gather-vd4v4/must-gather-l68qt"
Jul 28 18:13:10.372872 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:10.372847    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/182e2255-3dab-4354-9234-b4bef505e3b9-must-gather-output\") pod \"must-gather-l68qt\" (UID: \"182e2255-3dab-4354-9234-b4bef505e3b9\") " pod="openshift-must-gather-vd4v4/must-gather-l68qt"
Jul 28 18:13:10.372998 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:10.372890    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6f49\" (UniqueName: \"kubernetes.io/projected/182e2255-3dab-4354-9234-b4bef505e3b9-kube-api-access-z6f49\") pod \"must-gather-l68qt\" (UID: \"182e2255-3dab-4354-9234-b4bef505e3b9\") " pod="openshift-must-gather-vd4v4/must-gather-l68qt"
Jul 28 18:13:10.373199 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:10.373180    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/182e2255-3dab-4354-9234-b4bef505e3b9-must-gather-output\") pod \"must-gather-l68qt\" (UID: \"182e2255-3dab-4354-9234-b4bef505e3b9\") " pod="openshift-must-gather-vd4v4/must-gather-l68qt"
Jul 28 18:13:10.381875 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:10.381850    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6f49\" (UniqueName: \"kubernetes.io/projected/182e2255-3dab-4354-9234-b4bef505e3b9-kube-api-access-z6f49\") pod \"must-gather-l68qt\" (UID: \"182e2255-3dab-4354-9234-b4bef505e3b9\") " pod="openshift-must-gather-vd4v4/must-gather-l68qt"
Jul 28 18:13:10.473751 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:10.473725    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vd4v4/must-gather-l68qt"
Jul 28 18:13:10.599383 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:10.599360    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vd4v4/must-gather-l68qt"]
Jul 28 18:13:10.601718 ip-10-0-131-233 kubenswrapper[2566]: W0728 18:13:10.601692    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod182e2255_3dab_4354_9234_b4bef505e3b9.slice/crio-01ed5fc99ea6a488719a6b5104a6cc0250701eef9e08193dca0ed82f096da475 WatchSource:0}: Error finding container 01ed5fc99ea6a488719a6b5104a6cc0250701eef9e08193dca0ed82f096da475: Status 404 returned error can't find the container with id 01ed5fc99ea6a488719a6b5104a6cc0250701eef9e08193dca0ed82f096da475
Jul 28 18:13:10.603527 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:10.603510    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 18:13:10.739755 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:10.739668    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vd4v4/must-gather-l68qt" event={"ID":"182e2255-3dab-4354-9234-b4bef505e3b9","Type":"ContainerStarted","Data":"01ed5fc99ea6a488719a6b5104a6cc0250701eef9e08193dca0ed82f096da475"}
Jul 28 18:13:11.745820 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:11.745783    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vd4v4/must-gather-l68qt" event={"ID":"182e2255-3dab-4354-9234-b4bef505e3b9","Type":"ContainerStarted","Data":"46b0318f85fd5137f71d6af6b96bed99b42221a8a19f89cb29739c56a9833a48"}
Jul 28 18:13:11.745820 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:11.745825    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vd4v4/must-gather-l68qt" event={"ID":"182e2255-3dab-4354-9234-b4bef505e3b9","Type":"ContainerStarted","Data":"46636363a930ca251878c7b30c783f8cf20956fa4ba12e9beb0584871e3945c4"}
Jul 28 18:13:11.765607 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:11.765561    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vd4v4/must-gather-l68qt" podStartSLOduration=1.027028878 podStartE2EDuration="1.765545872s" totalImagesPullingTime="738.516994ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:13:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:13:11.76406423 +0000 UTC m=+3042.074229735" watchObservedRunningTime="2026-07-28 18:13:11.765545872 +0000 UTC m=+3042.075711377"
Jul 28 18:13:12.909200 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:12.909168    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dpv42_3b881ad9-ece5-40ab-b106-32e6d520c37e/global-pull-secret-syncer/0.log"
Jul 28 18:13:13.211778 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:13.211741    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xsptw_51a23dc6-82ad-4ab6-a528-82ad3146e7c6/konnectivity-agent/0.log"
Jul 28 18:13:13.243761 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:13.243731    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-233.ec2.internal_a2d32026394497da64d771aa0e11975c/haproxy/0.log"
Jul 28 18:13:17.553511 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:17.553479    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-588f9d6645-ftg7q_f9b80343-ffd9-457d-a1d7-e40f1f467a3a/metrics-server/0.log"
Jul 28 18:13:17.642478 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:17.642425    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7fzw8_fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc/node-exporter/0.log"
Jul 28 18:13:17.674750 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:17.674726    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7fzw8_fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc/kube-rbac-proxy/0.log"
Jul 28 18:13:17.708013 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:17.707979    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7fzw8_fdd5bfe5-8b39-4f6d-a5ec-85009c390ebc/init-textfile/0.log"
Jul 28 18:13:20.859261 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:20.859227    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vd4v4/perf-node-gather-daemonset-mghtr"]
Jul 28 18:13:20.864612 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:20.864588    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vd4v4/perf-node-gather-daemonset-mghtr"
Jul 28 18:13:20.875650 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:20.875623    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vd4v4/perf-node-gather-daemonset-mghtr"]
Jul 28 18:13:20.960986 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:20.960954    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/32d2634c-6752-48bc-b2e0-24c640c805d3-podres\") pod \"perf-node-gather-daemonset-mghtr\" (UID: \"32d2634c-6752-48bc-b2e0-24c640c805d3\") " pod="openshift-must-gather-vd4v4/perf-node-gather-daemonset-mghtr"
Jul 28 18:13:20.961174 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:20.960995    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32d2634c-6752-48bc-b2e0-24c640c805d3-sys\") pod \"perf-node-gather-daemonset-mghtr\" (UID: \"32d2634c-6752-48bc-b2e0-24c640c805d3\") " pod="openshift-must-gather-vd4v4/perf-node-gather-daemonset-mghtr"
Jul 28 18:13:20.961174 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:20.961016    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32d2634c-6752-48bc-b2e0-24c640c805d3-lib-modules\") pod \"perf-node-gather-daemonset-mghtr\" (UID: \"32d2634c-6752-48bc-b2e0-24c640c805d3\") " pod="openshift-must-gather-vd4v4/perf-node-gather-daemonset-mghtr"
Jul 28 18:13:20.961174 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:20.961072    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcb55\" (UniqueName: \"kubernetes.io/projected/32d2634c-6752-48bc-b2e0-24c640c805d3-kube-api-access-hcb55\") pod \"perf-node-gather-daemonset-mghtr\" (UID: \"32d2634c-6752-48bc-b2e0-24c640c805d3\") " pod="openshift-must-gather-vd4v4/perf-node-gather-daemonset-mghtr"
Jul 28 18:13:20.961174 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:20.961133    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/32d2634c-6752-48bc-b2e0-24c640c805d3-proc\") pod \"perf-node-gather-daemonset-mghtr\" (UID: \"32d2634c-6752-48bc-b2e0-24c640c805d3\") " pod="openshift-must-gather-vd4v4/perf-node-gather-daemonset-mghtr"
Jul 28 18:13:21.062490 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:21.062451    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/32d2634c-6752-48bc-b2e0-24c640c805d3-proc\") pod \"perf-node-gather-daemonset-mghtr\" (UID: \"32d2634c-6752-48bc-b2e0-24c640c805d3\") " pod="openshift-must-gather-vd4v4/perf-node-gather-daemonset-mghtr"
Jul 28 18:13:21.062490 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:21.062497    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/32d2634c-6752-48bc-b2e0-24c640c805d3-podres\") pod \"perf-node-gather-daemonset-mghtr\" (UID: \"32d2634c-6752-48bc-b2e0-24c640c805d3\") " pod="openshift-must-gather-vd4v4/perf-node-gather-daemonset-mghtr"
Jul 28 18:13:21.062746 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:21.062535    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32d2634c-6752-48bc-b2e0-24c640c805d3-sys\") pod \"perf-node-gather-daemonset-mghtr\" (UID: \"32d2634c-6752-48bc-b2e0-24c640c805d3\") " pod="openshift-must-gather-vd4v4/perf-node-gather-daemonset-mghtr"
Jul 28 18:13:21.062746 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:21.062568    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32d2634c-6752-48bc-b2e0-24c640c805d3-lib-modules\") pod \"perf-node-gather-daemonset-mghtr\" (UID: \"32d2634c-6752-48bc-b2e0-24c640c805d3\") " pod="openshift-must-gather-vd4v4/perf-node-gather-daemonset-mghtr"
Jul 28 18:13:21.062746 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:21.062571    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/32d2634c-6752-48bc-b2e0-24c640c805d3-proc\") pod \"perf-node-gather-daemonset-mghtr\" (UID: \"32d2634c-6752-48bc-b2e0-24c640c805d3\") " pod="openshift-must-gather-vd4v4/perf-node-gather-daemonset-mghtr"
Jul 28 18:13:21.062746 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:21.062597    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcb55\" (UniqueName: \"kubernetes.io/projected/32d2634c-6752-48bc-b2e0-24c640c805d3-kube-api-access-hcb55\") pod \"perf-node-gather-daemonset-mghtr\" (UID: \"32d2634c-6752-48bc-b2e0-24c640c805d3\") " pod="openshift-must-gather-vd4v4/perf-node-gather-daemonset-mghtr"
Jul 28 18:13:21.062746 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:21.062643    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32d2634c-6752-48bc-b2e0-24c640c805d3-sys\") pod \"perf-node-gather-daemonset-mghtr\" (UID: \"32d2634c-6752-48bc-b2e0-24c640c805d3\") " pod="openshift-must-gather-vd4v4/perf-node-gather-daemonset-mghtr"
Jul 28 18:13:21.062746 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:21.062682    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/32d2634c-6752-48bc-b2e0-24c640c805d3-podres\") pod \"perf-node-gather-daemonset-mghtr\" (UID: \"32d2634c-6752-48bc-b2e0-24c640c805d3\") " pod="openshift-must-gather-vd4v4/perf-node-gather-daemonset-mghtr"
Jul 28 18:13:21.062746 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:21.062697    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32d2634c-6752-48bc-b2e0-24c640c805d3-lib-modules\") pod \"perf-node-gather-daemonset-mghtr\" (UID: \"32d2634c-6752-48bc-b2e0-24c640c805d3\") " pod="openshift-must-gather-vd4v4/perf-node-gather-daemonset-mghtr"
Jul 28 18:13:21.075664 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:21.075640    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcb55\" (UniqueName: \"kubernetes.io/projected/32d2634c-6752-48bc-b2e0-24c640c805d3-kube-api-access-hcb55\") pod \"perf-node-gather-daemonset-mghtr\" (UID: \"32d2634c-6752-48bc-b2e0-24c640c805d3\") " pod="openshift-must-gather-vd4v4/perf-node-gather-daemonset-mghtr"
Jul 28 18:13:21.176483 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:21.176405    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vd4v4/perf-node-gather-daemonset-mghtr"
Jul 28 18:13:21.357206 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:21.357170    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vd4v4/perf-node-gather-daemonset-mghtr"]
Jul 28 18:13:21.360866 ip-10-0-131-233 kubenswrapper[2566]: W0728 18:13:21.360840    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod32d2634c_6752_48bc_b2e0_24c640c805d3.slice/crio-ce5337d173d60f5cd3d7f500d2d5b7fdcdd48e105e84a564f27e9374430035b7 WatchSource:0}: Error finding container ce5337d173d60f5cd3d7f500d2d5b7fdcdd48e105e84a564f27e9374430035b7: Status 404 returned error can't find the container with id ce5337d173d60f5cd3d7f500d2d5b7fdcdd48e105e84a564f27e9374430035b7
Jul 28 18:13:21.792715 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:21.792677    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vd4v4/perf-node-gather-daemonset-mghtr" event={"ID":"32d2634c-6752-48bc-b2e0-24c640c805d3","Type":"ContainerStarted","Data":"2ce1cc45ca88116abdde838fce38e814296c873404ee574862aa5cae1b384ca8"}
Jul 28 18:13:21.792876 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:21.792743    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vd4v4/perf-node-gather-daemonset-mghtr" event={"ID":"32d2634c-6752-48bc-b2e0-24c640c805d3","Type":"ContainerStarted","Data":"ce5337d173d60f5cd3d7f500d2d5b7fdcdd48e105e84a564f27e9374430035b7"}
Jul 28 18:13:21.792876 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:21.792773    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vd4v4/perf-node-gather-daemonset-mghtr"
Jul 28 18:13:21.813500 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:21.813452    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vd4v4/perf-node-gather-daemonset-mghtr" podStartSLOduration=1.8134364490000001 podStartE2EDuration="1.813436449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:13:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:13:21.812108243 +0000 UTC m=+3052.122273750" watchObservedRunningTime="2026-07-28 18:13:21.813436449 +0000 UTC m=+3052.123601953"
Jul 28 18:13:22.721845 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:22.721819    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tpb27_e17b0fe1-b406-4126-b193-f673a5846bd1/dns/0.log"
Jul 28 18:13:22.753524 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:22.753495    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tpb27_e17b0fe1-b406-4126-b193-f673a5846bd1/kube-rbac-proxy/0.log"
Jul 28 18:13:22.815834 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:22.815810    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s7j5k_091666fc-49ce-46bc-9a7d-5815676ea342/dns-node-resolver/0.log"
Jul 28 18:13:23.367845 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:23.367812    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5bhch_630e1b18-a044-454e-9be4-359de1817e2b/node-ca/0.log"
Jul 28 18:13:24.553633 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:24.553603    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b67m9_b244121d-33d8-46d2-aa87-b451301005c8/serve-healthcheck-canary/0.log"
Jul 28 18:13:25.016081 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:25.016055    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-79zvc_3a9203ae-f509-43f1-ae1e-2841014548a4/kube-rbac-proxy/0.log"
Jul 28 18:13:25.051227 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:25.051196    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-79zvc_3a9203ae-f509-43f1-ae1e-2841014548a4/exporter/0.log"
Jul 28 18:13:25.105711 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:25.105683    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-79zvc_3a9203ae-f509-43f1-ae1e-2841014548a4/extractor/0.log"
Jul 28 18:13:27.740362 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:27.740311    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-654c557856-9k58d_8312b01c-e651-40f5-bf99-d5fbd4359133/manager/0.log"
Jul 28 18:13:27.804757 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:27.804729    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vd4v4/perf-node-gather-daemonset-mghtr"
Jul 28 18:13:34.188118 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:34.188081    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-48d52_15fc65f5-e122-489d-b56c-1999d101223f/kube-multus-additional-cni-plugins/0.log"
Jul 28 18:13:34.215571 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:34.215549    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-48d52_15fc65f5-e122-489d-b56c-1999d101223f/egress-router-binary-copy/0.log"
Jul 28 18:13:34.246885 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:34.246855    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-48d52_15fc65f5-e122-489d-b56c-1999d101223f/cni-plugins/0.log"
Jul 28 18:13:34.285160 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:34.285129    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-48d52_15fc65f5-e122-489d-b56c-1999d101223f/bond-cni-plugin/0.log"
Jul 28 18:13:34.321209 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:34.321186    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-48d52_15fc65f5-e122-489d-b56c-1999d101223f/routeoverride-cni/0.log"
Jul 28 18:13:34.355446 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:34.355421    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-48d52_15fc65f5-e122-489d-b56c-1999d101223f/whereabouts-cni-bincopy/0.log"
Jul 28 18:13:34.378712 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:34.378692    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-48d52_15fc65f5-e122-489d-b56c-1999d101223f/whereabouts-cni/0.log"
Jul 28 18:13:34.924343 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:34.924305    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-xnpc6_795058ed-0974-4328-b462-32e755000a70/kube-multus/0.log"
Jul 28 18:13:35.102967 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:35.102925    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-znptx_dedc8b78-7469-4e8d-a7b8-b11c741186d7/network-metrics-daemon/0.log"
Jul 28 18:13:35.132526 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:35.132499    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-znptx_dedc8b78-7469-4e8d-a7b8-b11c741186d7/kube-rbac-proxy/0.log"
Jul 28 18:13:35.931956 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:35.931924    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sdnx_6e018f9d-b031-40ad-9846-f1fb5b4e21e8/ovn-controller/0.log"
Jul 28 18:13:35.959500 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:35.959477    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sdnx_6e018f9d-b031-40ad-9846-f1fb5b4e21e8/ovn-acl-logging/0.log"
Jul 28 18:13:36.000424 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:36.000401    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sdnx_6e018f9d-b031-40ad-9846-f1fb5b4e21e8/kube-rbac-proxy-node/0.log"
Jul 28 18:13:36.030949 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:36.030922    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sdnx_6e018f9d-b031-40ad-9846-f1fb5b4e21e8/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 18:13:36.054443 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:36.054417    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sdnx_6e018f9d-b031-40ad-9846-f1fb5b4e21e8/northd/0.log"
Jul 28 18:13:36.079524 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:36.079495    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sdnx_6e018f9d-b031-40ad-9846-f1fb5b4e21e8/nbdb/0.log"
Jul 28 18:13:36.104422 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:36.104398    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sdnx_6e018f9d-b031-40ad-9846-f1fb5b4e21e8/sbdb/0.log"
Jul 28 18:13:36.162849 ip-10-0-131-233 kubenswrapper[2566]: I0728 18:13:36.162819    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sdnx_6e018f9d-b031-40ad-9846-f1fb5b4e21e8/ovnkube-controller/0.log"