Jul 01 17:13:23.662588 ip-10-0-141-213 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 01 17:13:23.662600 ip-10-0-141-213 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 01 17:13:23.662607 ip-10-0-141-213 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 01 17:13:23.662832 ip-10-0-141-213 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 01 17:13:33.465549 ip-10-0-141-213 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Jul 01 17:13:33.465564 ip-10-0-141-213 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 6d898a7a10ef4b17978d5955b37454af --
Jul 01 17:15:10.461819 ip-10-0-141-213 systemd[1]: Starting Kubernetes Kubelet...
Jul 01 17:15:10.919143 ip-10-0-141-213 kubenswrapper[2585]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 17:15:10.919143 ip-10-0-141-213 kubenswrapper[2585]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 01 17:15:10.919143 ip-10-0-141-213 kubenswrapper[2585]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 17:15:10.919143 ip-10-0-141-213 kubenswrapper[2585]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 01 17:15:10.919143 ip-10-0-141-213 kubenswrapper[2585]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 17:15:10.920333 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.920100    2585 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 01 17:15:10.925163 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925147    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 17:15:10.925163 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925162    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 17:15:10.925163 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925165    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 17:15:10.925293 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925168    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 17:15:10.925293 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925173    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 17:15:10.925293 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925175    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 17:15:10.925293 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925178    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 17:15:10.925293 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925182    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 17:15:10.925293 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925186    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 17:15:10.925293 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925190    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 17:15:10.925293 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925194    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 17:15:10.925293 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925197    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 17:15:10.925293 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925201    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 17:15:10.925293 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925204    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 17:15:10.925293 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925207    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 17:15:10.925293 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925210    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 17:15:10.925293 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925212    2585 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 17:15:10.925293 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925215    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 17:15:10.925293 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925219    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 17:15:10.925293 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925222    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 17:15:10.925293 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925225    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 17:15:10.925743 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925227    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 17:15:10.925743 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925230    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 17:15:10.925743 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925232    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 17:15:10.925743 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925235    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 17:15:10.925743 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925238    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 17:15:10.925743 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925240    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 17:15:10.925743 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925243    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 17:15:10.925743 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925245    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 17:15:10.925743 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925248    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 17:15:10.925743 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925251    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 17:15:10.925743 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925254    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 17:15:10.925743 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925257    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 17:15:10.925743 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925261    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 17:15:10.925743 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925264    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 17:15:10.925743 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925266    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 17:15:10.925743 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925269    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 17:15:10.925743 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925272    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 17:15:10.925743 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925274    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 17:15:10.925743 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925277    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 17:15:10.925743 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925279    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 17:15:10.926216 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925282    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 17:15:10.926216 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925284    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 17:15:10.926216 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925287    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 17:15:10.926216 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925290    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 17:15:10.926216 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925292    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 17:15:10.926216 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925295    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 17:15:10.926216 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925298    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 17:15:10.926216 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925300    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 17:15:10.926216 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925303    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 17:15:10.926216 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925306    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 17:15:10.926216 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925308    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 17:15:10.926216 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925311    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 17:15:10.926216 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925313    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 17:15:10.926216 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925315    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 17:15:10.926216 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925318    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 17:15:10.926216 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925320    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 17:15:10.926216 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925322    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 17:15:10.926216 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925325    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 17:15:10.926216 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925327    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 17:15:10.926703 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925329    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 17:15:10.926703 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925332    2585 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 17:15:10.926703 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925335    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 17:15:10.926703 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925338    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 17:15:10.926703 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925341    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 17:15:10.926703 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925344    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 17:15:10.926703 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925347    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 17:15:10.926703 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925349    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 17:15:10.926703 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925353    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 17:15:10.926703 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925356    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 17:15:10.926703 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925358    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 17:15:10.926703 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925361    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 17:15:10.926703 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925363    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 17:15:10.926703 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925366    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 17:15:10.926703 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925368    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 17:15:10.926703 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925371    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 17:15:10.926703 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925373    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 17:15:10.926703 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925376    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 17:15:10.926703 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925378    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 17:15:10.926703 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925381    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 17:15:10.927184 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925383    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 17:15:10.927184 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925386    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 17:15:10.927184 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925388    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 17:15:10.927184 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925390    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 17:15:10.927184 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925393    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 17:15:10.927184 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925395    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 17:15:10.927184 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925397    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 17:15:10.927184 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925400    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 17:15:10.927184 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925402    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 17:15:10.927184 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925405    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 17:15:10.927184 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925407    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 17:15:10.927184 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925410    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 17:15:10.927184 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925412    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 17:15:10.927184 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925415    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 17:15:10.927184 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925417    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 17:15:10.927184 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925420    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 17:15:10.927184 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925422    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 17:15:10.927184 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925424    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 17:15:10.927184 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.925427    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 17:15:10.927184 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926151    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 17:15:10.927679 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926158    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 17:15:10.927679 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926161    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 17:15:10.927679 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926163    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 17:15:10.927679 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926167    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 17:15:10.927679 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926170    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 17:15:10.927679 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926172    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 17:15:10.927679 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926174    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 17:15:10.927679 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926177    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 17:15:10.927679 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926180    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 17:15:10.927679 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926182    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 17:15:10.927679 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926185    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 17:15:10.927679 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926188    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 17:15:10.927679 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926190    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 17:15:10.927679 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926193    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 17:15:10.927679 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926196    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 17:15:10.927679 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926198    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 17:15:10.927679 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926201    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 17:15:10.927679 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926203    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 17:15:10.927679 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926206    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 17:15:10.928175 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926208    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 17:15:10.928175 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926210    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 17:15:10.928175 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926213    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 17:15:10.928175 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926215    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 17:15:10.928175 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926218    2585 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 17:15:10.928175 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926221    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 17:15:10.928175 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926225    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 17:15:10.928175 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926227    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 17:15:10.928175 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926230    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 17:15:10.928175 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926232    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 17:15:10.928175 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926235    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 17:15:10.928175 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926237    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 17:15:10.928175 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926240    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 17:15:10.928175 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926242    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 17:15:10.928175 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926245    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 17:15:10.928175 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926247    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 17:15:10.928175 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926250    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 17:15:10.928175 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926252    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 17:15:10.928175 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926254    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 17:15:10.928175 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926257    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 17:15:10.928677 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926259    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 17:15:10.928677 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926261    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 17:15:10.928677 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926264    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 17:15:10.928677 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926266    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 17:15:10.928677 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926270    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 17:15:10.928677 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926272    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 17:15:10.928677 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926274    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 17:15:10.928677 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926277    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 17:15:10.928677 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926279    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 17:15:10.928677 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926283    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 17:15:10.928677 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926286    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 17:15:10.928677 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926289    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 17:15:10.928677 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926291    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 17:15:10.928677 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926294    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 17:15:10.928677 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926296    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 17:15:10.928677 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926298    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 17:15:10.928677 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926307    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 17:15:10.928677 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926309    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 17:15:10.928677 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926312    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 17:15:10.929139 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926315    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 17:15:10.929139 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926318    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 17:15:10.929139 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926320    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 17:15:10.929139 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926323    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 17:15:10.929139 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926325    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 17:15:10.929139 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926328    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 17:15:10.929139 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926330    2585 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 17:15:10.929139 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926333    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 17:15:10.929139 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926336    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 17:15:10.929139 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926338    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 17:15:10.929139 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926341    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 17:15:10.929139 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926343    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 17:15:10.929139 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926346    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 17:15:10.929139 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926348    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 17:15:10.929139 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926351    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 17:15:10.929139 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926353    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 17:15:10.929139 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926356    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 17:15:10.929139 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926358    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 17:15:10.929139 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926361    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 17:15:10.929139 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926363    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 17:15:10.929640 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926366    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 17:15:10.929640 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926368    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 17:15:10.929640 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926371    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 17:15:10.929640 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926373    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 17:15:10.929640 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926375    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 17:15:10.929640 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926378    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 17:15:10.929640 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926380    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 17:15:10.929640 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926382    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 17:15:10.929640 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926386    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 17:15:10.929640 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926389    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 17:15:10.929640 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926391    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 17:15:10.929640 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926394    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 17:15:10.929640 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926396    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 17:15:10.929640 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926398    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 17:15:10.929640 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926401    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 17:15:10.929640 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926403    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 17:15:10.929640 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926407    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 17:15:10.929640 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926411    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 17:15:10.929640 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926416    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 17:15:10.930093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926419    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 17:15:10.930093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926827    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 17:15:10.930093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926833    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 17:15:10.930093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926835    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 17:15:10.930093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926838    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 17:15:10.930093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926841    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 17:15:10.930093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926844    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 17:15:10.930093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926846    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 17:15:10.930093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926848    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 17:15:10.930093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926851    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 17:15:10.930093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926854    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 17:15:10.930093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926857    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 17:15:10.930093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926859    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 17:15:10.930093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926862    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 17:15:10.930093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926865    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 17:15:10.930093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926867    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 17:15:10.930093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926869    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 17:15:10.930093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926872    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 17:15:10.930093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926874    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 17:15:10.930093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926876    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 17:15:10.930634 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926879    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 17:15:10.930634 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926881    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 17:15:10.930634 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926884    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 17:15:10.930634 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926886    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 17:15:10.930634 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926889    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 17:15:10.930634 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926891    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 17:15:10.930634 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926894    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 17:15:10.930634 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926896    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 17:15:10.930634 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926899    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 17:15:10.930634 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926901    2585 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 17:15:10.930634 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926904    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 17:15:10.930634 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926908    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 17:15:10.930634 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926911    2585 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 17:15:10.930634 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926914    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 17:15:10.930634 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926916    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 17:15:10.930634 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926919    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 17:15:10.930634 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926923    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 17:15:10.930634 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926926    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 17:15:10.930634 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926928    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 17:15:10.931093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926931    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 17:15:10.931093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926933    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 17:15:10.931093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926935    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 17:15:10.931093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926938    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 17:15:10.931093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926940    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 17:15:10.931093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926943    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 17:15:10.931093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926946    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 17:15:10.931093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926948    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 17:15:10.931093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926950    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 17:15:10.931093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926952    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 17:15:10.931093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926955    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 17:15:10.931093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926957    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 17:15:10.931093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926960    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 17:15:10.931093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926962    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 17:15:10.931093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926965    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 17:15:10.931093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926967    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 17:15:10.931093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926969    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 17:15:10.931093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926973    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 17:15:10.931093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926975    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 17:15:10.931093 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926977    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 17:15:10.931663 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926980    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 17:15:10.931663 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926982    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 17:15:10.931663 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926984    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 17:15:10.931663 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926987    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 17:15:10.931663 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926990    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 17:15:10.931663 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926993    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 17:15:10.931663 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926996    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 17:15:10.931663 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.926998    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 17:15:10.931663 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927001    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 17:15:10.931663 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927003    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 17:15:10.931663 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927005    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 17:15:10.931663 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927008    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 17:15:10.931663 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927010    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 17:15:10.931663 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927013    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 17:15:10.931663 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927015    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 17:15:10.931663 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927018    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 17:15:10.931663 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927020    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 17:15:10.931663 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927022    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 17:15:10.931663 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927026    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 17:15:10.931663 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927028    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 17:15:10.932144 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927031    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 17:15:10.932144 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927033    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 17:15:10.932144 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927035    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 17:15:10.932144 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927038    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 17:15:10.932144 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927040    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 17:15:10.932144 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927042    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 17:15:10.932144 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927045    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 17:15:10.932144 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927047    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 17:15:10.932144 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927051    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 17:15:10.932144 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927055    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 17:15:10.932144 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927057    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 17:15:10.932144 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927060    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 17:15:10.932144 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927063    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 17:15:10.932144 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927065    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 17:15:10.932144 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927068    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 17:15:10.932144 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927071    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 17:15:10.932144 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927076    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 17:15:10.932144 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927079    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 17:15:10.932144 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927082    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 17:15:10.932604 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927085    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 17:15:10.932604 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927087    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 17:15:10.932604 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927165    2585 flags.go:64] FLAG: --address="0.0.0.0"
Jul 01 17:15:10.932604 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927175    2585 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 01 17:15:10.932604 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927185    2585 flags.go:64] FLAG: --anonymous-auth="true"
Jul 01 17:15:10.932604 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927192    2585 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 01 17:15:10.932604 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927197    2585 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 01 17:15:10.932604 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927201    2585 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 01 17:15:10.932604 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927205    2585 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 01 17:15:10.932604 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927209    2585 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 01 17:15:10.932604 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927212    2585 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 01 17:15:10.932604 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927215    2585 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 01 17:15:10.932604 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927219    2585 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 01 17:15:10.932604 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927223    2585 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 01 17:15:10.932604 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927228    2585 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 01 17:15:10.932604 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927231    2585 flags.go:64] FLAG: --cgroup-root=""
Jul 01 17:15:10.932604 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927234    2585 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 01 17:15:10.932604 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927236    2585 flags.go:64] FLAG: --client-ca-file=""
Jul 01 17:15:10.932604 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927239    2585 flags.go:64] FLAG: --cloud-provider="external"
Jul 01 17:15:10.932604 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927242    2585 flags.go:64] FLAG: --cluster-dns="[]"
Jul 01 17:15:10.932604 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927246    2585 flags.go:64] FLAG: --cluster-domain=""
Jul 01 17:15:10.932604 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927249    2585 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 01 17:15:10.932604 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927252    2585 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 01 17:15:10.933191 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927255    2585 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 01 17:15:10.933191 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927258    2585 flags.go:64] FLAG: --container-log-max-files="5"
Jul 01 17:15:10.933191 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927262    2585 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 01 17:15:10.933191 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927265    2585 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 01 17:15:10.933191 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927268    2585 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 01 17:15:10.933191 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927271    2585 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 01 17:15:10.933191 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927274    2585 flags.go:64] FLAG: --contention-profiling="false"
Jul 01 17:15:10.933191 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927277    2585 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 01 17:15:10.933191 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927280    2585 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 01 17:15:10.933191 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927283    2585 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 01 17:15:10.933191 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927285    2585 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 01 17:15:10.933191 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927290    2585 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 01 17:15:10.933191 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927293    2585 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 01 17:15:10.933191 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927296    2585 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 01 17:15:10.933191 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927299    2585 flags.go:64] FLAG: --enable-load-reader="false"
Jul 01 17:15:10.933191 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927302    2585 flags.go:64] FLAG: --enable-server="true"
Jul 01 17:15:10.933191 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927305    2585 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 01 17:15:10.933191 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927309    2585 flags.go:64] FLAG: --event-burst="100"
Jul 01 17:15:10.933191 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927312    2585 flags.go:64] FLAG: --event-qps="50"
Jul 01 17:15:10.933191 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927315    2585 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 01 17:15:10.933191 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927318    2585 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 01 17:15:10.933191 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927321    2585 flags.go:64] FLAG: --eviction-hard=""
Jul 01 17:15:10.933191 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927325    2585 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 01 17:15:10.933191 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927330    2585 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 01 17:15:10.933797 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927332    2585 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 01 17:15:10.933797 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927335    2585 flags.go:64] FLAG: --eviction-soft=""
Jul 01 17:15:10.933797 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927338    2585 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 01 17:15:10.933797 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927340    2585 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 01 17:15:10.933797 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927343    2585 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 01 17:15:10.933797 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927346    2585 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 01 17:15:10.933797 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927348    2585 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 01 17:15:10.933797 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927351    2585 flags.go:64] FLAG: --fail-swap-on="true"
Jul 01 17:15:10.933797 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927354    2585 flags.go:64] FLAG: --feature-gates=""
Jul 01 17:15:10.933797 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927357    2585 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 01 17:15:10.933797 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927360    2585 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 01 17:15:10.933797 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927363    2585 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 01 17:15:10.933797 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927366    2585 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 01 17:15:10.933797 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927368    2585 flags.go:64] FLAG: --healthz-port="10248"
Jul 01 17:15:10.933797 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927371    2585 flags.go:64] FLAG: --help="false"
Jul 01 17:15:10.933797 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927375    2585 flags.go:64] FLAG: --hostname-override="ip-10-0-141-213.ec2.internal"
Jul 01 17:15:10.933797 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927378    2585 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 01 17:15:10.933797 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927381    2585 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 01 17:15:10.933797 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927384    2585 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 01 17:15:10.933797 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927387    2585 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 01 17:15:10.933797 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927390    2585 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 01 17:15:10.933797 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927393    2585 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 01 17:15:10.933797 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927395    2585 flags.go:64] FLAG: --image-service-endpoint=""
Jul 01 17:15:10.934374 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927398    2585 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 01 17:15:10.934374 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927401    2585 flags.go:64] FLAG: --kube-api-burst="100"
Jul 01 17:15:10.934374 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927404    2585 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 01 17:15:10.934374 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927407    2585 flags.go:64] FLAG: --kube-api-qps="50"
Jul 01 17:15:10.934374 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927409    2585 flags.go:64] FLAG: --kube-reserved=""
Jul 01 17:15:10.934374 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927412    2585 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 01 17:15:10.934374 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927415    2585 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 01 17:15:10.934374 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927418    2585 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 01 17:15:10.934374 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927422    2585 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 01 17:15:10.934374 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927424    2585 flags.go:64] FLAG: --lock-file=""
Jul 01 17:15:10.934374 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927427    2585 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 01 17:15:10.934374 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927430    2585 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 01 17:15:10.934374 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927433    2585 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 01 17:15:10.934374 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927438    2585 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 01 17:15:10.934374 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927441    2585 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 01 17:15:10.934374 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927444    2585 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 01 17:15:10.934374 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927446    2585 flags.go:64] FLAG: --logging-format="text"
Jul 01 17:15:10.934374 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927449    2585 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 01 17:15:10.934374 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927452    2585 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 01 17:15:10.934374 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927455    2585 flags.go:64] FLAG: --manifest-url=""
Jul 01 17:15:10.934374 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927458    2585 flags.go:64] FLAG: --manifest-url-header=""
Jul 01 17:15:10.934374 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927462    2585 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 01 17:15:10.934374 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927464    2585 flags.go:64] FLAG: --max-open-files="1000000"
Jul 01 17:15:10.934374 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927468    2585 flags.go:64] FLAG: --max-pods="110"
Jul 01 17:15:10.934374 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927471    2585 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 01 17:15:10.935329 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927474    2585 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 01 17:15:10.935329 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927477    2585 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 01 17:15:10.935329 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927480    2585 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 01 17:15:10.935329 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927482    2585 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 01 17:15:10.935329 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927485    2585 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 01 17:15:10.935329 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927488    2585 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 01 17:15:10.935329 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927494    2585 flags.go:64] FLAG: --node-status-max-images="50"
Jul 01 17:15:10.935329 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927497    2585 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 01 17:15:10.935329 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927500    2585 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 01 17:15:10.935329 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927503    2585 flags.go:64] FLAG: --pod-cidr=""
Jul 01 17:15:10.935329 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927505    2585 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04"
Jul 01 17:15:10.935329 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927511    2585 flags.go:64] FLAG: --pod-manifest-path=""
Jul 01 17:15:10.935329 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927514    2585 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 01 17:15:10.935329 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927517    2585 flags.go:64] FLAG: --pods-per-core="0"
Jul 01 17:15:10.935329 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927519    2585 flags.go:64] FLAG: --port="10250"
Jul 01 17:15:10.935329 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927524    2585 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 01 17:15:10.935329 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927527    2585 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a0b72a5bc08a0a18"
Jul 01 17:15:10.935329 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927530    2585 flags.go:64] FLAG: --qos-reserved=""
Jul 01 17:15:10.935329 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927533    2585 flags.go:64] FLAG: --read-only-port="10255"
Jul 01 17:15:10.935329 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927536    2585 flags.go:64] FLAG: --register-node="true"
Jul 01 17:15:10.935329 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927538    2585 flags.go:64] FLAG: --register-with-taints=""
Jul 01 17:15:10.935329 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927541    2585 flags.go:64] FLAG: --registry-burst="10"
Jul 01 17:15:10.935329 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927544    2585 flags.go:64] FLAG: --registry-qps="5"
Jul 01 17:15:10.935329 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927547    2585 flags.go:64] FLAG: --reserved-cpus=""
Jul 01 17:15:10.936325 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927549    2585 flags.go:64] FLAG: --reserved-memory=""
Jul 01 17:15:10.936325 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927553    2585 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 01 17:15:10.936325 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927555    2585 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 01 17:15:10.936325 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927570    2585 flags.go:64] FLAG: --rotate-certificates="false"
Jul 01 17:15:10.936325 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927573    2585 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 01 17:15:10.936325 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927576    2585 flags.go:64] FLAG: --runonce="false"
Jul 01 17:15:10.936325 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927578    2585 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 01 17:15:10.936325 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927581    2585 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 01 17:15:10.936325 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927584    2585 flags.go:64] FLAG: --seccomp-default="false"
Jul 01 17:15:10.936325 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927587    2585 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 01 17:15:10.936325 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927590    2585 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 01 17:15:10.936325 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927593    2585 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 01 17:15:10.936325 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927595    2585 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 01 17:15:10.936325 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927598    2585 flags.go:64] FLAG: --storage-driver-password="root"
Jul 01 17:15:10.936325 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927601    2585 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 01 17:15:10.936325 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927604    2585 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 01 17:15:10.936325 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927606    2585 flags.go:64] FLAG: --storage-driver-user="root"
Jul 01 17:15:10.936325 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927609    2585 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 01 17:15:10.936325 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927612    2585 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 01 17:15:10.936325 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927615    2585 flags.go:64] FLAG: --system-cgroups=""
Jul 01 17:15:10.936325 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927617    2585 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 01 17:15:10.936325 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927622    2585 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 01 17:15:10.936325 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927625    2585 flags.go:64] FLAG: --tls-cert-file=""
Jul 01 17:15:10.936325 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927629    2585 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 01 17:15:10.936325 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927633    2585 flags.go:64] FLAG: --tls-min-version=""
Jul 01 17:15:10.937394 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927636    2585 flags.go:64] FLAG: --tls-private-key-file=""
Jul 01 17:15:10.937394 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927638    2585 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 01 17:15:10.937394 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927641    2585 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 01 17:15:10.937394 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927644    2585 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 01 17:15:10.937394 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927646    2585 flags.go:64] FLAG: --v="2"
Jul 01 17:15:10.937394 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927650    2585 flags.go:64] FLAG: --version="false"
Jul 01 17:15:10.937394 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927654    2585 flags.go:64] FLAG: --vmodule=""
Jul 01 17:15:10.937394 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927664    2585 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 01 17:15:10.937394 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.927667    2585 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 01 17:15:10.937394 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927784    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 17:15:10.937394 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927788    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 17:15:10.937394 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927791    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 17:15:10.937394 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927794    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 17:15:10.937394 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927797    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 17:15:10.937394 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927802    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 17:15:10.937394 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927805    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 17:15:10.937394 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927809    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 17:15:10.937394 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927812    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 17:15:10.937394 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927814    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 17:15:10.937394 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927818    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 17:15:10.937394 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927822    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 17:15:10.938425 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927825    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 17:15:10.938425 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927827    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 17:15:10.938425 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927830    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 17:15:10.938425 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927833    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 17:15:10.938425 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927837    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 17:15:10.938425 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927840    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 17:15:10.938425 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927843    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 17:15:10.938425 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927846    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 17:15:10.938425 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927849    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 17:15:10.938425 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927853    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 17:15:10.938425 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927856    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 17:15:10.938425 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927858    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 17:15:10.938425 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927861    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 17:15:10.938425 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927864    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 17:15:10.938425 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927866    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 17:15:10.938425 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927869    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 17:15:10.938425 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927873    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 17:15:10.938425 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927876    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 17:15:10.938425 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927878    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 17:15:10.939060 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927881    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 17:15:10.939060 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927884    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 17:15:10.939060 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927886    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 17:15:10.939060 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927889    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 17:15:10.939060 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927891    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 17:15:10.939060 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927894    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 17:15:10.939060 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927897    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 17:15:10.939060 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927900    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 17:15:10.939060 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927902    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 17:15:10.939060 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927905    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 17:15:10.939060 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927907    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 17:15:10.939060 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927910    2585 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 17:15:10.939060 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927912    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 17:15:10.939060 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927915    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 17:15:10.939060 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927917    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 17:15:10.939060 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927919    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 17:15:10.939060 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927922    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 17:15:10.939060 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927924    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 17:15:10.939060 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927926    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 17:15:10.939060 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927929    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 17:15:10.939701 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927931    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 17:15:10.939701 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927934    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 17:15:10.939701 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927937    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 17:15:10.939701 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927939    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 17:15:10.939701 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927942    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 17:15:10.939701 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927944    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 17:15:10.939701 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927947    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 17:15:10.939701 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927949    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 17:15:10.939701 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927952    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 17:15:10.939701 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927954    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 17:15:10.939701 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927956    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 17:15:10.939701 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927959    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 17:15:10.939701 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927961    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 17:15:10.939701 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927963    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 17:15:10.939701 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927966    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 17:15:10.939701 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927968    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 17:15:10.939701 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927970    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 17:15:10.939701 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927973    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 17:15:10.939701 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927977    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 17:15:10.939701 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927979    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 17:15:10.940163 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927982    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 17:15:10.940163 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927984    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 17:15:10.940163 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927987    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 17:15:10.940163 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927989    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 17:15:10.940163 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927991    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 17:15:10.940163 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927994    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 17:15:10.940163 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927996    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 17:15:10.940163 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.927999    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 17:15:10.940163 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.928001    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 17:15:10.940163 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.928003    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 17:15:10.940163 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.928006    2585 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 17:15:10.940163 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.928008    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 17:15:10.940163 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.928010    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 17:15:10.940163 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.928013    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 17:15:10.940163 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.928017    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 17:15:10.940163 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.928019    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 17:15:10.940163 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.928022    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 17:15:10.940163 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.928024    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 17:15:10.940163 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.928027    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 17:15:10.940163 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.928029    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 17:15:10.940904 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.928032    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 17:15:10.940904 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.928034    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 17:15:10.940904 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.928036    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 17:15:10.940904 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.928039    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 17:15:10.940904 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.928041    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 17:15:10.940904 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.928044    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 17:15:10.940904 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.928046    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 17:15:10.940904 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.928048    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 17:15:10.940904 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.928054    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 17:15:10.940904 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.936794    2585 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 01 17:15:10.940904 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.936814    2585 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 01 17:15:10.940904 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.936841    2585 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 01 17:15:10.940904 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.936849    2585 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 01 17:15:10.940904 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.936935    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 17:15:10.940904 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.936943    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 17:15:10.941467 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.936950    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 17:15:10.941467 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.936955    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 17:15:10.941467 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.936960    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 17:15:10.941467 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.936965    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 17:15:10.941467 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.936970    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 17:15:10.941467 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.936974    2585 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 17:15:10.941467 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.936978    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 17:15:10.941467 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.936983    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 17:15:10.941467 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.936989    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 17:15:10.941467 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.936996    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 17:15:10.941467 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937001    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 17:15:10.941467 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937005    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 17:15:10.941467 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937009    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 17:15:10.941467 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937014    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 17:15:10.941467 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937018    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 17:15:10.941467 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937022    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 17:15:10.941467 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937025    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 17:15:10.941467 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937030    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 17:15:10.941467 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937037    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 17:15:10.942031 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937044    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 17:15:10.942031 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937048    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 17:15:10.942031 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937054    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 17:15:10.942031 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937058    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 17:15:10.942031 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937062    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 17:15:10.942031 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937066    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 17:15:10.942031 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937072    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 17:15:10.942031 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937076    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 17:15:10.942031 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937081    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 17:15:10.942031 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937085    2585 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 17:15:10.942031 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937090    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 17:15:10.942031 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937094    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 17:15:10.942031 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937099    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 17:15:10.942031 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937103    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 17:15:10.942031 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937107    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 17:15:10.942031 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937111    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 17:15:10.942031 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937115    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 17:15:10.942031 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937119    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 17:15:10.942031 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937123    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 17:15:10.942031 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937127    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 17:15:10.942714 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937131    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 17:15:10.942714 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937135    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 17:15:10.942714 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937140    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 17:15:10.942714 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937144    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 17:15:10.942714 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937148    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 17:15:10.942714 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937152    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 17:15:10.942714 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937157    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 17:15:10.942714 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937161    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 17:15:10.942714 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937165    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 17:15:10.942714 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937169    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 17:15:10.942714 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937173    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 17:15:10.942714 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937177    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 17:15:10.942714 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937182    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 17:15:10.942714 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937186    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 17:15:10.942714 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937190    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 17:15:10.942714 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937194    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 17:15:10.942714 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937199    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 17:15:10.942714 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937203    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 17:15:10.942714 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937207    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 17:15:10.942714 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937211    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 17:15:10.943220 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937215    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 17:15:10.943220 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937219    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 17:15:10.943220 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937223    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 17:15:10.943220 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937230    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 17:15:10.943220 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937235    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 17:15:10.943220 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937240    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 17:15:10.943220 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937245    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 17:15:10.943220 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937249    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 17:15:10.943220 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937253    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 17:15:10.943220 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937258    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 17:15:10.943220 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937262    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 17:15:10.943220 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937267    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 17:15:10.943220 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937271    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 17:15:10.943220 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937275    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 17:15:10.943220 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937279    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 17:15:10.943220 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937284    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 17:15:10.943220 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937288    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 17:15:10.943220 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937292    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 17:15:10.943220 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937297    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 17:15:10.943220 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937301    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 17:15:10.943715 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937305    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 17:15:10.943715 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937309    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 17:15:10.943715 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937313    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 17:15:10.943715 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937317    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 17:15:10.943715 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937322    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 17:15:10.943715 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937326    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 17:15:10.943715 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937331    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 17:15:10.943715 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937334    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 17:15:10.943715 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937339    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 17:15:10.943715 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937343    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 17:15:10.943715 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937347    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 17:15:10.943715 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937352    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 17:15:10.943715 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937356    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 17:15:10.943715 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937360    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 17:15:10.943715 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937364    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 17:15:10.943715 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937368    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 17:15:10.943715 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937373    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 17:15:10.943715 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937377    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 17:15:10.944208 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.937385    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 17:15:10.944208 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937619    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 17:15:10.944208 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937630    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 17:15:10.944208 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937634    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 17:15:10.944208 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937638    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 17:15:10.944208 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937642    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 17:15:10.944208 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937646    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 17:15:10.944208 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937650    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 17:15:10.944208 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937653    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 17:15:10.944208 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937657    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 17:15:10.944208 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937661    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 17:15:10.944208 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937665    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 17:15:10.944208 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937668    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 17:15:10.944208 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937673    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 17:15:10.944208 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937678    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 17:15:10.944605 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937682    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 17:15:10.944605 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937686    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 17:15:10.944605 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937693    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 17:15:10.944605 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937699    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 17:15:10.944605 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937704    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 17:15:10.944605 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937709    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 17:15:10.944605 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937714    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 17:15:10.944605 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937717    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 17:15:10.944605 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937722    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 17:15:10.944605 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937728    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 17:15:10.944605 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937732    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 17:15:10.944605 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937736    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 17:15:10.944605 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937740    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 17:15:10.944605 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937744    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 17:15:10.944605 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937748    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 17:15:10.944605 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937752    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 17:15:10.944605 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937756    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 17:15:10.944605 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937760    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 17:15:10.944605 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937764    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 17:15:10.945129 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937768    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 17:15:10.945129 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937772    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 17:15:10.945129 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937776    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 17:15:10.945129 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937781    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 17:15:10.945129 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937787    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 17:15:10.945129 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937793    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 17:15:10.945129 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937798    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 17:15:10.945129 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937802    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 17:15:10.945129 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937807    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 17:15:10.945129 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937811    2585 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 17:15:10.945129 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937816    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 17:15:10.945129 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937820    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 17:15:10.945129 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937825    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 17:15:10.945129 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937829    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 17:15:10.945129 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937834    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 17:15:10.945129 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937838    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 17:15:10.945129 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937842    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 17:15:10.945129 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937846    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 17:15:10.945129 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937851    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 17:15:10.945617 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937855    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 17:15:10.945617 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937860    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 17:15:10.945617 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937864    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 17:15:10.945617 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937868    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 17:15:10.945617 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937872    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 17:15:10.945617 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937877    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 17:15:10.945617 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937881    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 17:15:10.945617 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937885    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 17:15:10.945617 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937889    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 17:15:10.945617 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937893    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 17:15:10.945617 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937898    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 17:15:10.945617 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937902    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 17:15:10.945617 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937906    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 17:15:10.945617 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937910    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 17:15:10.945617 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937914    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 17:15:10.945617 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937918    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 17:15:10.945617 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937922    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 17:15:10.945617 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937926    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 17:15:10.945617 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937930    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 17:15:10.945617 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937934    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 17:15:10.946130 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937938    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 17:15:10.946130 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937942    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 17:15:10.946130 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937946    2585 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 17:15:10.946130 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937951    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 17:15:10.946130 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937955    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 17:15:10.946130 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937959    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 17:15:10.946130 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937964    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 17:15:10.946130 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937968    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 17:15:10.946130 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937972    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 17:15:10.946130 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937976    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 17:15:10.946130 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937980    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 17:15:10.946130 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937985    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 17:15:10.946130 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937988    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 17:15:10.946130 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937992    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 17:15:10.946130 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.937997    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 17:15:10.946130 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.938000    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 17:15:10.946130 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.938004    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 17:15:10.946130 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.938008    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 17:15:10.946130 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.938012    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 17:15:10.946130 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.938016    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 17:15:10.946788 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.938020    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 17:15:10.946788 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.938025    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 17:15:10.946788 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.938029    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 17:15:10.946788 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.938033    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 17:15:10.946788 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.938037    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 17:15:10.946788 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.938043    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 17:15:10.946788 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:10.938048    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 17:15:10.946788 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.938056    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 17:15:10.946788 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.938226    2585 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 01 17:15:10.946788 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.942100    2585 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 01 17:15:10.946788 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.943818    2585 server.go:1019] "Starting client certificate rotation"
Jul 01 17:15:10.946788 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.943921    2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 17:15:10.946788 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.943957    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 17:15:10.947317 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.947094    2585 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 17:15:10.948408 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.948386    2585 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 17:15:10.958449 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.958432    2585 log.go:25] "Validated CRI v1 runtime API"
Jul 01 17:15:10.963158 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.963143    2585 log.go:25] "Validated CRI v1 image API"
Jul 01 17:15:10.964751 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.964730    2585 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 01 17:15:10.967180 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.967163    2585 fs.go:135] Filesystem UUIDs: map[0a4539bd-bd89-493a-a8e4-a9b3a9cfe1c7:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8af8b62d-4807-42fc-8385-40d29baa2ab8:/dev/nvme0n1p4]
Jul 01 17:15:10.967234 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.967182    2585 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 01 17:15:10.972407 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.972300    2585 manager.go:216] Machine: {Timestamp:2026-07-01 17:15:10.970680069 +0000 UTC m=+0.375891669 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3134800 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27fe968dcff4de09e0cb8bac442639 SystemUUID:ec27fe96-8dcf-f4de-09e0-cb8bac442639 BootID:6d898a7a-10ef-4b17-978d-5955b37454af Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:05:8d:c9:6f:15 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:05:8d:c9:6f:15 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8e:7e:b5:da:6d:25 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 01 17:15:10.972407 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.972405    2585 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 01 17:15:10.972516 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.972482    2585 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 01 17:15:10.974835 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.974787    2585 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 01 17:15:10.975361 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.974840    2585 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-213.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 01 17:15:10.976178 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.976166    2585 topology_manager.go:138] "Creating topology manager with none policy"
Jul 01 17:15:10.976228 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.976181    2585 container_manager_linux.go:309] "Creating device plugin manager"
Jul 01 17:15:10.976228 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.976189    2585 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 17:15:10.977098 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.977088    2585 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 17:15:10.977323 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.977308    2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 17:15:10.977966 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.977955    2585 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 01 17:15:10.980901 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.980889    2585 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 17:15:10.981011 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.981002    2585 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 01 17:15:10.981046 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.981032    2585 kubelet.go:486] "Attempting to sync node with API server"
Jul 01 17:15:10.981046 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.981042    2585 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 01 17:15:10.981101 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.981052    2585 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 01 17:15:10.981101 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.981061    2585 kubelet.go:398] "Adding apiserver pod source"
Jul 01 17:15:10.981101 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.981069    2585 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 01 17:15:10.981466 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.981455    2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 17:15:10.981503 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.981475    2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 17:15:10.983781 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.983763    2585 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 01 17:15:10.985253 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.985237    2585 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 01 17:15:10.985338 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.985275    2585 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 01 17:15:10.986505 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.986483    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 01 17:15:10.986594 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.986511    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 01 17:15:10.986594 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.986521    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 01 17:15:10.986594 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.986529    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 01 17:15:10.986594 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.986543    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 01 17:15:10.986594 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.986553    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 01 17:15:10.986594 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.986580    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 01 17:15:10.986594 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.986589    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 01 17:15:10.986812 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.986601    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 01 17:15:10.986812 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.986610    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 01 17:15:10.986812 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.986623    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 01 17:15:10.986812 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.986631    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 01 17:15:10.987502 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.987490    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 01 17:15:10.987577 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.987506    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 01 17:15:10.990693 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.990678    2585 server.go:1268] "Started kubelet"
Jul 01 17:15:10.990803 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.990772    2585 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 01 17:15:10.990884 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.990824    2585 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 01 17:15:10.990926 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.990910    2585 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 01 17:15:10.991694 ip-10-0-141-213 systemd[1]: Started Kubernetes Kubelet.
Jul 01 17:15:10.992343 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.992274    2585 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 01 17:15:10.993084 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.993070    2585 server.go:310] "Adding debug handlers to kubelet server"
Jul 01 17:15:10.994973 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.994954    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-213.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 01 17:15:10.995465 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:10.995429    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-213.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 01 17:15:10.995465 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:10.995429    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 01 17:15:10.995588 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:10.995492    2585 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 01 17:15:10.998683 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.998666    2585 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 01 17:15:10.998683 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.998679    2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 01 17:15:10.998882 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:10.998865    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-213.ec2.internal\" not found"
Jul 01 17:15:10.999616 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.999600    2585 factory.go:153] Registering CRI-O factory
Jul 01 17:15:10.999706 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.999621    2585 factory.go:223] Registration of the crio container factory successfully
Jul 01 17:15:10.999706 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.999691    2585 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 01 17:15:10.999706 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.999698    2585 factory.go:55] Registering systemd factory
Jul 01 17:15:10.999706 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.999704    2585 factory.go:223] Registration of the systemd container factory successfully
Jul 01 17:15:10.999877 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.999722    2585 factory.go:103] Registering Raw factory
Jul 01 17:15:10.999877 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.999734    2585 manager.go:1196] Started watching for new ooms in manager
Jul 01 17:15:10.999877 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.999831    2585 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 01 17:15:10.999877 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.999833    2585 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 01 17:15:10.999877 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.999877    2585 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 01 17:15:11.000174 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.999994    2585 reconstruct.go:97] "Volume reconstruction finished"
Jul 01 17:15:11.000174 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:10.999999    2585 reconciler.go:29] "Reconciler: start to sync state"
Jul 01 17:15:11.000174 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.000156    2585 manager.go:349] Starting recovery of all containers
Jul 01 17:15:11.005921 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:11.005885    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 01 17:15:11.006037 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:11.005962    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-213.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 01 17:15:11.010044 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:11.005959    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-213.ec2.internal.18be39da025dd0cd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-213.ec2.internal,UID:ip-10-0-141-213.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-213.ec2.internal,},FirstTimestamp:2026-07-01 17:15:10.990651597 +0000 UTC m=+0.395863197,LastTimestamp:2026-07-01 17:15:10.990651597 +0000 UTC m=+0.395863197,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-213.ec2.internal,}"
Jul 01 17:15:11.013163 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.013146    2585 manager.go:354] Recovery completed
Jul 01 17:15:11.017276 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.017263    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 17:15:11.019328 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.019309    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 17:15:11.019402 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.019341    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 17:15:11.019402 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.019354    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasSufficientPID"
Jul 01 17:15:11.020010 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.019996    2585 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 01 17:15:11.020010 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.020009    2585 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 01 17:15:11.020113 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.020013    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 17:15:11.020113 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.020024    2585 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 17:15:11.020713 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.020698    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 17:15:11.020789 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.020729    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 17:15:11.020789 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.020743    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasSufficientPID"
Jul 01 17:15:11.021463 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.021451    2585 policy_none.go:49] "None policy: Start"
Jul 01 17:15:11.021463 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.021466    2585 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 01 17:15:11.021619 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.021476    2585 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 01 17:15:11.021801 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:11.021683    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-213.ec2.internal.18be39da041357c4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-213.ec2.internal,UID:ip-10-0-141-213.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-213.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-213.ec2.internal,},FirstTimestamp:2026-07-01 17:15:11.01932538 +0000 UTC m=+0.424536981,LastTimestamp:2026-07-01 17:15:11.01932538 +0000 UTC m=+0.424536981,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-213.ec2.internal,}"
Jul 01 17:15:11.022601 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.022589    2585 policy_none.go:47] "Start"
Jul 01 17:15:11.029388 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.029371    2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xtgqx"
Jul 01 17:15:11.030655 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:11.030578    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-213.ec2.internal.18be39da0413b0cd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-213.ec2.internal,UID:ip-10-0-141-213.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-213.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-213.ec2.internal,},FirstTimestamp:2026-07-01 17:15:11.019348173 +0000 UTC m=+0.424559774,LastTimestamp:2026-07-01 17:15:11.019348173 +0000 UTC m=+0.424559774,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-213.ec2.internal,}"
Jul 01 17:15:11.034852 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:11.034788    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-213.ec2.internal.18be39da0413d7d1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-213.ec2.internal,UID:ip-10-0-141-213.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-141-213.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-141-213.ec2.internal,},FirstTimestamp:2026-07-01 17:15:11.019358161 +0000 UTC m=+0.424569763,LastTimestamp:2026-07-01 17:15:11.019358161 +0000 UTC m=+0.424569763,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-213.ec2.internal,}"
Jul 01 17:15:11.036514 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.036498    2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xtgqx"
Jul 01 17:15:11.041237 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:11.041178    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-141-213.ec2.internal.18be39da041357c4\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-213.ec2.internal.18be39da041357c4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-213.ec2.internal,UID:ip-10-0-141-213.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-213.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-213.ec2.internal,},FirstTimestamp:2026-07-01 17:15:11.01932538 +0000 UTC m=+0.424536981,LastTimestamp:2026-07-01 17:15:11.020712693 +0000 UTC m=+0.425924298,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-213.ec2.internal,}"
Jul 01 17:15:11.063296 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.063229    2585 manager.go:337] "Starting Device Plugin manager"
Jul 01 17:15:11.063296 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:11.063271    2585 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 01 17:15:11.063296 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.063286    2585 server.go:87] "Starting device plugin registration server"
Jul 01 17:15:11.071711 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.063581    2585 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 01 17:15:11.071711 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.063635    2585 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 01 17:15:11.071711 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.063809    2585 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 01 17:15:11.071711 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.063889    2585 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 01 17:15:11.071711 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.063896    2585 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 01 17:15:11.071711 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:11.064769    2585 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 01 17:15:11.071711 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:11.064835    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-213.ec2.internal\" not found"
Jul 01 17:15:11.071711 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.065332    2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 01 17:15:11.071711 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.066784    2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 01 17:15:11.071711 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.066826    2585 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 01 17:15:11.071711 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.066854    2585 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 01 17:15:11.071711 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:11.066890    2585 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 01 17:15:11.071711 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.069228    2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 17:15:11.164055 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.164019    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 17:15:11.166944 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.166923    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 17:15:11.167050 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.166926    2585 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-213.ec2.internal"]
Jul 01 17:15:11.167050 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.167046    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 17:15:11.167149 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.166962    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 17:15:11.167149 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.167105    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasSufficientPID"
Jul 01 17:15:11.167149 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.167132    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-213.ec2.internal"
Jul 01 17:15:11.168703 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.168690    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 17:15:11.168759 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.168715    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 17:15:11.168759 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.168727    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasSufficientPID"
Jul 01 17:15:11.170374 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.170312    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 17:15:11.170479 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.170407    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal"
Jul 01 17:15:11.170479 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.170443    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 17:15:11.171831 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.171815    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 17:15:11.171910 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.171819    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 17:15:11.171910 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.171840    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 17:15:11.171910 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.171855    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasSufficientPID"
Jul 01 17:15:11.171910 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.171859    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 17:15:11.171910 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.171871    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasSufficientPID"
Jul 01 17:15:11.173099 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.173083    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-213.ec2.internal"
Jul 01 17:15:11.173187 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.173107    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 17:15:11.173825 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.173812    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 17:15:11.173921 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.173843    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 17:15:11.173921 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.173857    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasSufficientPID"
Jul 01 17:15:11.175829 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.175815    2585 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-213.ec2.internal"
Jul 01 17:15:11.175889 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:11.175834    2585 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-213.ec2.internal\": node \"ip-10-0-141-213.ec2.internal\" not found"
Jul 01 17:15:11.199409 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:11.199384    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-213.ec2.internal\" not found" node="ip-10-0-141-213.ec2.internal"
Jul 01 17:15:11.200964 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.200943    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/749b61ceb1d13c63af40ebe9b4481208-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal\" (UID: \"749b61ceb1d13c63af40ebe9b4481208\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal"
Jul 01 17:15:11.201065 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.200994    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/749b61ceb1d13c63af40ebe9b4481208-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal\" (UID: \"749b61ceb1d13c63af40ebe9b4481208\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal"
Jul 01 17:15:11.201065 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.201019    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/608b014fc4a8976e8619472b298b39c8-config\") pod \"kube-apiserver-proxy-ip-10-0-141-213.ec2.internal\" (UID: \"608b014fc4a8976e8619472b298b39c8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-213.ec2.internal"
Jul 01 17:15:11.201596 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:11.201557    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-213.ec2.internal\" not found"
Jul 01 17:15:11.202966 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:11.202952    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-213.ec2.internal\" not found" node="ip-10-0-141-213.ec2.internal"
Jul 01 17:15:11.301685 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:11.301634    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-213.ec2.internal\" not found"
Jul 01 17:15:11.301685 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.301686    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/749b61ceb1d13c63af40ebe9b4481208-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal\" (UID: \"749b61ceb1d13c63af40ebe9b4481208\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal"
Jul 01 17:15:11.301899 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.301715    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/749b61ceb1d13c63af40ebe9b4481208-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal\" (UID: \"749b61ceb1d13c63af40ebe9b4481208\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal"
Jul 01 17:15:11.301899 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.301771    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/749b61ceb1d13c63af40ebe9b4481208-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal\" (UID: \"749b61ceb1d13c63af40ebe9b4481208\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal"
Jul 01 17:15:11.301899 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.301782    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/749b61ceb1d13c63af40ebe9b4481208-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal\" (UID: \"749b61ceb1d13c63af40ebe9b4481208\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal"
Jul 01 17:15:11.301899 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.301831    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/608b014fc4a8976e8619472b298b39c8-config\") pod \"kube-apiserver-proxy-ip-10-0-141-213.ec2.internal\" (UID: \"608b014fc4a8976e8619472b298b39c8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-213.ec2.internal"
Jul 01 17:15:11.301899 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:11.301855    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/608b014fc4a8976e8619472b298b39c8-config\") pod \"kube-apiserver-proxy-ip-10-0-141-213.ec2.internal\" (UID: \"608b014fc4a8976e8619472b298b39c8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-213.ec2.internal"
Jul 01 17:15:11.402629 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:11.402597    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-213.ec2.internal\" not found"
Jul 01 17:15:12.603346 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:12.603316    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal"
Jul 01 17:15:12.603435 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:12.603315    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-213.ec2.internal\" not found"
Jul 01 17:15:12.606841 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:12.606804    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-213.ec2.internal"
Jul 01 17:15:12.703688 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:12.703655    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-213.ec2.internal\" not found"
Jul 01 17:15:12.804142 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:12.804109    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-213.ec2.internal\" not found"
Jul 01 17:15:12.904637 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:12.904555    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-213.ec2.internal\" not found"
Jul 01 17:15:12.959363 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:12.959334    2585 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 17:15:12.999559 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:12.999527    2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal"
Jul 01 17:15:13.015523 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.015493    2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 17:15:13.018219 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.018195    2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-213.ec2.internal"
Jul 01 17:15:13.028848 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.028827    2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 17:15:13.042956 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.042922    2585 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 01 17:15:13.043143 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.043119    2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 01 17:15:13.081275 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.081248    2585 apiserver.go:52] "Watching apiserver"
Jul 01 17:15:13.091102 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.091073    2585 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 01 17:15:13.091430 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.091409    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-mgc4x","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9zcq","openshift-network-diagnostics/network-check-target-2x2qg","openshift-network-operator/iptables-alerter-kb8gh","openshift-ovn-kubernetes/ovnkube-node-v8tsw","kube-system/kube-apiserver-proxy-ip-10-0-141-213.ec2.internal","openshift-cluster-node-tuning-operator/tuned-v8wlm","openshift-dns/node-resolver-jwvgk","openshift-image-registry/node-ca-r54bn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal","openshift-multus/multus-89p2n","openshift-multus/multus-additional-cni-plugins-hjngx","openshift-multus/network-metrics-daemon-4rrth","kube-system/global-pull-secret-syncer-knvdq"]
Jul 01 17:15:13.092878 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.092857    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.093893 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.093873    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:15:13.093995 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:13.093950    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rrth" podUID="5026d3ed-589a-4390-a2b7-d5f64c06e687"
Jul 01 17:15:13.095038 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.095022    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x2qg"
Jul 01 17:15:13.095121 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:13.095078    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x2qg" podUID="415f427a-adac-47bb-9cad-3befc6a711b6"
Jul 01 17:15:13.095481 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.095462    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pxn68\""
Jul 01 17:15:13.096075 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.096044    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 01 17:15:13.096166 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.096052    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 01 17:15:13.096247 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.096227    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kb8gh"
Jul 01 17:15:13.097678 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.097658    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.098834 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.098815    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 01 17:15:13.098834 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.098832    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 01 17:15:13.099046 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.099030    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knvdq"
Jul 01 17:15:13.099110 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:13.099092    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knvdq" podUID="3defaf9e-f5ef-45ea-a02e-4729f15db90c"
Jul 01 17:15:13.099358 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.099340    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 01 17:15:13.099754 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.099737    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-47q82\""
Jul 01 17:15:13.099821 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.099755    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 01 17:15:13.100303 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.100287    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r54bn"
Jul 01 17:15:13.101533 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.101504    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 01 17:15:13.101648 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.101510    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 01 17:15:13.101710 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.101652    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mgc4x"
Jul 01 17:15:13.101767 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.101725    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 01 17:15:13.101767 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.101746    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 01 17:15:13.101865 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.101798    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 01 17:15:13.102038 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.102025    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 01 17:15:13.102197 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.102183    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qxl8n\""
Jul 01 17:15:13.103213 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.103195    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9zcq"
Jul 01 17:15:13.104600 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.104567    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jwvgk"
Jul 01 17:15:13.105880 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.105862    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4bccceba-0d8e-402a-a82f-accee4a1be82-etc-sysctl-d\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.105982 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.105889    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/500d2a90-2d21-42b3-9584-b04c14becaec-ovnkube-config\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.105982 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.105907    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k6bd\" (UniqueName: \"kubernetes.io/projected/500d2a90-2d21-42b3-9584-b04c14becaec-kube-api-access-2k6bd\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.105982 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.105924    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3defaf9e-f5ef-45ea-a02e-4729f15db90c-dbus\") pod \"global-pull-secret-syncer-knvdq\" (UID: \"3defaf9e-f5ef-45ea-a02e-4729f15db90c\") " pod="kube-system/global-pull-secret-syncer-knvdq"
Jul 01 17:15:13.105982 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.105931    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.105982 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.105939    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d6614097-0891-470c-b830-26bf595ff80e-serviceca\") pod \"node-ca-r54bn\" (UID: \"d6614097-0891-470c-b830-26bf595ff80e\") " pod="openshift-image-registry/node-ca-r54bn"
Jul 01 17:15:13.105982 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.105957    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5db90618-fcdb-4ea6-ac95-a2d535908e0e-konnectivity-ca\") pod \"konnectivity-agent-mgc4x\" (UID: \"5db90618-fcdb-4ea6-ac95-a2d535908e0e\") " pod="kube-system/konnectivity-agent-mgc4x"
Jul 01 17:15:13.106261 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.105995    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-run-openvswitch\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.106261 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106025    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bccceba-0d8e-402a-a82f-accee4a1be82-sys\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.106261 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106049    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4bccceba-0d8e-402a-a82f-accee4a1be82-etc-tuned\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.106261 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106070    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkjw2\" (UniqueName: \"kubernetes.io/projected/4bccceba-0d8e-402a-a82f-accee4a1be82-kube-api-access-rkjw2\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.106261 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106095    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-host-slash\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.106261 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106118    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-host-cni-bin\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.106261 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106152    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.106261 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106199    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3af4c82a-7a73-4de0-8f7b-25768775f79e-iptables-alerter-script\") pod \"iptables-alerter-kb8gh\" (UID: \"3af4c82a-7a73-4de0-8f7b-25768775f79e\") " pod="openshift-network-operator/iptables-alerter-kb8gh"
Jul 01 17:15:13.106261 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106232    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4bccceba-0d8e-402a-a82f-accee4a1be82-run\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.106261 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106258    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bccceba-0d8e-402a-a82f-accee4a1be82-var-lib-kubelet\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.106722 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106281    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-host-run-netns\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.106722 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106302    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-node-log\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.106722 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106338    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d6614097-0891-470c-b830-26bf595ff80e-host\") pod \"node-ca-r54bn\" (UID: \"d6614097-0891-470c-b830-26bf595ff80e\") " pod="openshift-image-registry/node-ca-r54bn"
Jul 01 17:15:13.106722 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106367    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bccceba-0d8e-402a-a82f-accee4a1be82-etc-kubernetes\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.106722 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106397    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f6dj\" (UniqueName: \"kubernetes.io/projected/415f427a-adac-47bb-9cad-3befc6a711b6-kube-api-access-7f6dj\") pod \"network-check-target-2x2qg\" (UID: \"415f427a-adac-47bb-9cad-3befc6a711b6\") " pod="openshift-network-diagnostics/network-check-target-2x2qg"
Jul 01 17:15:13.106722 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106422    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-systemd-units\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.106722 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106439    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 01 17:15:13.106722 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106444    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-584mt\" (UniqueName: \"kubernetes.io/projected/d6614097-0891-470c-b830-26bf595ff80e-kube-api-access-584mt\") pod \"node-ca-r54bn\" (UID: \"d6614097-0891-470c-b830-26bf595ff80e\") " pod="openshift-image-registry/node-ca-r54bn"
Jul 01 17:15:13.106722 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106474    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 01 17:15:13.106722 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106562    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3af4c82a-7a73-4de0-8f7b-25768775f79e-host-slash\") pod \"iptables-alerter-kb8gh\" (UID: \"3af4c82a-7a73-4de0-8f7b-25768775f79e\") " pod="openshift-network-operator/iptables-alerter-kb8gh"
Jul 01 17:15:13.106722 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106613    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5026d3ed-589a-4390-a2b7-d5f64c06e687-metrics-certs\") pod \"network-metrics-daemon-4rrth\" (UID: \"5026d3ed-589a-4390-a2b7-d5f64c06e687\") " pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:15:13.106722 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106639    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-host-kubelet\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.106722 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106661    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-host-run-ovn-kubernetes\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.106722 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106684    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3defaf9e-f5ef-45ea-a02e-4729f15db90c-kubelet-config\") pod \"global-pull-secret-syncer-knvdq\" (UID: \"3defaf9e-f5ef-45ea-a02e-4729f15db90c\") " pod="kube-system/global-pull-secret-syncer-knvdq"
Jul 01 17:15:13.106722 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106712    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3defaf9e-f5ef-45ea-a02e-4729f15db90c-original-pull-secret\") pod \"global-pull-secret-syncer-knvdq\" (UID: \"3defaf9e-f5ef-45ea-a02e-4729f15db90c\") " pod="kube-system/global-pull-secret-syncer-knvdq"
Jul 01 17:15:13.107395 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106735    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z44vc\" (UniqueName: \"kubernetes.io/projected/3af4c82a-7a73-4de0-8f7b-25768775f79e-kube-api-access-z44vc\") pod \"iptables-alerter-kb8gh\" (UID: \"3af4c82a-7a73-4de0-8f7b-25768775f79e\") " pod="openshift-network-operator/iptables-alerter-kb8gh"
Jul 01 17:15:13.107395 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106760    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4bccceba-0d8e-402a-a82f-accee4a1be82-etc-sysconfig\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.107395 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106780    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4bccceba-0d8e-402a-a82f-accee4a1be82-etc-sysctl-conf\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.107395 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106803    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4bccceba-0d8e-402a-a82f-accee4a1be82-etc-systemd\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.107395 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106826    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4bccceba-0d8e-402a-a82f-accee4a1be82-host\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.107395 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106844    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bccceba-0d8e-402a-a82f-accee4a1be82-tmp\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.107395 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106867    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 01 17:15:13.107395 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106873    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-run-systemd\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.107395 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106896    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-var-lib-openvswitch\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.107395 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106974    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/500d2a90-2d21-42b3-9584-b04c14becaec-ovn-node-metrics-cert\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.107395 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106783    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 01 17:15:13.107395 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106986    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pbzgr\""
Jul 01 17:15:13.107395 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.106849    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 01 17:15:13.107395 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.107032    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4bccceba-0d8e-402a-a82f-accee4a1be82-etc-modprobe-d\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.107395 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.107083    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvjzz\" (UniqueName: \"kubernetes.io/projected/5026d3ed-589a-4390-a2b7-d5f64c06e687-kube-api-access-qvjzz\") pod \"network-metrics-daemon-4rrth\" (UID: \"5026d3ed-589a-4390-a2b7-d5f64c06e687\") " pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:15:13.107395 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.107108    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-run-ovn\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.107395 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.107131    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 01 17:15:13.107395 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.107205    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 01 17:15:13.107395 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.107133    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-host-cni-netd\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.107395 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.107246    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/500d2a90-2d21-42b3-9584-b04c14becaec-ovnkube-script-lib\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.108364 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.107273    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 01 17:15:13.108364 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.107274    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4bccceba-0d8e-402a-a82f-accee4a1be82-lib-modules\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.108364 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.107317    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-etc-openvswitch\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.108364 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.107333    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gcgqx\""
Jul 01 17:15:13.108364 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.107338    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-log-socket\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.108364 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.107704    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/500d2a90-2d21-42b3-9584-b04c14becaec-env-overrides\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.108364 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.107838    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5db90618-fcdb-4ea6-ac95-a2d535908e0e-agent-certs\") pod \"konnectivity-agent-mgc4x\" (UID: \"5db90618-fcdb-4ea6-ac95-a2d535908e0e\") " pod="kube-system/konnectivity-agent-mgc4x"
Jul 01 17:15:13.108364 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.107960    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-55nkr\""
Jul 01 17:15:13.108364 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.108252    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 01 17:15:13.108796 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.108370    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 01 17:15:13.108796 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.108421    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 01 17:15:13.108796 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.108489    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7j995\""
Jul 01 17:15:13.109147 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.109129    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hjngx"
Jul 01 17:15:13.110847 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.110825    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 01 17:15:13.111071 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.110868    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 01 17:15:13.111071 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.110949    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 01 17:15:13.111218 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.111103    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hbk4m\""
Jul 01 17:15:13.112985 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.112967    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 01 17:15:13.114299 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.114282    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 01 17:15:13.114418 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.114331    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nfqvz\""
Jul 01 17:15:13.118005 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.117987    2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 17:15:13.141414 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.141360    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-30 17:10:12 +0000 UTC" deadline="2028-03-26 22:54:57.212884044 +0000 UTC"
Jul 01 17:15:13.141503 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.141416    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15221h39m44.071472625s"
Jul 01 17:15:13.151689 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.151666    2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wh8pm"
Jul 01 17:15:13.159077 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.159057    2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wh8pm"
Jul 01 17:15:13.200820 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.200797    2585 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 01 17:15:13.208241 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.208216    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/388a1ebd-458e-4573-8242-b4da8bb8a77a-socket-dir\") pod \"aws-ebs-csi-driver-node-b9zcq\" (UID: \"388a1ebd-458e-4573-8242-b4da8bb8a77a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9zcq"
Jul 01 17:15:13.208367 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.208249    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ff59413-27fa-4837-8549-f2a6898826a9-tmp-dir\") pod \"node-resolver-jwvgk\" (UID: \"3ff59413-27fa-4837-8549-f2a6898826a9\") " pod="openshift-dns/node-resolver-jwvgk"
Jul 01 17:15:13.208367 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.208273    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ebd82416-a8de-490c-a980-74946452dc40-cni-binary-copy\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.208367 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.208298    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4bccceba-0d8e-402a-a82f-accee4a1be82-host\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.208367 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.208319    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-run-systemd\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.208367 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.208340    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-var-lib-openvswitch\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.208367 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.208360    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/388a1ebd-458e-4573-8242-b4da8bb8a77a-device-dir\") pod \"aws-ebs-csi-driver-node-b9zcq\" (UID: \"388a1ebd-458e-4573-8242-b4da8bb8a77a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9zcq"
Jul 01 17:15:13.208653 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.208383    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3ff59413-27fa-4837-8549-f2a6898826a9-hosts-file\") pod \"node-resolver-jwvgk\" (UID: \"3ff59413-27fa-4837-8549-f2a6898826a9\") " pod="openshift-dns/node-resolver-jwvgk"
Jul 01 17:15:13.208653 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.208405    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-hostroot\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.208653 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.208401    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-var-lib-openvswitch\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.208653 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.208420    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4bccceba-0d8e-402a-a82f-accee4a1be82-host\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.208653 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.208403    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-run-systemd\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.208653 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.208447    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zf6q\" (UniqueName: \"kubernetes.io/projected/ebd82416-a8de-490c-a980-74946452dc40-kube-api-access-9zf6q\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.208653 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.208474    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7bab8af3-8766-4098-95ab-720089120d08-cnibin\") pod \"multus-additional-cni-plugins-hjngx\" (UID: \"7bab8af3-8766-4098-95ab-720089120d08\") " pod="openshift-multus/multus-additional-cni-plugins-hjngx"
Jul 01 17:15:13.208653 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.208513    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4bccceba-0d8e-402a-a82f-accee4a1be82-etc-modprobe-d\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.208653 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.208593    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-run-ovn\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.208653 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.208619    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-host-run-netns\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.208653 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.208634    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-run-ovn\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.208653 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.208636    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7bab8af3-8766-4098-95ab-720089120d08-os-release\") pod \"multus-additional-cni-plugins-hjngx\" (UID: \"7bab8af3-8766-4098-95ab-720089120d08\") " pod="openshift-multus/multus-additional-cni-plugins-hjngx"
Jul 01 17:15:13.209163 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.208659    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4bccceba-0d8e-402a-a82f-accee4a1be82-etc-modprobe-d\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.209163 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.208672    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7bab8af3-8766-4098-95ab-720089120d08-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hjngx\" (UID: \"7bab8af3-8766-4098-95ab-720089120d08\") " pod="openshift-multus/multus-additional-cni-plugins-hjngx"
Jul 01 17:15:13.209163 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.208699    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjnqd\" (UniqueName: \"kubernetes.io/projected/7bab8af3-8766-4098-95ab-720089120d08-kube-api-access-pjnqd\") pod \"multus-additional-cni-plugins-hjngx\" (UID: \"7bab8af3-8766-4098-95ab-720089120d08\") " pod="openshift-multus/multus-additional-cni-plugins-hjngx"
Jul 01 17:15:13.209163 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.208717    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-log-socket\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.209163 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.208735    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/500d2a90-2d21-42b3-9584-b04c14becaec-env-overrides\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.209163 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.208778    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7bab8af3-8766-4098-95ab-720089120d08-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hjngx\" (UID: \"7bab8af3-8766-4098-95ab-720089120d08\") " pod="openshift-multus/multus-additional-cni-plugins-hjngx"
Jul 01 17:15:13.209163 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.208781    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-log-socket\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.209163 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.208813    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4bccceba-0d8e-402a-a82f-accee4a1be82-etc-sysctl-d\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.209163 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.208838    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/500d2a90-2d21-42b3-9584-b04c14becaec-ovnkube-config\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.209163 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.208859    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k6bd\" (UniqueName: \"kubernetes.io/projected/500d2a90-2d21-42b3-9584-b04c14becaec-kube-api-access-2k6bd\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.209163 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.208882    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-host-var-lib-cni-multus\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.209163 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.208910    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7bab8af3-8766-4098-95ab-720089120d08-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hjngx\" (UID: \"7bab8af3-8766-4098-95ab-720089120d08\") " pod="openshift-multus/multus-additional-cni-plugins-hjngx"
Jul 01 17:15:13.209163 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.208935    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-run-openvswitch\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.209163 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.208959    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bccceba-0d8e-402a-a82f-accee4a1be82-sys\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.209163 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.208982    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4bccceba-0d8e-402a-a82f-accee4a1be82-etc-tuned\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.209163 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209005    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-host-slash\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.209163 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209014    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4bccceba-0d8e-402a-a82f-accee4a1be82-etc-sysctl-d\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.209882 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209040    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-run-openvswitch\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.209882 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209054    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.209882 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209092    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-host-slash\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.209882 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209101    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bccceba-0d8e-402a-a82f-accee4a1be82-sys\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.209882 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209104    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-system-cni-dir\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.209882 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209147    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.209882 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209153    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-host-run-k8s-cni-cncf-io\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.209882 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209198    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7bab8af3-8766-4098-95ab-720089120d08-cni-binary-copy\") pod \"multus-additional-cni-plugins-hjngx\" (UID: \"7bab8af3-8766-4098-95ab-720089120d08\") " pod="openshift-multus/multus-additional-cni-plugins-hjngx"
Jul 01 17:15:13.209882 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209214    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/500d2a90-2d21-42b3-9584-b04c14becaec-env-overrides\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.209882 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209223    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4bccceba-0d8e-402a-a82f-accee4a1be82-run\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.209882 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209248    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-host-run-netns\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.209882 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209276    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-host-var-lib-cni-bin\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.209882 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209299    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4bccceba-0d8e-402a-a82f-accee4a1be82-run\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.209882 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209307    2585 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 01 17:15:13.209882 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209318    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-host-run-netns\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.209882 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209337    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bccceba-0d8e-402a-a82f-accee4a1be82-etc-kubernetes\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.209882 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209364    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-systemd-units\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.209882 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209411    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-systemd-units\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.210743 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209424    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bccceba-0d8e-402a-a82f-accee4a1be82-etc-kubernetes\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.210743 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209452    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3af4c82a-7a73-4de0-8f7b-25768775f79e-host-slash\") pod \"iptables-alerter-kb8gh\" (UID: \"3af4c82a-7a73-4de0-8f7b-25768775f79e\") " pod="openshift-network-operator/iptables-alerter-kb8gh"
Jul 01 17:15:13.210743 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209477    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-etc-kubernetes\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.210743 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209503    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3defaf9e-f5ef-45ea-a02e-4729f15db90c-kubelet-config\") pod \"global-pull-secret-syncer-knvdq\" (UID: \"3defaf9e-f5ef-45ea-a02e-4729f15db90c\") " pod="kube-system/global-pull-secret-syncer-knvdq"
Jul 01 17:15:13.210743 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209554    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3af4c82a-7a73-4de0-8f7b-25768775f79e-host-slash\") pod \"iptables-alerter-kb8gh\" (UID: \"3af4c82a-7a73-4de0-8f7b-25768775f79e\") " pod="openshift-network-operator/iptables-alerter-kb8gh"
Jul 01 17:15:13.210743 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209570    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3defaf9e-f5ef-45ea-a02e-4729f15db90c-kubelet-config\") pod \"global-pull-secret-syncer-knvdq\" (UID: \"3defaf9e-f5ef-45ea-a02e-4729f15db90c\") " pod="kube-system/global-pull-secret-syncer-knvdq"
Jul 01 17:15:13.210743 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209602    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z44vc\" (UniqueName: \"kubernetes.io/projected/3af4c82a-7a73-4de0-8f7b-25768775f79e-kube-api-access-z44vc\") pod \"iptables-alerter-kb8gh\" (UID: \"3af4c82a-7a73-4de0-8f7b-25768775f79e\") " pod="openshift-network-operator/iptables-alerter-kb8gh"
Jul 01 17:15:13.210743 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209633    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/500d2a90-2d21-42b3-9584-b04c14becaec-ovnkube-config\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.210743 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209686    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-592jn\" (UniqueName: \"kubernetes.io/projected/388a1ebd-458e-4573-8242-b4da8bb8a77a-kube-api-access-592jn\") pod \"aws-ebs-csi-driver-node-b9zcq\" (UID: \"388a1ebd-458e-4573-8242-b4da8bb8a77a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9zcq"
Jul 01 17:15:13.210743 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209775    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-os-release\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.210743 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209793    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-host-run-multus-certs\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.210743 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209837    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4bccceba-0d8e-402a-a82f-accee4a1be82-etc-sysconfig\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.210743 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209866    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4bccceba-0d8e-402a-a82f-accee4a1be82-etc-sysctl-conf\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.210743 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209889    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4bccceba-0d8e-402a-a82f-accee4a1be82-etc-systemd\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.210743 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209912    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bccceba-0d8e-402a-a82f-accee4a1be82-tmp\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.210743 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209913    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4bccceba-0d8e-402a-a82f-accee4a1be82-etc-sysconfig\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.210743 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209952    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4bccceba-0d8e-402a-a82f-accee4a1be82-etc-systemd\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.211533 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209937    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/500d2a90-2d21-42b3-9584-b04c14becaec-ovn-node-metrics-cert\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.211533 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.209992    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/388a1ebd-458e-4573-8242-b4da8bb8a77a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b9zcq\" (UID: \"388a1ebd-458e-4573-8242-b4da8bb8a77a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9zcq"
Jul 01 17:15:13.211533 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210022    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4bccceba-0d8e-402a-a82f-accee4a1be82-etc-sysctl-conf\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.211533 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210016    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-multus-socket-dir-parent\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.211533 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210058    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvjzz\" (UniqueName: \"kubernetes.io/projected/5026d3ed-589a-4390-a2b7-d5f64c06e687-kube-api-access-qvjzz\") pod \"network-metrics-daemon-4rrth\" (UID: \"5026d3ed-589a-4390-a2b7-d5f64c06e687\") " pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:15:13.211533 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210074    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-host-cni-netd\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.211533 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210089    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/500d2a90-2d21-42b3-9584-b04c14becaec-ovnkube-script-lib\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.211533 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210107    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/388a1ebd-458e-4573-8242-b4da8bb8a77a-etc-selinux\") pod \"aws-ebs-csi-driver-node-b9zcq\" (UID: \"388a1ebd-458e-4573-8242-b4da8bb8a77a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9zcq"
Jul 01 17:15:13.211533 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210140    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-host-cni-netd\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.211533 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210148    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4bccceba-0d8e-402a-a82f-accee4a1be82-lib-modules\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.211533 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210175    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-etc-openvswitch\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.211533 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210203    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5db90618-fcdb-4ea6-ac95-a2d535908e0e-agent-certs\") pod \"konnectivity-agent-mgc4x\" (UID: \"5db90618-fcdb-4ea6-ac95-a2d535908e0e\") " pod="kube-system/konnectivity-agent-mgc4x"
Jul 01 17:15:13.211533 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210226    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5dhq\" (UniqueName: \"kubernetes.io/projected/3ff59413-27fa-4837-8549-f2a6898826a9-kube-api-access-w5dhq\") pod \"node-resolver-jwvgk\" (UID: \"3ff59413-27fa-4837-8549-f2a6898826a9\") " pod="openshift-dns/node-resolver-jwvgk"
Jul 01 17:15:13.211533 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210249    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-cnibin\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.211533 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210255    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4bccceba-0d8e-402a-a82f-accee4a1be82-lib-modules\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.211533 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210270    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3defaf9e-f5ef-45ea-a02e-4729f15db90c-dbus\") pod \"global-pull-secret-syncer-knvdq\" (UID: \"3defaf9e-f5ef-45ea-a02e-4729f15db90c\") " pod="kube-system/global-pull-secret-syncer-knvdq"
Jul 01 17:15:13.211533 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210294    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d6614097-0891-470c-b830-26bf595ff80e-serviceca\") pod \"node-ca-r54bn\" (UID: \"d6614097-0891-470c-b830-26bf595ff80e\") " pod="openshift-image-registry/node-ca-r54bn"
Jul 01 17:15:13.212742 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210309    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-etc-openvswitch\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.212742 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210332    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5db90618-fcdb-4ea6-ac95-a2d535908e0e-konnectivity-ca\") pod \"konnectivity-agent-mgc4x\" (UID: \"5db90618-fcdb-4ea6-ac95-a2d535908e0e\") " pod="kube-system/konnectivity-agent-mgc4x"
Jul 01 17:15:13.212742 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210363    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/388a1ebd-458e-4573-8242-b4da8bb8a77a-sys-fs\") pod \"aws-ebs-csi-driver-node-b9zcq\" (UID: \"388a1ebd-458e-4573-8242-b4da8bb8a77a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9zcq"
Jul 01 17:15:13.212742 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210390    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bab8af3-8766-4098-95ab-720089120d08-system-cni-dir\") pod \"multus-additional-cni-plugins-hjngx\" (UID: \"7bab8af3-8766-4098-95ab-720089120d08\") " pod="openshift-multus/multus-additional-cni-plugins-hjngx"
Jul 01 17:15:13.212742 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210409    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3defaf9e-f5ef-45ea-a02e-4729f15db90c-dbus\") pod \"global-pull-secret-syncer-knvdq\" (UID: \"3defaf9e-f5ef-45ea-a02e-4729f15db90c\") " pod="kube-system/global-pull-secret-syncer-knvdq"
Jul 01 17:15:13.212742 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210417    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkjw2\" (UniqueName: \"kubernetes.io/projected/4bccceba-0d8e-402a-a82f-accee4a1be82-kube-api-access-rkjw2\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.212742 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210448    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-host-cni-bin\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.212742 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210476    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3af4c82a-7a73-4de0-8f7b-25768775f79e-iptables-alerter-script\") pod \"iptables-alerter-kb8gh\" (UID: \"3af4c82a-7a73-4de0-8f7b-25768775f79e\") " pod="openshift-network-operator/iptables-alerter-kb8gh"
Jul 01 17:15:13.212742 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210512    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-multus-cni-dir\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.212742 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210539    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-multus-conf-dir\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.212742 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210564    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ebd82416-a8de-490c-a980-74946452dc40-multus-daemon-config\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.212742 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210611    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bccceba-0d8e-402a-a82f-accee4a1be82-var-lib-kubelet\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.212742 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210643    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/500d2a90-2d21-42b3-9584-b04c14becaec-ovnkube-script-lib\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.212742 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210639    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-node-log\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.212742 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210712    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d6614097-0891-470c-b830-26bf595ff80e-serviceca\") pod \"node-ca-r54bn\" (UID: \"d6614097-0891-470c-b830-26bf595ff80e\") " pod="openshift-image-registry/node-ca-r54bn"
Jul 01 17:15:13.212742 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210716    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-host-cni-bin\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.212742 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210743    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d6614097-0891-470c-b830-26bf595ff80e-host\") pod \"node-ca-r54bn\" (UID: \"d6614097-0891-470c-b830-26bf595ff80e\") " pod="openshift-image-registry/node-ca-r54bn"
Jul 01 17:15:13.213591 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210771    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-host-var-lib-kubelet\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.213591 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210806    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-node-log\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.213591 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210812    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f6dj\" (UniqueName: \"kubernetes.io/projected/415f427a-adac-47bb-9cad-3befc6a711b6-kube-api-access-7f6dj\") pod \"network-check-target-2x2qg\" (UID: \"415f427a-adac-47bb-9cad-3befc6a711b6\") " pod="openshift-network-diagnostics/network-check-target-2x2qg"
Jul 01 17:15:13.213591 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210879    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-584mt\" (UniqueName: \"kubernetes.io/projected/d6614097-0891-470c-b830-26bf595ff80e-kube-api-access-584mt\") pod \"node-ca-r54bn\" (UID: \"d6614097-0891-470c-b830-26bf595ff80e\") " pod="openshift-image-registry/node-ca-r54bn"
Jul 01 17:15:13.213591 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210908    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/388a1ebd-458e-4573-8242-b4da8bb8a77a-registration-dir\") pod \"aws-ebs-csi-driver-node-b9zcq\" (UID: \"388a1ebd-458e-4573-8242-b4da8bb8a77a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9zcq"
Jul 01 17:15:13.213591 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210936    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5026d3ed-589a-4390-a2b7-d5f64c06e687-metrics-certs\") pod \"network-metrics-daemon-4rrth\" (UID: \"5026d3ed-589a-4390-a2b7-d5f64c06e687\") " pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:15:13.213591 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.210963    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bccceba-0d8e-402a-a82f-accee4a1be82-var-lib-kubelet\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.213591 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.211013    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-host-kubelet\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.213591 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.211041    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-host-run-ovn-kubernetes\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.213591 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:13.211059    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 17:15:13.213591 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.211067    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3defaf9e-f5ef-45ea-a02e-4729f15db90c-original-pull-secret\") pod \"global-pull-secret-syncer-knvdq\" (UID: \"3defaf9e-f5ef-45ea-a02e-4729f15db90c\") " pod="kube-system/global-pull-secret-syncer-knvdq"
Jul 01 17:15:13.213591 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:13.211160    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5026d3ed-589a-4390-a2b7-d5f64c06e687-metrics-certs podName:5026d3ed-589a-4390-a2b7-d5f64c06e687 nodeName:}" failed. No retries permitted until 2026-07-01 17:15:13.711139764 +0000 UTC m=+2.016337920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5026d3ed-589a-4390-a2b7-d5f64c06e687-metrics-certs") pod "network-metrics-daemon-4rrth" (UID: "5026d3ed-589a-4390-a2b7-d5f64c06e687") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 17:15:13.213591 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.211199    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5db90618-fcdb-4ea6-ac95-a2d535908e0e-konnectivity-ca\") pod \"konnectivity-agent-mgc4x\" (UID: \"5db90618-fcdb-4ea6-ac95-a2d535908e0e\") " pod="kube-system/konnectivity-agent-mgc4x"
Jul 01 17:15:13.213591 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.211202    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d6614097-0891-470c-b830-26bf595ff80e-host\") pod \"node-ca-r54bn\" (UID: \"d6614097-0891-470c-b830-26bf595ff80e\") " pod="openshift-image-registry/node-ca-r54bn"
Jul 01 17:15:13.213591 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.211259    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-host-kubelet\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.213591 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.211341    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/500d2a90-2d21-42b3-9584-b04c14becaec-host-run-ovn-kubernetes\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.213591 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:13.211794    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 17:15:13.214296 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:13.211880    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3defaf9e-f5ef-45ea-a02e-4729f15db90c-original-pull-secret podName:3defaf9e-f5ef-45ea-a02e-4729f15db90c nodeName:}" failed. No retries permitted until 2026-07-01 17:15:13.711863394 +0000 UTC m=+2.017061552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3defaf9e-f5ef-45ea-a02e-4729f15db90c-original-pull-secret") pod "global-pull-secret-syncer-knvdq" (UID: "3defaf9e-f5ef-45ea-a02e-4729f15db90c") : object "kube-system"/"original-pull-secret" not registered
Jul 01 17:15:13.214296 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.212059    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3af4c82a-7a73-4de0-8f7b-25768775f79e-iptables-alerter-script\") pod \"iptables-alerter-kb8gh\" (UID: \"3af4c82a-7a73-4de0-8f7b-25768775f79e\") " pod="openshift-network-operator/iptables-alerter-kb8gh"
Jul 01 17:15:13.214296 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.213226    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4bccceba-0d8e-402a-a82f-accee4a1be82-etc-tuned\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.214296 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.213280    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bccceba-0d8e-402a-a82f-accee4a1be82-tmp\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.214296 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.213475    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5db90618-fcdb-4ea6-ac95-a2d535908e0e-agent-certs\") pod \"konnectivity-agent-mgc4x\" (UID: \"5db90618-fcdb-4ea6-ac95-a2d535908e0e\") " pod="kube-system/konnectivity-agent-mgc4x"
Jul 01 17:15:13.214296 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.214279    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/500d2a90-2d21-42b3-9584-b04c14becaec-ovn-node-metrics-cert\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.220176 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:13.220065    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 17:15:13.220176 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:13.220096    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 17:15:13.220176 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:13.220110    2585 projected.go:196] Error preparing data for projected volume kube-api-access-7f6dj for pod openshift-network-diagnostics/network-check-target-2x2qg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 17:15:13.220374 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:13.220184    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/415f427a-adac-47bb-9cad-3befc6a711b6-kube-api-access-7f6dj podName:415f427a-adac-47bb-9cad-3befc6a711b6 nodeName:}" failed. No retries permitted until 2026-07-01 17:15:13.720159729 +0000 UTC m=+2.025357897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7f6dj" (UniqueName: "kubernetes.io/projected/415f427a-adac-47bb-9cad-3befc6a711b6-kube-api-access-7f6dj") pod "network-check-target-2x2qg" (UID: "415f427a-adac-47bb-9cad-3befc6a711b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 17:15:13.221464 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.221439    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvjzz\" (UniqueName: \"kubernetes.io/projected/5026d3ed-589a-4390-a2b7-d5f64c06e687-kube-api-access-qvjzz\") pod \"network-metrics-daemon-4rrth\" (UID: \"5026d3ed-589a-4390-a2b7-d5f64c06e687\") " pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:15:13.221841 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.221821    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z44vc\" (UniqueName: \"kubernetes.io/projected/3af4c82a-7a73-4de0-8f7b-25768775f79e-kube-api-access-z44vc\") pod \"iptables-alerter-kb8gh\" (UID: \"3af4c82a-7a73-4de0-8f7b-25768775f79e\") " pod="openshift-network-operator/iptables-alerter-kb8gh"
Jul 01 17:15:13.222147 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.222129    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-584mt\" (UniqueName: \"kubernetes.io/projected/d6614097-0891-470c-b830-26bf595ff80e-kube-api-access-584mt\") pod \"node-ca-r54bn\" (UID: \"d6614097-0891-470c-b830-26bf595ff80e\") " pod="openshift-image-registry/node-ca-r54bn"
Jul 01 17:15:13.222621 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.222600    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k6bd\" (UniqueName: \"kubernetes.io/projected/500d2a90-2d21-42b3-9584-b04c14becaec-kube-api-access-2k6bd\") pod \"ovnkube-node-v8tsw\" (UID: \"500d2a90-2d21-42b3-9584-b04c14becaec\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.222743 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.222722    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkjw2\" (UniqueName: \"kubernetes.io/projected/4bccceba-0d8e-402a-a82f-accee4a1be82-kube-api-access-rkjw2\") pod \"tuned-v8wlm\" (UID: \"4bccceba-0d8e-402a-a82f-accee4a1be82\") " pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.250903 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.250882    2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 17:15:13.256347 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:13.256319    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod749b61ceb1d13c63af40ebe9b4481208.slice/crio-a50c704241ad61348015d5b265769ba5b3387b9c09f8dc39605cce7fd16798a0 WatchSource:0}: Error finding container a50c704241ad61348015d5b265769ba5b3387b9c09f8dc39605cce7fd16798a0: Status 404 returned error can't find the container with id a50c704241ad61348015d5b265769ba5b3387b9c09f8dc39605cce7fd16798a0
Jul 01 17:15:13.256605 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:13.256567    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod608b014fc4a8976e8619472b298b39c8.slice/crio-86be634731e515017557cfb483cff52d8992ac7982578c4dfcf7f36a8ed76272 WatchSource:0}: Error finding container 86be634731e515017557cfb483cff52d8992ac7982578c4dfcf7f36a8ed76272: Status 404 returned error can't find the container with id 86be634731e515017557cfb483cff52d8992ac7982578c4dfcf7f36a8ed76272
Jul 01 17:15:13.261313 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.261297    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 17:15:13.278451 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.278430    2585 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 17:15:13.311466 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.311439    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-multus-cni-dir\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.311655 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.311474    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-multus-conf-dir\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.311655 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.311490    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ebd82416-a8de-490c-a980-74946452dc40-multus-daemon-config\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.311655 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.311511    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-host-var-lib-kubelet\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.311655 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.311555    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-host-var-lib-kubelet\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.311655 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.311555    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-multus-conf-dir\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.311655 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.311564    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-multus-cni-dir\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.311655 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.311641    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/388a1ebd-458e-4573-8242-b4da8bb8a77a-registration-dir\") pod \"aws-ebs-csi-driver-node-b9zcq\" (UID: \"388a1ebd-458e-4573-8242-b4da8bb8a77a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9zcq"
Jul 01 17:15:13.312006 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.311689    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/388a1ebd-458e-4573-8242-b4da8bb8a77a-socket-dir\") pod \"aws-ebs-csi-driver-node-b9zcq\" (UID: \"388a1ebd-458e-4573-8242-b4da8bb8a77a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9zcq"
Jul 01 17:15:13.312006 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.311800    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ff59413-27fa-4837-8549-f2a6898826a9-tmp-dir\") pod \"node-resolver-jwvgk\" (UID: \"3ff59413-27fa-4837-8549-f2a6898826a9\") " pod="openshift-dns/node-resolver-jwvgk"
Jul 01 17:15:13.312006 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.311812    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/388a1ebd-458e-4573-8242-b4da8bb8a77a-socket-dir\") pod \"aws-ebs-csi-driver-node-b9zcq\" (UID: \"388a1ebd-458e-4573-8242-b4da8bb8a77a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9zcq"
Jul 01 17:15:13.312006 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.311816    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/388a1ebd-458e-4573-8242-b4da8bb8a77a-registration-dir\") pod \"aws-ebs-csi-driver-node-b9zcq\" (UID: \"388a1ebd-458e-4573-8242-b4da8bb8a77a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9zcq"
Jul 01 17:15:13.312006 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.311828    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ebd82416-a8de-490c-a980-74946452dc40-cni-binary-copy\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.312006 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.311855    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/388a1ebd-458e-4573-8242-b4da8bb8a77a-device-dir\") pod \"aws-ebs-csi-driver-node-b9zcq\" (UID: \"388a1ebd-458e-4573-8242-b4da8bb8a77a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9zcq"
Jul 01 17:15:13.312006 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.311871    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3ff59413-27fa-4837-8549-f2a6898826a9-hosts-file\") pod \"node-resolver-jwvgk\" (UID: \"3ff59413-27fa-4837-8549-f2a6898826a9\") " pod="openshift-dns/node-resolver-jwvgk"
Jul 01 17:15:13.312006 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.311886    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-hostroot\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.312006 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.311902    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zf6q\" (UniqueName: \"kubernetes.io/projected/ebd82416-a8de-490c-a980-74946452dc40-kube-api-access-9zf6q\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.312006 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.311918    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7bab8af3-8766-4098-95ab-720089120d08-cnibin\") pod \"multus-additional-cni-plugins-hjngx\" (UID: \"7bab8af3-8766-4098-95ab-720089120d08\") " pod="openshift-multus/multus-additional-cni-plugins-hjngx"
Jul 01 17:15:13.312006 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.311938    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-host-run-netns\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.312006 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.311961    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7bab8af3-8766-4098-95ab-720089120d08-os-release\") pod \"multus-additional-cni-plugins-hjngx\" (UID: \"7bab8af3-8766-4098-95ab-720089120d08\") " pod="openshift-multus/multus-additional-cni-plugins-hjngx"
Jul 01 17:15:13.312006 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.311972    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-hostroot\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.312006 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.311976    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3ff59413-27fa-4837-8549-f2a6898826a9-hosts-file\") pod \"node-resolver-jwvgk\" (UID: \"3ff59413-27fa-4837-8549-f2a6898826a9\") " pod="openshift-dns/node-resolver-jwvgk"
Jul 01 17:15:13.312006 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.311991    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7bab8af3-8766-4098-95ab-720089120d08-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hjngx\" (UID: \"7bab8af3-8766-4098-95ab-720089120d08\") " pod="openshift-multus/multus-additional-cni-plugins-hjngx"
Jul 01 17:15:13.312781 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312039    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjnqd\" (UniqueName: \"kubernetes.io/projected/7bab8af3-8766-4098-95ab-720089120d08-kube-api-access-pjnqd\") pod \"multus-additional-cni-plugins-hjngx\" (UID: \"7bab8af3-8766-4098-95ab-720089120d08\") " pod="openshift-multus/multus-additional-cni-plugins-hjngx"
Jul 01 17:15:13.312781 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312041    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ebd82416-a8de-490c-a980-74946452dc40-multus-daemon-config\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.312781 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312077    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-host-run-netns\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.312781 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312113    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7bab8af3-8766-4098-95ab-720089120d08-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hjngx\" (UID: \"7bab8af3-8766-4098-95ab-720089120d08\") " pod="openshift-multus/multus-additional-cni-plugins-hjngx"
Jul 01 17:15:13.312781 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312124    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ff59413-27fa-4837-8549-f2a6898826a9-tmp-dir\") pod \"node-resolver-jwvgk\" (UID: \"3ff59413-27fa-4837-8549-f2a6898826a9\") " pod="openshift-dns/node-resolver-jwvgk"
Jul 01 17:15:13.312781 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312136    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-host-var-lib-cni-multus\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.312781 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312151    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7bab8af3-8766-4098-95ab-720089120d08-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hjngx\" (UID: \"7bab8af3-8766-4098-95ab-720089120d08\") " pod="openshift-multus/multus-additional-cni-plugins-hjngx"
Jul 01 17:15:13.312781 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312152    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7bab8af3-8766-4098-95ab-720089120d08-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hjngx\" (UID: \"7bab8af3-8766-4098-95ab-720089120d08\") " pod="openshift-multus/multus-additional-cni-plugins-hjngx"
Jul 01 17:15:13.312781 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312187    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/388a1ebd-458e-4573-8242-b4da8bb8a77a-device-dir\") pod \"aws-ebs-csi-driver-node-b9zcq\" (UID: \"388a1ebd-458e-4573-8242-b4da8bb8a77a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9zcq"
Jul 01 17:15:13.312781 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312194    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-system-cni-dir\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.312781 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312225    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-host-run-k8s-cni-cncf-io\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.312781 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312253    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7bab8af3-8766-4098-95ab-720089120d08-cni-binary-copy\") pod \"multus-additional-cni-plugins-hjngx\" (UID: \"7bab8af3-8766-4098-95ab-720089120d08\") " pod="openshift-multus/multus-additional-cni-plugins-hjngx"
Jul 01 17:15:13.312781 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312264    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7bab8af3-8766-4098-95ab-720089120d08-os-release\") pod \"multus-additional-cni-plugins-hjngx\" (UID: \"7bab8af3-8766-4098-95ab-720089120d08\") " pod="openshift-multus/multus-additional-cni-plugins-hjngx"
Jul 01 17:15:13.312781 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312278    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-host-var-lib-cni-bin\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.312781 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312306    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-etc-kubernetes\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.312781 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312308    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-host-run-k8s-cni-cncf-io\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.312781 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312324    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ebd82416-a8de-490c-a980-74946452dc40-cni-binary-copy\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.313366 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312337    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-592jn\" (UniqueName: \"kubernetes.io/projected/388a1ebd-458e-4573-8242-b4da8bb8a77a-kube-api-access-592jn\") pod \"aws-ebs-csi-driver-node-b9zcq\" (UID: \"388a1ebd-458e-4573-8242-b4da8bb8a77a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9zcq"
Jul 01 17:15:13.313366 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312358    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-host-var-lib-cni-bin\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.313366 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312397    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-os-release\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.313366 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312424    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-host-run-multus-certs\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.313366 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312307    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7bab8af3-8766-4098-95ab-720089120d08-cnibin\") pod \"multus-additional-cni-plugins-hjngx\" (UID: \"7bab8af3-8766-4098-95ab-720089120d08\") " pod="openshift-multus/multus-additional-cni-plugins-hjngx"
Jul 01 17:15:13.313366 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312267    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-system-cni-dir\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.313366 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312399    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-host-var-lib-cni-multus\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.313366 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312486    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-host-run-multus-certs\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.313366 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312497    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-os-release\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.313366 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312429    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-etc-kubernetes\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.313366 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312531    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/388a1ebd-458e-4573-8242-b4da8bb8a77a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b9zcq\" (UID: \"388a1ebd-458e-4573-8242-b4da8bb8a77a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9zcq"
Jul 01 17:15:13.313366 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312570    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7bab8af3-8766-4098-95ab-720089120d08-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hjngx\" (UID: \"7bab8af3-8766-4098-95ab-720089120d08\") " pod="openshift-multus/multus-additional-cni-plugins-hjngx"
Jul 01 17:15:13.313366 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312597    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/388a1ebd-458e-4573-8242-b4da8bb8a77a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b9zcq\" (UID: \"388a1ebd-458e-4573-8242-b4da8bb8a77a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9zcq"
Jul 01 17:15:13.313366 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312605    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-multus-socket-dir-parent\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.313366 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312646    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/388a1ebd-458e-4573-8242-b4da8bb8a77a-etc-selinux\") pod \"aws-ebs-csi-driver-node-b9zcq\" (UID: \"388a1ebd-458e-4573-8242-b4da8bb8a77a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9zcq"
Jul 01 17:15:13.313366 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312677    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5dhq\" (UniqueName: \"kubernetes.io/projected/3ff59413-27fa-4837-8549-f2a6898826a9-kube-api-access-w5dhq\") pod \"node-resolver-jwvgk\" (UID: \"3ff59413-27fa-4837-8549-f2a6898826a9\") " pod="openshift-dns/node-resolver-jwvgk"
Jul 01 17:15:13.313366 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312709    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-cnibin\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.313366 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312714    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-multus-socket-dir-parent\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.313866 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312732    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7bab8af3-8766-4098-95ab-720089120d08-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hjngx\" (UID: \"7bab8af3-8766-4098-95ab-720089120d08\") " pod="openshift-multus/multus-additional-cni-plugins-hjngx"
Jul 01 17:15:13.313866 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312737    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/388a1ebd-458e-4573-8242-b4da8bb8a77a-sys-fs\") pod \"aws-ebs-csi-driver-node-b9zcq\" (UID: \"388a1ebd-458e-4573-8242-b4da8bb8a77a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9zcq"
Jul 01 17:15:13.313866 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312756    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/388a1ebd-458e-4573-8242-b4da8bb8a77a-etc-selinux\") pod \"aws-ebs-csi-driver-node-b9zcq\" (UID: \"388a1ebd-458e-4573-8242-b4da8bb8a77a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9zcq"
Jul 01 17:15:13.313866 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312764    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bab8af3-8766-4098-95ab-720089120d08-system-cni-dir\") pod \"multus-additional-cni-plugins-hjngx\" (UID: \"7bab8af3-8766-4098-95ab-720089120d08\") " pod="openshift-multus/multus-additional-cni-plugins-hjngx"
Jul 01 17:15:13.313866 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312785    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ebd82416-a8de-490c-a980-74946452dc40-cnibin\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.313866 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312807    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7bab8af3-8766-4098-95ab-720089120d08-cni-binary-copy\") pod \"multus-additional-cni-plugins-hjngx\" (UID: \"7bab8af3-8766-4098-95ab-720089120d08\") " pod="openshift-multus/multus-additional-cni-plugins-hjngx"
Jul 01 17:15:13.313866 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312820    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/388a1ebd-458e-4573-8242-b4da8bb8a77a-sys-fs\") pod \"aws-ebs-csi-driver-node-b9zcq\" (UID: \"388a1ebd-458e-4573-8242-b4da8bb8a77a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9zcq"
Jul 01 17:15:13.313866 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.312834    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bab8af3-8766-4098-95ab-720089120d08-system-cni-dir\") pod \"multus-additional-cni-plugins-hjngx\" (UID: \"7bab8af3-8766-4098-95ab-720089120d08\") " pod="openshift-multus/multus-additional-cni-plugins-hjngx"
Jul 01 17:15:13.321515 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.321497    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5dhq\" (UniqueName: \"kubernetes.io/projected/3ff59413-27fa-4837-8549-f2a6898826a9-kube-api-access-w5dhq\") pod \"node-resolver-jwvgk\" (UID: \"3ff59413-27fa-4837-8549-f2a6898826a9\") " pod="openshift-dns/node-resolver-jwvgk"
Jul 01 17:15:13.321833 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.321814    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zf6q\" (UniqueName: \"kubernetes.io/projected/ebd82416-a8de-490c-a980-74946452dc40-kube-api-access-9zf6q\") pod \"multus-89p2n\" (UID: \"ebd82416-a8de-490c-a980-74946452dc40\") " pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.321878 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.321850    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-592jn\" (UniqueName: \"kubernetes.io/projected/388a1ebd-458e-4573-8242-b4da8bb8a77a-kube-api-access-592jn\") pod \"aws-ebs-csi-driver-node-b9zcq\" (UID: \"388a1ebd-458e-4573-8242-b4da8bb8a77a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9zcq"
Jul 01 17:15:13.321921 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.321908    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjnqd\" (UniqueName: \"kubernetes.io/projected/7bab8af3-8766-4098-95ab-720089120d08-kube-api-access-pjnqd\") pod \"multus-additional-cni-plugins-hjngx\" (UID: \"7bab8af3-8766-4098-95ab-720089120d08\") " pod="openshift-multus/multus-additional-cni-plugins-hjngx"
Jul 01 17:15:13.436397 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.436327    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v8wlm"
Jul 01 17:15:13.443149 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.443130    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kb8gh"
Jul 01 17:15:13.444585 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:13.444530    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bccceba_0d8e_402a_a82f_accee4a1be82.slice/crio-d2e288caaa1e0495f062c86ad1fadff981062f863bc10ab26dbbefb1139cd7bf WatchSource:0}: Error finding container d2e288caaa1e0495f062c86ad1fadff981062f863bc10ab26dbbefb1139cd7bf: Status 404 returned error can't find the container with id d2e288caaa1e0495f062c86ad1fadff981062f863bc10ab26dbbefb1139cd7bf
Jul 01 17:15:13.451210 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:13.451184    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af4c82a_7a73_4de0_8f7b_25768775f79e.slice/crio-8defe8daf30bf304d2c6012d32c65b166b0b19be365db359a95ef6b16d106805 WatchSource:0}: Error finding container 8defe8daf30bf304d2c6012d32c65b166b0b19be365db359a95ef6b16d106805: Status 404 returned error can't find the container with id 8defe8daf30bf304d2c6012d32c65b166b0b19be365db359a95ef6b16d106805
Jul 01 17:15:13.461249 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.461233    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:13.468017 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:13.467981    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod500d2a90_2d21_42b3_9584_b04c14becaec.slice/crio-4d8d122a63277b9aad7a2526a6f929a9d5d918fa274a6237292a27bbbb8410ef WatchSource:0}: Error finding container 4d8d122a63277b9aad7a2526a6f929a9d5d918fa274a6237292a27bbbb8410ef: Status 404 returned error can't find the container with id 4d8d122a63277b9aad7a2526a6f929a9d5d918fa274a6237292a27bbbb8410ef
Jul 01 17:15:13.479833 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.479815    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r54bn"
Jul 01 17:15:13.486819 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:13.486797    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6614097_0891_470c_b830_26bf595ff80e.slice/crio-f1f69134b849181aaa7c53fe85e9b06491cca244717de44d6458e11c2cea9e29 WatchSource:0}: Error finding container f1f69134b849181aaa7c53fe85e9b06491cca244717de44d6458e11c2cea9e29: Status 404 returned error can't find the container with id f1f69134b849181aaa7c53fe85e9b06491cca244717de44d6458e11c2cea9e29
Jul 01 17:15:13.492791 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.492772    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mgc4x"
Jul 01 17:15:13.499907 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:13.499886    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5db90618_fcdb_4ea6_ac95_a2d535908e0e.slice/crio-7c6e7db5040cd3ac060014bdb9f5d2d951fbe5e6a421041783bb9a1070a7d534 WatchSource:0}: Error finding container 7c6e7db5040cd3ac060014bdb9f5d2d951fbe5e6a421041783bb9a1070a7d534: Status 404 returned error can't find the container with id 7c6e7db5040cd3ac060014bdb9f5d2d951fbe5e6a421041783bb9a1070a7d534
Jul 01 17:15:13.513952 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.513931    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9zcq"
Jul 01 17:15:13.521138 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:13.521113    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod388a1ebd_458e_4573_8242_b4da8bb8a77a.slice/crio-0ad04b37ca5650e1db91b2e3044a80f6b23d8864cd16d70850d4ab4ca2dbaccd WatchSource:0}: Error finding container 0ad04b37ca5650e1db91b2e3044a80f6b23d8864cd16d70850d4ab4ca2dbaccd: Status 404 returned error can't find the container with id 0ad04b37ca5650e1db91b2e3044a80f6b23d8864cd16d70850d4ab4ca2dbaccd
Jul 01 17:15:13.540995 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.540953    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jwvgk"
Jul 01 17:15:13.548556 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:13.548528    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff59413_27fa_4837_8549_f2a6898826a9.slice/crio-8affae2acc055c1f411877bcc5f775abbcb7d6803deecba2b87acd9b70af2a1f WatchSource:0}: Error finding container 8affae2acc055c1f411877bcc5f775abbcb7d6803deecba2b87acd9b70af2a1f: Status 404 returned error can't find the container with id 8affae2acc055c1f411877bcc5f775abbcb7d6803deecba2b87acd9b70af2a1f
Jul 01 17:15:13.557685 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.557657    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-89p2n"
Jul 01 17:15:13.562102 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.562078    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hjngx"
Jul 01 17:15:13.565621 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:13.565591    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd82416_a8de_490c_a980_74946452dc40.slice/crio-74371164e40adb7d0fd4de45a3083ce04b4b85772bde9108c6c102917e15cebb WatchSource:0}: Error finding container 74371164e40adb7d0fd4de45a3083ce04b4b85772bde9108c6c102917e15cebb: Status 404 returned error can't find the container with id 74371164e40adb7d0fd4de45a3083ce04b4b85772bde9108c6c102917e15cebb
Jul 01 17:15:13.570213 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:13.570193    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bab8af3_8766_4098_95ab_720089120d08.slice/crio-ed7aeb5781e8b91cef2a3dff59aa6a89f20ef19c392f15d451b601541cab89d5 WatchSource:0}: Error finding container ed7aeb5781e8b91cef2a3dff59aa6a89f20ef19c392f15d451b601541cab89d5: Status 404 returned error can't find the container with id ed7aeb5781e8b91cef2a3dff59aa6a89f20ef19c392f15d451b601541cab89d5
Jul 01 17:15:13.716365 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.716274    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5026d3ed-589a-4390-a2b7-d5f64c06e687-metrics-certs\") pod \"network-metrics-daemon-4rrth\" (UID: \"5026d3ed-589a-4390-a2b7-d5f64c06e687\") " pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:15:13.716365 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.716309    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3defaf9e-f5ef-45ea-a02e-4729f15db90c-original-pull-secret\") pod \"global-pull-secret-syncer-knvdq\" (UID: \"3defaf9e-f5ef-45ea-a02e-4729f15db90c\") " pod="kube-system/global-pull-secret-syncer-knvdq"
Jul 01 17:15:13.716604 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:13.716391    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 17:15:13.716604 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:13.716406    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 17:15:13.716604 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:13.716454    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5026d3ed-589a-4390-a2b7-d5f64c06e687-metrics-certs podName:5026d3ed-589a-4390-a2b7-d5f64c06e687 nodeName:}" failed. No retries permitted until 2026-07-01 17:15:14.716438445 +0000 UTC m=+3.021636600 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5026d3ed-589a-4390-a2b7-d5f64c06e687-metrics-certs") pod "network-metrics-daemon-4rrth" (UID: "5026d3ed-589a-4390-a2b7-d5f64c06e687") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 17:15:13.716604 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:13.716468    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3defaf9e-f5ef-45ea-a02e-4729f15db90c-original-pull-secret podName:3defaf9e-f5ef-45ea-a02e-4729f15db90c nodeName:}" failed. No retries permitted until 2026-07-01 17:15:14.716461877 +0000 UTC m=+3.021660033 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3defaf9e-f5ef-45ea-a02e-4729f15db90c-original-pull-secret") pod "global-pull-secret-syncer-knvdq" (UID: "3defaf9e-f5ef-45ea-a02e-4729f15db90c") : object "kube-system"/"original-pull-secret" not registered
Jul 01 17:15:13.816799 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:13.816752    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f6dj\" (UniqueName: \"kubernetes.io/projected/415f427a-adac-47bb-9cad-3befc6a711b6-kube-api-access-7f6dj\") pod \"network-check-target-2x2qg\" (UID: \"415f427a-adac-47bb-9cad-3befc6a711b6\") " pod="openshift-network-diagnostics/network-check-target-2x2qg"
Jul 01 17:15:13.816961 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:13.816938    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 17:15:13.817036 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:13.816964    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 17:15:13.817036 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:13.816978    2585 projected.go:196] Error preparing data for projected volume kube-api-access-7f6dj for pod openshift-network-diagnostics/network-check-target-2x2qg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 17:15:13.817140 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:13.817042    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/415f427a-adac-47bb-9cad-3befc6a711b6-kube-api-access-7f6dj podName:415f427a-adac-47bb-9cad-3befc6a711b6 nodeName:}" failed. No retries permitted until 2026-07-01 17:15:14.817025242 +0000 UTC m=+3.122223410 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7f6dj" (UniqueName: "kubernetes.io/projected/415f427a-adac-47bb-9cad-3befc6a711b6-kube-api-access-7f6dj") pod "network-check-target-2x2qg" (UID: "415f427a-adac-47bb-9cad-3befc6a711b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 17:15:14.159810 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:14.159549    2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 17:15:14.160973 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:14.160839    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 17:10:13 +0000 UTC" deadline="2028-03-03 02:08:08.089959383 +0000 UTC"
Jul 01 17:15:14.160973 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:14.160900    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14648h52m53.929063204s"
Jul 01 17:15:14.175659 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:14.175543    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal" event={"ID":"749b61ceb1d13c63af40ebe9b4481208","Type":"ContainerStarted","Data":"a50c704241ad61348015d5b265769ba5b3387b9c09f8dc39605cce7fd16798a0"}
Jul 01 17:15:14.182351 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:14.182276    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-213.ec2.internal" event={"ID":"608b014fc4a8976e8619472b298b39c8","Type":"ContainerStarted","Data":"86be634731e515017557cfb483cff52d8992ac7982578c4dfcf7f36a8ed76272"}
Jul 01 17:15:14.184049 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:14.183974    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-89p2n" event={"ID":"ebd82416-a8de-490c-a980-74946452dc40","Type":"ContainerStarted","Data":"74371164e40adb7d0fd4de45a3083ce04b4b85772bde9108c6c102917e15cebb"}
Jul 01 17:15:14.185449 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:14.185376    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jwvgk" event={"ID":"3ff59413-27fa-4837-8549-f2a6898826a9","Type":"ContainerStarted","Data":"8affae2acc055c1f411877bcc5f775abbcb7d6803deecba2b87acd9b70af2a1f"}
Jul 01 17:15:14.187128 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:14.187047    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9zcq" event={"ID":"388a1ebd-458e-4573-8242-b4da8bb8a77a","Type":"ContainerStarted","Data":"0ad04b37ca5650e1db91b2e3044a80f6b23d8864cd16d70850d4ab4ca2dbaccd"}
Jul 01 17:15:14.188647 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:14.188564    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mgc4x" event={"ID":"5db90618-fcdb-4ea6-ac95-a2d535908e0e","Type":"ContainerStarted","Data":"7c6e7db5040cd3ac060014bdb9f5d2d951fbe5e6a421041783bb9a1070a7d534"}
Jul 01 17:15:14.204482 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:14.204447    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r54bn" event={"ID":"d6614097-0891-470c-b830-26bf595ff80e","Type":"ContainerStarted","Data":"f1f69134b849181aaa7c53fe85e9b06491cca244717de44d6458e11c2cea9e29"}
Jul 01 17:15:14.208492 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:14.208453    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw" event={"ID":"500d2a90-2d21-42b3-9584-b04c14becaec","Type":"ContainerStarted","Data":"4d8d122a63277b9aad7a2526a6f929a9d5d918fa274a6237292a27bbbb8410ef"}
Jul 01 17:15:14.217139 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:14.217107    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kb8gh" event={"ID":"3af4c82a-7a73-4de0-8f7b-25768775f79e","Type":"ContainerStarted","Data":"8defe8daf30bf304d2c6012d32c65b166b0b19be365db359a95ef6b16d106805"}
Jul 01 17:15:14.221086 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:14.221057    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v8wlm" event={"ID":"4bccceba-0d8e-402a-a82f-accee4a1be82","Type":"ContainerStarted","Data":"d2e288caaa1e0495f062c86ad1fadff981062f863bc10ab26dbbefb1139cd7bf"}
Jul 01 17:15:14.222779 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:14.222754    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hjngx" event={"ID":"7bab8af3-8766-4098-95ab-720089120d08","Type":"ContainerStarted","Data":"ed7aeb5781e8b91cef2a3dff59aa6a89f20ef19c392f15d451b601541cab89d5"}
Jul 01 17:15:14.723916 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:14.723821    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5026d3ed-589a-4390-a2b7-d5f64c06e687-metrics-certs\") pod \"network-metrics-daemon-4rrth\" (UID: \"5026d3ed-589a-4390-a2b7-d5f64c06e687\") " pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:15:14.723916 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:14.723874    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3defaf9e-f5ef-45ea-a02e-4729f15db90c-original-pull-secret\") pod \"global-pull-secret-syncer-knvdq\" (UID: \"3defaf9e-f5ef-45ea-a02e-4729f15db90c\") " pod="kube-system/global-pull-secret-syncer-knvdq"
Jul 01 17:15:14.724176 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:14.724052    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 17:15:14.724176 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:14.724117    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3defaf9e-f5ef-45ea-a02e-4729f15db90c-original-pull-secret podName:3defaf9e-f5ef-45ea-a02e-4729f15db90c nodeName:}" failed. No retries permitted until 2026-07-01 17:15:16.724097653 +0000 UTC m=+5.029295814 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3defaf9e-f5ef-45ea-a02e-4729f15db90c-original-pull-secret") pod "global-pull-secret-syncer-knvdq" (UID: "3defaf9e-f5ef-45ea-a02e-4729f15db90c") : object "kube-system"/"original-pull-secret" not registered
Jul 01 17:15:14.724538 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:14.724521    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 17:15:14.724627 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:14.724591    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5026d3ed-589a-4390-a2b7-d5f64c06e687-metrics-certs podName:5026d3ed-589a-4390-a2b7-d5f64c06e687 nodeName:}" failed. No retries permitted until 2026-07-01 17:15:16.724561351 +0000 UTC m=+5.029759512 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5026d3ed-589a-4390-a2b7-d5f64c06e687-metrics-certs") pod "network-metrics-daemon-4rrth" (UID: "5026d3ed-589a-4390-a2b7-d5f64c06e687") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 17:15:14.824277 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:14.824240    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f6dj\" (UniqueName: \"kubernetes.io/projected/415f427a-adac-47bb-9cad-3befc6a711b6-kube-api-access-7f6dj\") pod \"network-check-target-2x2qg\" (UID: \"415f427a-adac-47bb-9cad-3befc6a711b6\") " pod="openshift-network-diagnostics/network-check-target-2x2qg"
Jul 01 17:15:14.824465 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:14.824435    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 17:15:14.824465 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:14.824454    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 17:15:14.824465 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:14.824466    2585 projected.go:196] Error preparing data for projected volume kube-api-access-7f6dj for pod openshift-network-diagnostics/network-check-target-2x2qg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 17:15:14.824651 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:14.824525    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/415f427a-adac-47bb-9cad-3befc6a711b6-kube-api-access-7f6dj podName:415f427a-adac-47bb-9cad-3befc6a711b6 nodeName:}" failed. No retries permitted until 2026-07-01 17:15:16.824507593 +0000 UTC m=+5.129705765 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7f6dj" (UniqueName: "kubernetes.io/projected/415f427a-adac-47bb-9cad-3befc6a711b6-kube-api-access-7f6dj") pod "network-check-target-2x2qg" (UID: "415f427a-adac-47bb-9cad-3befc6a711b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 17:15:15.162069 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:15.161962    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 17:10:13 +0000 UTC" deadline="2028-04-12 21:50:35.281589678 +0000 UTC"
Jul 01 17:15:15.162069 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:15.162007    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15628h35m20.119587913s"
Jul 01 17:15:15.167635 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:15.167609    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knvdq"
Jul 01 17:15:15.167805 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:15.167737    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knvdq" podUID="3defaf9e-f5ef-45ea-a02e-4729f15db90c"
Jul 01 17:15:15.168170 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:15.168148    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:15:15.168285 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:15.168264    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rrth" podUID="5026d3ed-589a-4390-a2b7-d5f64c06e687"
Jul 01 17:15:15.168355 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:15.168336    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x2qg"
Jul 01 17:15:15.168426 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:15.168406    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x2qg" podUID="415f427a-adac-47bb-9cad-3befc6a711b6"
Jul 01 17:15:16.743141 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:16.743009    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5026d3ed-589a-4390-a2b7-d5f64c06e687-metrics-certs\") pod \"network-metrics-daemon-4rrth\" (UID: \"5026d3ed-589a-4390-a2b7-d5f64c06e687\") " pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:15:16.743141 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:16.743063    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3defaf9e-f5ef-45ea-a02e-4729f15db90c-original-pull-secret\") pod \"global-pull-secret-syncer-knvdq\" (UID: \"3defaf9e-f5ef-45ea-a02e-4729f15db90c\") " pod="kube-system/global-pull-secret-syncer-knvdq"
Jul 01 17:15:16.743698 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:16.743215    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 17:15:16.743698 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:16.743226    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 17:15:16.743698 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:16.743274    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5026d3ed-589a-4390-a2b7-d5f64c06e687-metrics-certs podName:5026d3ed-589a-4390-a2b7-d5f64c06e687 nodeName:}" failed. No retries permitted until 2026-07-01 17:15:20.743257432 +0000 UTC m=+9.048455587 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5026d3ed-589a-4390-a2b7-d5f64c06e687-metrics-certs") pod "network-metrics-daemon-4rrth" (UID: "5026d3ed-589a-4390-a2b7-d5f64c06e687") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 17:15:16.743698 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:16.743292    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3defaf9e-f5ef-45ea-a02e-4729f15db90c-original-pull-secret podName:3defaf9e-f5ef-45ea-a02e-4729f15db90c nodeName:}" failed. No retries permitted until 2026-07-01 17:15:20.743282214 +0000 UTC m=+9.048480373 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3defaf9e-f5ef-45ea-a02e-4729f15db90c-original-pull-secret") pod "global-pull-secret-syncer-knvdq" (UID: "3defaf9e-f5ef-45ea-a02e-4729f15db90c") : object "kube-system"/"original-pull-secret" not registered
Jul 01 17:15:16.843915 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:16.843877    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f6dj\" (UniqueName: \"kubernetes.io/projected/415f427a-adac-47bb-9cad-3befc6a711b6-kube-api-access-7f6dj\") pod \"network-check-target-2x2qg\" (UID: \"415f427a-adac-47bb-9cad-3befc6a711b6\") " pod="openshift-network-diagnostics/network-check-target-2x2qg"
Jul 01 17:15:16.844081 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:16.844057    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 17:15:16.844081 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:16.844079    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 17:15:16.844207 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:16.844092    2585 projected.go:196] Error preparing data for projected volume kube-api-access-7f6dj for pod openshift-network-diagnostics/network-check-target-2x2qg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 17:15:16.844207 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:16.844148    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/415f427a-adac-47bb-9cad-3befc6a711b6-kube-api-access-7f6dj podName:415f427a-adac-47bb-9cad-3befc6a711b6 nodeName:}" failed. No retries permitted until 2026-07-01 17:15:20.844130219 +0000 UTC m=+9.149328380 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7f6dj" (UniqueName: "kubernetes.io/projected/415f427a-adac-47bb-9cad-3befc6a711b6-kube-api-access-7f6dj") pod "network-check-target-2x2qg" (UID: "415f427a-adac-47bb-9cad-3befc6a711b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 17:15:17.167648 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:17.167503    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knvdq"
Jul 01 17:15:17.167803 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:17.167648    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knvdq" podUID="3defaf9e-f5ef-45ea-a02e-4729f15db90c"
Jul 01 17:15:17.167803 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:17.167726    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:15:17.167945 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:17.167810    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rrth" podUID="5026d3ed-589a-4390-a2b7-d5f64c06e687"
Jul 01 17:15:17.167945 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:17.167852    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x2qg"
Jul 01 17:15:17.167945 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:17.167925    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x2qg" podUID="415f427a-adac-47bb-9cad-3befc6a711b6"
Jul 01 17:15:19.167136 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:19.167098    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knvdq"
Jul 01 17:15:19.167558 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:19.167146    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:15:19.167558 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:19.167146    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x2qg"
Jul 01 17:15:19.167558 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:19.167239    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knvdq" podUID="3defaf9e-f5ef-45ea-a02e-4729f15db90c"
Jul 01 17:15:19.167558 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:19.167351    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x2qg" podUID="415f427a-adac-47bb-9cad-3befc6a711b6"
Jul 01 17:15:19.167558 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:19.167461    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rrth" podUID="5026d3ed-589a-4390-a2b7-d5f64c06e687"
Jul 01 17:15:20.774172 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:20.773927    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5026d3ed-589a-4390-a2b7-d5f64c06e687-metrics-certs\") pod \"network-metrics-daemon-4rrth\" (UID: \"5026d3ed-589a-4390-a2b7-d5f64c06e687\") " pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:15:20.774172 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:20.773981    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3defaf9e-f5ef-45ea-a02e-4729f15db90c-original-pull-secret\") pod \"global-pull-secret-syncer-knvdq\" (UID: \"3defaf9e-f5ef-45ea-a02e-4729f15db90c\") " pod="kube-system/global-pull-secret-syncer-knvdq"
Jul 01 17:15:20.774172 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:20.774103    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 17:15:20.774172 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:20.774122    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 17:15:20.774172 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:20.774184    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5026d3ed-589a-4390-a2b7-d5f64c06e687-metrics-certs podName:5026d3ed-589a-4390-a2b7-d5f64c06e687 nodeName:}" failed. No retries permitted until 2026-07-01 17:15:28.774164297 +0000 UTC m=+17.079362458 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5026d3ed-589a-4390-a2b7-d5f64c06e687-metrics-certs") pod "network-metrics-daemon-4rrth" (UID: "5026d3ed-589a-4390-a2b7-d5f64c06e687") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 17:15:20.774828 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:20.774206    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3defaf9e-f5ef-45ea-a02e-4729f15db90c-original-pull-secret podName:3defaf9e-f5ef-45ea-a02e-4729f15db90c nodeName:}" failed. No retries permitted until 2026-07-01 17:15:28.774196243 +0000 UTC m=+17.079394402 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3defaf9e-f5ef-45ea-a02e-4729f15db90c-original-pull-secret") pod "global-pull-secret-syncer-knvdq" (UID: "3defaf9e-f5ef-45ea-a02e-4729f15db90c") : object "kube-system"/"original-pull-secret" not registered
Jul 01 17:15:20.874891 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:20.874852    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f6dj\" (UniqueName: \"kubernetes.io/projected/415f427a-adac-47bb-9cad-3befc6a711b6-kube-api-access-7f6dj\") pod \"network-check-target-2x2qg\" (UID: \"415f427a-adac-47bb-9cad-3befc6a711b6\") " pod="openshift-network-diagnostics/network-check-target-2x2qg"
Jul 01 17:15:20.875062 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:20.875028    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 17:15:20.875062 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:20.875051    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 17:15:20.875186 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:20.875067    2585 projected.go:196] Error preparing data for projected volume kube-api-access-7f6dj for pod openshift-network-diagnostics/network-check-target-2x2qg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 17:15:20.875186 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:20.875111    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/415f427a-adac-47bb-9cad-3befc6a711b6-kube-api-access-7f6dj podName:415f427a-adac-47bb-9cad-3befc6a711b6 nodeName:}" failed. No retries permitted until 2026-07-01 17:15:28.87509869 +0000 UTC m=+17.180296857 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7f6dj" (UniqueName: "kubernetes.io/projected/415f427a-adac-47bb-9cad-3befc6a711b6-kube-api-access-7f6dj") pod "network-check-target-2x2qg" (UID: "415f427a-adac-47bb-9cad-3befc6a711b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 17:15:21.168005 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:21.167221    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knvdq"
Jul 01 17:15:21.168005 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:21.167345    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knvdq" podUID="3defaf9e-f5ef-45ea-a02e-4729f15db90c"
Jul 01 17:15:21.168005 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:21.167708    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:15:21.168005 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:21.167801    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rrth" podUID="5026d3ed-589a-4390-a2b7-d5f64c06e687"
Jul 01 17:15:21.168005 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:21.167849    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x2qg"
Jul 01 17:15:21.168005 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:21.167913    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x2qg" podUID="415f427a-adac-47bb-9cad-3befc6a711b6"
Jul 01 17:15:23.167214 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:23.167093    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knvdq"
Jul 01 17:15:23.167673 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:23.167217    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:15:23.167673 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:23.167233    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knvdq" podUID="3defaf9e-f5ef-45ea-a02e-4729f15db90c"
Jul 01 17:15:23.167673 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:23.167312    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x2qg"
Jul 01 17:15:23.167673 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:23.167406    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x2qg" podUID="415f427a-adac-47bb-9cad-3befc6a711b6"
Jul 01 17:15:23.167673 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:23.167484    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rrth" podUID="5026d3ed-589a-4390-a2b7-d5f64c06e687"
Jul 01 17:15:25.167688 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:25.167647    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:15:25.167688 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:25.167673    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x2qg"
Jul 01 17:15:25.167688 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:25.167661    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knvdq"
Jul 01 17:15:25.168122 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:25.167760    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x2qg" podUID="415f427a-adac-47bb-9cad-3befc6a711b6"
Jul 01 17:15:25.168122 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:25.167810    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knvdq" podUID="3defaf9e-f5ef-45ea-a02e-4729f15db90c"
Jul 01 17:15:25.168122 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:25.167881    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rrth" podUID="5026d3ed-589a-4390-a2b7-d5f64c06e687"
Jul 01 17:15:27.167167 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:27.167135    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knvdq"
Jul 01 17:15:27.167167 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:27.167149    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:15:27.167681 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:27.167230    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knvdq" podUID="3defaf9e-f5ef-45ea-a02e-4729f15db90c"
Jul 01 17:15:27.167681 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:27.167245    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x2qg"
Jul 01 17:15:27.167681 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:27.167382    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rrth" podUID="5026d3ed-589a-4390-a2b7-d5f64c06e687"
Jul 01 17:15:27.167681 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:27.167458    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x2qg" podUID="415f427a-adac-47bb-9cad-3befc6a711b6"
Jul 01 17:15:28.830769 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:28.830729    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5026d3ed-589a-4390-a2b7-d5f64c06e687-metrics-certs\") pod \"network-metrics-daemon-4rrth\" (UID: \"5026d3ed-589a-4390-a2b7-d5f64c06e687\") " pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:15:28.830769 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:28.830787    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3defaf9e-f5ef-45ea-a02e-4729f15db90c-original-pull-secret\") pod \"global-pull-secret-syncer-knvdq\" (UID: \"3defaf9e-f5ef-45ea-a02e-4729f15db90c\") " pod="kube-system/global-pull-secret-syncer-knvdq"
Jul 01 17:15:28.831282 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:28.830924    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 17:15:28.831282 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:28.830934    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 17:15:28.831282 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:28.830992    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5026d3ed-589a-4390-a2b7-d5f64c06e687-metrics-certs podName:5026d3ed-589a-4390-a2b7-d5f64c06e687 nodeName:}" failed. No retries permitted until 2026-07-01 17:15:44.830976457 +0000 UTC m=+33.136174612 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5026d3ed-589a-4390-a2b7-d5f64c06e687-metrics-certs") pod "network-metrics-daemon-4rrth" (UID: "5026d3ed-589a-4390-a2b7-d5f64c06e687") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 17:15:28.831282 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:28.831013    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3defaf9e-f5ef-45ea-a02e-4729f15db90c-original-pull-secret podName:3defaf9e-f5ef-45ea-a02e-4729f15db90c nodeName:}" failed. No retries permitted until 2026-07-01 17:15:44.831001727 +0000 UTC m=+33.136199883 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3defaf9e-f5ef-45ea-a02e-4729f15db90c-original-pull-secret") pod "global-pull-secret-syncer-knvdq" (UID: "3defaf9e-f5ef-45ea-a02e-4729f15db90c") : object "kube-system"/"original-pull-secret" not registered
Jul 01 17:15:28.931629 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:28.931570    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f6dj\" (UniqueName: \"kubernetes.io/projected/415f427a-adac-47bb-9cad-3befc6a711b6-kube-api-access-7f6dj\") pod \"network-check-target-2x2qg\" (UID: \"415f427a-adac-47bb-9cad-3befc6a711b6\") " pod="openshift-network-diagnostics/network-check-target-2x2qg"
Jul 01 17:15:28.931807 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:28.931777    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 17:15:28.931807 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:28.931802    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 17:15:28.931919 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:28.931816    2585 projected.go:196] Error preparing data for projected volume kube-api-access-7f6dj for pod openshift-network-diagnostics/network-check-target-2x2qg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 17:15:28.931919 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:28.931878    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/415f427a-adac-47bb-9cad-3befc6a711b6-kube-api-access-7f6dj podName:415f427a-adac-47bb-9cad-3befc6a711b6 nodeName:}" failed. No retries permitted until 2026-07-01 17:15:44.931860153 +0000 UTC m=+33.237058327 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7f6dj" (UniqueName: "kubernetes.io/projected/415f427a-adac-47bb-9cad-3befc6a711b6-kube-api-access-7f6dj") pod "network-check-target-2x2qg" (UID: "415f427a-adac-47bb-9cad-3befc6a711b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 17:15:29.167664 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:29.167567    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knvdq"
Jul 01 17:15:29.167664 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:29.167619    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:15:29.167664 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:29.167619    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x2qg"
Jul 01 17:15:29.167907 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:29.167717    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knvdq" podUID="3defaf9e-f5ef-45ea-a02e-4729f15db90c"
Jul 01 17:15:29.167907 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:29.167805    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rrth" podUID="5026d3ed-589a-4390-a2b7-d5f64c06e687"
Jul 01 17:15:29.167907 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:29.167875    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x2qg" podUID="415f427a-adac-47bb-9cad-3befc6a711b6"
Jul 01 17:15:31.167808 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:31.167773    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x2qg"
Jul 01 17:15:31.168319 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:31.167773    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knvdq"
Jul 01 17:15:31.168319 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:31.167889    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x2qg" podUID="415f427a-adac-47bb-9cad-3befc6a711b6"
Jul 01 17:15:31.168319 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:31.167773    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:15:31.168319 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:31.167965    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knvdq" podUID="3defaf9e-f5ef-45ea-a02e-4729f15db90c"
Jul 01 17:15:31.168319 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:31.168061    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rrth" podUID="5026d3ed-589a-4390-a2b7-d5f64c06e687"
Jul 01 17:15:33.167288 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:33.167259    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:15:33.167812 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:33.167358    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rrth" podUID="5026d3ed-589a-4390-a2b7-d5f64c06e687"
Jul 01 17:15:33.167812 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:33.167381    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knvdq"
Jul 01 17:15:33.167812 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:33.167265    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x2qg"
Jul 01 17:15:33.167812 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:33.167480    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knvdq" podUID="3defaf9e-f5ef-45ea-a02e-4729f15db90c"
Jul 01 17:15:33.167812 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:33.167529    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x2qg" podUID="415f427a-adac-47bb-9cad-3befc6a711b6"
Jul 01 17:15:34.275696 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:34.275480    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8tsw_500d2a90-2d21-42b3-9584-b04c14becaec/ovn-acl-logging/0.log"
Jul 01 17:15:34.276486 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:34.276038    2585 generic.go:356] "Generic (PLEG): container finished" podID="500d2a90-2d21-42b3-9584-b04c14becaec" containerID="1aaa03a662b9c93313b88b2f707e02a23dc589bc52b54bd1cc1d8d8e28ee0e9d" exitCode=1
Jul 01 17:15:34.276486 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:34.276109    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw" event={"ID":"500d2a90-2d21-42b3-9584-b04c14becaec","Type":"ContainerStarted","Data":"920f225675c7739734624371fdd78bacddaa1156cd568e686142bcfed57e741f"}
Jul 01 17:15:34.276486 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:34.276238    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw" event={"ID":"500d2a90-2d21-42b3-9584-b04c14becaec","Type":"ContainerDied","Data":"1aaa03a662b9c93313b88b2f707e02a23dc589bc52b54bd1cc1d8d8e28ee0e9d"}
Jul 01 17:15:34.276486 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:34.276278    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw" event={"ID":"500d2a90-2d21-42b3-9584-b04c14becaec","Type":"ContainerStarted","Data":"621fc1683371cfd0e600feed6d71916329120245001d274f0762e40ec9f0ac61"}
Jul 01 17:15:34.277720 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:34.277691    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v8wlm" event={"ID":"4bccceba-0d8e-402a-a82f-accee4a1be82","Type":"ContainerStarted","Data":"cf38e89035cd7e14191a6ffb99c968a529b18c56ea44a3a07d2a86c2ddf7a7f2"}
Jul 01 17:15:34.279743 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:34.279722    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-213.ec2.internal" event={"ID":"608b014fc4a8976e8619472b298b39c8","Type":"ContainerStarted","Data":"ec7ff7980de44738562ded27a0cc63caade33028e5d04d9b8832291037c8d66c"}
Jul 01 17:15:34.282256 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:34.282233    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-89p2n" event={"ID":"ebd82416-a8de-490c-a980-74946452dc40","Type":"ContainerStarted","Data":"81b1bf1aab9cd2196f83204cb8eec70f549fb6fe054de8b8b75f302481c51165"}
Jul 01 17:15:34.296176 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:34.296117    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-v8wlm" podStartSLOduration=1.922720669 podStartE2EDuration="22.296103161s" totalImagesPullingTime="20.373382492s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 17:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:15:34.295836191 +0000 UTC m=+22.601034368" watchObservedRunningTime="2026-07-01 17:15:34.296103161 +0000 UTC m=+22.601301337"
Jul 01 17:15:34.313633 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:34.313592    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-89p2n" podStartSLOduration=1.786370002 podStartE2EDuration="22.313561113s" totalImagesPullingTime="20.527191111s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 17:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:15:34.313346099 +0000 UTC m=+22.618544277" watchObservedRunningTime="2026-07-01 17:15:34.313561113 +0000 UTC m=+22.618759290"
Jul 01 17:15:35.167632 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:35.167329    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x2qg"
Jul 01 17:15:35.167632 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:35.167369    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:15:35.167844 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:35.167642    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x2qg" podUID="415f427a-adac-47bb-9cad-3befc6a711b6"
Jul 01 17:15:35.167844 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:35.167385    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knvdq"
Jul 01 17:15:35.167844 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:35.167704    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rrth" podUID="5026d3ed-589a-4390-a2b7-d5f64c06e687"
Jul 01 17:15:35.167844 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:35.167748    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knvdq" podUID="3defaf9e-f5ef-45ea-a02e-4729f15db90c"
Jul 01 17:15:35.285093 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:35.285063    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jwvgk" event={"ID":"3ff59413-27fa-4837-8549-f2a6898826a9","Type":"ContainerStarted","Data":"d564f7c463b72e07e04a8a8387ce1c10f7784b88c36d99b000e27e8ec7e5e309"}
Jul 01 17:15:35.286649 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:35.286623    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9zcq" event={"ID":"388a1ebd-458e-4573-8242-b4da8bb8a77a","Type":"ContainerStarted","Data":"f6de223664975967dcbd2982829a432540336c78fb5683250903c93b8e8b9a79"}
Jul 01 17:15:35.287956 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:35.287932    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mgc4x" event={"ID":"5db90618-fcdb-4ea6-ac95-a2d535908e0e","Type":"ContainerStarted","Data":"a3795df18ec1291ae17d0069efb57a8c4b2de49943f11c60c3deddb65e98591e"}
Jul 01 17:15:35.289449 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:35.289428    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r54bn" event={"ID":"d6614097-0891-470c-b830-26bf595ff80e","Type":"ContainerStarted","Data":"f3cfe0eb29f9e766104d554358defb676b342dae84317c1f3ffce84352b1f9e6"}
Jul 01 17:15:35.292072 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:35.292055    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8tsw_500d2a90-2d21-42b3-9584-b04c14becaec/ovn-acl-logging/0.log"
Jul 01 17:15:35.292462 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:35.292446    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw" event={"ID":"500d2a90-2d21-42b3-9584-b04c14becaec","Type":"ContainerStarted","Data":"50819cc58e85e56f07291bdb28663885b2b1e3bf8bb17267b528136cf873c4bb"}
Jul 01 17:15:35.292529 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:35.292514    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw" event={"ID":"500d2a90-2d21-42b3-9584-b04c14becaec","Type":"ContainerStarted","Data":"88fde9b10cc1919da6ef18063f9b81aa6e9f01404252e913162aea572109cb65"}
Jul 01 17:15:35.292610 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:35.292568    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw" event={"ID":"500d2a90-2d21-42b3-9584-b04c14becaec","Type":"ContainerStarted","Data":"910a2e128665eed1041581794b32ac97e47c0fb10ce08bc18f8c6cdef2cb331a"}
Jul 01 17:15:35.293838 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:35.293816    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kb8gh" event={"ID":"3af4c82a-7a73-4de0-8f7b-25768775f79e","Type":"ContainerStarted","Data":"c75925f9296fa9c2503bb623570a042d75553020d71a01e8e2251cf0a8224732"}
Jul 01 17:15:35.295209 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:35.295191    2585 generic.go:356] "Generic (PLEG): container finished" podID="7bab8af3-8766-4098-95ab-720089120d08" containerID="4e24277421237c15038444280723e2d54be1eb75765affae5623f4003a48b074" exitCode=0
Jul 01 17:15:35.295275 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:35.295233    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hjngx" event={"ID":"7bab8af3-8766-4098-95ab-720089120d08","Type":"ContainerDied","Data":"4e24277421237c15038444280723e2d54be1eb75765affae5623f4003a48b074"}
Jul 01 17:15:35.299567 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:35.299515    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-213.ec2.internal" podStartSLOduration=22.299503383 podStartE2EDuration="22.299503383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 17:15:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:15:34.33570182 +0000 UTC m=+22.640899997" watchObservedRunningTime="2026-07-01 17:15:35.299503383 +0000 UTC m=+23.604701551"
Jul 01 17:15:35.299769 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:35.299748    2585 generic.go:356] "Generic (PLEG): container finished" podID="749b61ceb1d13c63af40ebe9b4481208" containerID="7497eefc75c9d516da8b859edb6801630856365a02362cb70d54224a16f5e2b7" exitCode=0
Jul 01 17:15:35.299832 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:35.299749    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jwvgk" podStartSLOduration=3.064216496 podStartE2EDuration="23.299739368s" totalImagesPullingTime="20.235522872s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 17:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:15:35.299477475 +0000 UTC m=+23.604675652" watchObservedRunningTime="2026-07-01 17:15:35.299739368 +0000 UTC m=+23.604937544"
Jul 01 17:15:35.299884 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:35.299849    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal" event={"ID":"749b61ceb1d13c63af40ebe9b4481208","Type":"ContainerDied","Data":"7497eefc75c9d516da8b859edb6801630856365a02362cb70d54224a16f5e2b7"}
Jul 01 17:15:35.313180 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:35.313146    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mgc4x" podStartSLOduration=2.998956997 podStartE2EDuration="23.313134611s" totalImagesPullingTime="20.314177614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:15:35.312857541 +0000 UTC m=+23.618055717" watchObservedRunningTime="2026-07-01 17:15:35.313134611 +0000 UTC m=+23.618332787"
Jul 01 17:15:35.327779 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:35.327743    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kb8gh" podStartSLOduration=2.962496298 podStartE2EDuration="23.327728033s" totalImagesPullingTime="20.365231735s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 17:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:15:35.327728707 +0000 UTC m=+23.632926884" watchObservedRunningTime="2026-07-01 17:15:35.327728033 +0000 UTC m=+23.632926200"
Jul 01 17:15:35.341904 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:35.341852    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-r54bn" podStartSLOduration=3.069164893 podStartE2EDuration="23.34183168s" totalImagesPullingTime="20.272666787s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 17:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:15:35.340912245 +0000 UTC m=+23.646110422" watchObservedRunningTime="2026-07-01 17:15:35.34183168 +0000 UTC m=+23.647029858"
Jul 01 17:15:35.985547 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:35.985464    2585 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 01 17:15:36.174511 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:36.174344    2585 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-01T17:15:35.985477063Z","UUID":"e4dafa34-7f39-4b84-bafa-112dd0ccff7a","Handler":null,"Name":"","Endpoint":""}
Jul 01 17:15:36.175834 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:36.175811    2585 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 01 17:15:36.175955 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:36.175852    2585 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 01 17:15:36.304306 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:36.304269    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9zcq" event={"ID":"388a1ebd-458e-4573-8242-b4da8bb8a77a","Type":"ContainerStarted","Data":"874f7eeec2282db81951eecab3a8b1c258c85baeca5195d7265b8d2b85903794"}
Jul 01 17:15:36.306376 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:36.306326    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal" event={"ID":"749b61ceb1d13c63af40ebe9b4481208","Type":"ContainerStarted","Data":"d35c917da732706c6e59dabf1fd989b561bafc372b974243644fcb9e5f896500"}
Jul 01 17:15:37.167462 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:37.167426    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x2qg"
Jul 01 17:15:37.167462 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:37.167446    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knvdq"
Jul 01 17:15:37.167853 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:37.167548    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:15:37.167853 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:37.167547    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x2qg" podUID="415f427a-adac-47bb-9cad-3befc6a711b6"
Jul 01 17:15:37.167853 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:37.167666    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knvdq" podUID="3defaf9e-f5ef-45ea-a02e-4729f15db90c"
Jul 01 17:15:37.167853 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:37.167750    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rrth" podUID="5026d3ed-589a-4390-a2b7-d5f64c06e687"
Jul 01 17:15:37.314254 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:37.314226    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8tsw_500d2a90-2d21-42b3-9584-b04c14becaec/ovn-acl-logging/0.log"
Jul 01 17:15:37.314741 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:37.314712    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw" event={"ID":"500d2a90-2d21-42b3-9584-b04c14becaec","Type":"ContainerStarted","Data":"094834b9c3784b63faec8799a8d08cb9c52d47e8aeb3ad1922053728e5277709"}
Jul 01 17:15:38.319970 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:38.319935    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9zcq" event={"ID":"388a1ebd-458e-4573-8242-b4da8bb8a77a","Type":"ContainerStarted","Data":"c1c443c7397d0800d26739446f4ad4fcd56e49305cd0a4bf26578858568729d5"}
Jul 01 17:15:38.338720 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:38.338658    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal" podStartSLOduration=25.338641046 podStartE2EDuration="25.338641046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 17:15:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:15:36.322836759 +0000 UTC m=+24.628034942" watchObservedRunningTime="2026-07-01 17:15:38.338641046 +0000 UTC m=+26.643839222"
Jul 01 17:15:39.167440 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:39.167411    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:15:39.167638 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:39.167408    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knvdq"
Jul 01 17:15:39.167638 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:39.167537    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rrth" podUID="5026d3ed-589a-4390-a2b7-d5f64c06e687"
Jul 01 17:15:39.167638 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:39.167593    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x2qg"
Jul 01 17:15:39.167773 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:39.167704    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knvdq" podUID="3defaf9e-f5ef-45ea-a02e-4729f15db90c"
Jul 01 17:15:39.167832 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:39.167792    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x2qg" podUID="415f427a-adac-47bb-9cad-3befc6a711b6"
Jul 01 17:15:39.710244 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:39.710095    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mgc4x"
Jul 01 17:15:39.710845 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:39.710826    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mgc4x"
Jul 01 17:15:39.726350 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:39.726311    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9zcq" podStartSLOduration=5.1428009 podStartE2EDuration="27.726295236s" totalImagesPullingTime="22.583494336s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 17:15:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:15:38.339313496 +0000 UTC m=+26.644511674" watchObservedRunningTime="2026-07-01 17:15:39.726295236 +0000 UTC m=+28.031493413"
Jul 01 17:15:40.327778 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:40.327755    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8tsw_500d2a90-2d21-42b3-9584-b04c14becaec/ovn-acl-logging/0.log"
Jul 01 17:15:40.328142 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:40.328116    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw" event={"ID":"500d2a90-2d21-42b3-9584-b04c14becaec","Type":"ContainerStarted","Data":"9a7c8997517b9ede16eb69a2629224e376d75ed38fdf9f6239b537b3f32d3062"}
Jul 01 17:15:40.328395 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:40.328372    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:40.328559 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:40.328544    2585 scope.go:117] "RemoveContainer" containerID="1aaa03a662b9c93313b88b2f707e02a23dc589bc52b54bd1cc1d8d8e28ee0e9d"
Jul 01 17:15:40.330264 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:40.330241    2585 generic.go:356] "Generic (PLEG): container finished" podID="7bab8af3-8766-4098-95ab-720089120d08" containerID="7cca2ee9c8a1cb7aafd6039f5e83e77bdc879d4a270ed0f2930c550a8d7c67ca" exitCode=0
Jul 01 17:15:40.330376 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:40.330306    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hjngx" event={"ID":"7bab8af3-8766-4098-95ab-720089120d08","Type":"ContainerDied","Data":"7cca2ee9c8a1cb7aafd6039f5e83e77bdc879d4a270ed0f2930c550a8d7c67ca"}
Jul 01 17:15:40.330505 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:40.330480    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mgc4x"
Jul 01 17:15:40.331062 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:40.330988    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mgc4x"
Jul 01 17:15:40.345313 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:40.345293    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:41.149116 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:41.148863    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:41.167121 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:41.167094    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knvdq"
Jul 01 17:15:41.167121 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:41.167113    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:15:41.167327 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:41.167094    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x2qg"
Jul 01 17:15:41.167327 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:41.167214    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knvdq" podUID="3defaf9e-f5ef-45ea-a02e-4729f15db90c"
Jul 01 17:15:41.167416 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:41.167324    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rrth" podUID="5026d3ed-589a-4390-a2b7-d5f64c06e687"
Jul 01 17:15:41.167473 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:41.167446    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x2qg" podUID="415f427a-adac-47bb-9cad-3befc6a711b6"
Jul 01 17:15:41.336055 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:41.336031    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8tsw_500d2a90-2d21-42b3-9584-b04c14becaec/ovn-acl-logging/0.log"
Jul 01 17:15:41.336595 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:41.336558    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw" event={"ID":"500d2a90-2d21-42b3-9584-b04c14becaec","Type":"ContainerStarted","Data":"979ea6c8607a248fb3d840cdd6c0cd4d0d86f5dfb3f952aab76afd05af7de8c1"}
Jul 01 17:15:41.336837 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:41.336806    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:41.352736 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:41.352710    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:15:41.371663 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:41.371460    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw" podStartSLOduration=8.974464051 podStartE2EDuration="29.371438413s" totalImagesPullingTime="20.396974362s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 17:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:15:41.369801989 +0000 UTC m=+29.675000180" watchObservedRunningTime="2026-07-01 17:15:41.371438413 +0000 UTC m=+29.676636568"
Jul 01 17:15:41.665231 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:41.665200    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2x2qg"]
Jul 01 17:15:41.665410 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:41.665344    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x2qg"
Jul 01 17:15:41.665497 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:41.665456    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x2qg" podUID="415f427a-adac-47bb-9cad-3befc6a711b6"
Jul 01 17:15:41.668442 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:41.668422    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-knvdq"]
Jul 01 17:15:41.668615 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:41.668519    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knvdq"
Jul 01 17:15:41.668683 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:41.668608    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knvdq" podUID="3defaf9e-f5ef-45ea-a02e-4729f15db90c"
Jul 01 17:15:41.669041 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:41.669018    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4rrth"]
Jul 01 17:15:41.669143 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:41.669130    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:15:41.669258 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:41.669237    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rrth" podUID="5026d3ed-589a-4390-a2b7-d5f64c06e687"
Jul 01 17:15:42.340866 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:42.340829    2585 generic.go:356] "Generic (PLEG): container finished" podID="7bab8af3-8766-4098-95ab-720089120d08" containerID="e2d7151cf5a63303350ec639c7034ceaa850186bfbb992b11ca8950ae6432673" exitCode=0
Jul 01 17:15:42.341250 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:42.340907    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hjngx" event={"ID":"7bab8af3-8766-4098-95ab-720089120d08","Type":"ContainerDied","Data":"e2d7151cf5a63303350ec639c7034ceaa850186bfbb992b11ca8950ae6432673"}
Jul 01 17:15:43.167501 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:43.167422    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knvdq"
Jul 01 17:15:43.167662 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:43.167524    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:15:43.167662 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:43.167530    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knvdq" podUID="3defaf9e-f5ef-45ea-a02e-4729f15db90c"
Jul 01 17:15:43.167662 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:43.167556    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x2qg"
Jul 01 17:15:43.167809 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:43.167672    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rrth" podUID="5026d3ed-589a-4390-a2b7-d5f64c06e687"
Jul 01 17:15:43.167862 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:43.167777    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x2qg" podUID="415f427a-adac-47bb-9cad-3befc6a711b6"
Jul 01 17:15:44.350255 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:44.350216    2585 generic.go:356] "Generic (PLEG): container finished" podID="7bab8af3-8766-4098-95ab-720089120d08" containerID="3d1f0767a68fb8e6e26635072fe3a218637f9f8cfc99f8ce5039ed528f794750" exitCode=0
Jul 01 17:15:44.350675 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:44.350273    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hjngx" event={"ID":"7bab8af3-8766-4098-95ab-720089120d08","Type":"ContainerDied","Data":"3d1f0767a68fb8e6e26635072fe3a218637f9f8cfc99f8ce5039ed528f794750"}
Jul 01 17:15:44.853341 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:44.853309    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5026d3ed-589a-4390-a2b7-d5f64c06e687-metrics-certs\") pod \"network-metrics-daemon-4rrth\" (UID: \"5026d3ed-589a-4390-a2b7-d5f64c06e687\") " pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:15:44.853341 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:44.853348    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3defaf9e-f5ef-45ea-a02e-4729f15db90c-original-pull-secret\") pod \"global-pull-secret-syncer-knvdq\" (UID: \"3defaf9e-f5ef-45ea-a02e-4729f15db90c\") " pod="kube-system/global-pull-secret-syncer-knvdq"
Jul 01 17:15:44.853568 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:44.853470    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 17:15:44.853568 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:44.853474    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 17:15:44.853568 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:44.853538    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3defaf9e-f5ef-45ea-a02e-4729f15db90c-original-pull-secret podName:3defaf9e-f5ef-45ea-a02e-4729f15db90c nodeName:}" failed. No retries permitted until 2026-07-01 17:16:16.853520385 +0000 UTC m=+65.158718541 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3defaf9e-f5ef-45ea-a02e-4729f15db90c-original-pull-secret") pod "global-pull-secret-syncer-knvdq" (UID: "3defaf9e-f5ef-45ea-a02e-4729f15db90c") : object "kube-system"/"original-pull-secret" not registered
Jul 01 17:15:44.853568 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:44.853558    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5026d3ed-589a-4390-a2b7-d5f64c06e687-metrics-certs podName:5026d3ed-589a-4390-a2b7-d5f64c06e687 nodeName:}" failed. No retries permitted until 2026-07-01 17:16:16.853548809 +0000 UTC m=+65.158746968 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5026d3ed-589a-4390-a2b7-d5f64c06e687-metrics-certs") pod "network-metrics-daemon-4rrth" (UID: "5026d3ed-589a-4390-a2b7-d5f64c06e687") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 17:15:44.954655 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:44.954622    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f6dj\" (UniqueName: \"kubernetes.io/projected/415f427a-adac-47bb-9cad-3befc6a711b6-kube-api-access-7f6dj\") pod \"network-check-target-2x2qg\" (UID: \"415f427a-adac-47bb-9cad-3befc6a711b6\") " pod="openshift-network-diagnostics/network-check-target-2x2qg"
Jul 01 17:15:44.954847 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:44.954826    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 17:15:44.954903 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:44.954860    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 17:15:44.954903 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:44.954872    2585 projected.go:196] Error preparing data for projected volume kube-api-access-7f6dj for pod openshift-network-diagnostics/network-check-target-2x2qg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 17:15:44.954982 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:44.954936    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/415f427a-adac-47bb-9cad-3befc6a711b6-kube-api-access-7f6dj podName:415f427a-adac-47bb-9cad-3befc6a711b6 nodeName:}" failed. No retries permitted until 2026-07-01 17:16:16.954920466 +0000 UTC m=+65.260118621 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7f6dj" (UniqueName: "kubernetes.io/projected/415f427a-adac-47bb-9cad-3befc6a711b6-kube-api-access-7f6dj") pod "network-check-target-2x2qg" (UID: "415f427a-adac-47bb-9cad-3befc6a711b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 17:15:45.167952 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:45.167868    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:15:45.167952 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:45.167905    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x2qg"
Jul 01 17:15:45.168158 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:45.167868    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knvdq"
Jul 01 17:15:45.168158 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:45.167994    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rrth" podUID="5026d3ed-589a-4390-a2b7-d5f64c06e687"
Jul 01 17:15:45.168158 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:45.168045    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x2qg" podUID="415f427a-adac-47bb-9cad-3befc6a711b6"
Jul 01 17:15:45.168158 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:45.168117    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knvdq" podUID="3defaf9e-f5ef-45ea-a02e-4729f15db90c"
Jul 01 17:15:47.002993 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.002815    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeReady"
Jul 01 17:15:47.003433 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.003104    2585 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 01 17:15:47.041885 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.041848    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-59d56646cd-98p6s"]
Jul 01 17:15:47.061667 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.061627    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59d56646cd-98p6s"]
Jul 01 17:15:47.061835 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.061719    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:15:47.064013 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.063443    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-phc4p"]
Jul 01 17:15:47.066278 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.066184    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 01 17:15:47.066278 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.066215    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 01 17:15:47.066793 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.066223    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 01 17:15:47.066793 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.066453    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jkmfz\""
Jul 01 17:15:47.071283 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.071263    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 01 17:15:47.077682 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.077651    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674f6d459d-8fs25"]
Jul 01 17:15:47.077833 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.077776    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-phc4p"
Jul 01 17:15:47.081968 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.081947    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-2klvb\""
Jul 01 17:15:47.082383 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.082365    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 01 17:15:47.083440 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.083405    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 01 17:15:47.093923 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.093897    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c957b7d94-9xj75"]
Jul 01 17:15:47.094047 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.094029    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674f6d459d-8fs25"
Jul 01 17:15:47.097655 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.097636    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 01 17:15:47.097789 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.097679    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-4j2wh\""
Jul 01 17:15:47.097789 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.097634    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 01 17:15:47.097972 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.097953    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 01 17:15:47.098116 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.098081    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 01 17:15:47.109791 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.109768    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74ff8b899b-6zbhz"]
Jul 01 17:15:47.109902 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.109888    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c957b7d94-9xj75"
Jul 01 17:15:47.113092 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.113075    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 01 17:15:47.130146 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.130123    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-n2fz9"]
Jul 01 17:15:47.130285 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.130267    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74ff8b899b-6zbhz"
Jul 01 17:15:47.132861 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.132840    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 01 17:15:47.133098 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.133084    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 01 17:15:47.133867 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.133841    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 01 17:15:47.137658 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.137639    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 01 17:15:47.148170 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.148144    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jq2fj"]
Jul 01 17:15:47.148281 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.148261    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n2fz9"
Jul 01 17:15:47.150974 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.150952    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pcm4c\""
Jul 01 17:15:47.151325 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.151307    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 01 17:15:47.151428 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.151313    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 01 17:15:47.168269 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.168250    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-phc4p"]
Jul 01 17:15:47.168369 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.168280    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674f6d459d-8fs25"]
Jul 01 17:15:47.168369 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.168294    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c957b7d94-9xj75"]
Jul 01 17:15:47.168369 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.168308    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jq2fj"]
Jul 01 17:15:47.168369 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.168321    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74ff8b899b-6zbhz"]
Jul 01 17:15:47.168369 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.168334    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n2fz9"]
Jul 01 17:15:47.168369 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.168345    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:15:47.168706 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.168349    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jq2fj"
Jul 01 17:15:47.168706 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.168346    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x2qg"
Jul 01 17:15:47.168706 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.168510    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knvdq"
Jul 01 17:15:47.171285 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.171266    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gnmdh\""
Jul 01 17:15:47.171435 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.171405    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5zq94\""
Jul 01 17:15:47.171537 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.171494    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 17:15:47.171615 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.171553    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dece537c-3a74-44b7-b4bf-ef9fbd0f1d01-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-phc4p\" (UID: \"dece537c-3a74-44b7-b4bf-ef9fbd0f1d01\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-phc4p"
Jul 01 17:15:47.171673 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.171613    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 17:15:47.171673 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.171608    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/33dc1940-f452-4f4e-a62b-c07096cee771-registry-certificates\") pod \"image-registry-59d56646cd-98p6s\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") " pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:15:47.171769 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.171681    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dece537c-3a74-44b7-b4bf-ef9fbd0f1d01-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-phc4p\" (UID: \"dece537c-3a74-44b7-b4bf-ef9fbd0f1d01\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-phc4p"
Jul 01 17:15:47.171769 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.171726    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/33dc1940-f452-4f4e-a62b-c07096cee771-image-registry-private-configuration\") pod \"image-registry-59d56646cd-98p6s\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") " pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:15:47.171769 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.171730    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 17:15:47.171769 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.171747    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-bound-sa-token\") pod \"image-registry-59d56646cd-98p6s\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") " pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:15:47.171769 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.171759    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 01 17:15:47.172004 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.171793    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4clc\" (UniqueName: \"kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-kube-api-access-t4clc\") pod \"image-registry-59d56646cd-98p6s\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") " pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:15:47.172004 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.171851    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33dc1940-f452-4f4e-a62b-c07096cee771-trusted-ca\") pod \"image-registry-59d56646cd-98p6s\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") " pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:15:47.172004 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.171920    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 01 17:15:47.172004 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.171970    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 17:15:47.172004 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.171984    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-875gr\""
Jul 01 17:15:47.172233 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.172045    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 01 17:15:47.172233 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.171922    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-registry-tls\") pod \"image-registry-59d56646cd-98p6s\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") " pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:15:47.172233 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.172098    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/33dc1940-f452-4f4e-a62b-c07096cee771-ca-trust-extracted\") pod \"image-registry-59d56646cd-98p6s\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") " pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:15:47.172233 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.172125    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/33dc1940-f452-4f4e-a62b-c07096cee771-installation-pull-secrets\") pod \"image-registry-59d56646cd-98p6s\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") " pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:15:47.172233 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.172168    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc7xj\" (UniqueName: \"kubernetes.io/projected/816f87d3-b783-4ee7-82bd-d68ff690c9f6-kube-api-access-nc7xj\") pod \"managed-serviceaccount-addon-agent-674f6d459d-8fs25\" (UID: \"816f87d3-b783-4ee7-82bd-d68ff690c9f6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674f6d459d-8fs25"
Jul 01 17:15:47.172233 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.172199    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/816f87d3-b783-4ee7-82bd-d68ff690c9f6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-674f6d459d-8fs25\" (UID: \"816f87d3-b783-4ee7-82bd-d68ff690c9f6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674f6d459d-8fs25"
Jul 01 17:15:47.273494 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.273397    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dece537c-3a74-44b7-b4bf-ef9fbd0f1d01-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-phc4p\" (UID: \"dece537c-3a74-44b7-b4bf-ef9fbd0f1d01\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-phc4p"
Jul 01 17:15:47.273494 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.273453    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-config-volume\") pod \"dns-default-n2fz9\" (UID: \"67bb5c8b-7164-4be3-a11a-9d6755b5a6be\") " pod="openshift-dns/dns-default-n2fz9"
Jul 01 17:15:47.273749 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.273512    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69mgz\" (UniqueName: \"kubernetes.io/projected/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-kube-api-access-69mgz\") pod \"dns-default-n2fz9\" (UID: \"67bb5c8b-7164-4be3-a11a-9d6755b5a6be\") " pod="openshift-dns/dns-default-n2fz9"
Jul 01 17:15:47.273749 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.273548    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4601677c-35cd-4c70-95c6-1d3d6670235f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-74ff8b899b-6zbhz\" (UID: \"4601677c-35cd-4c70-95c6-1d3d6670235f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74ff8b899b-6zbhz"
Jul 01 17:15:47.273749 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.273588    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-bound-sa-token\") pod \"image-registry-59d56646cd-98p6s\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") " pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:15:47.273749 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.273611    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4clc\" (UniqueName: \"kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-kube-api-access-t4clc\") pod \"image-registry-59d56646cd-98p6s\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") " pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:15:47.273749 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.273641    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33dc1940-f452-4f4e-a62b-c07096cee771-trusted-ca\") pod \"image-registry-59d56646cd-98p6s\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") " pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:15:47.273749 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.273663    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mblr\" (UniqueName: \"kubernetes.io/projected/e5892b19-9801-4f95-aaae-6e8328bdca92-kube-api-access-5mblr\") pod \"klusterlet-addon-workmgr-5c957b7d94-9xj75\" (UID: \"e5892b19-9801-4f95-aaae-6e8328bdca92\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c957b7d94-9xj75"
Jul 01 17:15:47.273749 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.273703    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-tmp-dir\") pod \"dns-default-n2fz9\" (UID: \"67bb5c8b-7164-4be3-a11a-9d6755b5a6be\") " pod="openshift-dns/dns-default-n2fz9"
Jul 01 17:15:47.273749 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.273724    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-registry-tls\") pod \"image-registry-59d56646cd-98p6s\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") " pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:15:47.273749 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.273739    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4601677c-35cd-4c70-95c6-1d3d6670235f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-74ff8b899b-6zbhz\" (UID: \"4601677c-35cd-4c70-95c6-1d3d6670235f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74ff8b899b-6zbhz"
Jul 01 17:15:47.274151 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.273790    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4601677c-35cd-4c70-95c6-1d3d6670235f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-74ff8b899b-6zbhz\" (UID: \"4601677c-35cd-4c70-95c6-1d3d6670235f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74ff8b899b-6zbhz"
Jul 01 17:15:47.274151 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.273814    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d75jl\" (UniqueName: \"kubernetes.io/projected/6ec33cc2-606d-4445-9f9b-4fe556dfbc6e-kube-api-access-d75jl\") pod \"ingress-canary-jq2fj\" (UID: \"6ec33cc2-606d-4445-9f9b-4fe556dfbc6e\") " pod="openshift-ingress-canary/ingress-canary-jq2fj"
Jul 01 17:15:47.274151 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.273838    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/33dc1940-f452-4f4e-a62b-c07096cee771-registry-certificates\") pod \"image-registry-59d56646cd-98p6s\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") " pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:15:47.274151 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.273856    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dece537c-3a74-44b7-b4bf-ef9fbd0f1d01-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-phc4p\" (UID: \"dece537c-3a74-44b7-b4bf-ef9fbd0f1d01\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-phc4p"
Jul 01 17:15:47.274151 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.273880    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4601677c-35cd-4c70-95c6-1d3d6670235f-ca\") pod \"cluster-proxy-proxy-agent-74ff8b899b-6zbhz\" (UID: \"4601677c-35cd-4c70-95c6-1d3d6670235f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74ff8b899b-6zbhz"
Jul 01 17:15:47.274151 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:47.273895    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 17:15:47.274151 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:47.273913    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59d56646cd-98p6s: secret "image-registry-tls" not found
Jul 01 17:15:47.274151 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:47.273972    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-registry-tls podName:33dc1940-f452-4f4e-a62b-c07096cee771 nodeName:}" failed. No retries permitted until 2026-07-01 17:15:47.77395376 +0000 UTC m=+36.079151942 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-registry-tls") pod "image-registry-59d56646cd-98p6s" (UID: "33dc1940-f452-4f4e-a62b-c07096cee771") : secret "image-registry-tls" not found
Jul 01 17:15:47.274151 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.273902    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/33dc1940-f452-4f4e-a62b-c07096cee771-image-registry-private-configuration\") pod \"image-registry-59d56646cd-98p6s\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") " pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:15:47.274151 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.274074    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5892b19-9801-4f95-aaae-6e8328bdca92-tmp\") pod \"klusterlet-addon-workmgr-5c957b7d94-9xj75\" (UID: \"e5892b19-9801-4f95-aaae-6e8328bdca92\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c957b7d94-9xj75"
Jul 01 17:15:47.274151 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.274097    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e5892b19-9801-4f95-aaae-6e8328bdca92-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c957b7d94-9xj75\" (UID: \"e5892b19-9801-4f95-aaae-6e8328bdca92\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c957b7d94-9xj75"
Jul 01 17:15:47.274151 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.274119    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ec33cc2-606d-4445-9f9b-4fe556dfbc6e-cert\") pod \"ingress-canary-jq2fj\" (UID: \"6ec33cc2-606d-4445-9f9b-4fe556dfbc6e\") " pod="openshift-ingress-canary/ingress-canary-jq2fj"
Jul 01 17:15:47.274151 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.274142    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4601677c-35cd-4c70-95c6-1d3d6670235f-hub\") pod \"cluster-proxy-proxy-agent-74ff8b899b-6zbhz\" (UID: \"4601677c-35cd-4c70-95c6-1d3d6670235f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74ff8b899b-6zbhz"
Jul 01 17:15:47.274151 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.274157    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-metrics-tls\") pod \"dns-default-n2fz9\" (UID: \"67bb5c8b-7164-4be3-a11a-9d6755b5a6be\") " pod="openshift-dns/dns-default-n2fz9"
Jul 01 17:15:47.274855 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.274188    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/33dc1940-f452-4f4e-a62b-c07096cee771-ca-trust-extracted\") pod \"image-registry-59d56646cd-98p6s\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") " pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:15:47.274855 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.274213    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/33dc1940-f452-4f4e-a62b-c07096cee771-installation-pull-secrets\") pod \"image-registry-59d56646cd-98p6s\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") " pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:15:47.274855 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.274232    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc7xj\" (UniqueName: \"kubernetes.io/projected/816f87d3-b783-4ee7-82bd-d68ff690c9f6-kube-api-access-nc7xj\") pod \"managed-serviceaccount-addon-agent-674f6d459d-8fs25\" (UID: \"816f87d3-b783-4ee7-82bd-d68ff690c9f6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674f6d459d-8fs25"
Jul 01 17:15:47.274855 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:47.274251    2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 17:15:47.274855 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:47.274301    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dece537c-3a74-44b7-b4bf-ef9fbd0f1d01-networking-console-plugin-cert podName:dece537c-3a74-44b7-b4bf-ef9fbd0f1d01 nodeName:}" failed. No retries permitted until 2026-07-01 17:15:47.774285852 +0000 UTC m=+36.079484010 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dece537c-3a74-44b7-b4bf-ef9fbd0f1d01-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-phc4p" (UID: "dece537c-3a74-44b7-b4bf-ef9fbd0f1d01") : secret "networking-console-plugin-cert" not found
Jul 01 17:15:47.274855 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.274250    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/816f87d3-b783-4ee7-82bd-d68ff690c9f6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-674f6d459d-8fs25\" (UID: \"816f87d3-b783-4ee7-82bd-d68ff690c9f6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674f6d459d-8fs25"
Jul 01 17:15:47.274855 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.274554    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcqgc\" (UniqueName: \"kubernetes.io/projected/4601677c-35cd-4c70-95c6-1d3d6670235f-kube-api-access-kcqgc\") pod \"cluster-proxy-proxy-agent-74ff8b899b-6zbhz\" (UID: \"4601677c-35cd-4c70-95c6-1d3d6670235f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74ff8b899b-6zbhz"
Jul 01 17:15:47.274855 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.274595    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dece537c-3a74-44b7-b4bf-ef9fbd0f1d01-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-phc4p\" (UID: \"dece537c-3a74-44b7-b4bf-ef9fbd0f1d01\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-phc4p"
Jul 01 17:15:47.274855 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.274778    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/33dc1940-f452-4f4e-a62b-c07096cee771-registry-certificates\") pod \"image-registry-59d56646cd-98p6s\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") " pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:15:47.275268 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.274903    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33dc1940-f452-4f4e-a62b-c07096cee771-trusted-ca\") pod \"image-registry-59d56646cd-98p6s\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") " pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:15:47.275268 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.274923    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/33dc1940-f452-4f4e-a62b-c07096cee771-ca-trust-extracted\") pod \"image-registry-59d56646cd-98p6s\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") " pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:15:47.278524 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.278495    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/33dc1940-f452-4f4e-a62b-c07096cee771-installation-pull-secrets\") pod \"image-registry-59d56646cd-98p6s\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") " pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:15:47.278722 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.278645    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/816f87d3-b783-4ee7-82bd-d68ff690c9f6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-674f6d459d-8fs25\" (UID: \"816f87d3-b783-4ee7-82bd-d68ff690c9f6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674f6d459d-8fs25"
Jul 01 17:15:47.279727 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.279696    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/33dc1940-f452-4f4e-a62b-c07096cee771-image-registry-private-configuration\") pod \"image-registry-59d56646cd-98p6s\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") " pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:15:47.282598 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.282559    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4clc\" (UniqueName: \"kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-kube-api-access-t4clc\") pod \"image-registry-59d56646cd-98p6s\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") " pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:15:47.282695 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.282600    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-bound-sa-token\") pod \"image-registry-59d56646cd-98p6s\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") " pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:15:47.283336 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.283311    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc7xj\" (UniqueName: \"kubernetes.io/projected/816f87d3-b783-4ee7-82bd-d68ff690c9f6-kube-api-access-nc7xj\") pod \"managed-serviceaccount-addon-agent-674f6d459d-8fs25\" (UID: \"816f87d3-b783-4ee7-82bd-d68ff690c9f6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674f6d459d-8fs25"
Jul 01 17:15:47.375811 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.375769    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-config-volume\") pod \"dns-default-n2fz9\" (UID: \"67bb5c8b-7164-4be3-a11a-9d6755b5a6be\") " pod="openshift-dns/dns-default-n2fz9"
Jul 01 17:15:47.375811 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.375815    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69mgz\" (UniqueName: \"kubernetes.io/projected/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-kube-api-access-69mgz\") pod \"dns-default-n2fz9\" (UID: \"67bb5c8b-7164-4be3-a11a-9d6755b5a6be\") " pod="openshift-dns/dns-default-n2fz9"
Jul 01 17:15:47.376066 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.375844    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4601677c-35cd-4c70-95c6-1d3d6670235f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-74ff8b899b-6zbhz\" (UID: \"4601677c-35cd-4c70-95c6-1d3d6670235f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74ff8b899b-6zbhz"
Jul 01 17:15:47.376066 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.375884    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mblr\" (UniqueName: \"kubernetes.io/projected/e5892b19-9801-4f95-aaae-6e8328bdca92-kube-api-access-5mblr\") pod \"klusterlet-addon-workmgr-5c957b7d94-9xj75\" (UID: \"e5892b19-9801-4f95-aaae-6e8328bdca92\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c957b7d94-9xj75"
Jul 01 17:15:47.376066 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.375907    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-tmp-dir\") pod \"dns-default-n2fz9\" (UID: \"67bb5c8b-7164-4be3-a11a-9d6755b5a6be\") " pod="openshift-dns/dns-default-n2fz9"
Jul 01 17:15:47.376066 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.375936    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4601677c-35cd-4c70-95c6-1d3d6670235f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-74ff8b899b-6zbhz\" (UID: \"4601677c-35cd-4c70-95c6-1d3d6670235f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74ff8b899b-6zbhz"
Jul 01 17:15:47.376066 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.375984    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4601677c-35cd-4c70-95c6-1d3d6670235f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-74ff8b899b-6zbhz\" (UID: \"4601677c-35cd-4c70-95c6-1d3d6670235f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74ff8b899b-6zbhz"
Jul 01 17:15:47.376066 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.376006    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d75jl\" (UniqueName: \"kubernetes.io/projected/6ec33cc2-606d-4445-9f9b-4fe556dfbc6e-kube-api-access-d75jl\") pod \"ingress-canary-jq2fj\" (UID: \"6ec33cc2-606d-4445-9f9b-4fe556dfbc6e\") " pod="openshift-ingress-canary/ingress-canary-jq2fj"
Jul 01 17:15:47.376066 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.376040    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4601677c-35cd-4c70-95c6-1d3d6670235f-ca\") pod \"cluster-proxy-proxy-agent-74ff8b899b-6zbhz\" (UID: \"4601677c-35cd-4c70-95c6-1d3d6670235f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74ff8b899b-6zbhz"
Jul 01 17:15:47.376066 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.376066    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5892b19-9801-4f95-aaae-6e8328bdca92-tmp\") pod \"klusterlet-addon-workmgr-5c957b7d94-9xj75\" (UID: \"e5892b19-9801-4f95-aaae-6e8328bdca92\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c957b7d94-9xj75"
Jul 01 17:15:47.376420 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.376089    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e5892b19-9801-4f95-aaae-6e8328bdca92-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c957b7d94-9xj75\" (UID: \"e5892b19-9801-4f95-aaae-6e8328bdca92\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c957b7d94-9xj75"
Jul 01 17:15:47.376420 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.376108    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ec33cc2-606d-4445-9f9b-4fe556dfbc6e-cert\") pod \"ingress-canary-jq2fj\" (UID: \"6ec33cc2-606d-4445-9f9b-4fe556dfbc6e\") " pod="openshift-ingress-canary/ingress-canary-jq2fj"
Jul 01 17:15:47.376420 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.376122    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4601677c-35cd-4c70-95c6-1d3d6670235f-hub\") pod \"cluster-proxy-proxy-agent-74ff8b899b-6zbhz\" (UID: \"4601677c-35cd-4c70-95c6-1d3d6670235f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74ff8b899b-6zbhz"
Jul 01 17:15:47.376420 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.376138    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-metrics-tls\") pod \"dns-default-n2fz9\" (UID: \"67bb5c8b-7164-4be3-a11a-9d6755b5a6be\") " pod="openshift-dns/dns-default-n2fz9"
Jul 01 17:15:47.376420 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.376180    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcqgc\" (UniqueName: \"kubernetes.io/projected/4601677c-35cd-4c70-95c6-1d3d6670235f-kube-api-access-kcqgc\") pod \"cluster-proxy-proxy-agent-74ff8b899b-6zbhz\" (UID: \"4601677c-35cd-4c70-95c6-1d3d6670235f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74ff8b899b-6zbhz"
Jul 01 17:15:47.377749 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.377119    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-config-volume\") pod \"dns-default-n2fz9\" (UID: \"67bb5c8b-7164-4be3-a11a-9d6755b5a6be\") " pod="openshift-dns/dns-default-n2fz9"
Jul 01 17:15:47.377749 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.377221    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-tmp-dir\") pod \"dns-default-n2fz9\" (UID: \"67bb5c8b-7164-4be3-a11a-9d6755b5a6be\") " pod="openshift-dns/dns-default-n2fz9"
Jul 01 17:15:47.377749 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.377377    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5892b19-9801-4f95-aaae-6e8328bdca92-tmp\") pod \"klusterlet-addon-workmgr-5c957b7d94-9xj75\" (UID: \"e5892b19-9801-4f95-aaae-6e8328bdca92\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c957b7d94-9xj75"
Jul 01 17:15:47.377749 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.377475    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4601677c-35cd-4c70-95c6-1d3d6670235f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-74ff8b899b-6zbhz\" (UID: \"4601677c-35cd-4c70-95c6-1d3d6670235f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74ff8b899b-6zbhz"
Jul 01 17:15:47.377749 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:47.377487    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 17:15:47.377749 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:47.377725    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-metrics-tls podName:67bb5c8b-7164-4be3-a11a-9d6755b5a6be nodeName:}" failed. No retries permitted until 2026-07-01 17:15:47.877661481 +0000 UTC m=+36.182859637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-metrics-tls") pod "dns-default-n2fz9" (UID: "67bb5c8b-7164-4be3-a11a-9d6755b5a6be") : secret "dns-default-metrics-tls" not found
Jul 01 17:15:47.380068 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.380044    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4601677c-35cd-4c70-95c6-1d3d6670235f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-74ff8b899b-6zbhz\" (UID: \"4601677c-35cd-4c70-95c6-1d3d6670235f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74ff8b899b-6zbhz"
Jul 01 17:15:47.380551 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.380501    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ec33cc2-606d-4445-9f9b-4fe556dfbc6e-cert\") pod \"ingress-canary-jq2fj\" (UID: \"6ec33cc2-606d-4445-9f9b-4fe556dfbc6e\") " pod="openshift-ingress-canary/ingress-canary-jq2fj"
Jul 01 17:15:47.380664 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.380617    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4601677c-35cd-4c70-95c6-1d3d6670235f-hub\") pod \"cluster-proxy-proxy-agent-74ff8b899b-6zbhz\" (UID: \"4601677c-35cd-4c70-95c6-1d3d6670235f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74ff8b899b-6zbhz"
Jul 01 17:15:47.380745 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.380667    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4601677c-35cd-4c70-95c6-1d3d6670235f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-74ff8b899b-6zbhz\" (UID: \"4601677c-35cd-4c70-95c6-1d3d6670235f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74ff8b899b-6zbhz"
Jul 01 17:15:47.380801 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.380766    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e5892b19-9801-4f95-aaae-6e8328bdca92-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c957b7d94-9xj75\" (UID: \"e5892b19-9801-4f95-aaae-6e8328bdca92\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c957b7d94-9xj75"
Jul 01 17:15:47.380856 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.380795    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4601677c-35cd-4c70-95c6-1d3d6670235f-ca\") pod \"cluster-proxy-proxy-agent-74ff8b899b-6zbhz\" (UID: \"4601677c-35cd-4c70-95c6-1d3d6670235f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74ff8b899b-6zbhz"
Jul 01 17:15:47.386135 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.386113    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d75jl\" (UniqueName: \"kubernetes.io/projected/6ec33cc2-606d-4445-9f9b-4fe556dfbc6e-kube-api-access-d75jl\") pod \"ingress-canary-jq2fj\" (UID: \"6ec33cc2-606d-4445-9f9b-4fe556dfbc6e\") " pod="openshift-ingress-canary/ingress-canary-jq2fj"
Jul 01 17:15:47.386228 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.386114    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcqgc\" (UniqueName: \"kubernetes.io/projected/4601677c-35cd-4c70-95c6-1d3d6670235f-kube-api-access-kcqgc\") pod \"cluster-proxy-proxy-agent-74ff8b899b-6zbhz\" (UID: \"4601677c-35cd-4c70-95c6-1d3d6670235f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74ff8b899b-6zbhz"
Jul 01 17:15:47.386292 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.386271    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69mgz\" (UniqueName: \"kubernetes.io/projected/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-kube-api-access-69mgz\") pod \"dns-default-n2fz9\" (UID: \"67bb5c8b-7164-4be3-a11a-9d6755b5a6be\") " pod="openshift-dns/dns-default-n2fz9"
Jul 01 17:15:47.387623 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.387556    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mblr\" (UniqueName: \"kubernetes.io/projected/e5892b19-9801-4f95-aaae-6e8328bdca92-kube-api-access-5mblr\") pod \"klusterlet-addon-workmgr-5c957b7d94-9xj75\" (UID: \"e5892b19-9801-4f95-aaae-6e8328bdca92\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c957b7d94-9xj75"
Jul 01 17:15:47.416652 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.416622    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674f6d459d-8fs25"
Jul 01 17:15:47.425817 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.425780    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c957b7d94-9xj75"
Jul 01 17:15:47.440011 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.439975    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74ff8b899b-6zbhz"
Jul 01 17:15:47.486461 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.486433    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jq2fj"
Jul 01 17:15:47.780524 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.780488    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dece537c-3a74-44b7-b4bf-ef9fbd0f1d01-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-phc4p\" (UID: \"dece537c-3a74-44b7-b4bf-ef9fbd0f1d01\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-phc4p"
Jul 01 17:15:47.780719 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.780621    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-registry-tls\") pod \"image-registry-59d56646cd-98p6s\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") " pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:15:47.780719 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:47.780661    2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 17:15:47.780719 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:47.780709    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 17:15:47.780719 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:47.780721    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59d56646cd-98p6s: secret "image-registry-tls" not found
Jul 01 17:15:47.780895 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:47.780734    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dece537c-3a74-44b7-b4bf-ef9fbd0f1d01-networking-console-plugin-cert podName:dece537c-3a74-44b7-b4bf-ef9fbd0f1d01 nodeName:}" failed. No retries permitted until 2026-07-01 17:15:48.780713511 +0000 UTC m=+37.085911678 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dece537c-3a74-44b7-b4bf-ef9fbd0f1d01-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-phc4p" (UID: "dece537c-3a74-44b7-b4bf-ef9fbd0f1d01") : secret "networking-console-plugin-cert" not found
Jul 01 17:15:47.780895 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:47.780754    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-registry-tls podName:33dc1940-f452-4f4e-a62b-c07096cee771 nodeName:}" failed. No retries permitted until 2026-07-01 17:15:48.780743301 +0000 UTC m=+37.085941470 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-registry-tls") pod "image-registry-59d56646cd-98p6s" (UID: "33dc1940-f452-4f4e-a62b-c07096cee771") : secret "image-registry-tls" not found
Jul 01 17:15:47.881642 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:47.881610    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-metrics-tls\") pod \"dns-default-n2fz9\" (UID: \"67bb5c8b-7164-4be3-a11a-9d6755b5a6be\") " pod="openshift-dns/dns-default-n2fz9"
Jul 01 17:15:47.881801 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:47.881751    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 17:15:47.881856 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:47.881809    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-metrics-tls podName:67bb5c8b-7164-4be3-a11a-9d6755b5a6be nodeName:}" failed. No retries permitted until 2026-07-01 17:15:48.881796122 +0000 UTC m=+37.186994277 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-metrics-tls") pod "dns-default-n2fz9" (UID: "67bb5c8b-7164-4be3-a11a-9d6755b5a6be") : secret "dns-default-metrics-tls" not found
Jul 01 17:15:48.788000 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:48.787967    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-registry-tls\") pod \"image-registry-59d56646cd-98p6s\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") " pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:15:48.788619 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:48.788029    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dece537c-3a74-44b7-b4bf-ef9fbd0f1d01-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-phc4p\" (UID: \"dece537c-3a74-44b7-b4bf-ef9fbd0f1d01\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-phc4p"
Jul 01 17:15:48.788619 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:48.788114    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 17:15:48.788619 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:48.788131    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59d56646cd-98p6s: secret "image-registry-tls" not found
Jul 01 17:15:48.788619 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:48.788132    2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 17:15:48.788619 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:48.788182    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dece537c-3a74-44b7-b4bf-ef9fbd0f1d01-networking-console-plugin-cert podName:dece537c-3a74-44b7-b4bf-ef9fbd0f1d01 nodeName:}" failed. No retries permitted until 2026-07-01 17:15:50.788169209 +0000 UTC m=+39.093367368 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dece537c-3a74-44b7-b4bf-ef9fbd0f1d01-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-phc4p" (UID: "dece537c-3a74-44b7-b4bf-ef9fbd0f1d01") : secret "networking-console-plugin-cert" not found
Jul 01 17:15:48.788619 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:48.788194    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-registry-tls podName:33dc1940-f452-4f4e-a62b-c07096cee771 nodeName:}" failed. No retries permitted until 2026-07-01 17:15:50.788188745 +0000 UTC m=+39.093386899 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-registry-tls") pod "image-registry-59d56646cd-98p6s" (UID: "33dc1940-f452-4f4e-a62b-c07096cee771") : secret "image-registry-tls" not found
Jul 01 17:15:48.796682 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:48.796643    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674f6d459d-8fs25"]
Jul 01 17:15:48.797625 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:48.797599    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jq2fj"]
Jul 01 17:15:48.809927 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:48.809909    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c957b7d94-9xj75"]
Jul 01 17:15:48.812777 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:48.812759    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74ff8b899b-6zbhz"]
Jul 01 17:15:48.817131 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:48.817105    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ec33cc2_606d_4445_9f9b_4fe556dfbc6e.slice/crio-65a9d315644c5c0b9e5792d6d3b2e5648504674f47758f48aca1aac48a433a90 WatchSource:0}: Error finding container 65a9d315644c5c0b9e5792d6d3b2e5648504674f47758f48aca1aac48a433a90: Status 404 returned error can't find the container with id 65a9d315644c5c0b9e5792d6d3b2e5648504674f47758f48aca1aac48a433a90
Jul 01 17:15:48.817494 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:48.817443    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod816f87d3_b783_4ee7_82bd_d68ff690c9f6.slice/crio-e8007715966dace4010a13c8a8a889e0cf29d8ed04b66142b3a950982c03bbea WatchSource:0}: Error finding container e8007715966dace4010a13c8a8a889e0cf29d8ed04b66142b3a950982c03bbea: Status 404 returned error can't find the container with id e8007715966dace4010a13c8a8a889e0cf29d8ed04b66142b3a950982c03bbea
Jul 01 17:15:48.818718 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:48.818698    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5892b19_9801_4f95_aaae_6e8328bdca92.slice/crio-bdb1cc05e52f5b996a8b024aa5d8174be3f4f3cf678702da1f59847d89abc8af WatchSource:0}: Error finding container bdb1cc05e52f5b996a8b024aa5d8174be3f4f3cf678702da1f59847d89abc8af: Status 404 returned error can't find the container with id bdb1cc05e52f5b996a8b024aa5d8174be3f4f3cf678702da1f59847d89abc8af
Jul 01 17:15:48.819087 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:15:48.819063    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4601677c_35cd_4c70_95c6_1d3d6670235f.slice/crio-9cd3aa689dfaae2b7e4169a9400cfdcb358cd6def3a8b8e92c636d80f9be4eb6 WatchSource:0}: Error finding container 9cd3aa689dfaae2b7e4169a9400cfdcb358cd6def3a8b8e92c636d80f9be4eb6: Status 404 returned error can't find the container with id 9cd3aa689dfaae2b7e4169a9400cfdcb358cd6def3a8b8e92c636d80f9be4eb6
Jul 01 17:15:48.888899 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:48.888861    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-metrics-tls\") pod \"dns-default-n2fz9\" (UID: \"67bb5c8b-7164-4be3-a11a-9d6755b5a6be\") " pod="openshift-dns/dns-default-n2fz9"
Jul 01 17:15:48.889051 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:48.889032    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 17:15:48.889112 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:48.889094    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-metrics-tls podName:67bb5c8b-7164-4be3-a11a-9d6755b5a6be nodeName:}" failed. No retries permitted until 2026-07-01 17:15:50.889078332 +0000 UTC m=+39.194276487 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-metrics-tls") pod "dns-default-n2fz9" (UID: "67bb5c8b-7164-4be3-a11a-9d6755b5a6be") : secret "dns-default-metrics-tls" not found
Jul 01 17:15:49.365045 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:49.364790    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674f6d459d-8fs25" event={"ID":"816f87d3-b783-4ee7-82bd-d68ff690c9f6","Type":"ContainerStarted","Data":"e8007715966dace4010a13c8a8a889e0cf29d8ed04b66142b3a950982c03bbea"}
Jul 01 17:15:49.365837 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:49.365812    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c957b7d94-9xj75" event={"ID":"e5892b19-9801-4f95-aaae-6e8328bdca92","Type":"ContainerStarted","Data":"bdb1cc05e52f5b996a8b024aa5d8174be3f4f3cf678702da1f59847d89abc8af"}
Jul 01 17:15:49.366767 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:49.366744    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jq2fj" event={"ID":"6ec33cc2-606d-4445-9f9b-4fe556dfbc6e","Type":"ContainerStarted","Data":"65a9d315644c5c0b9e5792d6d3b2e5648504674f47758f48aca1aac48a433a90"}
Jul 01 17:15:49.367677 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:49.367660    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74ff8b899b-6zbhz" event={"ID":"4601677c-35cd-4c70-95c6-1d3d6670235f","Type":"ContainerStarted","Data":"9cd3aa689dfaae2b7e4169a9400cfdcb358cd6def3a8b8e92c636d80f9be4eb6"}
Jul 01 17:15:49.370103 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:49.370083    2585 generic.go:356] "Generic (PLEG): container finished" podID="7bab8af3-8766-4098-95ab-720089120d08" containerID="f14f71918f8b51e3b55be2337033caa4db840803a99b29608173d9d767796579" exitCode=0
Jul 01 17:15:49.370169 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:49.370109    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hjngx" event={"ID":"7bab8af3-8766-4098-95ab-720089120d08","Type":"ContainerDied","Data":"f14f71918f8b51e3b55be2337033caa4db840803a99b29608173d9d767796579"}
Jul 01 17:15:50.384414 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:50.384375    2585 generic.go:356] "Generic (PLEG): container finished" podID="7bab8af3-8766-4098-95ab-720089120d08" containerID="8d4407137d8cfec995f643504928d63d04c7287e09d3c3cce9bd61d97d11c0c1" exitCode=0
Jul 01 17:15:50.384878 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:50.384465    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hjngx" event={"ID":"7bab8af3-8766-4098-95ab-720089120d08","Type":"ContainerDied","Data":"8d4407137d8cfec995f643504928d63d04c7287e09d3c3cce9bd61d97d11c0c1"}
Jul 01 17:15:50.805256 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:50.804942    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-registry-tls\") pod \"image-registry-59d56646cd-98p6s\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") " pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:15:50.805256 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:50.805030    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dece537c-3a74-44b7-b4bf-ef9fbd0f1d01-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-phc4p\" (UID: \"dece537c-3a74-44b7-b4bf-ef9fbd0f1d01\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-phc4p"
Jul 01 17:15:50.805256 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:50.805180    2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 17:15:50.805256 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:50.805245    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dece537c-3a74-44b7-b4bf-ef9fbd0f1d01-networking-console-plugin-cert podName:dece537c-3a74-44b7-b4bf-ef9fbd0f1d01 nodeName:}" failed. No retries permitted until 2026-07-01 17:15:54.805225748 +0000 UTC m=+43.110423926 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dece537c-3a74-44b7-b4bf-ef9fbd0f1d01-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-phc4p" (UID: "dece537c-3a74-44b7-b4bf-ef9fbd0f1d01") : secret "networking-console-plugin-cert" not found
Jul 01 17:15:50.806057 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:50.805671    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 17:15:50.806057 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:50.805695    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59d56646cd-98p6s: secret "image-registry-tls" not found
Jul 01 17:15:50.806057 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:50.805742    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-registry-tls podName:33dc1940-f452-4f4e-a62b-c07096cee771 nodeName:}" failed. No retries permitted until 2026-07-01 17:15:54.805725704 +0000 UTC m=+43.110923883 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-registry-tls") pod "image-registry-59d56646cd-98p6s" (UID: "33dc1940-f452-4f4e-a62b-c07096cee771") : secret "image-registry-tls" not found
Jul 01 17:15:50.906888 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:50.906226    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-metrics-tls\") pod \"dns-default-n2fz9\" (UID: \"67bb5c8b-7164-4be3-a11a-9d6755b5a6be\") " pod="openshift-dns/dns-default-n2fz9"
Jul 01 17:15:50.906888 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:50.906424    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 17:15:50.906888 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:50.906498    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-metrics-tls podName:67bb5c8b-7164-4be3-a11a-9d6755b5a6be nodeName:}" failed. No retries permitted until 2026-07-01 17:15:54.906470127 +0000 UTC m=+43.211668299 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-metrics-tls") pod "dns-default-n2fz9" (UID: "67bb5c8b-7164-4be3-a11a-9d6755b5a6be") : secret "dns-default-metrics-tls" not found
Jul 01 17:15:51.398636 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:51.398224    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hjngx" event={"ID":"7bab8af3-8766-4098-95ab-720089120d08","Type":"ContainerStarted","Data":"78f958e855d05afd0f3bb4ecfcf6005cf91df45648c22ad894ebc6597a00c566"}
Jul 01 17:15:51.428012 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:51.427788    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hjngx" podStartSLOduration=7.445552974 podStartE2EDuration="39.427766582s" totalImagesPullingTime="31.982213608s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 17:15:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:15:51.4265379 +0000 UTC m=+39.731736083" watchObservedRunningTime="2026-07-01 17:15:51.427766582 +0000 UTC m=+39.732964761"
Jul 01 17:15:54.842944 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:54.842902    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-registry-tls\") pod \"image-registry-59d56646cd-98p6s\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") " pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:15:54.843445 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:54.842980    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dece537c-3a74-44b7-b4bf-ef9fbd0f1d01-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-phc4p\" (UID: \"dece537c-3a74-44b7-b4bf-ef9fbd0f1d01\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-phc4p"
Jul 01 17:15:54.843445 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:54.843075    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 17:15:54.843445 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:54.843093    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59d56646cd-98p6s: secret "image-registry-tls" not found
Jul 01 17:15:54.843445 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:54.843095    2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 17:15:54.843445 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:54.843156    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-registry-tls podName:33dc1940-f452-4f4e-a62b-c07096cee771 nodeName:}" failed. No retries permitted until 2026-07-01 17:16:02.843136625 +0000 UTC m=+51.148334780 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-registry-tls") pod "image-registry-59d56646cd-98p6s" (UID: "33dc1940-f452-4f4e-a62b-c07096cee771") : secret "image-registry-tls" not found
Jul 01 17:15:54.843445 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:54.843169    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dece537c-3a74-44b7-b4bf-ef9fbd0f1d01-networking-console-plugin-cert podName:dece537c-3a74-44b7-b4bf-ef9fbd0f1d01 nodeName:}" failed. No retries permitted until 2026-07-01 17:16:02.843163724 +0000 UTC m=+51.148361879 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dece537c-3a74-44b7-b4bf-ef9fbd0f1d01-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-phc4p" (UID: "dece537c-3a74-44b7-b4bf-ef9fbd0f1d01") : secret "networking-console-plugin-cert" not found
Jul 01 17:15:54.943507 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:54.943470    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-metrics-tls\") pod \"dns-default-n2fz9\" (UID: \"67bb5c8b-7164-4be3-a11a-9d6755b5a6be\") " pod="openshift-dns/dns-default-n2fz9"
Jul 01 17:15:54.943704 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:54.943637    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 17:15:54.943704 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:15:54.943705    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-metrics-tls podName:67bb5c8b-7164-4be3-a11a-9d6755b5a6be nodeName:}" failed. No retries permitted until 2026-07-01 17:16:02.943686266 +0000 UTC m=+51.248884434 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-metrics-tls") pod "dns-default-n2fz9" (UID: "67bb5c8b-7164-4be3-a11a-9d6755b5a6be") : secret "dns-default-metrics-tls" not found
Jul 01 17:15:58.431739 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:58.431700    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674f6d459d-8fs25" event={"ID":"816f87d3-b783-4ee7-82bd-d68ff690c9f6","Type":"ContainerStarted","Data":"03c92c999e28edfa028ac4988778816a3928df11d465d3df7684469e940ed2c6"}
Jul 01 17:15:58.433168 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:58.433135    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c957b7d94-9xj75" event={"ID":"e5892b19-9801-4f95-aaae-6e8328bdca92","Type":"ContainerStarted","Data":"4796156abd9904a59e355fcc6ed8a1f38620a593aade2d4b75277843f742565c"}
Jul 01 17:15:58.433296 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:58.433188    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c957b7d94-9xj75"
Jul 01 17:15:58.434485 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:58.434449    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jq2fj" event={"ID":"6ec33cc2-606d-4445-9f9b-4fe556dfbc6e","Type":"ContainerStarted","Data":"4b7061898f3c87319d0ceef1f37b9b3a90d8278621f056f097f816ba5cb5bc35"}
Jul 01 17:15:58.435229 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:58.435209    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c957b7d94-9xj75"
Jul 01 17:15:58.436519 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:58.436498    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74ff8b899b-6zbhz" event={"ID":"4601677c-35cd-4c70-95c6-1d3d6670235f","Type":"ContainerStarted","Data":"a1e0cb810b5928f820eee1889ce9cfbe2dc93aca3a2eacfcf26b5581244007ed"}
Jul 01 17:15:58.436659 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:58.436529    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74ff8b899b-6zbhz" event={"ID":"4601677c-35cd-4c70-95c6-1d3d6670235f","Type":"ContainerStarted","Data":"98ee9a49f737330621519eb98be49af0055411ffd10b486b6ba64b515ad8a4cd"}
Jul 01 17:15:58.436659 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:58.436548    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74ff8b899b-6zbhz" event={"ID":"4601677c-35cd-4c70-95c6-1d3d6670235f","Type":"ContainerStarted","Data":"1698986164f0fcd3b67839ee87c473e50d294a03293a457ac17fdad8be928838"}
Jul 01 17:15:58.447271 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:58.447233    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674f6d459d-8fs25" podStartSLOduration=5.986967487 podStartE2EDuration="14.447220953s" totalImagesPullingTime="8.460253466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:15:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:15:58.446941602 +0000 UTC m=+46.752139854" watchObservedRunningTime="2026-07-01 17:15:58.447220953 +0000 UTC m=+46.752419130"
Jul 01 17:15:58.464221 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:58.464180    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c957b7d94-9xj75" podStartSLOduration=6.00276358 podStartE2EDuration="14.464166999s" totalImagesPullingTime="8.461403419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:15:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:15:58.46318192 +0000 UTC m=+46.768380096" watchObservedRunningTime="2026-07-01 17:15:58.464166999 +0000 UTC m=+46.769365154"
Jul 01 17:15:58.481164 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:58.481127    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74ff8b899b-6zbhz" podStartSLOduration=6.00407834 podStartE2EDuration="14.481114665s" totalImagesPullingTime="8.477036325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:15:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:15:58.480625815 +0000 UTC m=+46.785823994" watchObservedRunningTime="2026-07-01 17:15:58.481114665 +0000 UTC m=+46.786312842"
Jul 01 17:15:58.496738 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:15:58.496700    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jq2fj" podStartSLOduration=3.033854253 podStartE2EDuration="11.496687888s" totalImagesPullingTime="8.462833635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:15:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:15:58.495952888 +0000 UTC m=+46.801151064" watchObservedRunningTime="2026-07-01 17:15:58.496687888 +0000 UTC m=+46.801886065"
Jul 01 17:16:02.905895 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:16:02.905851    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dece537c-3a74-44b7-b4bf-ef9fbd0f1d01-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-phc4p\" (UID: \"dece537c-3a74-44b7-b4bf-ef9fbd0f1d01\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-phc4p"
Jul 01 17:16:02.906275 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:16:02.905929    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-registry-tls\") pod \"image-registry-59d56646cd-98p6s\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") " pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:16:02.906275 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:16:02.906012    2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 17:16:02.906275 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:16:02.906022    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 17:16:02.906275 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:16:02.906037    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59d56646cd-98p6s: secret "image-registry-tls" not found
Jul 01 17:16:02.906275 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:16:02.906078    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-registry-tls podName:33dc1940-f452-4f4e-a62b-c07096cee771 nodeName:}" failed. No retries permitted until 2026-07-01 17:16:18.906065825 +0000 UTC m=+67.211263984 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-registry-tls") pod "image-registry-59d56646cd-98p6s" (UID: "33dc1940-f452-4f4e-a62b-c07096cee771") : secret "image-registry-tls" not found
Jul 01 17:16:02.906275 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:16:02.906091    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dece537c-3a74-44b7-b4bf-ef9fbd0f1d01-networking-console-plugin-cert podName:dece537c-3a74-44b7-b4bf-ef9fbd0f1d01 nodeName:}" failed. No retries permitted until 2026-07-01 17:16:18.906083911 +0000 UTC m=+67.211282065 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dece537c-3a74-44b7-b4bf-ef9fbd0f1d01-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-phc4p" (UID: "dece537c-3a74-44b7-b4bf-ef9fbd0f1d01") : secret "networking-console-plugin-cert" not found
Jul 01 17:16:03.006707 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:16:03.006673    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-metrics-tls\") pod \"dns-default-n2fz9\" (UID: \"67bb5c8b-7164-4be3-a11a-9d6755b5a6be\") " pod="openshift-dns/dns-default-n2fz9"
Jul 01 17:16:03.006862 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:16:03.006819    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 17:16:03.006907 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:16:03.006883    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-metrics-tls podName:67bb5c8b-7164-4be3-a11a-9d6755b5a6be nodeName:}" failed. No retries permitted until 2026-07-01 17:16:19.006868654 +0000 UTC m=+67.312066812 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-metrics-tls") pod "dns-default-n2fz9" (UID: "67bb5c8b-7164-4be3-a11a-9d6755b5a6be") : secret "dns-default-metrics-tls" not found
Jul 01 17:16:13.356771 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:16:13.356742    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v8tsw"
Jul 01 17:16:16.917788 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:16:16.917734    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3defaf9e-f5ef-45ea-a02e-4729f15db90c-original-pull-secret\") pod \"global-pull-secret-syncer-knvdq\" (UID: \"3defaf9e-f5ef-45ea-a02e-4729f15db90c\") " pod="kube-system/global-pull-secret-syncer-knvdq"
Jul 01 17:16:16.918240 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:16:16.917827    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5026d3ed-589a-4390-a2b7-d5f64c06e687-metrics-certs\") pod \"network-metrics-daemon-4rrth\" (UID: \"5026d3ed-589a-4390-a2b7-d5f64c06e687\") " pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:16:16.921058 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:16:16.921038    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 17:16:16.921058 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:16:16.921053    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 17:16:16.928737 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:16:16.928712    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 01 17:16:16.928841 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:16:16.928782    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5026d3ed-589a-4390-a2b7-d5f64c06e687-metrics-certs podName:5026d3ed-589a-4390-a2b7-d5f64c06e687 nodeName:}" failed. No retries permitted until 2026-07-01 17:17:20.928765316 +0000 UTC m=+129.233963471 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5026d3ed-589a-4390-a2b7-d5f64c06e687-metrics-certs") pod "network-metrics-daemon-4rrth" (UID: "5026d3ed-589a-4390-a2b7-d5f64c06e687") : secret "metrics-daemon-secret" not found
Jul 01 17:16:16.932069 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:16:16.932047    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3defaf9e-f5ef-45ea-a02e-4729f15db90c-original-pull-secret\") pod \"global-pull-secret-syncer-knvdq\" (UID: \"3defaf9e-f5ef-45ea-a02e-4729f15db90c\") " pod="kube-system/global-pull-secret-syncer-knvdq"
Jul 01 17:16:17.018519 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:16:17.018463    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f6dj\" (UniqueName: \"kubernetes.io/projected/415f427a-adac-47bb-9cad-3befc6a711b6-kube-api-access-7f6dj\") pod \"network-check-target-2x2qg\" (UID: \"415f427a-adac-47bb-9cad-3befc6a711b6\") " pod="openshift-network-diagnostics/network-check-target-2x2qg"
Jul 01 17:16:17.021460 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:16:17.021443    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 17:16:17.031321 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:16:17.031300    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 17:16:17.042016 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:16:17.041988    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f6dj\" (UniqueName: \"kubernetes.io/projected/415f427a-adac-47bb-9cad-3befc6a711b6-kube-api-access-7f6dj\") pod \"network-check-target-2x2qg\" (UID: \"415f427a-adac-47bb-9cad-3befc6a711b6\") " pod="openshift-network-diagnostics/network-check-target-2x2qg"
Jul 01 17:16:17.196490 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:16:17.196413    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-875gr\""
Jul 01 17:16:17.199752 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:16:17.199737    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knvdq"
Jul 01 17:16:17.204532 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:16:17.204517    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x2qg"
Jul 01 17:16:17.332648 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:16:17.332612    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-knvdq"]
Jul 01 17:16:17.335507 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:16:17.335480    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3defaf9e_f5ef_45ea_a02e_4729f15db90c.slice/crio-aedd51e11a836b74bd86a970dc7b69a147ae93339e0ac990230aa4f968ccb93e WatchSource:0}: Error finding container aedd51e11a836b74bd86a970dc7b69a147ae93339e0ac990230aa4f968ccb93e: Status 404 returned error can't find the container with id aedd51e11a836b74bd86a970dc7b69a147ae93339e0ac990230aa4f968ccb93e
Jul 01 17:16:17.348354 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:16:17.348330    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2x2qg"]
Jul 01 17:16:17.351837 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:16:17.351811    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod415f427a_adac_47bb_9cad_3befc6a711b6.slice/crio-0f5efafede9fbd4200817cccc412b1f0d1df0a5709faedbcd9fcf53901a63052 WatchSource:0}: Error finding container 0f5efafede9fbd4200817cccc412b1f0d1df0a5709faedbcd9fcf53901a63052: Status 404 returned error can't find the container with id 0f5efafede9fbd4200817cccc412b1f0d1df0a5709faedbcd9fcf53901a63052
Jul 01 17:16:17.491116 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:16:17.491038    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2x2qg" event={"ID":"415f427a-adac-47bb-9cad-3befc6a711b6","Type":"ContainerStarted","Data":"0f5efafede9fbd4200817cccc412b1f0d1df0a5709faedbcd9fcf53901a63052"}
Jul 01 17:16:17.492206 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:16:17.492176    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-knvdq" event={"ID":"3defaf9e-f5ef-45ea-a02e-4729f15db90c","Type":"ContainerStarted","Data":"aedd51e11a836b74bd86a970dc7b69a147ae93339e0ac990230aa4f968ccb93e"}
Jul 01 17:16:18.934414 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:16:18.934379    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-registry-tls\") pod \"image-registry-59d56646cd-98p6s\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") " pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:16:18.935022 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:16:18.934445    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dece537c-3a74-44b7-b4bf-ef9fbd0f1d01-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-phc4p\" (UID: \"dece537c-3a74-44b7-b4bf-ef9fbd0f1d01\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-phc4p"
Jul 01 17:16:18.935022 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:16:18.934614    2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 17:16:18.935022 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:16:18.934672    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dece537c-3a74-44b7-b4bf-ef9fbd0f1d01-networking-console-plugin-cert podName:dece537c-3a74-44b7-b4bf-ef9fbd0f1d01 nodeName:}" failed. No retries permitted until 2026-07-01 17:16:50.934653891 +0000 UTC m=+99.239852064 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dece537c-3a74-44b7-b4bf-ef9fbd0f1d01-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-phc4p" (UID: "dece537c-3a74-44b7-b4bf-ef9fbd0f1d01") : secret "networking-console-plugin-cert" not found
Jul 01 17:16:18.935022 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:16:18.934993    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 17:16:18.935022 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:16:18.935009    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59d56646cd-98p6s: secret "image-registry-tls" not found
Jul 01 17:16:18.935275 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:16:18.935047    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-registry-tls podName:33dc1940-f452-4f4e-a62b-c07096cee771 nodeName:}" failed. No retries permitted until 2026-07-01 17:16:50.935037448 +0000 UTC m=+99.240235607 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-registry-tls") pod "image-registry-59d56646cd-98p6s" (UID: "33dc1940-f452-4f4e-a62b-c07096cee771") : secret "image-registry-tls" not found
Jul 01 17:16:19.035211 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:16:19.035165    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-metrics-tls\") pod \"dns-default-n2fz9\" (UID: \"67bb5c8b-7164-4be3-a11a-9d6755b5a6be\") " pod="openshift-dns/dns-default-n2fz9"
Jul 01 17:16:19.035389 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:16:19.035337    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 17:16:19.035442 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:16:19.035414    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-metrics-tls podName:67bb5c8b-7164-4be3-a11a-9d6755b5a6be nodeName:}" failed. No retries permitted until 2026-07-01 17:16:51.03539332 +0000 UTC m=+99.340591490 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-metrics-tls") pod "dns-default-n2fz9" (UID: "67bb5c8b-7164-4be3-a11a-9d6755b5a6be") : secret "dns-default-metrics-tls" not found
Jul 01 17:16:20.505367 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:16:20.505273    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2x2qg" event={"ID":"415f427a-adac-47bb-9cad-3befc6a711b6","Type":"ContainerStarted","Data":"c5d68b04b57113bf6359fb8d2ce6fb05e91fe01716c5b3102219b116b5d91f79"}
Jul 01 17:16:20.505821 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:16:20.505413    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2x2qg"
Jul 01 17:16:20.524971 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:16:20.524918    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2x2qg" podStartSLOduration=65.672469497 podStartE2EDuration="1m8.524903162s" totalImagesPullingTime="2.852433665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:16:20.523705598 +0000 UTC m=+68.828903776" watchObservedRunningTime="2026-07-01 17:16:20.524903162 +0000 UTC m=+68.830101340"
Jul 01 17:16:22.512805 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:16:22.512764    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-knvdq" event={"ID":"3defaf9e-f5ef-45ea-a02e-4729f15db90c","Type":"ContainerStarted","Data":"d7d59014e4748ddf636f2b53f03d154fbfe6c28230269d6532327c0a8f669b3e"}
Jul 01 17:16:22.534377 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:16:22.534316    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-knvdq" podStartSLOduration=65.820582549 podStartE2EDuration="1m10.534302153s" totalImagesPullingTime="4.713719604s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 17:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:16:22.533883396 +0000 UTC m=+70.839081567" watchObservedRunningTime="2026-07-01 17:16:22.534302153 +0000 UTC m=+70.839500367"
Jul 01 17:16:50.977993 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:16:50.977944    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-registry-tls\") pod \"image-registry-59d56646cd-98p6s\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") " pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:16:50.978493 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:16:50.978015    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dece537c-3a74-44b7-b4bf-ef9fbd0f1d01-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-phc4p\" (UID: \"dece537c-3a74-44b7-b4bf-ef9fbd0f1d01\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-phc4p"
Jul 01 17:16:50.978493 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:16:50.978088    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 17:16:50.978493 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:16:50.978112    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59d56646cd-98p6s: secret "image-registry-tls" not found
Jul 01 17:16:50.978493 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:16:50.978157    2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 17:16:50.978493 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:16:50.978180    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-registry-tls podName:33dc1940-f452-4f4e-a62b-c07096cee771 nodeName:}" failed. No retries permitted until 2026-07-01 17:17:54.978164096 +0000 UTC m=+163.283362271 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-registry-tls") pod "image-registry-59d56646cd-98p6s" (UID: "33dc1940-f452-4f4e-a62b-c07096cee771") : secret "image-registry-tls" not found
Jul 01 17:16:50.978493 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:16:50.978215    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dece537c-3a74-44b7-b4bf-ef9fbd0f1d01-networking-console-plugin-cert podName:dece537c-3a74-44b7-b4bf-ef9fbd0f1d01 nodeName:}" failed. No retries permitted until 2026-07-01 17:17:54.97820251 +0000 UTC m=+163.283400666 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dece537c-3a74-44b7-b4bf-ef9fbd0f1d01-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-phc4p" (UID: "dece537c-3a74-44b7-b4bf-ef9fbd0f1d01") : secret "networking-console-plugin-cert" not found
Jul 01 17:16:51.078504 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:16:51.078463    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-metrics-tls\") pod \"dns-default-n2fz9\" (UID: \"67bb5c8b-7164-4be3-a11a-9d6755b5a6be\") " pod="openshift-dns/dns-default-n2fz9"
Jul 01 17:16:51.078697 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:16:51.078660    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 17:16:51.078747 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:16:51.078736    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-metrics-tls podName:67bb5c8b-7164-4be3-a11a-9d6755b5a6be nodeName:}" failed. No retries permitted until 2026-07-01 17:17:55.078718039 +0000 UTC m=+163.383916198 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-metrics-tls") pod "dns-default-n2fz9" (UID: "67bb5c8b-7164-4be3-a11a-9d6755b5a6be") : secret "dns-default-metrics-tls" not found
Jul 01 17:16:51.511063 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:16:51.511032    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2x2qg"
Jul 01 17:17:21.004224 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:17:21.004181    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5026d3ed-589a-4390-a2b7-d5f64c06e687-metrics-certs\") pod \"network-metrics-daemon-4rrth\" (UID: \"5026d3ed-589a-4390-a2b7-d5f64c06e687\") " pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:17:21.004741 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:17:21.004353    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 01 17:17:21.004741 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:17:21.004429    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5026d3ed-589a-4390-a2b7-d5f64c06e687-metrics-certs podName:5026d3ed-589a-4390-a2b7-d5f64c06e687 nodeName:}" failed. No retries permitted until 2026-07-01 17:19:23.004412557 +0000 UTC m=+251.309610712 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5026d3ed-589a-4390-a2b7-d5f64c06e687-metrics-certs") pod "network-metrics-daemon-4rrth" (UID: "5026d3ed-589a-4390-a2b7-d5f64c06e687") : secret "metrics-daemon-secret" not found
Jul 01 17:17:50.074705 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:17:50.074656    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-59d56646cd-98p6s" podUID="33dc1940-f452-4f4e-a62b-c07096cee771"
Jul 01 17:17:50.092975 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:17:50.092927    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-phc4p" podUID="dece537c-3a74-44b7-b4bf-ef9fbd0f1d01"
Jul 01 17:17:50.167308 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:17:50.167272    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-n2fz9" podUID="67bb5c8b-7164-4be3-a11a-9d6755b5a6be"
Jul 01 17:17:50.179563 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:17:50.179531    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-4rrth" podUID="5026d3ed-589a-4390-a2b7-d5f64c06e687"
Jul 01 17:17:50.770886 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:17:50.770853    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:17:50.771064 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:17:50.770886    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-phc4p"
Jul 01 17:17:50.771133 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:17:50.771097    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n2fz9"
Jul 01 17:17:55.063644 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:17:55.063599    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-registry-tls\") pod \"image-registry-59d56646cd-98p6s\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") " pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:17:55.064081 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:17:55.063662    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dece537c-3a74-44b7-b4bf-ef9fbd0f1d01-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-phc4p\" (UID: \"dece537c-3a74-44b7-b4bf-ef9fbd0f1d01\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-phc4p"
Jul 01 17:17:55.064081 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:17:55.063763    2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 17:17:55.064081 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:17:55.063776    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 17:17:55.064081 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:17:55.063800    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59d56646cd-98p6s: secret "image-registry-tls" not found
Jul 01 17:17:55.064081 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:17:55.063824    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dece537c-3a74-44b7-b4bf-ef9fbd0f1d01-networking-console-plugin-cert podName:dece537c-3a74-44b7-b4bf-ef9fbd0f1d01 nodeName:}" failed. No retries permitted until 2026-07-01 17:19:57.063811586 +0000 UTC m=+285.369009741 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dece537c-3a74-44b7-b4bf-ef9fbd0f1d01-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-phc4p" (UID: "dece537c-3a74-44b7-b4bf-ef9fbd0f1d01") : secret "networking-console-plugin-cert" not found
Jul 01 17:17:55.064081 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:17:55.063863    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-registry-tls podName:33dc1940-f452-4f4e-a62b-c07096cee771 nodeName:}" failed. No retries permitted until 2026-07-01 17:19:57.06384537 +0000 UTC m=+285.369043525 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-registry-tls") pod "image-registry-59d56646cd-98p6s" (UID: "33dc1940-f452-4f4e-a62b-c07096cee771") : secret "image-registry-tls" not found
Jul 01 17:17:55.164600 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:17:55.164540    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-metrics-tls\") pod \"dns-default-n2fz9\" (UID: \"67bb5c8b-7164-4be3-a11a-9d6755b5a6be\") " pod="openshift-dns/dns-default-n2fz9"
Jul 01 17:17:55.164780 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:17:55.164697    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 17:17:55.164780 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:17:55.164773    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-metrics-tls podName:67bb5c8b-7164-4be3-a11a-9d6755b5a6be nodeName:}" failed. No retries permitted until 2026-07-01 17:19:57.164757983 +0000 UTC m=+285.469956138 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-metrics-tls") pod "dns-default-n2fz9" (UID: "67bb5c8b-7164-4be3-a11a-9d6755b5a6be") : secret "dns-default-metrics-tls" not found
Jul 01 17:17:57.793689 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:17:57.793594    2585 generic.go:356] "Generic (PLEG): container finished" podID="e5892b19-9801-4f95-aaae-6e8328bdca92" containerID="4796156abd9904a59e355fcc6ed8a1f38620a593aade2d4b75277843f742565c" exitCode=1
Jul 01 17:17:57.793689 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:17:57.793653    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c957b7d94-9xj75" event={"ID":"e5892b19-9801-4f95-aaae-6e8328bdca92","Type":"ContainerDied","Data":"4796156abd9904a59e355fcc6ed8a1f38620a593aade2d4b75277843f742565c"}
Jul 01 17:17:57.794184 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:17:57.794029    2585 scope.go:117] "RemoveContainer" containerID="4796156abd9904a59e355fcc6ed8a1f38620a593aade2d4b75277843f742565c"
Jul 01 17:17:57.795212 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:17:57.795189    2585 generic.go:356] "Generic (PLEG): container finished" podID="816f87d3-b783-4ee7-82bd-d68ff690c9f6" containerID="03c92c999e28edfa028ac4988778816a3928df11d465d3df7684469e940ed2c6" exitCode=255
Jul 01 17:17:57.795302 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:17:57.795220    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674f6d459d-8fs25" event={"ID":"816f87d3-b783-4ee7-82bd-d68ff690c9f6","Type":"ContainerDied","Data":"03c92c999e28edfa028ac4988778816a3928df11d465d3df7684469e940ed2c6"}
Jul 01 17:17:57.795454 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:17:57.795441    2585 scope.go:117] "RemoveContainer" containerID="03c92c999e28edfa028ac4988778816a3928df11d465d3df7684469e940ed2c6"
Jul 01 17:17:58.433557 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:17:58.433515    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c957b7d94-9xj75"
Jul 01 17:17:58.800092 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:17:58.800005    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674f6d459d-8fs25" event={"ID":"816f87d3-b783-4ee7-82bd-d68ff690c9f6","Type":"ContainerStarted","Data":"cd60dd2fe767ab4b5b8b0c6938c8d1839ddc9d1557b25da9c8cb18f2c6dca9c0"}
Jul 01 17:17:58.801657 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:17:58.801633    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c957b7d94-9xj75" event={"ID":"e5892b19-9801-4f95-aaae-6e8328bdca92","Type":"ContainerStarted","Data":"839268290cc4623d0f3b5701b46976d56f353bf84f33fce9e98ad9cc01e39655"}
Jul 01 17:17:58.801849 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:17:58.801830    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c957b7d94-9xj75"
Jul 01 17:17:58.802414 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:17:58.802396    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c957b7d94-9xj75"
Jul 01 17:18:05.167442 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:18:05.167340    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:19:23.035444 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:23.035401    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5026d3ed-589a-4390-a2b7-d5f64c06e687-metrics-certs\") pod \"network-metrics-daemon-4rrth\" (UID: \"5026d3ed-589a-4390-a2b7-d5f64c06e687\") " pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:19:23.036398 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:19:23.036375    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 01 17:19:23.036600 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:19:23.036567    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5026d3ed-589a-4390-a2b7-d5f64c06e687-metrics-certs podName:5026d3ed-589a-4390-a2b7-d5f64c06e687 nodeName:}" failed. No retries permitted until 2026-07-01 17:21:25.036543453 +0000 UTC m=+373.341741625 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5026d3ed-589a-4390-a2b7-d5f64c06e687-metrics-certs") pod "network-metrics-daemon-4rrth" (UID: "5026d3ed-589a-4390-a2b7-d5f64c06e687") : secret "metrics-daemon-secret" not found
Jul 01 17:19:42.153806 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:42.153772    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bzvrj"]
Jul 01 17:19:42.156211 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:42.156195    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bzvrj"
Jul 01 17:19:42.159249 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:42.159225    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 01 17:19:42.159361 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:42.159232    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 01 17:19:42.160417 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:42.160393    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b7kxs\""
Jul 01 17:19:42.160527 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:42.160397    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 01 17:19:42.160593 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:42.160399    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 01 17:19:42.171704 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:42.171683    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bzvrj"]
Jul 01 17:19:42.262658 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:42.262626    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b513528-3295-478a-92da-b184d6708e46-data-volume\") pod \"insights-runtime-extractor-bzvrj\" (UID: \"0b513528-3295-478a-92da-b184d6708e46\") " pod="openshift-insights/insights-runtime-extractor-bzvrj"
Jul 01 17:19:42.262658 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:42.262673    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b513528-3295-478a-92da-b184d6708e46-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzvrj\" (UID: \"0b513528-3295-478a-92da-b184d6708e46\") " pod="openshift-insights/insights-runtime-extractor-bzvrj"
Jul 01 17:19:42.262877 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:42.262706    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b513528-3295-478a-92da-b184d6708e46-crio-socket\") pod \"insights-runtime-extractor-bzvrj\" (UID: \"0b513528-3295-478a-92da-b184d6708e46\") " pod="openshift-insights/insights-runtime-extractor-bzvrj"
Jul 01 17:19:42.262877 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:42.262761    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b513528-3295-478a-92da-b184d6708e46-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzvrj\" (UID: \"0b513528-3295-478a-92da-b184d6708e46\") " pod="openshift-insights/insights-runtime-extractor-bzvrj"
Jul 01 17:19:42.262877 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:42.262780    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f66d2\" (UniqueName: \"kubernetes.io/projected/0b513528-3295-478a-92da-b184d6708e46-kube-api-access-f66d2\") pod \"insights-runtime-extractor-bzvrj\" (UID: \"0b513528-3295-478a-92da-b184d6708e46\") " pod="openshift-insights/insights-runtime-extractor-bzvrj"
Jul 01 17:19:42.363281 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:42.363249    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b513528-3295-478a-92da-b184d6708e46-data-volume\") pod \"insights-runtime-extractor-bzvrj\" (UID: \"0b513528-3295-478a-92da-b184d6708e46\") " pod="openshift-insights/insights-runtime-extractor-bzvrj"
Jul 01 17:19:42.363461 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:42.363293    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b513528-3295-478a-92da-b184d6708e46-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzvrj\" (UID: \"0b513528-3295-478a-92da-b184d6708e46\") " pod="openshift-insights/insights-runtime-extractor-bzvrj"
Jul 01 17:19:42.363461 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:42.363326    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b513528-3295-478a-92da-b184d6708e46-crio-socket\") pod \"insights-runtime-extractor-bzvrj\" (UID: \"0b513528-3295-478a-92da-b184d6708e46\") " pod="openshift-insights/insights-runtime-extractor-bzvrj"
Jul 01 17:19:42.363461 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:42.363371    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b513528-3295-478a-92da-b184d6708e46-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzvrj\" (UID: \"0b513528-3295-478a-92da-b184d6708e46\") " pod="openshift-insights/insights-runtime-extractor-bzvrj"
Jul 01 17:19:42.363461 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:42.363391    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f66d2\" (UniqueName: \"kubernetes.io/projected/0b513528-3295-478a-92da-b184d6708e46-kube-api-access-f66d2\") pod \"insights-runtime-extractor-bzvrj\" (UID: \"0b513528-3295-478a-92da-b184d6708e46\") " pod="openshift-insights/insights-runtime-extractor-bzvrj"
Jul 01 17:19:42.363461 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:19:42.363427    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 17:19:42.363735 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:42.363473    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b513528-3295-478a-92da-b184d6708e46-crio-socket\") pod \"insights-runtime-extractor-bzvrj\" (UID: \"0b513528-3295-478a-92da-b184d6708e46\") " pod="openshift-insights/insights-runtime-extractor-bzvrj"
Jul 01 17:19:42.363735 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:19:42.363501    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 17:19:42.363735 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:19:42.363518    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b513528-3295-478a-92da-b184d6708e46-insights-runtime-extractor-tls podName:0b513528-3295-478a-92da-b184d6708e46 nodeName:}" failed. No retries permitted until 2026-07-01 17:19:42.863497563 +0000 UTC m=+271.168695722 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b513528-3295-478a-92da-b184d6708e46-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bzvrj" (UID: "0b513528-3295-478a-92da-b184d6708e46") : secret "insights-runtime-extractor-tls" not found
Jul 01 17:19:42.363735 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:19:42.363570    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0b513528-3295-478a-92da-b184d6708e46-kube-rbac-proxy-cm podName:0b513528-3295-478a-92da-b184d6708e46 nodeName:}" failed. No retries permitted until 2026-07-01 17:19:42.863556467 +0000 UTC m=+271.168754622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0b513528-3295-478a-92da-b184d6708e46-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bzvrj" (UID: "0b513528-3295-478a-92da-b184d6708e46") : configmap "kube-rbac-proxy" not found
Jul 01 17:19:42.363735 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:42.363613    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b513528-3295-478a-92da-b184d6708e46-data-volume\") pod \"insights-runtime-extractor-bzvrj\" (UID: \"0b513528-3295-478a-92da-b184d6708e46\") " pod="openshift-insights/insights-runtime-extractor-bzvrj"
Jul 01 17:19:42.371355 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:42.371327    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f66d2\" (UniqueName: \"kubernetes.io/projected/0b513528-3295-478a-92da-b184d6708e46-kube-api-access-f66d2\") pod \"insights-runtime-extractor-bzvrj\" (UID: \"0b513528-3295-478a-92da-b184d6708e46\") " pod="openshift-insights/insights-runtime-extractor-bzvrj"
Jul 01 17:19:42.866724 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:42.866685    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b513528-3295-478a-92da-b184d6708e46-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzvrj\" (UID: \"0b513528-3295-478a-92da-b184d6708e46\") " pod="openshift-insights/insights-runtime-extractor-bzvrj"
Jul 01 17:19:42.866901 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:42.866775    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b513528-3295-478a-92da-b184d6708e46-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzvrj\" (UID: \"0b513528-3295-478a-92da-b184d6708e46\") " pod="openshift-insights/insights-runtime-extractor-bzvrj"
Jul 01 17:19:42.866901 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:19:42.866817    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 17:19:42.866901 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:19:42.866888    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0b513528-3295-478a-92da-b184d6708e46-kube-rbac-proxy-cm podName:0b513528-3295-478a-92da-b184d6708e46 nodeName:}" failed. No retries permitted until 2026-07-01 17:19:43.866871997 +0000 UTC m=+272.172070152 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0b513528-3295-478a-92da-b184d6708e46-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bzvrj" (UID: "0b513528-3295-478a-92da-b184d6708e46") : configmap "kube-rbac-proxy" not found
Jul 01 17:19:42.867006 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:19:42.866920    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 17:19:42.867006 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:19:42.866973    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b513528-3295-478a-92da-b184d6708e46-insights-runtime-extractor-tls podName:0b513528-3295-478a-92da-b184d6708e46 nodeName:}" failed. No retries permitted until 2026-07-01 17:19:43.866960457 +0000 UTC m=+272.172158612 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b513528-3295-478a-92da-b184d6708e46-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bzvrj" (UID: "0b513528-3295-478a-92da-b184d6708e46") : secret "insights-runtime-extractor-tls" not found
Jul 01 17:19:43.874791 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:43.874738    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b513528-3295-478a-92da-b184d6708e46-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzvrj\" (UID: \"0b513528-3295-478a-92da-b184d6708e46\") " pod="openshift-insights/insights-runtime-extractor-bzvrj"
Jul 01 17:19:43.875290 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:43.874835    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b513528-3295-478a-92da-b184d6708e46-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzvrj\" (UID: \"0b513528-3295-478a-92da-b184d6708e46\") " pod="openshift-insights/insights-runtime-extractor-bzvrj"
Jul 01 17:19:43.875290 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:19:43.874883    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 17:19:43.875290 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:19:43.874949    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 17:19:43.875290 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:19:43.874974    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0b513528-3295-478a-92da-b184d6708e46-kube-rbac-proxy-cm podName:0b513528-3295-478a-92da-b184d6708e46 nodeName:}" failed. No retries permitted until 2026-07-01 17:19:45.874950314 +0000 UTC m=+274.180148475 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0b513528-3295-478a-92da-b184d6708e46-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bzvrj" (UID: "0b513528-3295-478a-92da-b184d6708e46") : configmap "kube-rbac-proxy" not found
Jul 01 17:19:43.875290 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:19:43.874994    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b513528-3295-478a-92da-b184d6708e46-insights-runtime-extractor-tls podName:0b513528-3295-478a-92da-b184d6708e46 nodeName:}" failed. No retries permitted until 2026-07-01 17:19:45.874982919 +0000 UTC m=+274.180181073 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b513528-3295-478a-92da-b184d6708e46-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bzvrj" (UID: "0b513528-3295-478a-92da-b184d6708e46") : secret "insights-runtime-extractor-tls" not found
Jul 01 17:19:44.363902 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:44.363875    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jwvgk_3ff59413-27fa-4837-8549-f2a6898826a9/dns-node-resolver/0.log"
Jul 01 17:19:45.376711 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:45.376680    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r54bn_d6614097-0891-470c-b830-26bf595ff80e/node-ca/0.log"
Jul 01 17:19:45.891139 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:45.891097    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b513528-3295-478a-92da-b184d6708e46-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzvrj\" (UID: \"0b513528-3295-478a-92da-b184d6708e46\") " pod="openshift-insights/insights-runtime-extractor-bzvrj"
Jul 01 17:19:45.891306 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:45.891170    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b513528-3295-478a-92da-b184d6708e46-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzvrj\" (UID: \"0b513528-3295-478a-92da-b184d6708e46\") " pod="openshift-insights/insights-runtime-extractor-bzvrj"
Jul 01 17:19:45.891306 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:19:45.891226    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 17:19:45.891306 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:19:45.891252    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 17:19:45.891306 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:19:45.891291    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0b513528-3295-478a-92da-b184d6708e46-kube-rbac-proxy-cm podName:0b513528-3295-478a-92da-b184d6708e46 nodeName:}" failed. No retries permitted until 2026-07-01 17:19:49.891275129 +0000 UTC m=+278.196473283 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0b513528-3295-478a-92da-b184d6708e46-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bzvrj" (UID: "0b513528-3295-478a-92da-b184d6708e46") : configmap "kube-rbac-proxy" not found
Jul 01 17:19:45.891306 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:19:45.891306    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b513528-3295-478a-92da-b184d6708e46-insights-runtime-extractor-tls podName:0b513528-3295-478a-92da-b184d6708e46 nodeName:}" failed. No retries permitted until 2026-07-01 17:19:49.891300259 +0000 UTC m=+278.196498414 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b513528-3295-478a-92da-b184d6708e46-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bzvrj" (UID: "0b513528-3295-478a-92da-b184d6708e46") : secret "insights-runtime-extractor-tls" not found
Jul 01 17:19:45.965871 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:45.965841    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jq2fj_6ec33cc2-606d-4445-9f9b-4fe556dfbc6e/serve-healthcheck-canary/0.log"
Jul 01 17:19:49.924534 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:49.924489    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b513528-3295-478a-92da-b184d6708e46-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzvrj\" (UID: \"0b513528-3295-478a-92da-b184d6708e46\") " pod="openshift-insights/insights-runtime-extractor-bzvrj"
Jul 01 17:19:49.925029 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:49.924568    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b513528-3295-478a-92da-b184d6708e46-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzvrj\" (UID: \"0b513528-3295-478a-92da-b184d6708e46\") " pod="openshift-insights/insights-runtime-extractor-bzvrj"
Jul 01 17:19:49.925029 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:19:49.924629    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 17:19:49.925029 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:19:49.924711    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0b513528-3295-478a-92da-b184d6708e46-kube-rbac-proxy-cm podName:0b513528-3295-478a-92da-b184d6708e46 nodeName:}" failed. No retries permitted until 2026-07-01 17:19:57.924694198 +0000 UTC m=+286.229892353 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0b513528-3295-478a-92da-b184d6708e46-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bzvrj" (UID: "0b513528-3295-478a-92da-b184d6708e46") : configmap "kube-rbac-proxy" not found
Jul 01 17:19:49.925029 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:19:49.924723    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 17:19:49.925029 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:19:49.924798    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b513528-3295-478a-92da-b184d6708e46-insights-runtime-extractor-tls podName:0b513528-3295-478a-92da-b184d6708e46 nodeName:}" failed. No retries permitted until 2026-07-01 17:19:57.924783222 +0000 UTC m=+286.229981383 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b513528-3295-478a-92da-b184d6708e46-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bzvrj" (UID: "0b513528-3295-478a-92da-b184d6708e46") : secret "insights-runtime-extractor-tls" not found
Jul 01 17:19:53.771474 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:19:53.771428    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-phc4p" podUID="dece537c-3a74-44b7-b4bf-ef9fbd0f1d01"
Jul 01 17:19:53.771474 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:19:53.771463    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-n2fz9" podUID="67bb5c8b-7164-4be3-a11a-9d6755b5a6be"
Jul 01 17:19:53.771474 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:19:53.771429    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-59d56646cd-98p6s" podUID="33dc1940-f452-4f4e-a62b-c07096cee771"
Jul 01 17:19:54.131956 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:54.131928    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-phc4p"
Jul 01 17:19:54.131956 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:54.131941    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n2fz9"
Jul 01 17:19:54.132156 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:54.131928    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:19:57.081810 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:57.081767    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dece537c-3a74-44b7-b4bf-ef9fbd0f1d01-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-phc4p\" (UID: \"dece537c-3a74-44b7-b4bf-ef9fbd0f1d01\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-phc4p"
Jul 01 17:19:57.082185 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:57.081845    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-registry-tls\") pod \"image-registry-59d56646cd-98p6s\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") " pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:19:57.084261 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:57.084235    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dece537c-3a74-44b7-b4bf-ef9fbd0f1d01-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-phc4p\" (UID: \"dece537c-3a74-44b7-b4bf-ef9fbd0f1d01\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-phc4p"
Jul 01 17:19:57.084406 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:57.084384    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-registry-tls\") pod \"image-registry-59d56646cd-98p6s\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") " pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:19:57.137155 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:57.137123    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jkmfz\""
Jul 01 17:19:57.137155 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:57.137123    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-2klvb\""
Jul 01 17:19:57.143467 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:57.143438    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:19:57.143467 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:57.143454    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-phc4p"
Jul 01 17:19:57.183114 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:57.182610    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-metrics-tls\") pod \"dns-default-n2fz9\" (UID: \"67bb5c8b-7164-4be3-a11a-9d6755b5a6be\") " pod="openshift-dns/dns-default-n2fz9"
Jul 01 17:19:57.186763 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:57.186735    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67bb5c8b-7164-4be3-a11a-9d6755b5a6be-metrics-tls\") pod \"dns-default-n2fz9\" (UID: \"67bb5c8b-7164-4be3-a11a-9d6755b5a6be\") " pod="openshift-dns/dns-default-n2fz9"
Jul 01 17:19:57.279964 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:57.279930    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-phc4p"]
Jul 01 17:19:57.283270 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:19:57.283243    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddece537c_3a74_44b7_b4bf_ef9fbd0f1d01.slice/crio-51267f859e7332403e0931ef392c5656f03ae2ca1164d309438f8e9653d4a83a WatchSource:0}: Error finding container 51267f859e7332403e0931ef392c5656f03ae2ca1164d309438f8e9653d4a83a: Status 404 returned error can't find the container with id 51267f859e7332403e0931ef392c5656f03ae2ca1164d309438f8e9653d4a83a
Jul 01 17:19:57.296673 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:57.296649    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59d56646cd-98p6s"]
Jul 01 17:19:57.299714 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:19:57.299690    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33dc1940_f452_4f4e_a62b_c07096cee771.slice/crio-3581d65f8743d7932c62887257f21e8003b3c6109a8c5cc921e44e1cf82b2924 WatchSource:0}: Error finding container 3581d65f8743d7932c62887257f21e8003b3c6109a8c5cc921e44e1cf82b2924: Status 404 returned error can't find the container with id 3581d65f8743d7932c62887257f21e8003b3c6109a8c5cc921e44e1cf82b2924
Jul 01 17:19:57.435905 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:57.435878    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pcm4c\""
Jul 01 17:19:57.443860 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:57.443838    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n2fz9"
Jul 01 17:19:57.568783 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:57.568752    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n2fz9"]
Jul 01 17:19:57.571535 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:19:57.571504    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67bb5c8b_7164_4be3_a11a_9d6755b5a6be.slice/crio-90af15fc4855e1d6963e113ceb790dfb135e4e58d1cd6329f363c5dec74aa789 WatchSource:0}: Error finding container 90af15fc4855e1d6963e113ceb790dfb135e4e58d1cd6329f363c5dec74aa789: Status 404 returned error can't find the container with id 90af15fc4855e1d6963e113ceb790dfb135e4e58d1cd6329f363c5dec74aa789
Jul 01 17:19:57.989087 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:57.989052    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b513528-3295-478a-92da-b184d6708e46-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzvrj\" (UID: \"0b513528-3295-478a-92da-b184d6708e46\") " pod="openshift-insights/insights-runtime-extractor-bzvrj"
Jul 01 17:19:57.989258 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:57.989127    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b513528-3295-478a-92da-b184d6708e46-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzvrj\" (UID: \"0b513528-3295-478a-92da-b184d6708e46\") " pod="openshift-insights/insights-runtime-extractor-bzvrj"
Jul 01 17:19:57.989258 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:19:57.989233    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 17:19:57.989353 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:19:57.989309    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b513528-3295-478a-92da-b184d6708e46-insights-runtime-extractor-tls podName:0b513528-3295-478a-92da-b184d6708e46 nodeName:}" failed. No retries permitted until 2026-07-01 17:20:13.989289583 +0000 UTC m=+302.294487739 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b513528-3295-478a-92da-b184d6708e46-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bzvrj" (UID: "0b513528-3295-478a-92da-b184d6708e46") : secret "insights-runtime-extractor-tls" not found
Jul 01 17:19:57.989775 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:57.989753    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b513528-3295-478a-92da-b184d6708e46-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzvrj\" (UID: \"0b513528-3295-478a-92da-b184d6708e46\") " pod="openshift-insights/insights-runtime-extractor-bzvrj"
Jul 01 17:19:58.143684 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:58.143596    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59d56646cd-98p6s" event={"ID":"33dc1940-f452-4f4e-a62b-c07096cee771","Type":"ContainerStarted","Data":"47cd56f4600af830ed70a2b346cebe605ac8d652229621dea469b32fafc091f4"}
Jul 01 17:19:58.143684 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:58.143654    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59d56646cd-98p6s" event={"ID":"33dc1940-f452-4f4e-a62b-c07096cee771","Type":"ContainerStarted","Data":"3581d65f8743d7932c62887257f21e8003b3c6109a8c5cc921e44e1cf82b2924"}
Jul 01 17:19:58.144125 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:58.143689    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:19:58.145222 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:58.145201    2585 generic.go:356] "Generic (PLEG): container finished" podID="816f87d3-b783-4ee7-82bd-d68ff690c9f6" containerID="cd60dd2fe767ab4b5b8b0c6938c8d1839ddc9d1557b25da9c8cb18f2c6dca9c0" exitCode=255
Jul 01 17:19:58.145337 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:58.145274    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674f6d459d-8fs25" event={"ID":"816f87d3-b783-4ee7-82bd-d68ff690c9f6","Type":"ContainerDied","Data":"cd60dd2fe767ab4b5b8b0c6938c8d1839ddc9d1557b25da9c8cb18f2c6dca9c0"}
Jul 01 17:19:58.145337 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:58.145328    2585 scope.go:117] "RemoveContainer" containerID="03c92c999e28edfa028ac4988778816a3928df11d465d3df7684469e940ed2c6"
Jul 01 17:19:58.145652 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:58.145639    2585 scope.go:117] "RemoveContainer" containerID="cd60dd2fe767ab4b5b8b0c6938c8d1839ddc9d1557b25da9c8cb18f2c6dca9c0"
Jul 01 17:19:58.145802 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:19:58.145785    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-674f6d459d-8fs25_open-cluster-management-agent-addon(816f87d3-b783-4ee7-82bd-d68ff690c9f6)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674f6d459d-8fs25" podUID="816f87d3-b783-4ee7-82bd-d68ff690c9f6"
Jul 01 17:19:58.147048 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:58.147028    2585 generic.go:356] "Generic (PLEG): container finished" podID="e5892b19-9801-4f95-aaae-6e8328bdca92" containerID="839268290cc4623d0f3b5701b46976d56f353bf84f33fce9e98ad9cc01e39655" exitCode=1
Jul 01 17:19:58.147144 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:58.147072    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c957b7d94-9xj75" event={"ID":"e5892b19-9801-4f95-aaae-6e8328bdca92","Type":"ContainerDied","Data":"839268290cc4623d0f3b5701b46976d56f353bf84f33fce9e98ad9cc01e39655"}
Jul 01 17:19:58.147451 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:58.147434    2585 scope.go:117] "RemoveContainer" containerID="839268290cc4623d0f3b5701b46976d56f353bf84f33fce9e98ad9cc01e39655"
Jul 01 17:19:58.147698 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:19:58.147674    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-5c957b7d94-9xj75_open-cluster-management-agent-addon(e5892b19-9801-4f95-aaae-6e8328bdca92)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c957b7d94-9xj75" podUID="e5892b19-9801-4f95-aaae-6e8328bdca92"
Jul 01 17:19:58.148316 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:58.148288    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n2fz9" event={"ID":"67bb5c8b-7164-4be3-a11a-9d6755b5a6be","Type":"ContainerStarted","Data":"90af15fc4855e1d6963e113ceb790dfb135e4e58d1cd6329f363c5dec74aa789"}
Jul 01 17:19:58.149499 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:58.149478    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-phc4p" event={"ID":"dece537c-3a74-44b7-b4bf-ef9fbd0f1d01","Type":"ContainerStarted","Data":"51267f859e7332403e0931ef392c5656f03ae2ca1164d309438f8e9653d4a83a"}
Jul 01 17:19:58.155711 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:58.155693    2585 scope.go:117] "RemoveContainer" containerID="4796156abd9904a59e355fcc6ed8a1f38620a593aade2d4b75277843f742565c"
Jul 01 17:19:58.165825 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:58.165762    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-59d56646cd-98p6s" podStartSLOduration=286.165750048 podStartE2EDuration="4m46.165750048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:15:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:19:58.165281288 +0000 UTC m=+286.470479460" watchObservedRunningTime="2026-07-01 17:19:58.165750048 +0000 UTC m=+286.470948202"
Jul 01 17:19:58.802108 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:58.802078    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c957b7d94-9xj75"
Jul 01 17:19:59.155303 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:59.155276    2585 scope.go:117] "RemoveContainer" containerID="839268290cc4623d0f3b5701b46976d56f353bf84f33fce9e98ad9cc01e39655"
Jul 01 17:19:59.155776 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:19:59.155474    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-5c957b7d94-9xj75_open-cluster-management-agent-addon(e5892b19-9801-4f95-aaae-6e8328bdca92)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c957b7d94-9xj75" podUID="e5892b19-9801-4f95-aaae-6e8328bdca92"
Jul 01 17:19:59.156652 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:59.156627    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-phc4p" event={"ID":"dece537c-3a74-44b7-b4bf-ef9fbd0f1d01","Type":"ContainerStarted","Data":"a09e0ab96df2f3fa5a7047199a8a45e4db5bbeea255431d5bd964495eb46b255"}
Jul 01 17:19:59.205498 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:19:59.205436    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-phc4p" podStartSLOduration=258.888496911 podStartE2EDuration="4m20.205416984s" totalImagesPullingTime="1.316920073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:19:59.205025225 +0000 UTC m=+287.510223393" watchObservedRunningTime="2026-07-01 17:19:59.205416984 +0000 UTC m=+287.510615160"
Jul 01 17:20:00.162984 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:00.162947    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n2fz9" event={"ID":"67bb5c8b-7164-4be3-a11a-9d6755b5a6be","Type":"ContainerStarted","Data":"bbff6e4bd1f44cbc0188ac12f8ab785b089b4c3bdd2350851714df4d063dd405"}
Jul 01 17:20:00.163355 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:00.163033    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n2fz9" event={"ID":"67bb5c8b-7164-4be3-a11a-9d6755b5a6be","Type":"ContainerStarted","Data":"866e113364fbcb34936250a507fa53672eb2a5330561ea26d913fbc6fdddd038"}
Jul 01 17:20:00.181340 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:00.181294    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-n2fz9" podStartSLOduration=251.013581983 podStartE2EDuration="4m13.181276984s" totalImagesPullingTime="2.167695001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:15:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:20:00.179554732 +0000 UTC m=+288.484752910" watchObservedRunningTime="2026-07-01 17:20:00.181276984 +0000 UTC m=+288.486475162"
Jul 01 17:20:01.167059 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:01.167027    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-n2fz9"
Jul 01 17:20:07.417200 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:07.417165    2585 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674f6d459d-8fs25"
Jul 01 17:20:07.417570 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:07.417510    2585 scope.go:117] "RemoveContainer" containerID="cd60dd2fe767ab4b5b8b0c6938c8d1839ddc9d1557b25da9c8cb18f2c6dca9c0"
Jul 01 17:20:07.417682 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:20:07.417663    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-674f6d459d-8fs25_open-cluster-management-agent-addon(816f87d3-b783-4ee7-82bd-d68ff690c9f6)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674f6d459d-8fs25" podUID="816f87d3-b783-4ee7-82bd-d68ff690c9f6"
Jul 01 17:20:07.426258 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:07.426239    2585 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c957b7d94-9xj75"
Jul 01 17:20:07.426551 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:07.426539    2585 scope.go:117] "RemoveContainer" containerID="839268290cc4623d0f3b5701b46976d56f353bf84f33fce9e98ad9cc01e39655"
Jul 01 17:20:07.426739 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:20:07.426720    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-5c957b7d94-9xj75_open-cluster-management-agent-addon(e5892b19-9801-4f95-aaae-6e8328bdca92)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c957b7d94-9xj75" podUID="e5892b19-9801-4f95-aaae-6e8328bdca92"
Jul 01 17:20:08.168715 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:20:08.168673    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-4rrth" podUID="5026d3ed-589a-4390-a2b7-d5f64c06e687"
Jul 01 17:20:11.172814 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:11.172785    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-n2fz9"
Jul 01 17:20:12.110320 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:12.110287    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8tsw_500d2a90-2d21-42b3-9584-b04c14becaec/ovn-acl-logging/0.log"
Jul 01 17:20:12.112917 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:12.112896    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8tsw_500d2a90-2d21-42b3-9584-b04c14becaec/ovn-acl-logging/0.log"
Jul 01 17:20:12.113384 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:12.113365    2585 kubelet.go:1648] "Image garbage collection succeeded"
Jul 01 17:20:14.009695 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:14.009641    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b513528-3295-478a-92da-b184d6708e46-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzvrj\" (UID: \"0b513528-3295-478a-92da-b184d6708e46\") " pod="openshift-insights/insights-runtime-extractor-bzvrj"
Jul 01 17:20:14.012121 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:14.012098    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b513528-3295-478a-92da-b184d6708e46-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzvrj\" (UID: \"0b513528-3295-478a-92da-b184d6708e46\") " pod="openshift-insights/insights-runtime-extractor-bzvrj"
Jul 01 17:20:14.268248 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:14.268166    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b7kxs\""
Jul 01 17:20:14.275806 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:14.275774    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bzvrj"
Jul 01 17:20:14.397295 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:14.397265    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bzvrj"]
Jul 01 17:20:14.400815 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:20:14.400786    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b513528_3295_478a_92da_b184d6708e46.slice/crio-4affbb777c5715789edb2d41bed94d42417024b457f4bebf52a9158003a69133 WatchSource:0}: Error finding container 4affbb777c5715789edb2d41bed94d42417024b457f4bebf52a9158003a69133: Status 404 returned error can't find the container with id 4affbb777c5715789edb2d41bed94d42417024b457f4bebf52a9158003a69133
Jul 01 17:20:14.401967 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:14.401947    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 17:20:15.210669 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:15.210630    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bzvrj" event={"ID":"0b513528-3295-478a-92da-b184d6708e46","Type":"ContainerStarted","Data":"4affbb777c5715789edb2d41bed94d42417024b457f4bebf52a9158003a69133"}
Jul 01 17:20:17.148311 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:17.148275    2585 patch_prober.go:28] interesting pod/image-registry-59d56646cd-98p6s container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 01 17:20:17.148709 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:17.148329    2585 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-59d56646cd-98p6s" podUID="33dc1940-f452-4f4e-a62b-c07096cee771" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:20:18.222445 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:18.222411    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bzvrj" event={"ID":"0b513528-3295-478a-92da-b184d6708e46","Type":"ContainerStarted","Data":"f5e9a14f5fc5678523bf85f9143f472ae182187040c392bf62627442e16c32c7"}
Jul 01 17:20:19.162474 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:19.162443    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:20:19.226989 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:19.226954    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bzvrj" event={"ID":"0b513528-3295-478a-92da-b184d6708e46","Type":"ContainerStarted","Data":"ef8348edb7b13cfd310ce59934c467a9c3e75c1113c48f2fe4cf685a740bebfb"}
Jul 01 17:20:20.167967 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:20.167930    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:20:20.168285 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:20.168264    2585 scope.go:117] "RemoveContainer" containerID="cd60dd2fe767ab4b5b8b0c6938c8d1839ddc9d1557b25da9c8cb18f2c6dca9c0"
Jul 01 17:20:20.233952 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:20.233912    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674f6d459d-8fs25" event={"ID":"816f87d3-b783-4ee7-82bd-d68ff690c9f6","Type":"ContainerStarted","Data":"11fb741645448f4b482c57bb8c85bfa0753bd2f9d03004e81efe1c2ee88b9362"}
Jul 01 17:20:22.134114 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.134081    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2mb48"]
Jul 01 17:20:22.136379 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.136363    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2mb48"
Jul 01 17:20:22.139044 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.139023    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cd8st\""
Jul 01 17:20:22.139434 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.139414    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 01 17:20:22.139523 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.139414    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 01 17:20:22.139523 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.139480    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 01 17:20:22.140243 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.140225    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 01 17:20:22.140350 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.140244    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 01 17:20:22.140350 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.140246    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 01 17:20:22.167711 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.167694    2585 scope.go:117] "RemoveContainer" containerID="839268290cc4623d0f3b5701b46976d56f353bf84f33fce9e98ad9cc01e39655"
Jul 01 17:20:22.242526 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.242487    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bzvrj" event={"ID":"0b513528-3295-478a-92da-b184d6708e46","Type":"ContainerStarted","Data":"76eae96784fe277d82e134570f7cebe0a5088e6fa40739a78af807888e247734"}
Jul 01 17:20:22.244607 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.244205    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c957b7d94-9xj75" event={"ID":"e5892b19-9801-4f95-aaae-6e8328bdca92","Type":"ContainerStarted","Data":"1db864706045715a394b5bab7a2b321fcd6177aa1cf78dac1b20873298ed533f"}
Jul 01 17:20:22.244736 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.244645    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c957b7d94-9xj75"
Jul 01 17:20:22.245549 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.245521    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c957b7d94-9xj75" podUID="e5892b19-9801-4f95-aaae-6e8328bdca92" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.9:8000/readyz\": dial tcp 10.132.0.9:8000: connect: connection refused"
Jul 01 17:20:22.262503 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.262457    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bzvrj" podStartSLOduration=33.16953856 podStartE2EDuration="40.262442177s" totalImagesPullingTime="7.092903617s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 17:19:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:20:22.261488211 +0000 UTC m=+310.566686388" watchObservedRunningTime="2026-07-01 17:20:22.262442177 +0000 UTC m=+310.567640353"
Jul 01 17:20:22.275328 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.275299    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5d1f6d5-d5b2-456c-8a97-74fb30efc048-node-exporter-tls\") pod \"node-exporter-2mb48\" (UID: \"b5d1f6d5-d5b2-456c-8a97-74fb30efc048\") " pod="openshift-monitoring/node-exporter-2mb48"
Jul 01 17:20:22.275447 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.275366    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5d1f6d5-d5b2-456c-8a97-74fb30efc048-root\") pod \"node-exporter-2mb48\" (UID: \"b5d1f6d5-d5b2-456c-8a97-74fb30efc048\") " pod="openshift-monitoring/node-exporter-2mb48"
Jul 01 17:20:22.275556 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.275449    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5d1f6d5-d5b2-456c-8a97-74fb30efc048-metrics-client-ca\") pod \"node-exporter-2mb48\" (UID: \"b5d1f6d5-d5b2-456c-8a97-74fb30efc048\") " pod="openshift-monitoring/node-exporter-2mb48"
Jul 01 17:20:22.275556 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.275497    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5d1f6d5-d5b2-456c-8a97-74fb30efc048-sys\") pod \"node-exporter-2mb48\" (UID: \"b5d1f6d5-d5b2-456c-8a97-74fb30efc048\") " pod="openshift-monitoring/node-exporter-2mb48"
Jul 01 17:20:22.275689 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.275568    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5d1f6d5-d5b2-456c-8a97-74fb30efc048-node-exporter-accelerators-collector-config\") pod \"node-exporter-2mb48\" (UID: \"b5d1f6d5-d5b2-456c-8a97-74fb30efc048\") " pod="openshift-monitoring/node-exporter-2mb48"
Jul 01 17:20:22.275689 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.275624    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kk59\" (UniqueName: \"kubernetes.io/projected/b5d1f6d5-d5b2-456c-8a97-74fb30efc048-kube-api-access-4kk59\") pod \"node-exporter-2mb48\" (UID: \"b5d1f6d5-d5b2-456c-8a97-74fb30efc048\") " pod="openshift-monitoring/node-exporter-2mb48"
Jul 01 17:20:22.275689 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.275657    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5d1f6d5-d5b2-456c-8a97-74fb30efc048-node-exporter-textfile\") pod \"node-exporter-2mb48\" (UID: \"b5d1f6d5-d5b2-456c-8a97-74fb30efc048\") " pod="openshift-monitoring/node-exporter-2mb48"
Jul 01 17:20:22.275689 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.275682    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5d1f6d5-d5b2-456c-8a97-74fb30efc048-node-exporter-wtmp\") pod \"node-exporter-2mb48\" (UID: \"b5d1f6d5-d5b2-456c-8a97-74fb30efc048\") " pod="openshift-monitoring/node-exporter-2mb48"
Jul 01 17:20:22.275881 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.275777    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5d1f6d5-d5b2-456c-8a97-74fb30efc048-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2mb48\" (UID: \"b5d1f6d5-d5b2-456c-8a97-74fb30efc048\") " pod="openshift-monitoring/node-exporter-2mb48"
Jul 01 17:20:22.376561 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.376528    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5d1f6d5-d5b2-456c-8a97-74fb30efc048-root\") pod \"node-exporter-2mb48\" (UID: \"b5d1f6d5-d5b2-456c-8a97-74fb30efc048\") " pod="openshift-monitoring/node-exporter-2mb48"
Jul 01 17:20:22.376561 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.376569    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5d1f6d5-d5b2-456c-8a97-74fb30efc048-metrics-client-ca\") pod \"node-exporter-2mb48\" (UID: \"b5d1f6d5-d5b2-456c-8a97-74fb30efc048\") " pod="openshift-monitoring/node-exporter-2mb48"
Jul 01 17:20:22.376835 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.376606    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5d1f6d5-d5b2-456c-8a97-74fb30efc048-sys\") pod \"node-exporter-2mb48\" (UID: \"b5d1f6d5-d5b2-456c-8a97-74fb30efc048\") " pod="openshift-monitoring/node-exporter-2mb48"
Jul 01 17:20:22.376835 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.376655    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5d1f6d5-d5b2-456c-8a97-74fb30efc048-root\") pod \"node-exporter-2mb48\" (UID: \"b5d1f6d5-d5b2-456c-8a97-74fb30efc048\") " pod="openshift-monitoring/node-exporter-2mb48"
Jul 01 17:20:22.376835 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.376665    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5d1f6d5-d5b2-456c-8a97-74fb30efc048-node-exporter-accelerators-collector-config\") pod \"node-exporter-2mb48\" (UID: \"b5d1f6d5-d5b2-456c-8a97-74fb30efc048\") " pod="openshift-monitoring/node-exporter-2mb48"
Jul 01 17:20:22.376835 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.376700    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5d1f6d5-d5b2-456c-8a97-74fb30efc048-sys\") pod \"node-exporter-2mb48\" (UID: \"b5d1f6d5-d5b2-456c-8a97-74fb30efc048\") " pod="openshift-monitoring/node-exporter-2mb48"
Jul 01 17:20:22.376835 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.376715    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kk59\" (UniqueName: \"kubernetes.io/projected/b5d1f6d5-d5b2-456c-8a97-74fb30efc048-kube-api-access-4kk59\") pod \"node-exporter-2mb48\" (UID: \"b5d1f6d5-d5b2-456c-8a97-74fb30efc048\") " pod="openshift-monitoring/node-exporter-2mb48"
Jul 01 17:20:22.376835 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.376759    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5d1f6d5-d5b2-456c-8a97-74fb30efc048-node-exporter-textfile\") pod \"node-exporter-2mb48\" (UID: \"b5d1f6d5-d5b2-456c-8a97-74fb30efc048\") " pod="openshift-monitoring/node-exporter-2mb48"
Jul 01 17:20:22.376835 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.376783    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5d1f6d5-d5b2-456c-8a97-74fb30efc048-node-exporter-wtmp\") pod \"node-exporter-2mb48\" (UID: \"b5d1f6d5-d5b2-456c-8a97-74fb30efc048\") " pod="openshift-monitoring/node-exporter-2mb48"
Jul 01 17:20:22.376835 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.376836    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5d1f6d5-d5b2-456c-8a97-74fb30efc048-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2mb48\" (UID: \"b5d1f6d5-d5b2-456c-8a97-74fb30efc048\") " pod="openshift-monitoring/node-exporter-2mb48"
Jul 01 17:20:22.377223 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.376871    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5d1f6d5-d5b2-456c-8a97-74fb30efc048-node-exporter-tls\") pod \"node-exporter-2mb48\" (UID: \"b5d1f6d5-d5b2-456c-8a97-74fb30efc048\") " pod="openshift-monitoring/node-exporter-2mb48"
Jul 01 17:20:22.377275 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:20:22.377230    2585 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 01 17:20:22.377275 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.377249    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5d1f6d5-d5b2-456c-8a97-74fb30efc048-node-exporter-textfile\") pod \"node-exporter-2mb48\" (UID: \"b5d1f6d5-d5b2-456c-8a97-74fb30efc048\") " pod="openshift-monitoring/node-exporter-2mb48"
Jul 01 17:20:22.377364 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.377272    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5d1f6d5-d5b2-456c-8a97-74fb30efc048-node-exporter-wtmp\") pod \"node-exporter-2mb48\" (UID: \"b5d1f6d5-d5b2-456c-8a97-74fb30efc048\") " pod="openshift-monitoring/node-exporter-2mb48"
Jul 01 17:20:22.377364 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:20:22.377285    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5d1f6d5-d5b2-456c-8a97-74fb30efc048-node-exporter-tls podName:b5d1f6d5-d5b2-456c-8a97-74fb30efc048 nodeName:}" failed. No retries permitted until 2026-07-01 17:20:22.877266709 +0000 UTC m=+311.182464868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/b5d1f6d5-d5b2-456c-8a97-74fb30efc048-node-exporter-tls") pod "node-exporter-2mb48" (UID: "b5d1f6d5-d5b2-456c-8a97-74fb30efc048") : secret "node-exporter-tls" not found
Jul 01 17:20:22.377364 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.377280    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5d1f6d5-d5b2-456c-8a97-74fb30efc048-node-exporter-accelerators-collector-config\") pod \"node-exporter-2mb48\" (UID: \"b5d1f6d5-d5b2-456c-8a97-74fb30efc048\") " pod="openshift-monitoring/node-exporter-2mb48"
Jul 01 17:20:22.377364 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.377289    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5d1f6d5-d5b2-456c-8a97-74fb30efc048-metrics-client-ca\") pod \"node-exporter-2mb48\" (UID: \"b5d1f6d5-d5b2-456c-8a97-74fb30efc048\") " pod="openshift-monitoring/node-exporter-2mb48"
Jul 01 17:20:22.379430 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.379408    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5d1f6d5-d5b2-456c-8a97-74fb30efc048-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2mb48\" (UID: \"b5d1f6d5-d5b2-456c-8a97-74fb30efc048\") " pod="openshift-monitoring/node-exporter-2mb48"
Jul 01 17:20:22.386104 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.386049    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kk59\" (UniqueName: \"kubernetes.io/projected/b5d1f6d5-d5b2-456c-8a97-74fb30efc048-kube-api-access-4kk59\") pod \"node-exporter-2mb48\" (UID: \"b5d1f6d5-d5b2-456c-8a97-74fb30efc048\") " pod="openshift-monitoring/node-exporter-2mb48"
Jul 01 17:20:22.882165 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.882136    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5d1f6d5-d5b2-456c-8a97-74fb30efc048-node-exporter-tls\") pod \"node-exporter-2mb48\" (UID: \"b5d1f6d5-d5b2-456c-8a97-74fb30efc048\") " pod="openshift-monitoring/node-exporter-2mb48"
Jul 01 17:20:22.884694 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:22.884671    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5d1f6d5-d5b2-456c-8a97-74fb30efc048-node-exporter-tls\") pod \"node-exporter-2mb48\" (UID: \"b5d1f6d5-d5b2-456c-8a97-74fb30efc048\") " pod="openshift-monitoring/node-exporter-2mb48"
Jul 01 17:20:23.044857 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:23.044823    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2mb48"
Jul 01 17:20:23.055126 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:20:23.055082    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5d1f6d5_d5b2_456c_8a97_74fb30efc048.slice/crio-469cb2edffe5c0108faefe40d93ec423e6b7d9d7777cb9883aa891eea0e2afc3 WatchSource:0}: Error finding container 469cb2edffe5c0108faefe40d93ec423e6b7d9d7777cb9883aa891eea0e2afc3: Status 404 returned error can't find the container with id 469cb2edffe5c0108faefe40d93ec423e6b7d9d7777cb9883aa891eea0e2afc3
Jul 01 17:20:23.248547 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:23.248461    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2mb48" event={"ID":"b5d1f6d5-d5b2-456c-8a97-74fb30efc048","Type":"ContainerStarted","Data":"469cb2edffe5c0108faefe40d93ec423e6b7d9d7777cb9883aa891eea0e2afc3"}
Jul 01 17:20:23.249634 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:23.249616    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c957b7d94-9xj75"
Jul 01 17:20:24.254199 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:24.254123    2585 generic.go:356] "Generic (PLEG): container finished" podID="b5d1f6d5-d5b2-456c-8a97-74fb30efc048" containerID="4b2412ffd4383fdd6d045b84b2eaeb740d4e0da9e12be842f58c8f4905133392" exitCode=0
Jul 01 17:20:24.254605 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:24.254208    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2mb48" event={"ID":"b5d1f6d5-d5b2-456c-8a97-74fb30efc048","Type":"ContainerDied","Data":"4b2412ffd4383fdd6d045b84b2eaeb740d4e0da9e12be842f58c8f4905133392"}
Jul 01 17:20:25.260311 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:25.260275    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2mb48" event={"ID":"b5d1f6d5-d5b2-456c-8a97-74fb30efc048","Type":"ContainerStarted","Data":"5ce8e997d8a4007345b6efa1b4e6e474cc5bcb1094cd8331eecc84396ca7e2f0"}
Jul 01 17:20:25.260725 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:25.260356    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2mb48" event={"ID":"b5d1f6d5-d5b2-456c-8a97-74fb30efc048","Type":"ContainerStarted","Data":"e672ab59893283081a21729db0efd87ef7f6e5d06ff285fb9403f3c80109c3b6"}
Jul 01 17:20:25.282196 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:25.282148    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2mb48" podStartSLOduration=2.272647833 podStartE2EDuration="3.282129245s" totalImagesPullingTime="939.833457ms" totalInitContainerRuntime="69.647955ms" isStatefulPod=true podCreationTimestamp="2026-07-01 17:20:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:20:25.28064831 +0000 UTC m=+313.585846478" watchObservedRunningTime="2026-07-01 17:20:25.282129245 +0000 UTC m=+313.587327422"
Jul 01 17:20:29.374768 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:29.374727    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59d56646cd-98p6s"]
Jul 01 17:20:47.441330 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:47.441286    2585 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74ff8b899b-6zbhz" podUID="4601677c-35cd-4c70-95c6-1d3d6670235f" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 01 17:20:54.399169 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:54.399096    2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-59d56646cd-98p6s" podUID="33dc1940-f452-4f4e-a62b-c07096cee771" containerName="registry" containerID="cri-o://47cd56f4600af830ed70a2b346cebe605ac8d652229621dea469b32fafc091f4" gracePeriod=30
Jul 01 17:20:54.632834 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:54.632808    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:20:54.707422 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:54.707350    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/33dc1940-f452-4f4e-a62b-c07096cee771-ca-trust-extracted\") pod \"33dc1940-f452-4f4e-a62b-c07096cee771\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") "
Jul 01 17:20:54.707422 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:54.707385    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33dc1940-f452-4f4e-a62b-c07096cee771-trusted-ca\") pod \"33dc1940-f452-4f4e-a62b-c07096cee771\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") "
Jul 01 17:20:54.707422 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:54.707402    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/33dc1940-f452-4f4e-a62b-c07096cee771-installation-pull-secrets\") pod \"33dc1940-f452-4f4e-a62b-c07096cee771\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") "
Jul 01 17:20:54.707674 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:54.707445    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-registry-tls\") pod \"33dc1940-f452-4f4e-a62b-c07096cee771\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") "
Jul 01 17:20:54.707674 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:54.707466    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/33dc1940-f452-4f4e-a62b-c07096cee771-image-registry-private-configuration\") pod \"33dc1940-f452-4f4e-a62b-c07096cee771\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") "
Jul 01 17:20:54.707674 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:54.707497    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/33dc1940-f452-4f4e-a62b-c07096cee771-registry-certificates\") pod \"33dc1940-f452-4f4e-a62b-c07096cee771\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") "
Jul 01 17:20:54.707674 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:54.707525    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4clc\" (UniqueName: \"kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-kube-api-access-t4clc\") pod \"33dc1940-f452-4f4e-a62b-c07096cee771\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") "
Jul 01 17:20:54.707896 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:54.707870    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-bound-sa-token\") pod \"33dc1940-f452-4f4e-a62b-c07096cee771\" (UID: \"33dc1940-f452-4f4e-a62b-c07096cee771\") "
Jul 01 17:20:54.707952 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:54.707932    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33dc1940-f452-4f4e-a62b-c07096cee771-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "33dc1940-f452-4f4e-a62b-c07096cee771" (UID: "33dc1940-f452-4f4e-a62b-c07096cee771"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:20:54.708005 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:54.707981    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33dc1940-f452-4f4e-a62b-c07096cee771-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "33dc1940-f452-4f4e-a62b-c07096cee771" (UID: "33dc1940-f452-4f4e-a62b-c07096cee771"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:20:54.708136 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:54.708118    2585 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/33dc1940-f452-4f4e-a62b-c07096cee771-registry-certificates\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:20:54.708194 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:54.708141    2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33dc1940-f452-4f4e-a62b-c07096cee771-trusted-ca\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:20:54.710221 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:54.710184    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33dc1940-f452-4f4e-a62b-c07096cee771-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "33dc1940-f452-4f4e-a62b-c07096cee771" (UID: "33dc1940-f452-4f4e-a62b-c07096cee771"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:20:54.710339 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:54.710293    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "33dc1940-f452-4f4e-a62b-c07096cee771" (UID: "33dc1940-f452-4f4e-a62b-c07096cee771"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:20:54.710339 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:54.710296    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-kube-api-access-t4clc" (OuterVolumeSpecName: "kube-api-access-t4clc") pod "33dc1940-f452-4f4e-a62b-c07096cee771" (UID: "33dc1940-f452-4f4e-a62b-c07096cee771"). InnerVolumeSpecName "kube-api-access-t4clc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:20:54.710732 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:54.710704    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33dc1940-f452-4f4e-a62b-c07096cee771-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "33dc1940-f452-4f4e-a62b-c07096cee771" (UID: "33dc1940-f452-4f4e-a62b-c07096cee771"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:20:54.710777 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:54.710749    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "33dc1940-f452-4f4e-a62b-c07096cee771" (UID: "33dc1940-f452-4f4e-a62b-c07096cee771"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:20:54.715871 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:54.715846    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33dc1940-f452-4f4e-a62b-c07096cee771-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "33dc1940-f452-4f4e-a62b-c07096cee771" (UID: "33dc1940-f452-4f4e-a62b-c07096cee771"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 17:20:54.809234 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:54.809194    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4clc\" (UniqueName: \"kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-kube-api-access-t4clc\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:20:54.809234 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:54.809232    2585 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-bound-sa-token\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:20:54.809234 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:54.809241    2585 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/33dc1940-f452-4f4e-a62b-c07096cee771-ca-trust-extracted\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:20:54.809415 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:54.809249    2585 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/33dc1940-f452-4f4e-a62b-c07096cee771-installation-pull-secrets\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:20:54.809415 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:54.809257    2585 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33dc1940-f452-4f4e-a62b-c07096cee771-registry-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:20:54.809415 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:54.809266    2585 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/33dc1940-f452-4f4e-a62b-c07096cee771-image-registry-private-configuration\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:20:55.359054 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:55.359023    2585 generic.go:356] "Generic (PLEG): container finished" podID="33dc1940-f452-4f4e-a62b-c07096cee771" containerID="47cd56f4600af830ed70a2b346cebe605ac8d652229621dea469b32fafc091f4" exitCode=0
Jul 01 17:20:55.359260 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:55.359099    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59d56646cd-98p6s"
Jul 01 17:20:55.359260 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:55.359100    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59d56646cd-98p6s" event={"ID":"33dc1940-f452-4f4e-a62b-c07096cee771","Type":"ContainerDied","Data":"47cd56f4600af830ed70a2b346cebe605ac8d652229621dea469b32fafc091f4"}
Jul 01 17:20:55.359260 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:55.359225    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59d56646cd-98p6s" event={"ID":"33dc1940-f452-4f4e-a62b-c07096cee771","Type":"ContainerDied","Data":"3581d65f8743d7932c62887257f21e8003b3c6109a8c5cc921e44e1cf82b2924"}
Jul 01 17:20:55.359260 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:55.359246    2585 scope.go:117] "RemoveContainer" containerID="47cd56f4600af830ed70a2b346cebe605ac8d652229621dea469b32fafc091f4"
Jul 01 17:20:55.367305 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:55.367289    2585 scope.go:117] "RemoveContainer" containerID="47cd56f4600af830ed70a2b346cebe605ac8d652229621dea469b32fafc091f4"
Jul 01 17:20:55.367567 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:20:55.367547    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47cd56f4600af830ed70a2b346cebe605ac8d652229621dea469b32fafc091f4\": container with ID starting with 47cd56f4600af830ed70a2b346cebe605ac8d652229621dea469b32fafc091f4 not found: ID does not exist" containerID="47cd56f4600af830ed70a2b346cebe605ac8d652229621dea469b32fafc091f4"
Jul 01 17:20:55.367636 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:55.367594    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47cd56f4600af830ed70a2b346cebe605ac8d652229621dea469b32fafc091f4"} err="failed to get container status \"47cd56f4600af830ed70a2b346cebe605ac8d652229621dea469b32fafc091f4\": rpc error: code = NotFound desc = could not find container \"47cd56f4600af830ed70a2b346cebe605ac8d652229621dea469b32fafc091f4\": container with ID starting with 47cd56f4600af830ed70a2b346cebe605ac8d652229621dea469b32fafc091f4 not found: ID does not exist"
Jul 01 17:20:55.378452 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:55.378433    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59d56646cd-98p6s"]
Jul 01 17:20:55.382762 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:55.382740    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-59d56646cd-98p6s"]
Jul 01 17:20:56.171454 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:56.171420    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33dc1940-f452-4f4e-a62b-c07096cee771" path="/var/lib/kubelet/pods/33dc1940-f452-4f4e-a62b-c07096cee771/volumes"
Jul 01 17:20:57.441595 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:20:57.441538    2585 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74ff8b899b-6zbhz" podUID="4601677c-35cd-4c70-95c6-1d3d6670235f" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 01 17:21:07.440979 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:21:07.440901    2585 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74ff8b899b-6zbhz" podUID="4601677c-35cd-4c70-95c6-1d3d6670235f" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 01 17:21:07.440979 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:21:07.440973    2585 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74ff8b899b-6zbhz"
Jul 01 17:21:07.441391 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:21:07.441375    2585 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"a1e0cb810b5928f820eee1889ce9cfbe2dc93aca3a2eacfcf26b5581244007ed"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74ff8b899b-6zbhz" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 01 17:21:07.441430 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:21:07.441408    2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74ff8b899b-6zbhz" podUID="4601677c-35cd-4c70-95c6-1d3d6670235f" containerName="service-proxy" containerID="cri-o://a1e0cb810b5928f820eee1889ce9cfbe2dc93aca3a2eacfcf26b5581244007ed" gracePeriod=30
Jul 01 17:21:08.402886 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:21:08.402852    2585 generic.go:356] "Generic (PLEG): container finished" podID="4601677c-35cd-4c70-95c6-1d3d6670235f" containerID="a1e0cb810b5928f820eee1889ce9cfbe2dc93aca3a2eacfcf26b5581244007ed" exitCode=2
Jul 01 17:21:08.403063 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:21:08.402918    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74ff8b899b-6zbhz" event={"ID":"4601677c-35cd-4c70-95c6-1d3d6670235f","Type":"ContainerDied","Data":"a1e0cb810b5928f820eee1889ce9cfbe2dc93aca3a2eacfcf26b5581244007ed"}
Jul 01 17:21:08.403063 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:21:08.402950    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74ff8b899b-6zbhz" event={"ID":"4601677c-35cd-4c70-95c6-1d3d6670235f","Type":"ContainerStarted","Data":"c3e39fdd71ee043e97d32a0084630894acdb38fa667444b62d872876a7b71529"}
Jul 01 17:21:10.066103 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:21:10.066072    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2mb48_b5d1f6d5-d5b2-456c-8a97-74fb30efc048/init-textfile/0.log"
Jul 01 17:21:10.267201 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:21:10.267171    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2mb48_b5d1f6d5-d5b2-456c-8a97-74fb30efc048/node-exporter/0.log"
Jul 01 17:21:10.466297 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:21:10.466210    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2mb48_b5d1f6d5-d5b2-456c-8a97-74fb30efc048/kube-rbac-proxy/0.log"
Jul 01 17:21:16.271489 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:21:16.271461    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cdb5968b9-phc4p_dece537c-3a74-44b7-b4bf-ef9fbd0f1d01/networking-console-plugin/0.log"
Jul 01 17:21:25.129505 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:21:25.129466    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5026d3ed-589a-4390-a2b7-d5f64c06e687-metrics-certs\") pod \"network-metrics-daemon-4rrth\" (UID: \"5026d3ed-589a-4390-a2b7-d5f64c06e687\") " pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:21:25.131892 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:21:25.131863    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5026d3ed-589a-4390-a2b7-d5f64c06e687-metrics-certs\") pod \"network-metrics-daemon-4rrth\" (UID: \"5026d3ed-589a-4390-a2b7-d5f64c06e687\") " pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:21:25.271532 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:21:25.271500    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gnmdh\""
Jul 01 17:21:25.279179 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:21:25.279161    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rrth"
Jul 01 17:21:25.406006 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:21:25.405910    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4rrth"]
Jul 01 17:21:25.410173 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:21:25.410145    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5026d3ed_589a_4390_a2b7_d5f64c06e687.slice/crio-3d0c8d2f9a53416251ef3c5ad29f0c6017dc039fb6f3a8b83fff48a9e5227399 WatchSource:0}: Error finding container 3d0c8d2f9a53416251ef3c5ad29f0c6017dc039fb6f3a8b83fff48a9e5227399: Status 404 returned error can't find the container with id 3d0c8d2f9a53416251ef3c5ad29f0c6017dc039fb6f3a8b83fff48a9e5227399
Jul 01 17:21:25.461276 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:21:25.461243    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4rrth" event={"ID":"5026d3ed-589a-4390-a2b7-d5f64c06e687","Type":"ContainerStarted","Data":"3d0c8d2f9a53416251ef3c5ad29f0c6017dc039fb6f3a8b83fff48a9e5227399"}
Jul 01 17:21:27.470022 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:21:27.469984    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4rrth" event={"ID":"5026d3ed-589a-4390-a2b7-d5f64c06e687","Type":"ContainerStarted","Data":"bf56455efba6fd1654912400a35b7235bcfb3086db72db9a1cd66bb302801ea3"}
Jul 01 17:21:27.470387 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:21:27.470039    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4rrth" event={"ID":"5026d3ed-589a-4390-a2b7-d5f64c06e687","Type":"ContainerStarted","Data":"7662a3ea5911faa4b837f3536724f16a3ca496733903240f03289e96b7bd4a41"}
Jul 01 17:21:27.487966 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:21:27.487921    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4rrth" podStartSLOduration=374.13708014 podStartE2EDuration="6m15.487906293s" totalImagesPullingTime="1.350826153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:21:27.486082328 +0000 UTC m=+375.791280506" watchObservedRunningTime="2026-07-01 17:21:27.487906293 +0000 UTC m=+375.793104470"
Jul 01 17:23:18.597554 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:18.597522    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nsgnn"]
Jul 01 17:23:18.597988 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:18.597752    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33dc1940-f452-4f4e-a62b-c07096cee771" containerName="registry"
Jul 01 17:23:18.597988 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:18.597763    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="33dc1940-f452-4f4e-a62b-c07096cee771" containerName="registry"
Jul 01 17:23:18.597988 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:18.597816    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="33dc1940-f452-4f4e-a62b-c07096cee771" containerName="registry"
Jul 01 17:23:18.599863 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:18.599844    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nsgnn"
Jul 01 17:23:18.603860 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:18.603842    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 01 17:23:18.603980 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:18.603876    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 01 17:23:18.604405 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:18.604390    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-6p5zl\""
Jul 01 17:23:18.604744 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:18.604727    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 01 17:23:18.625044 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:18.625018    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nsgnn"]
Jul 01 17:23:18.695447 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:18.695409    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gplrn\" (UniqueName: \"kubernetes.io/projected/f5535988-7369-4874-b6b5-45af3daf11c5-kube-api-access-gplrn\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-nsgnn\" (UID: \"f5535988-7369-4874-b6b5-45af3daf11c5\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nsgnn"
Jul 01 17:23:18.695447 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:18.695447    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/f5535988-7369-4874-b6b5-45af3daf11c5-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-nsgnn\" (UID: \"f5535988-7369-4874-b6b5-45af3daf11c5\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nsgnn"
Jul 01 17:23:18.795876 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:18.795847    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gplrn\" (UniqueName: \"kubernetes.io/projected/f5535988-7369-4874-b6b5-45af3daf11c5-kube-api-access-gplrn\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-nsgnn\" (UID: \"f5535988-7369-4874-b6b5-45af3daf11c5\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nsgnn"
Jul 01 17:23:18.795876 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:18.795882    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/f5535988-7369-4874-b6b5-45af3daf11c5-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-nsgnn\" (UID: \"f5535988-7369-4874-b6b5-45af3daf11c5\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nsgnn"
Jul 01 17:23:18.798324 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:18.798299    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/f5535988-7369-4874-b6b5-45af3daf11c5-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-nsgnn\" (UID: \"f5535988-7369-4874-b6b5-45af3daf11c5\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nsgnn"
Jul 01 17:23:18.809403 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:18.809380    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gplrn\" (UniqueName: \"kubernetes.io/projected/f5535988-7369-4874-b6b5-45af3daf11c5-kube-api-access-gplrn\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-nsgnn\" (UID: \"f5535988-7369-4874-b6b5-45af3daf11c5\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nsgnn"
Jul 01 17:23:18.909493 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:18.909407    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nsgnn"
Jul 01 17:23:19.040487 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:19.040454    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nsgnn"]
Jul 01 17:23:19.044320 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:23:19.044290    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5535988_7369_4874_b6b5_45af3daf11c5.slice/crio-c4956b233ba035ecbac69a5dc01f163b3803f5996c66ed75d6ae5e09840b42fa WatchSource:0}: Error finding container c4956b233ba035ecbac69a5dc01f163b3803f5996c66ed75d6ae5e09840b42fa: Status 404 returned error can't find the container with id c4956b233ba035ecbac69a5dc01f163b3803f5996c66ed75d6ae5e09840b42fa
Jul 01 17:23:19.835818 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:19.835767    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nsgnn" event={"ID":"f5535988-7369-4874-b6b5-45af3daf11c5","Type":"ContainerStarted","Data":"c4956b233ba035ecbac69a5dc01f163b3803f5996c66ed75d6ae5e09840b42fa"}
Jul 01 17:23:22.848942 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:22.848896    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nsgnn" event={"ID":"f5535988-7369-4874-b6b5-45af3daf11c5","Type":"ContainerStarted","Data":"29fe38d3080e425149fea52972e44193f837ff136e0da64a9be3719444352dca"}
Jul 01 17:23:22.849347 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:22.849035    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nsgnn"
Jul 01 17:23:22.873289 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:22.873231    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nsgnn" podStartSLOduration=1.746233245 podStartE2EDuration="4.873211642s" totalImagesPullingTime="3.126978397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:23:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:23:22.871038631 +0000 UTC m=+491.176236807" watchObservedRunningTime="2026-07-01 17:23:22.873211642 +0000 UTC m=+491.178409826"
Jul 01 17:23:22.875540 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:22.875521    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-68bff59c-fxdlb"]
Jul 01 17:23:22.877489 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:22.877474    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-fxdlb"
Jul 01 17:23:22.884450 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:22.884430    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 01 17:23:22.884691 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:22.884670    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 01 17:23:22.884942 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:22.884930    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-hf5nl\""
Jul 01 17:23:22.889371 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:22.889349    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-fxdlb"]
Jul 01 17:23:22.927170 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:22.927138    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdgcb\" (UniqueName: \"kubernetes.io/projected/f79c1e16-6cac-40f9-9689-f96e55278418-kube-api-access-qdgcb\") pod \"keda-operator-68bff59c-fxdlb\" (UID: \"f79c1e16-6cac-40f9-9689-f96e55278418\") " pod="openshift-keda/keda-operator-68bff59c-fxdlb"
Jul 01 17:23:22.927318 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:22.927185    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f79c1e16-6cac-40f9-9689-f96e55278418-certificates\") pod \"keda-operator-68bff59c-fxdlb\" (UID: \"f79c1e16-6cac-40f9-9689-f96e55278418\") " pod="openshift-keda/keda-operator-68bff59c-fxdlb"
Jul 01 17:23:22.927318 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:22.927232    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/f79c1e16-6cac-40f9-9689-f96e55278418-cabundle0\") pod \"keda-operator-68bff59c-fxdlb\" (UID: \"f79c1e16-6cac-40f9-9689-f96e55278418\") " pod="openshift-keda/keda-operator-68bff59c-fxdlb"
Jul 01 17:23:23.028030 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:23.027999    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f79c1e16-6cac-40f9-9689-f96e55278418-certificates\") pod \"keda-operator-68bff59c-fxdlb\" (UID: \"f79c1e16-6cac-40f9-9689-f96e55278418\") " pod="openshift-keda/keda-operator-68bff59c-fxdlb"
Jul 01 17:23:23.028030 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:23.028033    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/f79c1e16-6cac-40f9-9689-f96e55278418-cabundle0\") pod \"keda-operator-68bff59c-fxdlb\" (UID: \"f79c1e16-6cac-40f9-9689-f96e55278418\") " pod="openshift-keda/keda-operator-68bff59c-fxdlb"
Jul 01 17:23:23.028259 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:23.028066    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdgcb\" (UniqueName: \"kubernetes.io/projected/f79c1e16-6cac-40f9-9689-f96e55278418-kube-api-access-qdgcb\") pod \"keda-operator-68bff59c-fxdlb\" (UID: \"f79c1e16-6cac-40f9-9689-f96e55278418\") " pod="openshift-keda/keda-operator-68bff59c-fxdlb"
Jul 01 17:23:23.028259 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:23:23.028159    2585 secret.go:281] references non-existent secret key: ca.crt
Jul 01 17:23:23.028259 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:23:23.028184    2585 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 01 17:23:23.028259 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:23:23.028194    2585 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-fxdlb: references non-existent secret key: ca.crt
Jul 01 17:23:23.028259 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:23:23.028254    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f79c1e16-6cac-40f9-9689-f96e55278418-certificates podName:f79c1e16-6cac-40f9-9689-f96e55278418 nodeName:}" failed. No retries permitted until 2026-07-01 17:23:23.528235272 +0000 UTC m=+491.833433430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/f79c1e16-6cac-40f9-9689-f96e55278418-certificates") pod "keda-operator-68bff59c-fxdlb" (UID: "f79c1e16-6cac-40f9-9689-f96e55278418") : references non-existent secret key: ca.crt
Jul 01 17:23:23.028793 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:23.028774    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/f79c1e16-6cac-40f9-9689-f96e55278418-cabundle0\") pod \"keda-operator-68bff59c-fxdlb\" (UID: \"f79c1e16-6cac-40f9-9689-f96e55278418\") " pod="openshift-keda/keda-operator-68bff59c-fxdlb"
Jul 01 17:23:23.037272 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:23.037253    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdgcb\" (UniqueName: \"kubernetes.io/projected/f79c1e16-6cac-40f9-9689-f96e55278418-kube-api-access-qdgcb\") pod \"keda-operator-68bff59c-fxdlb\" (UID: \"f79c1e16-6cac-40f9-9689-f96e55278418\") " pod="openshift-keda/keda-operator-68bff59c-fxdlb"
Jul 01 17:23:23.532833 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:23.532794    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f79c1e16-6cac-40f9-9689-f96e55278418-certificates\") pod \"keda-operator-68bff59c-fxdlb\" (UID: \"f79c1e16-6cac-40f9-9689-f96e55278418\") " pod="openshift-keda/keda-operator-68bff59c-fxdlb"
Jul 01 17:23:23.535376 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:23.535346    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f79c1e16-6cac-40f9-9689-f96e55278418-certificates\") pod \"keda-operator-68bff59c-fxdlb\" (UID: \"f79c1e16-6cac-40f9-9689-f96e55278418\") " pod="openshift-keda/keda-operator-68bff59c-fxdlb"
Jul 01 17:23:23.787138 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:23.787050    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-fxdlb"
Jul 01 17:23:23.915777 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:23.915744    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-fxdlb"]
Jul 01 17:23:23.919366 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:23:23.919334    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf79c1e16_6cac_40f9_9689_f96e55278418.slice/crio-da67b139d853286afbbd1913ba89c899d411f561022ecf1a450b48e02f2d29e5 WatchSource:0}: Error finding container da67b139d853286afbbd1913ba89c899d411f561022ecf1a450b48e02f2d29e5: Status 404 returned error can't find the container with id da67b139d853286afbbd1913ba89c899d411f561022ecf1a450b48e02f2d29e5
Jul 01 17:23:24.858057 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:24.858023    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-fxdlb" event={"ID":"f79c1e16-6cac-40f9-9689-f96e55278418","Type":"ContainerStarted","Data":"da67b139d853286afbbd1913ba89c899d411f561022ecf1a450b48e02f2d29e5"}
Jul 01 17:23:26.866003 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:26.865966    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-fxdlb" event={"ID":"f79c1e16-6cac-40f9-9689-f96e55278418","Type":"ContainerStarted","Data":"724818d236eedbd639cb3a21140bd5aaff67cc15d1eb275b27a863c3bf8473a4"}
Jul 01 17:23:26.866341 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:26.866059    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-68bff59c-fxdlb"
Jul 01 17:23:26.890379 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:26.890335    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-68bff59c-fxdlb" podStartSLOduration=2.017481461 podStartE2EDuration="4.890318227s" totalImagesPullingTime="2.872836766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:23:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:23:26.888533973 +0000 UTC m=+495.193732142" watchObservedRunningTime="2026-07-01 17:23:26.890318227 +0000 UTC m=+495.195516401"
Jul 01 17:23:43.855336 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:43.855306    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nsgnn"
Jul 01 17:23:47.871147 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:23:47.871109    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-68bff59c-fxdlb"
Jul 01 17:24:29.817468 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:24:29.817388    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-ls7gm"]
Jul 01 17:24:29.819853 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:24:29.819833    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-ls7gm"
Jul 01 17:24:29.822623 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:24:29.822600    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 01 17:24:29.822744 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:24:29.822624    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 01 17:24:29.822744 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:24:29.822646    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 01 17:24:29.822744 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:24:29.822679    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-sg2cj\""
Jul 01 17:24:29.831859 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:24:29.831840    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-ls7gm"]
Jul 01 17:24:29.902644 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:24:29.902613    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxt6m\" (UniqueName: \"kubernetes.io/projected/e39523e9-5d87-43ae-b7bb-63b0a5f8c9aa-kube-api-access-wxt6m\") pod \"seaweedfs-64568bcd49-ls7gm\" (UID: \"e39523e9-5d87-43ae-b7bb-63b0a5f8c9aa\") " pod="kserve/seaweedfs-64568bcd49-ls7gm"
Jul 01 17:24:29.902752 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:24:29.902661    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e39523e9-5d87-43ae-b7bb-63b0a5f8c9aa-data\") pod \"seaweedfs-64568bcd49-ls7gm\" (UID: \"e39523e9-5d87-43ae-b7bb-63b0a5f8c9aa\") " pod="kserve/seaweedfs-64568bcd49-ls7gm"
Jul 01 17:24:30.003922 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:24:30.003893    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxt6m\" (UniqueName: \"kubernetes.io/projected/e39523e9-5d87-43ae-b7bb-63b0a5f8c9aa-kube-api-access-wxt6m\") pod \"seaweedfs-64568bcd49-ls7gm\" (UID: \"e39523e9-5d87-43ae-b7bb-63b0a5f8c9aa\") " pod="kserve/seaweedfs-64568bcd49-ls7gm"
Jul 01 17:24:30.004048 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:24:30.003936    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e39523e9-5d87-43ae-b7bb-63b0a5f8c9aa-data\") pod \"seaweedfs-64568bcd49-ls7gm\" (UID: \"e39523e9-5d87-43ae-b7bb-63b0a5f8c9aa\") " pod="kserve/seaweedfs-64568bcd49-ls7gm"
Jul 01 17:24:30.004282 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:24:30.004266    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e39523e9-5d87-43ae-b7bb-63b0a5f8c9aa-data\") pod \"seaweedfs-64568bcd49-ls7gm\" (UID: \"e39523e9-5d87-43ae-b7bb-63b0a5f8c9aa\") " pod="kserve/seaweedfs-64568bcd49-ls7gm"
Jul 01 17:24:30.012115 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:24:30.012089    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxt6m\" (UniqueName: \"kubernetes.io/projected/e39523e9-5d87-43ae-b7bb-63b0a5f8c9aa-kube-api-access-wxt6m\") pod \"seaweedfs-64568bcd49-ls7gm\" (UID: \"e39523e9-5d87-43ae-b7bb-63b0a5f8c9aa\") " pod="kserve/seaweedfs-64568bcd49-ls7gm"
Jul 01 17:24:30.129105 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:24:30.129034    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-ls7gm"
Jul 01 17:24:30.252554 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:24:30.252523    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-ls7gm"]
Jul 01 17:24:30.255780 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:24:30.255750    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode39523e9_5d87_43ae_b7bb_63b0a5f8c9aa.slice/crio-fda1fab626c60dc1f92d01890107d6197c69db4d990c3284148e228806f3ee79 WatchSource:0}: Error finding container fda1fab626c60dc1f92d01890107d6197c69db4d990c3284148e228806f3ee79: Status 404 returned error can't find the container with id fda1fab626c60dc1f92d01890107d6197c69db4d990c3284148e228806f3ee79
Jul 01 17:24:31.080883 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:24:31.080840    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-ls7gm" event={"ID":"e39523e9-5d87-43ae-b7bb-63b0a5f8c9aa","Type":"ContainerStarted","Data":"fda1fab626c60dc1f92d01890107d6197c69db4d990c3284148e228806f3ee79"}
Jul 01 17:24:34.092545 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:24:34.092509    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-ls7gm" event={"ID":"e39523e9-5d87-43ae-b7bb-63b0a5f8c9aa","Type":"ContainerStarted","Data":"71799e5e08677edb80980655d128cd8bc2944ff300ac36172dce877098d2694d"}
Jul 01 17:24:34.092988 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:24:34.092643    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-ls7gm"
Jul 01 17:24:34.109784 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:24:34.109741    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-ls7gm" podStartSLOduration=1.6274980879999998 podStartE2EDuration="5.109728196s" totalImagesPullingTime="3.482230108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:24:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:24:34.108528363 +0000 UTC m=+562.413726533" watchObservedRunningTime="2026-07-01 17:24:34.109728196 +0000 UTC m=+562.414926372"
Jul 01 17:24:40.098326 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:24:40.098297    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-ls7gm"
Jul 01 17:25:12.137062 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:25:12.137037    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8tsw_500d2a90-2d21-42b3-9584-b04c14becaec/ovn-acl-logging/0.log"
Jul 01 17:25:12.138949 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:25:12.138928    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8tsw_500d2a90-2d21-42b3-9584-b04c14becaec/ovn-acl-logging/0.log"
Jul 01 17:25:39.601152 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:25:39.601122    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-lxr8b"]
Jul 01 17:25:39.603319 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:25:39.603295    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-lxr8b"
Jul 01 17:25:39.605870 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:25:39.605847    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-4rdhp\""
Jul 01 17:25:39.605982 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:25:39.605929    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 01 17:25:39.614365 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:25:39.614343    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-lxr8b"]
Jul 01 17:25:39.687535 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:25:39.687503    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqd4n\" (UniqueName: \"kubernetes.io/projected/21b36ae9-4319-4dee-8bb1-c10f6d0b64a0-kube-api-access-bqd4n\") pod \"odh-model-controller-84b54b7f97-lxr8b\" (UID: \"21b36ae9-4319-4dee-8bb1-c10f6d0b64a0\") " pod="kserve/odh-model-controller-84b54b7f97-lxr8b"
Jul 01 17:25:39.687721 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:25:39.687562    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21b36ae9-4319-4dee-8bb1-c10f6d0b64a0-cert\") pod \"odh-model-controller-84b54b7f97-lxr8b\" (UID: \"21b36ae9-4319-4dee-8bb1-c10f6d0b64a0\") " pod="kserve/odh-model-controller-84b54b7f97-lxr8b"
Jul 01 17:25:39.788639 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:25:39.788609    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqd4n\" (UniqueName: \"kubernetes.io/projected/21b36ae9-4319-4dee-8bb1-c10f6d0b64a0-kube-api-access-bqd4n\") pod \"odh-model-controller-84b54b7f97-lxr8b\" (UID: \"21b36ae9-4319-4dee-8bb1-c10f6d0b64a0\") " pod="kserve/odh-model-controller-84b54b7f97-lxr8b"
Jul 01 17:25:39.788779 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:25:39.788650    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21b36ae9-4319-4dee-8bb1-c10f6d0b64a0-cert\") pod \"odh-model-controller-84b54b7f97-lxr8b\" (UID: \"21b36ae9-4319-4dee-8bb1-c10f6d0b64a0\") " pod="kserve/odh-model-controller-84b54b7f97-lxr8b"
Jul 01 17:25:39.788779 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:25:39.788737    2585 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 01 17:25:39.788858 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:25:39.788787    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21b36ae9-4319-4dee-8bb1-c10f6d0b64a0-cert podName:21b36ae9-4319-4dee-8bb1-c10f6d0b64a0 nodeName:}" failed. No retries permitted until 2026-07-01 17:25:40.288772289 +0000 UTC m=+628.593970444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/21b36ae9-4319-4dee-8bb1-c10f6d0b64a0-cert") pod "odh-model-controller-84b54b7f97-lxr8b" (UID: "21b36ae9-4319-4dee-8bb1-c10f6d0b64a0") : secret "odh-model-controller-webhook-cert" not found
Jul 01 17:25:39.800141 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:25:39.800109    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqd4n\" (UniqueName: \"kubernetes.io/projected/21b36ae9-4319-4dee-8bb1-c10f6d0b64a0-kube-api-access-bqd4n\") pod \"odh-model-controller-84b54b7f97-lxr8b\" (UID: \"21b36ae9-4319-4dee-8bb1-c10f6d0b64a0\") " pod="kserve/odh-model-controller-84b54b7f97-lxr8b"
Jul 01 17:25:40.292790 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:25:40.292753    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21b36ae9-4319-4dee-8bb1-c10f6d0b64a0-cert\") pod \"odh-model-controller-84b54b7f97-lxr8b\" (UID: \"21b36ae9-4319-4dee-8bb1-c10f6d0b64a0\") " pod="kserve/odh-model-controller-84b54b7f97-lxr8b"
Jul 01 17:25:40.295314 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:25:40.295293    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21b36ae9-4319-4dee-8bb1-c10f6d0b64a0-cert\") pod \"odh-model-controller-84b54b7f97-lxr8b\" (UID: \"21b36ae9-4319-4dee-8bb1-c10f6d0b64a0\") " pod="kserve/odh-model-controller-84b54b7f97-lxr8b"
Jul 01 17:25:40.513216 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:25:40.513182    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-lxr8b"
Jul 01 17:25:40.633707 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:25:40.633670    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-lxr8b"]
Jul 01 17:25:40.637038 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:25:40.637011    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21b36ae9_4319_4dee_8bb1_c10f6d0b64a0.slice/crio-24fbd0e4c602f15fc03c86104e9c5ee0267434ff3cc710305f06a45d20ebe45c WatchSource:0}: Error finding container 24fbd0e4c602f15fc03c86104e9c5ee0267434ff3cc710305f06a45d20ebe45c: Status 404 returned error can't find the container with id 24fbd0e4c602f15fc03c86104e9c5ee0267434ff3cc710305f06a45d20ebe45c
Jul 01 17:25:40.638762 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:25:40.638747    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 17:25:41.316747 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:25:41.316698    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-lxr8b" event={"ID":"21b36ae9-4319-4dee-8bb1-c10f6d0b64a0","Type":"ContainerStarted","Data":"24fbd0e4c602f15fc03c86104e9c5ee0267434ff3cc710305f06a45d20ebe45c"}
Jul 01 17:25:44.329478 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:25:44.329446    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-lxr8b" event={"ID":"21b36ae9-4319-4dee-8bb1-c10f6d0b64a0","Type":"ContainerStarted","Data":"e9be984687e677bd4fa543de3f2d376e73371a23713186078cfd06f444f40b79"}
Jul 01 17:25:44.329910 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:25:44.329604    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-lxr8b"
Jul 01 17:25:44.345220 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:25:44.345166    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-lxr8b" podStartSLOduration=2.6519713940000003 podStartE2EDuration="5.345147886s" totalImagesPullingTime="2.693176492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:25:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:25:44.344256568 +0000 UTC m=+632.649454749" watchObservedRunningTime="2026-07-01 17:25:44.345147886 +0000 UTC m=+632.650346064"
Jul 01 17:25:55.334894 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:25:55.334854    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-lxr8b"
Jul 01 17:27:27.269669 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:27:27.269591    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58"]
Jul 01 17:27:27.272815 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:27:27.272795    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58"
Jul 01 17:27:27.275619 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:27:27.275534    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 17:27:27.275759 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:27:27.275629    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d0081-predictor-serving-cert\""
Jul 01 17:27:27.275759 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:27:27.275653    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d0081-kube-rbac-proxy-sar-config\""
Jul 01 17:27:27.275759 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:27:27.275634    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 17:27:27.276637 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:27:27.276623    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zg2kz\""
Jul 01 17:27:27.282983 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:27:27.282963    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58"]
Jul 01 17:27:27.360964 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:27:27.360927    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fltfw\" (UniqueName: \"kubernetes.io/projected/21d5fd76-11a2-4651-8eb6-ec9c022e59d3-kube-api-access-fltfw\") pod \"error-404-isvc-d0081-predictor-754fc45f7b-pkk58\" (UID: \"21d5fd76-11a2-4651-8eb6-ec9c022e59d3\") " pod="kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58"
Jul 01 17:27:27.361148 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:27:27.360965    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21d5fd76-11a2-4651-8eb6-ec9c022e59d3-proxy-tls\") pod \"error-404-isvc-d0081-predictor-754fc45f7b-pkk58\" (UID: \"21d5fd76-11a2-4651-8eb6-ec9c022e59d3\") " pod="kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58"
Jul 01 17:27:27.361148 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:27:27.361023    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21d5fd76-11a2-4651-8eb6-ec9c022e59d3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d0081-predictor-754fc45f7b-pkk58\" (UID: \"21d5fd76-11a2-4651-8eb6-ec9c022e59d3\") " pod="kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58"
Jul 01 17:27:27.462015 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:27:27.461982    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fltfw\" (UniqueName: \"kubernetes.io/projected/21d5fd76-11a2-4651-8eb6-ec9c022e59d3-kube-api-access-fltfw\") pod \"error-404-isvc-d0081-predictor-754fc45f7b-pkk58\" (UID: \"21d5fd76-11a2-4651-8eb6-ec9c022e59d3\") " pod="kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58"
Jul 01 17:27:27.462015 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:27:27.462017    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21d5fd76-11a2-4651-8eb6-ec9c022e59d3-proxy-tls\") pod \"error-404-isvc-d0081-predictor-754fc45f7b-pkk58\" (UID: \"21d5fd76-11a2-4651-8eb6-ec9c022e59d3\") " pod="kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58"
Jul 01 17:27:27.462239 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:27:27.462065    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21d5fd76-11a2-4651-8eb6-ec9c022e59d3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d0081-predictor-754fc45f7b-pkk58\" (UID: \"21d5fd76-11a2-4651-8eb6-ec9c022e59d3\") " pod="kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58"
Jul 01 17:27:27.462289 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:27:27.462232    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-d0081-predictor-serving-cert: secret "error-404-isvc-d0081-predictor-serving-cert" not found
Jul 01 17:27:27.462356 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:27:27.462338    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21d5fd76-11a2-4651-8eb6-ec9c022e59d3-proxy-tls podName:21d5fd76-11a2-4651-8eb6-ec9c022e59d3 nodeName:}" failed. No retries permitted until 2026-07-01 17:27:27.962313294 +0000 UTC m=+736.267511461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/21d5fd76-11a2-4651-8eb6-ec9c022e59d3-proxy-tls") pod "error-404-isvc-d0081-predictor-754fc45f7b-pkk58" (UID: "21d5fd76-11a2-4651-8eb6-ec9c022e59d3") : secret "error-404-isvc-d0081-predictor-serving-cert" not found
Jul 01 17:27:27.462791 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:27:27.462775    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21d5fd76-11a2-4651-8eb6-ec9c022e59d3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d0081-predictor-754fc45f7b-pkk58\" (UID: \"21d5fd76-11a2-4651-8eb6-ec9c022e59d3\") " pod="kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58"
Jul 01 17:27:27.473632 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:27:27.473607    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fltfw\" (UniqueName: \"kubernetes.io/projected/21d5fd76-11a2-4651-8eb6-ec9c022e59d3-kube-api-access-fltfw\") pod \"error-404-isvc-d0081-predictor-754fc45f7b-pkk58\" (UID: \"21d5fd76-11a2-4651-8eb6-ec9c022e59d3\") " pod="kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58"
Jul 01 17:27:27.965478 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:27:27.965442    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21d5fd76-11a2-4651-8eb6-ec9c022e59d3-proxy-tls\") pod \"error-404-isvc-d0081-predictor-754fc45f7b-pkk58\" (UID: \"21d5fd76-11a2-4651-8eb6-ec9c022e59d3\") " pod="kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58"
Jul 01 17:27:27.965674 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:27:27.965562    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-d0081-predictor-serving-cert: secret "error-404-isvc-d0081-predictor-serving-cert" not found
Jul 01 17:27:27.965674 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:27:27.965648    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21d5fd76-11a2-4651-8eb6-ec9c022e59d3-proxy-tls podName:21d5fd76-11a2-4651-8eb6-ec9c022e59d3 nodeName:}" failed. No retries permitted until 2026-07-01 17:27:28.96563314 +0000 UTC m=+737.270831299 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/21d5fd76-11a2-4651-8eb6-ec9c022e59d3-proxy-tls") pod "error-404-isvc-d0081-predictor-754fc45f7b-pkk58" (UID: "21d5fd76-11a2-4651-8eb6-ec9c022e59d3") : secret "error-404-isvc-d0081-predictor-serving-cert" not found
Jul 01 17:27:28.972419 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:27:28.972377    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21d5fd76-11a2-4651-8eb6-ec9c022e59d3-proxy-tls\") pod \"error-404-isvc-d0081-predictor-754fc45f7b-pkk58\" (UID: \"21d5fd76-11a2-4651-8eb6-ec9c022e59d3\") " pod="kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58"
Jul 01 17:27:28.975083 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:27:28.975056    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21d5fd76-11a2-4651-8eb6-ec9c022e59d3-proxy-tls\") pod \"error-404-isvc-d0081-predictor-754fc45f7b-pkk58\" (UID: \"21d5fd76-11a2-4651-8eb6-ec9c022e59d3\") " pod="kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58"
Jul 01 17:27:29.083364 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:27:29.083324    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58"
Jul 01 17:27:29.241677 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:27:29.241602    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58"]
Jul 01 17:27:29.244681 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:27:29.244653    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d5fd76_11a2_4651_8eb6_ec9c022e59d3.slice/crio-c8fcf1287bd48bec1cf0f677198fadadc9cef0cb7e1731e5e49b03319fe99f29 WatchSource:0}: Error finding container c8fcf1287bd48bec1cf0f677198fadadc9cef0cb7e1731e5e49b03319fe99f29: Status 404 returned error can't find the container with id c8fcf1287bd48bec1cf0f677198fadadc9cef0cb7e1731e5e49b03319fe99f29
Jul 01 17:27:29.689220 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:27:29.689179    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58" event={"ID":"21d5fd76-11a2-4651-8eb6-ec9c022e59d3","Type":"ContainerStarted","Data":"c8fcf1287bd48bec1cf0f677198fadadc9cef0cb7e1731e5e49b03319fe99f29"}
Jul 01 17:27:41.745153 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:27:41.745068    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58" event={"ID":"21d5fd76-11a2-4651-8eb6-ec9c022e59d3","Type":"ContainerStarted","Data":"e2a6b994f9dd46175264004b555c84dfd13682b06f5eb18b61bd283e88419d5e"}
Jul 01 17:27:43.754399 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:27:43.754359    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58" event={"ID":"21d5fd76-11a2-4651-8eb6-ec9c022e59d3","Type":"ContainerStarted","Data":"283784a7cc2bd4d79158dea0a663100920721177d6b497151c6227ac377a8889"}
Jul 01 17:27:43.754819 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:27:43.754538    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58"
Jul 01 17:27:43.775397 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:27:43.775346    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58" podStartSLOduration=2.807354436 podStartE2EDuration="16.775331958s" totalImagesPullingTime="13.967977522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:27:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:27:43.773983836 +0000 UTC m=+752.079182025" watchObservedRunningTime="2026-07-01 17:27:43.775331958 +0000 UTC m=+752.080530135"
Jul 01 17:27:44.758467 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:27:44.758432    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58"
Jul 01 17:27:44.759455 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:27:44.759428    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58" podUID="21d5fd76-11a2-4651-8eb6-ec9c022e59d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Jul 01 17:27:45.761852 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:27:45.761812    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58" podUID="21d5fd76-11a2-4651-8eb6-ec9c022e59d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Jul 01 17:27:50.766454 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:27:50.766424    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58"
Jul 01 17:27:50.766918 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:27:50.766896    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58" podUID="21d5fd76-11a2-4651-8eb6-ec9c022e59d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Jul 01 17:28:00.766938 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:28:00.766898    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58" podUID="21d5fd76-11a2-4651-8eb6-ec9c022e59d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Jul 01 17:28:10.767492 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:28:10.767454    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58" podUID="21d5fd76-11a2-4651-8eb6-ec9c022e59d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Jul 01 17:28:20.766914 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:28:20.766879    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58" podUID="21d5fd76-11a2-4651-8eb6-ec9c022e59d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Jul 01 17:28:30.768166 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:28:30.768137    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58"
Jul 01 17:28:46.900862 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:28:46.900828    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-d0081-685667769f-jhtmj"]
Jul 01 17:28:46.903146 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:28:46.903130    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-d0081-685667769f-jhtmj"
Jul 01 17:28:46.906137 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:28:46.906115    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-d0081-kube-rbac-proxy-sar-config\""
Jul 01 17:28:46.906251 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:28:46.906174    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-d0081-serving-cert\""
Jul 01 17:28:46.912285 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:28:46.912266    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-d0081-685667769f-jhtmj"]
Jul 01 17:28:47.044626 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:28:47.044589    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38567387-2772-4b34-b16f-259964e04a2f-openshift-service-ca-bundle\") pod \"switch-graph-d0081-685667769f-jhtmj\" (UID: \"38567387-2772-4b34-b16f-259964e04a2f\") " pod="kserve-ci-e2e-test/switch-graph-d0081-685667769f-jhtmj"
Jul 01 17:28:47.044626 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:28:47.044629    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38567387-2772-4b34-b16f-259964e04a2f-proxy-tls\") pod \"switch-graph-d0081-685667769f-jhtmj\" (UID: \"38567387-2772-4b34-b16f-259964e04a2f\") " pod="kserve-ci-e2e-test/switch-graph-d0081-685667769f-jhtmj"
Jul 01 17:28:47.145076 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:28:47.145044    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38567387-2772-4b34-b16f-259964e04a2f-openshift-service-ca-bundle\") pod \"switch-graph-d0081-685667769f-jhtmj\" (UID: \"38567387-2772-4b34-b16f-259964e04a2f\") " pod="kserve-ci-e2e-test/switch-graph-d0081-685667769f-jhtmj"
Jul 01 17:28:47.145076 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:28:47.145079    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38567387-2772-4b34-b16f-259964e04a2f-proxy-tls\") pod \"switch-graph-d0081-685667769f-jhtmj\" (UID: \"38567387-2772-4b34-b16f-259964e04a2f\") " pod="kserve-ci-e2e-test/switch-graph-d0081-685667769f-jhtmj"
Jul 01 17:28:47.145799 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:28:47.145780    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38567387-2772-4b34-b16f-259964e04a2f-openshift-service-ca-bundle\") pod \"switch-graph-d0081-685667769f-jhtmj\" (UID: \"38567387-2772-4b34-b16f-259964e04a2f\") " pod="kserve-ci-e2e-test/switch-graph-d0081-685667769f-jhtmj"
Jul 01 17:28:47.147546 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:28:47.147530    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38567387-2772-4b34-b16f-259964e04a2f-proxy-tls\") pod \"switch-graph-d0081-685667769f-jhtmj\" (UID: \"38567387-2772-4b34-b16f-259964e04a2f\") " pod="kserve-ci-e2e-test/switch-graph-d0081-685667769f-jhtmj"
Jul 01 17:28:47.212626 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:28:47.212527    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-d0081-685667769f-jhtmj"
Jul 01 17:28:47.334549 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:28:47.334486    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-d0081-685667769f-jhtmj"]
Jul 01 17:28:47.337284 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:28:47.337260    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38567387_2772_4b34_b16f_259964e04a2f.slice/crio-0d29c56e0b1525819c5e9142c1ecdfecfd682707123266ca79bea4b23817a96d WatchSource:0}: Error finding container 0d29c56e0b1525819c5e9142c1ecdfecfd682707123266ca79bea4b23817a96d: Status 404 returned error can't find the container with id 0d29c56e0b1525819c5e9142c1ecdfecfd682707123266ca79bea4b23817a96d
Jul 01 17:28:47.979349 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:28:47.979316    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-d0081-685667769f-jhtmj" event={"ID":"38567387-2772-4b34-b16f-259964e04a2f","Type":"ContainerStarted","Data":"0d29c56e0b1525819c5e9142c1ecdfecfd682707123266ca79bea4b23817a96d"}
Jul 01 17:28:50.994963 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:28:50.994922    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-d0081-685667769f-jhtmj" event={"ID":"38567387-2772-4b34-b16f-259964e04a2f","Type":"ContainerStarted","Data":"816ece37b20db966792778394d227b9217788a63cc4d9b433ec87b24a1507025"}
Jul 01 17:28:50.995366 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:28:50.995051    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-d0081-685667769f-jhtmj"
Jul 01 17:28:51.011741 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:28:51.011698    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-d0081-685667769f-jhtmj" podStartSLOduration=2.101334761 podStartE2EDuration="5.011686209s" totalImagesPullingTime="2.910351448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:28:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:28:51.011440857 +0000 UTC m=+819.316639034" watchObservedRunningTime="2026-07-01 17:28:51.011686209 +0000 UTC m=+819.316884406"
Jul 01 17:28:57.005108 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:28:57.005075    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-d0081-685667769f-jhtmj"
Jul 01 17:29:01.019996 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:01.019967    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-d0081-685667769f-jhtmj"]
Jul 01 17:29:01.020476 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:01.020143    2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-d0081-685667769f-jhtmj" podUID="38567387-2772-4b34-b16f-259964e04a2f" containerName="switch-graph-d0081" containerID="cri-o://816ece37b20db966792778394d227b9217788a63cc4d9b433ec87b24a1507025" gracePeriod=30
Jul 01 17:29:01.281624 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:01.281550    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g"]
Jul 01 17:29:01.284479 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:01.284461    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g"
Jul 01 17:29:01.287029 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:01.287012    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-6ac71-predictor-serving-cert\""
Jul 01 17:29:01.287134 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:01.287058    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-6ac71-kube-rbac-proxy-sar-config\""
Jul 01 17:29:01.297037 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:01.297015    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g"]
Jul 01 17:29:01.321273 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:01.321247    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58"]
Jul 01 17:29:01.321527 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:01.321490    2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58" podUID="21d5fd76-11a2-4651-8eb6-ec9c022e59d3" containerName="kserve-container" containerID="cri-o://e2a6b994f9dd46175264004b555c84dfd13682b06f5eb18b61bd283e88419d5e" gracePeriod=30
Jul 01 17:29:01.321644 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:01.321526    2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58" podUID="21d5fd76-11a2-4651-8eb6-ec9c022e59d3" containerName="kube-rbac-proxy" containerID="cri-o://283784a7cc2bd4d79158dea0a663100920721177d6b497151c6227ac377a8889" gracePeriod=30
Jul 01 17:29:01.450451 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:01.450423    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51c0df56-15c4-4eaa-988f-1c5b154853af-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g\" (UID: \"51c0df56-15c4-4eaa-988f-1c5b154853af\") " pod="kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g"
Jul 01 17:29:01.450591 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:01.450465    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcn5g\" (UniqueName: \"kubernetes.io/projected/51c0df56-15c4-4eaa-988f-1c5b154853af-kube-api-access-jcn5g\") pod \"error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g\" (UID: \"51c0df56-15c4-4eaa-988f-1c5b154853af\") " pod="kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g"
Jul 01 17:29:01.450591 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:01.450531    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51c0df56-15c4-4eaa-988f-1c5b154853af-proxy-tls\") pod \"error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g\" (UID: \"51c0df56-15c4-4eaa-988f-1c5b154853af\") " pod="kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g"
Jul 01 17:29:01.551972 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:01.551896    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51c0df56-15c4-4eaa-988f-1c5b154853af-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g\" (UID: \"51c0df56-15c4-4eaa-988f-1c5b154853af\") " pod="kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g"
Jul 01 17:29:01.551972 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:01.551933    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcn5g\" (UniqueName: \"kubernetes.io/projected/51c0df56-15c4-4eaa-988f-1c5b154853af-kube-api-access-jcn5g\") pod \"error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g\" (UID: \"51c0df56-15c4-4eaa-988f-1c5b154853af\") " pod="kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g"
Jul 01 17:29:01.552151 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:01.551978    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51c0df56-15c4-4eaa-988f-1c5b154853af-proxy-tls\") pod \"error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g\" (UID: \"51c0df56-15c4-4eaa-988f-1c5b154853af\") " pod="kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g"
Jul 01 17:29:01.552566 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:01.552534    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51c0df56-15c4-4eaa-988f-1c5b154853af-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g\" (UID: \"51c0df56-15c4-4eaa-988f-1c5b154853af\") " pod="kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g"
Jul 01 17:29:01.554497 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:01.554475    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51c0df56-15c4-4eaa-988f-1c5b154853af-proxy-tls\") pod \"error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g\" (UID: \"51c0df56-15c4-4eaa-988f-1c5b154853af\") " pod="kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g"
Jul 01 17:29:01.560148 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:01.560124    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcn5g\" (UniqueName: \"kubernetes.io/projected/51c0df56-15c4-4eaa-988f-1c5b154853af-kube-api-access-jcn5g\") pod \"error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g\" (UID: \"51c0df56-15c4-4eaa-988f-1c5b154853af\") " pod="kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g"
Jul 01 17:29:01.596299 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:01.596265    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g"
Jul 01 17:29:01.723417 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:01.723377    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g"]
Jul 01 17:29:01.726033 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:29:01.726011    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51c0df56_15c4_4eaa_988f_1c5b154853af.slice/crio-1940912da05bff3729b02213c1929adadffe6f9158da59bffe148cfb303e81a6 WatchSource:0}: Error finding container 1940912da05bff3729b02213c1929adadffe6f9158da59bffe148cfb303e81a6: Status 404 returned error can't find the container with id 1940912da05bff3729b02213c1929adadffe6f9158da59bffe148cfb303e81a6
Jul 01 17:29:02.003603 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:02.003541    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-d0081-685667769f-jhtmj" podUID="38567387-2772-4b34-b16f-259964e04a2f" containerName="switch-graph-d0081" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:29:02.037779 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:02.037748    2585 generic.go:356] "Generic (PLEG): container finished" podID="21d5fd76-11a2-4651-8eb6-ec9c022e59d3" containerID="283784a7cc2bd4d79158dea0a663100920721177d6b497151c6227ac377a8889" exitCode=2
Jul 01 17:29:02.038129 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:02.037822    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58" event={"ID":"21d5fd76-11a2-4651-8eb6-ec9c022e59d3","Type":"ContainerDied","Data":"283784a7cc2bd4d79158dea0a663100920721177d6b497151c6227ac377a8889"}
Jul 01 17:29:02.039566 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:02.039545    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g" event={"ID":"51c0df56-15c4-4eaa-988f-1c5b154853af","Type":"ContainerStarted","Data":"26d79261a6a3f208b5d25fafeacda6ee36676db2e6af5f35104a7fef951e9370"}
Jul 01 17:29:02.039709 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:02.039614    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g" event={"ID":"51c0df56-15c4-4eaa-988f-1c5b154853af","Type":"ContainerStarted","Data":"44a3cc08a02720c3590e95aae55233ce77776a1f3989d3c1cae806d8c37352b1"}
Jul 01 17:29:02.039709 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:02.039646    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g" event={"ID":"51c0df56-15c4-4eaa-988f-1c5b154853af","Type":"ContainerStarted","Data":"1940912da05bff3729b02213c1929adadffe6f9158da59bffe148cfb303e81a6"}
Jul 01 17:29:02.039832 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:02.039719    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g"
Jul 01 17:29:02.057198 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:02.057134    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g" podStartSLOduration=1.057119404 podStartE2EDuration="1.057119404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:29:02.055048357 +0000 UTC m=+830.360246538" watchObservedRunningTime="2026-07-01 17:29:02.057119404 +0000 UTC m=+830.362317634"
Jul 01 17:29:03.043712 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:03.043681    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g"
Jul 01 17:29:03.045000 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:03.044970    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g" podUID="51c0df56-15c4-4eaa-988f-1c5b154853af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 01 17:29:04.047279 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:04.047239    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g" podUID="51c0df56-15c4-4eaa-988f-1c5b154853af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 01 17:29:04.365862 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:04.365839    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58"
Jul 01 17:29:04.471231 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:04.471200    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21d5fd76-11a2-4651-8eb6-ec9c022e59d3-proxy-tls\") pod \"21d5fd76-11a2-4651-8eb6-ec9c022e59d3\" (UID: \"21d5fd76-11a2-4651-8eb6-ec9c022e59d3\") "
Jul 01 17:29:04.471365 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:04.471253    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fltfw\" (UniqueName: \"kubernetes.io/projected/21d5fd76-11a2-4651-8eb6-ec9c022e59d3-kube-api-access-fltfw\") pod \"21d5fd76-11a2-4651-8eb6-ec9c022e59d3\" (UID: \"21d5fd76-11a2-4651-8eb6-ec9c022e59d3\") "
Jul 01 17:29:04.471365 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:04.471274    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21d5fd76-11a2-4651-8eb6-ec9c022e59d3-kube-rbac-proxy-sar-config\") pod \"21d5fd76-11a2-4651-8eb6-ec9c022e59d3\" (UID: \"21d5fd76-11a2-4651-8eb6-ec9c022e59d3\") "
Jul 01 17:29:04.471650 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:04.471621    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21d5fd76-11a2-4651-8eb6-ec9c022e59d3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "21d5fd76-11a2-4651-8eb6-ec9c022e59d3" (UID: "21d5fd76-11a2-4651-8eb6-ec9c022e59d3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:29:04.473520 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:04.473490    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21d5fd76-11a2-4651-8eb6-ec9c022e59d3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "21d5fd76-11a2-4651-8eb6-ec9c022e59d3" (UID: "21d5fd76-11a2-4651-8eb6-ec9c022e59d3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:29:04.473630 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:04.473569    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21d5fd76-11a2-4651-8eb6-ec9c022e59d3-kube-api-access-fltfw" (OuterVolumeSpecName: "kube-api-access-fltfw") pod "21d5fd76-11a2-4651-8eb6-ec9c022e59d3" (UID: "21d5fd76-11a2-4651-8eb6-ec9c022e59d3"). InnerVolumeSpecName "kube-api-access-fltfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:29:04.572117 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:04.572089    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fltfw\" (UniqueName: \"kubernetes.io/projected/21d5fd76-11a2-4651-8eb6-ec9c022e59d3-kube-api-access-fltfw\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:29:04.572117 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:04.572114    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21d5fd76-11a2-4651-8eb6-ec9c022e59d3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:29:04.572267 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:04.572124    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21d5fd76-11a2-4651-8eb6-ec9c022e59d3-proxy-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:29:05.052352 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:05.052317    2585 generic.go:356] "Generic (PLEG): container finished" podID="21d5fd76-11a2-4651-8eb6-ec9c022e59d3" containerID="e2a6b994f9dd46175264004b555c84dfd13682b06f5eb18b61bd283e88419d5e" exitCode=0
Jul 01 17:29:05.052812 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:05.052353    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58" event={"ID":"21d5fd76-11a2-4651-8eb6-ec9c022e59d3","Type":"ContainerDied","Data":"e2a6b994f9dd46175264004b555c84dfd13682b06f5eb18b61bd283e88419d5e"}
Jul 01 17:29:05.052812 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:05.052424    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58" event={"ID":"21d5fd76-11a2-4651-8eb6-ec9c022e59d3","Type":"ContainerDied","Data":"c8fcf1287bd48bec1cf0f677198fadadc9cef0cb7e1731e5e49b03319fe99f29"}
Jul 01 17:29:05.052812 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:05.052360    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58"
Jul 01 17:29:05.052812 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:05.052465    2585 scope.go:117] "RemoveContainer" containerID="283784a7cc2bd4d79158dea0a663100920721177d6b497151c6227ac377a8889"
Jul 01 17:29:05.061295 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:05.061276    2585 scope.go:117] "RemoveContainer" containerID="e2a6b994f9dd46175264004b555c84dfd13682b06f5eb18b61bd283e88419d5e"
Jul 01 17:29:05.068226 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:05.068211    2585 scope.go:117] "RemoveContainer" containerID="283784a7cc2bd4d79158dea0a663100920721177d6b497151c6227ac377a8889"
Jul 01 17:29:05.068455 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:29:05.068441    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"283784a7cc2bd4d79158dea0a663100920721177d6b497151c6227ac377a8889\": container with ID starting with 283784a7cc2bd4d79158dea0a663100920721177d6b497151c6227ac377a8889 not found: ID does not exist" containerID="283784a7cc2bd4d79158dea0a663100920721177d6b497151c6227ac377a8889"
Jul 01 17:29:05.068490 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:05.068463    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"283784a7cc2bd4d79158dea0a663100920721177d6b497151c6227ac377a8889"} err="failed to get container status \"283784a7cc2bd4d79158dea0a663100920721177d6b497151c6227ac377a8889\": rpc error: code = NotFound desc = could not find container \"283784a7cc2bd4d79158dea0a663100920721177d6b497151c6227ac377a8889\": container with ID starting with 283784a7cc2bd4d79158dea0a663100920721177d6b497151c6227ac377a8889 not found: ID does not exist"
Jul 01 17:29:05.068490 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:05.068479    2585 scope.go:117] "RemoveContainer" containerID="e2a6b994f9dd46175264004b555c84dfd13682b06f5eb18b61bd283e88419d5e"
Jul 01 17:29:05.068694 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:29:05.068680    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2a6b994f9dd46175264004b555c84dfd13682b06f5eb18b61bd283e88419d5e\": container with ID starting with e2a6b994f9dd46175264004b555c84dfd13682b06f5eb18b61bd283e88419d5e not found: ID does not exist" containerID="e2a6b994f9dd46175264004b555c84dfd13682b06f5eb18b61bd283e88419d5e"
Jul 01 17:29:05.068735 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:05.068699    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2a6b994f9dd46175264004b555c84dfd13682b06f5eb18b61bd283e88419d5e"} err="failed to get container status \"e2a6b994f9dd46175264004b555c84dfd13682b06f5eb18b61bd283e88419d5e\": rpc error: code = NotFound desc = could not find container \"e2a6b994f9dd46175264004b555c84dfd13682b06f5eb18b61bd283e88419d5e\": container with ID starting with e2a6b994f9dd46175264004b555c84dfd13682b06f5eb18b61bd283e88419d5e not found: ID does not exist"
Jul 01 17:29:05.073590 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:05.073555    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58"]
Jul 01 17:29:05.077916 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:05.077895    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d0081-predictor-754fc45f7b-pkk58"]
Jul 01 17:29:06.172556 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:06.172521    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21d5fd76-11a2-4651-8eb6-ec9c022e59d3" path="/var/lib/kubelet/pods/21d5fd76-11a2-4651-8eb6-ec9c022e59d3/volumes"
Jul 01 17:29:07.003438 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:07.003404    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-d0081-685667769f-jhtmj" podUID="38567387-2772-4b34-b16f-259964e04a2f" containerName="switch-graph-d0081" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:29:09.051739 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:09.051707    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g"
Jul 01 17:29:09.052234 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:09.052208    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g" podUID="51c0df56-15c4-4eaa-988f-1c5b154853af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 01 17:29:12.003630 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:12.003595    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-d0081-685667769f-jhtmj" podUID="38567387-2772-4b34-b16f-259964e04a2f" containerName="switch-graph-d0081" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:29:12.003985 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:12.003696    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-d0081-685667769f-jhtmj"
Jul 01 17:29:17.003392 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:17.003350    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-d0081-685667769f-jhtmj" podUID="38567387-2772-4b34-b16f-259964e04a2f" containerName="switch-graph-d0081" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:29:19.052365 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:19.052318    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g" podUID="51c0df56-15c4-4eaa-988f-1c5b154853af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 01 17:29:22.004300 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:22.004265    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-d0081-685667769f-jhtmj" podUID="38567387-2772-4b34-b16f-259964e04a2f" containerName="switch-graph-d0081" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:29:27.003628 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:27.003590    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-d0081-685667769f-jhtmj" podUID="38567387-2772-4b34-b16f-259964e04a2f" containerName="switch-graph-d0081" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:29:29.052178 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:29.052140    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g" podUID="51c0df56-15c4-4eaa-988f-1c5b154853af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 01 17:29:31.147978 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:31.147946    2585 generic.go:356] "Generic (PLEG): container finished" podID="38567387-2772-4b34-b16f-259964e04a2f" containerID="816ece37b20db966792778394d227b9217788a63cc4d9b433ec87b24a1507025" exitCode=0
Jul 01 17:29:31.148329 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:31.148019    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-d0081-685667769f-jhtmj" event={"ID":"38567387-2772-4b34-b16f-259964e04a2f","Type":"ContainerDied","Data":"816ece37b20db966792778394d227b9217788a63cc4d9b433ec87b24a1507025"}
Jul 01 17:29:31.673038 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:31.673015    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-d0081-685667769f-jhtmj"
Jul 01 17:29:31.755417 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:31.755383    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38567387-2772-4b34-b16f-259964e04a2f-openshift-service-ca-bundle\") pod \"38567387-2772-4b34-b16f-259964e04a2f\" (UID: \"38567387-2772-4b34-b16f-259964e04a2f\") "
Jul 01 17:29:31.755598 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:31.755459    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38567387-2772-4b34-b16f-259964e04a2f-proxy-tls\") pod \"38567387-2772-4b34-b16f-259964e04a2f\" (UID: \"38567387-2772-4b34-b16f-259964e04a2f\") "
Jul 01 17:29:31.755813 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:31.755790    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38567387-2772-4b34-b16f-259964e04a2f-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "38567387-2772-4b34-b16f-259964e04a2f" (UID: "38567387-2772-4b34-b16f-259964e04a2f"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:29:31.757696 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:31.757660    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38567387-2772-4b34-b16f-259964e04a2f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "38567387-2772-4b34-b16f-259964e04a2f" (UID: "38567387-2772-4b34-b16f-259964e04a2f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:29:31.856815 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:31.856790    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38567387-2772-4b34-b16f-259964e04a2f-openshift-service-ca-bundle\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:29:31.856815 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:31.856815    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38567387-2772-4b34-b16f-259964e04a2f-proxy-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:29:32.152995 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:32.152916    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-d0081-685667769f-jhtmj"
Jul 01 17:29:32.152995 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:32.152935    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-d0081-685667769f-jhtmj" event={"ID":"38567387-2772-4b34-b16f-259964e04a2f","Type":"ContainerDied","Data":"0d29c56e0b1525819c5e9142c1ecdfecfd682707123266ca79bea4b23817a96d"}
Jul 01 17:29:32.153430 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:32.153046    2585 scope.go:117] "RemoveContainer" containerID="816ece37b20db966792778394d227b9217788a63cc4d9b433ec87b24a1507025"
Jul 01 17:29:32.173733 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:32.173706    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-d0081-685667769f-jhtmj"]
Jul 01 17:29:32.176896 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:32.176873    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-d0081-685667769f-jhtmj"]
Jul 01 17:29:34.172282 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:34.172251    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38567387-2772-4b34-b16f-259964e04a2f" path="/var/lib/kubelet/pods/38567387-2772-4b34-b16f-259964e04a2f/volumes"
Jul 01 17:29:36.848345 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:36.848311    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-5cf9f9c9dc-n4nq6"]
Jul 01 17:29:36.848721 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:36.848585    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21d5fd76-11a2-4651-8eb6-ec9c022e59d3" containerName="kserve-container"
Jul 01 17:29:36.848721 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:36.848596    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="21d5fd76-11a2-4651-8eb6-ec9c022e59d3" containerName="kserve-container"
Jul 01 17:29:36.848721 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:36.848614    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21d5fd76-11a2-4651-8eb6-ec9c022e59d3" containerName="kube-rbac-proxy"
Jul 01 17:29:36.848721 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:36.848620    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="21d5fd76-11a2-4651-8eb6-ec9c022e59d3" containerName="kube-rbac-proxy"
Jul 01 17:29:36.848721 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:36.848627    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38567387-2772-4b34-b16f-259964e04a2f" containerName="switch-graph-d0081"
Jul 01 17:29:36.848721 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:36.848634    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="38567387-2772-4b34-b16f-259964e04a2f" containerName="switch-graph-d0081"
Jul 01 17:29:36.848721 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:36.848674    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="21d5fd76-11a2-4651-8eb6-ec9c022e59d3" containerName="kube-rbac-proxy"
Jul 01 17:29:36.848721 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:36.848683    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="21d5fd76-11a2-4651-8eb6-ec9c022e59d3" containerName="kserve-container"
Jul 01 17:29:36.848721 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:36.848689    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="38567387-2772-4b34-b16f-259964e04a2f" containerName="switch-graph-d0081"
Jul 01 17:29:36.850885 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:36.850866    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-5cf9f9c9dc-n4nq6"
Jul 01 17:29:36.853699 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:36.853679    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
Jul 01 17:29:36.853798 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:36.853717    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
Jul 01 17:29:36.862618 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:36.862592    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5cf9f9c9dc-n4nq6"]
Jul 01 17:29:36.895028 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:36.895002    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/241b85d7-2d84-4b8a-9362-09ffbaa76939-openshift-service-ca-bundle\") pod \"model-chainer-5cf9f9c9dc-n4nq6\" (UID: \"241b85d7-2d84-4b8a-9362-09ffbaa76939\") " pod="kserve-ci-e2e-test/model-chainer-5cf9f9c9dc-n4nq6"
Jul 01 17:29:36.895131 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:36.895050    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/241b85d7-2d84-4b8a-9362-09ffbaa76939-proxy-tls\") pod \"model-chainer-5cf9f9c9dc-n4nq6\" (UID: \"241b85d7-2d84-4b8a-9362-09ffbaa76939\") " pod="kserve-ci-e2e-test/model-chainer-5cf9f9c9dc-n4nq6"
Jul 01 17:29:36.995796 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:36.995765    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/241b85d7-2d84-4b8a-9362-09ffbaa76939-proxy-tls\") pod \"model-chainer-5cf9f9c9dc-n4nq6\" (UID: \"241b85d7-2d84-4b8a-9362-09ffbaa76939\") " pod="kserve-ci-e2e-test/model-chainer-5cf9f9c9dc-n4nq6"
Jul 01 17:29:36.995930 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:36.995812    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/241b85d7-2d84-4b8a-9362-09ffbaa76939-openshift-service-ca-bundle\") pod \"model-chainer-5cf9f9c9dc-n4nq6\" (UID: \"241b85d7-2d84-4b8a-9362-09ffbaa76939\") " pod="kserve-ci-e2e-test/model-chainer-5cf9f9c9dc-n4nq6"
Jul 01 17:29:36.995930 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:29:36.995906    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-serving-cert: secret "model-chainer-serving-cert" not found
Jul 01 17:29:36.996008 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:29:36.995987    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/241b85d7-2d84-4b8a-9362-09ffbaa76939-proxy-tls podName:241b85d7-2d84-4b8a-9362-09ffbaa76939 nodeName:}" failed. No retries permitted until 2026-07-01 17:29:37.495971208 +0000 UTC m=+865.801169367 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/241b85d7-2d84-4b8a-9362-09ffbaa76939-proxy-tls") pod "model-chainer-5cf9f9c9dc-n4nq6" (UID: "241b85d7-2d84-4b8a-9362-09ffbaa76939") : secret "model-chainer-serving-cert" not found
Jul 01 17:29:36.996394 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:36.996378    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/241b85d7-2d84-4b8a-9362-09ffbaa76939-openshift-service-ca-bundle\") pod \"model-chainer-5cf9f9c9dc-n4nq6\" (UID: \"241b85d7-2d84-4b8a-9362-09ffbaa76939\") " pod="kserve-ci-e2e-test/model-chainer-5cf9f9c9dc-n4nq6"
Jul 01 17:29:37.498322 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:37.498285    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/241b85d7-2d84-4b8a-9362-09ffbaa76939-proxy-tls\") pod \"model-chainer-5cf9f9c9dc-n4nq6\" (UID: \"241b85d7-2d84-4b8a-9362-09ffbaa76939\") " pod="kserve-ci-e2e-test/model-chainer-5cf9f9c9dc-n4nq6"
Jul 01 17:29:37.500794 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:37.500771    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/241b85d7-2d84-4b8a-9362-09ffbaa76939-proxy-tls\") pod \"model-chainer-5cf9f9c9dc-n4nq6\" (UID: \"241b85d7-2d84-4b8a-9362-09ffbaa76939\") " pod="kserve-ci-e2e-test/model-chainer-5cf9f9c9dc-n4nq6"
Jul 01 17:29:37.762810 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:37.762730    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-5cf9f9c9dc-n4nq6"
Jul 01 17:29:37.888945 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:37.888907    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5cf9f9c9dc-n4nq6"]
Jul 01 17:29:37.891449 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:29:37.891424    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod241b85d7_2d84_4b8a_9362_09ffbaa76939.slice/crio-ede0ce111cc6491eb9866f4c8caeb16d8c9d80e3ea065454a68a6671275d6936 WatchSource:0}: Error finding container ede0ce111cc6491eb9866f4c8caeb16d8c9d80e3ea065454a68a6671275d6936: Status 404 returned error can't find the container with id ede0ce111cc6491eb9866f4c8caeb16d8c9d80e3ea065454a68a6671275d6936
Jul 01 17:29:38.176598 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:38.176549    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-5cf9f9c9dc-n4nq6" event={"ID":"241b85d7-2d84-4b8a-9362-09ffbaa76939","Type":"ContainerStarted","Data":"405bec264a6dcdd2b5bcce542a0a715e3cbb01318acf8bdfd948bd41a45a9efc"}
Jul 01 17:29:38.176779 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:38.176621    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-5cf9f9c9dc-n4nq6" event={"ID":"241b85d7-2d84-4b8a-9362-09ffbaa76939","Type":"ContainerStarted","Data":"ede0ce111cc6491eb9866f4c8caeb16d8c9d80e3ea065454a68a6671275d6936"}
Jul 01 17:29:38.176779 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:38.176701    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-5cf9f9c9dc-n4nq6"
Jul 01 17:29:38.193587 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:38.193531    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-5cf9f9c9dc-n4nq6" podStartSLOduration=2.193517187 podStartE2EDuration="2.193517187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:29:38.192763195 +0000 UTC m=+866.497961372" watchObservedRunningTime="2026-07-01 17:29:38.193517187 +0000 UTC m=+866.498715364"
Jul 01 17:29:39.052563 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:39.052528    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g" podUID="51c0df56-15c4-4eaa-988f-1c5b154853af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 01 17:29:44.186119 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:44.186090    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-5cf9f9c9dc-n4nq6"
Jul 01 17:29:46.951564 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:46.951533    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5cf9f9c9dc-n4nq6"]
Jul 01 17:29:46.951952 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:46.951730    2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-5cf9f9c9dc-n4nq6" podUID="241b85d7-2d84-4b8a-9362-09ffbaa76939" containerName="model-chainer" containerID="cri-o://405bec264a6dcdd2b5bcce542a0a715e3cbb01318acf8bdfd948bd41a45a9efc" gracePeriod=30
Jul 01 17:29:47.293242 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:47.293164    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7"]
Jul 01 17:29:47.295632 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:47.295615    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7"
Jul 01 17:29:47.299313 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:47.299290    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-766f3-predictor-serving-cert\""
Jul 01 17:29:47.299591 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:47.299566    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-766f3-kube-rbac-proxy-sar-config\""
Jul 01 17:29:47.311477 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:47.311457    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7"]
Jul 01 17:29:47.364658 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:47.364630    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8664e04d-5336-401b-9a09-6b159a33f1d7-proxy-tls\") pod \"error-404-isvc-766f3-predictor-6b6855b58b-dmhf7\" (UID: \"8664e04d-5336-401b-9a09-6b159a33f1d7\") " pod="kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7"
Jul 01 17:29:47.364787 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:47.364681    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2wx5\" (UniqueName: \"kubernetes.io/projected/8664e04d-5336-401b-9a09-6b159a33f1d7-kube-api-access-z2wx5\") pod \"error-404-isvc-766f3-predictor-6b6855b58b-dmhf7\" (UID: \"8664e04d-5336-401b-9a09-6b159a33f1d7\") " pod="kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7"
Jul 01 17:29:47.364787 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:47.364731    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8664e04d-5336-401b-9a09-6b159a33f1d7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-766f3-predictor-6b6855b58b-dmhf7\" (UID: \"8664e04d-5336-401b-9a09-6b159a33f1d7\") " pod="kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7"
Jul 01 17:29:47.465796 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:47.465764    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2wx5\" (UniqueName: \"kubernetes.io/projected/8664e04d-5336-401b-9a09-6b159a33f1d7-kube-api-access-z2wx5\") pod \"error-404-isvc-766f3-predictor-6b6855b58b-dmhf7\" (UID: \"8664e04d-5336-401b-9a09-6b159a33f1d7\") " pod="kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7"
Jul 01 17:29:47.465956 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:47.465805    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8664e04d-5336-401b-9a09-6b159a33f1d7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-766f3-predictor-6b6855b58b-dmhf7\" (UID: \"8664e04d-5336-401b-9a09-6b159a33f1d7\") " pod="kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7"
Jul 01 17:29:47.465956 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:47.465849    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8664e04d-5336-401b-9a09-6b159a33f1d7-proxy-tls\") pod \"error-404-isvc-766f3-predictor-6b6855b58b-dmhf7\" (UID: \"8664e04d-5336-401b-9a09-6b159a33f1d7\") " pod="kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7"
Jul 01 17:29:47.465956 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:29:47.465946    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-766f3-predictor-serving-cert: secret "error-404-isvc-766f3-predictor-serving-cert" not found
Jul 01 17:29:47.466081 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:29:47.466013    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8664e04d-5336-401b-9a09-6b159a33f1d7-proxy-tls podName:8664e04d-5336-401b-9a09-6b159a33f1d7 nodeName:}" failed. No retries permitted until 2026-07-01 17:29:47.965993652 +0000 UTC m=+876.271191831 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8664e04d-5336-401b-9a09-6b159a33f1d7-proxy-tls") pod "error-404-isvc-766f3-predictor-6b6855b58b-dmhf7" (UID: "8664e04d-5336-401b-9a09-6b159a33f1d7") : secret "error-404-isvc-766f3-predictor-serving-cert" not found
Jul 01 17:29:47.466526 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:47.466508    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8664e04d-5336-401b-9a09-6b159a33f1d7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-766f3-predictor-6b6855b58b-dmhf7\" (UID: \"8664e04d-5336-401b-9a09-6b159a33f1d7\") " pod="kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7"
Jul 01 17:29:47.474594 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:47.474544    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2wx5\" (UniqueName: \"kubernetes.io/projected/8664e04d-5336-401b-9a09-6b159a33f1d7-kube-api-access-z2wx5\") pod \"error-404-isvc-766f3-predictor-6b6855b58b-dmhf7\" (UID: \"8664e04d-5336-401b-9a09-6b159a33f1d7\") " pod="kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7"
Jul 01 17:29:47.968625 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:47.968593    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8664e04d-5336-401b-9a09-6b159a33f1d7-proxy-tls\") pod \"error-404-isvc-766f3-predictor-6b6855b58b-dmhf7\" (UID: \"8664e04d-5336-401b-9a09-6b159a33f1d7\") " pod="kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7"
Jul 01 17:29:47.970996 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:47.970975    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8664e04d-5336-401b-9a09-6b159a33f1d7-proxy-tls\") pod \"error-404-isvc-766f3-predictor-6b6855b58b-dmhf7\" (UID: \"8664e04d-5336-401b-9a09-6b159a33f1d7\") " pod="kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7"
Jul 01 17:29:48.205120 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:48.205070    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7"
Jul 01 17:29:48.333375 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:48.333231    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7"]
Jul 01 17:29:48.335518 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:29:48.335488    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8664e04d_5336_401b_9a09_6b159a33f1d7.slice/crio-3aa5085792830fb9f4539d3bbe215a8114c638e8a456333f7f35fae5fc4977da WatchSource:0}: Error finding container 3aa5085792830fb9f4539d3bbe215a8114c638e8a456333f7f35fae5fc4977da: Status 404 returned error can't find the container with id 3aa5085792830fb9f4539d3bbe215a8114c638e8a456333f7f35fae5fc4977da
Jul 01 17:29:49.052781 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:49.052755    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g"
Jul 01 17:29:49.184226 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:49.184197    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5cf9f9c9dc-n4nq6" podUID="241b85d7-2d84-4b8a-9362-09ffbaa76939" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:29:49.216397 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:49.216362    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7" event={"ID":"8664e04d-5336-401b-9a09-6b159a33f1d7","Type":"ContainerStarted","Data":"8aa476ebeb7b525cac913de165161469a50ae956c84df43b7903f29c0cb7b684"}
Jul 01 17:29:49.216547 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:49.216417    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7"
Jul 01 17:29:49.216547 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:49.216439    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7" event={"ID":"8664e04d-5336-401b-9a09-6b159a33f1d7","Type":"ContainerStarted","Data":"ae06692c478739eda6e871d6577b6c9262f8df2d593617978194df49926726c2"}
Jul 01 17:29:49.216683 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:49.216567    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7"
Jul 01 17:29:49.216683 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:49.216639    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7" event={"ID":"8664e04d-5336-401b-9a09-6b159a33f1d7","Type":"ContainerStarted","Data":"3aa5085792830fb9f4539d3bbe215a8114c638e8a456333f7f35fae5fc4977da"}
Jul 01 17:29:49.217828 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:49.217805    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7" podUID="8664e04d-5336-401b-9a09-6b159a33f1d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 17:29:49.234950 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:49.234911    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7" podStartSLOduration=2.234897614 podStartE2EDuration="2.234897614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:29:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:29:49.23380396 +0000 UTC m=+877.539002151" watchObservedRunningTime="2026-07-01 17:29:49.234897614 +0000 UTC m=+877.540095790"
Jul 01 17:29:50.220591 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:50.220536    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7" podUID="8664e04d-5336-401b-9a09-6b159a33f1d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 17:29:54.184039 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:54.184002    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5cf9f9c9dc-n4nq6" podUID="241b85d7-2d84-4b8a-9362-09ffbaa76939" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:29:55.225616 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:55.225570    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7"
Jul 01 17:29:55.226117 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:55.226088    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7" podUID="8664e04d-5336-401b-9a09-6b159a33f1d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 17:29:59.184477 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:59.184437    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5cf9f9c9dc-n4nq6" podUID="241b85d7-2d84-4b8a-9362-09ffbaa76939" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:29:59.184858 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:29:59.184532    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-5cf9f9c9dc-n4nq6"
Jul 01 17:30:01.241508 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:01.241475    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-6ac71-9d4758597-gcf7j"]
Jul 01 17:30:01.243699 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:01.243684    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-6ac71-9d4758597-gcf7j"
Jul 01 17:30:01.246347 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:01.246324    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-6ac71-serving-cert\""
Jul 01 17:30:01.246470 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:01.246356    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-6ac71-kube-rbac-proxy-sar-config\""
Jul 01 17:30:01.252493 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:01.252473    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-6ac71-9d4758597-gcf7j"]
Jul 01 17:30:01.374930 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:01.374900    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40577f87-46c7-483d-a615-428687ecf35b-proxy-tls\") pod \"switch-graph-6ac71-9d4758597-gcf7j\" (UID: \"40577f87-46c7-483d-a615-428687ecf35b\") " pod="kserve-ci-e2e-test/switch-graph-6ac71-9d4758597-gcf7j"
Jul 01 17:30:01.375088 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:01.374952    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40577f87-46c7-483d-a615-428687ecf35b-openshift-service-ca-bundle\") pod \"switch-graph-6ac71-9d4758597-gcf7j\" (UID: \"40577f87-46c7-483d-a615-428687ecf35b\") " pod="kserve-ci-e2e-test/switch-graph-6ac71-9d4758597-gcf7j"
Jul 01 17:30:01.476262 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:01.476221    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40577f87-46c7-483d-a615-428687ecf35b-proxy-tls\") pod \"switch-graph-6ac71-9d4758597-gcf7j\" (UID: \"40577f87-46c7-483d-a615-428687ecf35b\") " pod="kserve-ci-e2e-test/switch-graph-6ac71-9d4758597-gcf7j"
Jul 01 17:30:01.476447 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:01.476285    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40577f87-46c7-483d-a615-428687ecf35b-openshift-service-ca-bundle\") pod \"switch-graph-6ac71-9d4758597-gcf7j\" (UID: \"40577f87-46c7-483d-a615-428687ecf35b\") " pod="kserve-ci-e2e-test/switch-graph-6ac71-9d4758597-gcf7j"
Jul 01 17:30:01.476447 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:30:01.476364    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-6ac71-serving-cert: secret "switch-graph-6ac71-serving-cert" not found
Jul 01 17:30:01.476447 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:30:01.476432    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40577f87-46c7-483d-a615-428687ecf35b-proxy-tls podName:40577f87-46c7-483d-a615-428687ecf35b nodeName:}" failed. No retries permitted until 2026-07-01 17:30:01.976415984 +0000 UTC m=+890.281614139 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/40577f87-46c7-483d-a615-428687ecf35b-proxy-tls") pod "switch-graph-6ac71-9d4758597-gcf7j" (UID: "40577f87-46c7-483d-a615-428687ecf35b") : secret "switch-graph-6ac71-serving-cert" not found
Jul 01 17:30:01.476955 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:01.476932    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40577f87-46c7-483d-a615-428687ecf35b-openshift-service-ca-bundle\") pod \"switch-graph-6ac71-9d4758597-gcf7j\" (UID: \"40577f87-46c7-483d-a615-428687ecf35b\") " pod="kserve-ci-e2e-test/switch-graph-6ac71-9d4758597-gcf7j"
Jul 01 17:30:01.980241 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:01.980154    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40577f87-46c7-483d-a615-428687ecf35b-proxy-tls\") pod \"switch-graph-6ac71-9d4758597-gcf7j\" (UID: \"40577f87-46c7-483d-a615-428687ecf35b\") " pod="kserve-ci-e2e-test/switch-graph-6ac71-9d4758597-gcf7j"
Jul 01 17:30:01.982819 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:01.982793    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40577f87-46c7-483d-a615-428687ecf35b-proxy-tls\") pod \"switch-graph-6ac71-9d4758597-gcf7j\" (UID: \"40577f87-46c7-483d-a615-428687ecf35b\") " pod="kserve-ci-e2e-test/switch-graph-6ac71-9d4758597-gcf7j"
Jul 01 17:30:02.180857 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:02.180825    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-6ac71-9d4758597-gcf7j"
Jul 01 17:30:02.307386 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:02.307354    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-6ac71-9d4758597-gcf7j"]
Jul 01 17:30:02.310910 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:30:02.310880    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40577f87_46c7_483d_a615_428687ecf35b.slice/crio-375a4ada234ac5603698daa906c54228e61979e9355eb60401a96822d3b86a16 WatchSource:0}: Error finding container 375a4ada234ac5603698daa906c54228e61979e9355eb60401a96822d3b86a16: Status 404 returned error can't find the container with id 375a4ada234ac5603698daa906c54228e61979e9355eb60401a96822d3b86a16
Jul 01 17:30:03.281405 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:03.281370    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-6ac71-9d4758597-gcf7j" event={"ID":"40577f87-46c7-483d-a615-428687ecf35b","Type":"ContainerStarted","Data":"8059c76a6d9fc8040d8dbdf944c6dda57c49c6d3cc245bfdfc0f10db970afe2d"}
Jul 01 17:30:03.281631 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:03.281430    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-6ac71-9d4758597-gcf7j" event={"ID":"40577f87-46c7-483d-a615-428687ecf35b","Type":"ContainerStarted","Data":"375a4ada234ac5603698daa906c54228e61979e9355eb60401a96822d3b86a16"}
Jul 01 17:30:03.281631 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:03.281476    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-6ac71-9d4758597-gcf7j"
Jul 01 17:30:03.301402 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:03.301356    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-6ac71-9d4758597-gcf7j" podStartSLOduration=2.301340878 podStartE2EDuration="2.301340878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:30:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:30:03.300114866 +0000 UTC m=+891.605313055" watchObservedRunningTime="2026-07-01 17:30:03.301340878 +0000 UTC m=+891.606539055"
Jul 01 17:30:04.184497 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:04.184461    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5cf9f9c9dc-n4nq6" podUID="241b85d7-2d84-4b8a-9362-09ffbaa76939" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:30:05.226878 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:05.226841    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7" podUID="8664e04d-5336-401b-9a09-6b159a33f1d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 17:30:09.184081 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:09.184046    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5cf9f9c9dc-n4nq6" podUID="241b85d7-2d84-4b8a-9362-09ffbaa76939" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:30:09.291975 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:09.291942    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-6ac71-9d4758597-gcf7j"
Jul 01 17:30:12.161628 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:12.161599    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8tsw_500d2a90-2d21-42b3-9584-b04c14becaec/ovn-acl-logging/0.log"
Jul 01 17:30:12.162064 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:12.161673    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8tsw_500d2a90-2d21-42b3-9584-b04c14becaec/ovn-acl-logging/0.log"
Jul 01 17:30:14.184418 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:14.184374    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5cf9f9c9dc-n4nq6" podUID="241b85d7-2d84-4b8a-9362-09ffbaa76939" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:30:15.226441 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:15.226401    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7" podUID="8664e04d-5336-401b-9a09-6b159a33f1d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 17:30:17.093182 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:17.093154    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-5cf9f9c9dc-n4nq6"
Jul 01 17:30:17.195459 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:17.195419    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/241b85d7-2d84-4b8a-9362-09ffbaa76939-openshift-service-ca-bundle\") pod \"241b85d7-2d84-4b8a-9362-09ffbaa76939\" (UID: \"241b85d7-2d84-4b8a-9362-09ffbaa76939\") "
Jul 01 17:30:17.195682 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:17.195483    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/241b85d7-2d84-4b8a-9362-09ffbaa76939-proxy-tls\") pod \"241b85d7-2d84-4b8a-9362-09ffbaa76939\" (UID: \"241b85d7-2d84-4b8a-9362-09ffbaa76939\") "
Jul 01 17:30:17.195817 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:17.195790    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/241b85d7-2d84-4b8a-9362-09ffbaa76939-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "241b85d7-2d84-4b8a-9362-09ffbaa76939" (UID: "241b85d7-2d84-4b8a-9362-09ffbaa76939"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:30:17.197628 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:17.197598    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/241b85d7-2d84-4b8a-9362-09ffbaa76939-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "241b85d7-2d84-4b8a-9362-09ffbaa76939" (UID: "241b85d7-2d84-4b8a-9362-09ffbaa76939"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:30:17.296857 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:17.296766    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/241b85d7-2d84-4b8a-9362-09ffbaa76939-openshift-service-ca-bundle\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:30:17.296857 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:17.296799    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/241b85d7-2d84-4b8a-9362-09ffbaa76939-proxy-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:30:17.337148 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:17.337118    2585 generic.go:356] "Generic (PLEG): container finished" podID="241b85d7-2d84-4b8a-9362-09ffbaa76939" containerID="405bec264a6dcdd2b5bcce542a0a715e3cbb01318acf8bdfd948bd41a45a9efc" exitCode=0
Jul 01 17:30:17.337297 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:17.337164    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-5cf9f9c9dc-n4nq6"
Jul 01 17:30:17.337297 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:17.337213    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-5cf9f9c9dc-n4nq6" event={"ID":"241b85d7-2d84-4b8a-9362-09ffbaa76939","Type":"ContainerDied","Data":"405bec264a6dcdd2b5bcce542a0a715e3cbb01318acf8bdfd948bd41a45a9efc"}
Jul 01 17:30:17.337297 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:17.337264    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-5cf9f9c9dc-n4nq6" event={"ID":"241b85d7-2d84-4b8a-9362-09ffbaa76939","Type":"ContainerDied","Data":"ede0ce111cc6491eb9866f4c8caeb16d8c9d80e3ea065454a68a6671275d6936"}
Jul 01 17:30:17.337297 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:17.337285    2585 scope.go:117] "RemoveContainer" containerID="405bec264a6dcdd2b5bcce542a0a715e3cbb01318acf8bdfd948bd41a45a9efc"
Jul 01 17:30:17.345807 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:17.345789    2585 scope.go:117] "RemoveContainer" containerID="405bec264a6dcdd2b5bcce542a0a715e3cbb01318acf8bdfd948bd41a45a9efc"
Jul 01 17:30:17.346084 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:30:17.346067    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"405bec264a6dcdd2b5bcce542a0a715e3cbb01318acf8bdfd948bd41a45a9efc\": container with ID starting with 405bec264a6dcdd2b5bcce542a0a715e3cbb01318acf8bdfd948bd41a45a9efc not found: ID does not exist" containerID="405bec264a6dcdd2b5bcce542a0a715e3cbb01318acf8bdfd948bd41a45a9efc"
Jul 01 17:30:17.346139 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:17.346092    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"405bec264a6dcdd2b5bcce542a0a715e3cbb01318acf8bdfd948bd41a45a9efc"} err="failed to get container status \"405bec264a6dcdd2b5bcce542a0a715e3cbb01318acf8bdfd948bd41a45a9efc\": rpc error: code = NotFound desc = could not find container \"405bec264a6dcdd2b5bcce542a0a715e3cbb01318acf8bdfd948bd41a45a9efc\": container with ID starting with 405bec264a6dcdd2b5bcce542a0a715e3cbb01318acf8bdfd948bd41a45a9efc not found: ID does not exist"
Jul 01 17:30:17.357528 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:17.357508    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5cf9f9c9dc-n4nq6"]
Jul 01 17:30:17.360972 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:17.360953    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5cf9f9c9dc-n4nq6"]
Jul 01 17:30:18.172294 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:18.172257    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="241b85d7-2d84-4b8a-9362-09ffbaa76939" path="/var/lib/kubelet/pods/241b85d7-2d84-4b8a-9362-09ffbaa76939/volumes"
Jul 01 17:30:25.226176 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:25.226132    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7" podUID="8664e04d-5336-401b-9a09-6b159a33f1d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 17:30:35.227236 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:35.227199    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7"
Jul 01 17:30:47.217277 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:47.217246    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-766f3-f9786597b-xhzhn"]
Jul 01 17:30:47.219150 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:47.217534    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="241b85d7-2d84-4b8a-9362-09ffbaa76939" containerName="model-chainer"
Jul 01 17:30:47.219150 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:47.217546    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="241b85d7-2d84-4b8a-9362-09ffbaa76939" containerName="model-chainer"
Jul 01 17:30:47.219150 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:47.217635    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="241b85d7-2d84-4b8a-9362-09ffbaa76939" containerName="model-chainer"
Jul 01 17:30:47.219531 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:47.219516    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-766f3-f9786597b-xhzhn"
Jul 01 17:30:47.230303 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:47.230285    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-766f3-serving-cert\""
Jul 01 17:30:47.230724 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:47.230710    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-766f3-kube-rbac-proxy-sar-config\""
Jul 01 17:30:47.257311 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:47.257285    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-766f3-f9786597b-xhzhn"]
Jul 01 17:30:47.322138 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:47.322105    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe90fe7b-f8fd-4fcb-8962-f9a22544d016-proxy-tls\") pod \"sequence-graph-766f3-f9786597b-xhzhn\" (UID: \"fe90fe7b-f8fd-4fcb-8962-f9a22544d016\") " pod="kserve-ci-e2e-test/sequence-graph-766f3-f9786597b-xhzhn"
Jul 01 17:30:47.322316 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:47.322158    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe90fe7b-f8fd-4fcb-8962-f9a22544d016-openshift-service-ca-bundle\") pod \"sequence-graph-766f3-f9786597b-xhzhn\" (UID: \"fe90fe7b-f8fd-4fcb-8962-f9a22544d016\") " pod="kserve-ci-e2e-test/sequence-graph-766f3-f9786597b-xhzhn"
Jul 01 17:30:47.422875 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:47.422838    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe90fe7b-f8fd-4fcb-8962-f9a22544d016-openshift-service-ca-bundle\") pod \"sequence-graph-766f3-f9786597b-xhzhn\" (UID: \"fe90fe7b-f8fd-4fcb-8962-f9a22544d016\") " pod="kserve-ci-e2e-test/sequence-graph-766f3-f9786597b-xhzhn"
Jul 01 17:30:47.423039 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:47.422919    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe90fe7b-f8fd-4fcb-8962-f9a22544d016-proxy-tls\") pod \"sequence-graph-766f3-f9786597b-xhzhn\" (UID: \"fe90fe7b-f8fd-4fcb-8962-f9a22544d016\") " pod="kserve-ci-e2e-test/sequence-graph-766f3-f9786597b-xhzhn"
Jul 01 17:30:47.423039 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:30:47.423013    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-766f3-serving-cert: secret "sequence-graph-766f3-serving-cert" not found
Jul 01 17:30:47.423108 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:30:47.423088    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe90fe7b-f8fd-4fcb-8962-f9a22544d016-proxy-tls podName:fe90fe7b-f8fd-4fcb-8962-f9a22544d016 nodeName:}" failed. No retries permitted until 2026-07-01 17:30:47.923067565 +0000 UTC m=+936.228265733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fe90fe7b-f8fd-4fcb-8962-f9a22544d016-proxy-tls") pod "sequence-graph-766f3-f9786597b-xhzhn" (UID: "fe90fe7b-f8fd-4fcb-8962-f9a22544d016") : secret "sequence-graph-766f3-serving-cert" not found
Jul 01 17:30:47.423612 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:47.423596    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe90fe7b-f8fd-4fcb-8962-f9a22544d016-openshift-service-ca-bundle\") pod \"sequence-graph-766f3-f9786597b-xhzhn\" (UID: \"fe90fe7b-f8fd-4fcb-8962-f9a22544d016\") " pod="kserve-ci-e2e-test/sequence-graph-766f3-f9786597b-xhzhn"
Jul 01 17:30:47.927654 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:47.927613    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe90fe7b-f8fd-4fcb-8962-f9a22544d016-proxy-tls\") pod \"sequence-graph-766f3-f9786597b-xhzhn\" (UID: \"fe90fe7b-f8fd-4fcb-8962-f9a22544d016\") " pod="kserve-ci-e2e-test/sequence-graph-766f3-f9786597b-xhzhn"
Jul 01 17:30:47.930136 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:47.930105    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe90fe7b-f8fd-4fcb-8962-f9a22544d016-proxy-tls\") pod \"sequence-graph-766f3-f9786597b-xhzhn\" (UID: \"fe90fe7b-f8fd-4fcb-8962-f9a22544d016\") " pod="kserve-ci-e2e-test/sequence-graph-766f3-f9786597b-xhzhn"
Jul 01 17:30:48.128974 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:48.128940    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-766f3-f9786597b-xhzhn"
Jul 01 17:30:48.256356 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:48.256326    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-766f3-f9786597b-xhzhn"]
Jul 01 17:30:48.259264 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:30:48.259236    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe90fe7b_f8fd_4fcb_8962_f9a22544d016.slice/crio-dfcdc4aec08b126bbc16f24360585dcb03898b6adbaa7be9c63842a3e2854d09 WatchSource:0}: Error finding container dfcdc4aec08b126bbc16f24360585dcb03898b6adbaa7be9c63842a3e2854d09: Status 404 returned error can't find the container with id dfcdc4aec08b126bbc16f24360585dcb03898b6adbaa7be9c63842a3e2854d09
Jul 01 17:30:48.470126 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:48.470040    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-766f3-f9786597b-xhzhn" event={"ID":"fe90fe7b-f8fd-4fcb-8962-f9a22544d016","Type":"ContainerStarted","Data":"27a4c23ced9581cf29ac473ab6a72713ac17564af4c997aa3f5325f4b6584094"}
Jul 01 17:30:48.470126 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:48.470092    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-766f3-f9786597b-xhzhn" event={"ID":"fe90fe7b-f8fd-4fcb-8962-f9a22544d016","Type":"ContainerStarted","Data":"dfcdc4aec08b126bbc16f24360585dcb03898b6adbaa7be9c63842a3e2854d09"}
Jul 01 17:30:48.470305 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:48.470131    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-766f3-f9786597b-xhzhn"
Jul 01 17:30:48.487702 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:48.487640    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-766f3-f9786597b-xhzhn" podStartSLOduration=1.4876220199999999 podStartE2EDuration="1.48762202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:30:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:30:48.486831084 +0000 UTC m=+936.792029260" watchObservedRunningTime="2026-07-01 17:30:48.48762202 +0000 UTC m=+936.792820197"
Jul 01 17:30:54.480656 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:30:54.480626    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-766f3-f9786597b-xhzhn"
Jul 01 17:35:12.189735 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:35:12.189703    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8tsw_500d2a90-2d21-42b3-9584-b04c14becaec/ovn-acl-logging/0.log"
Jul 01 17:35:12.192543 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:35:12.192522    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8tsw_500d2a90-2d21-42b3-9584-b04c14becaec/ovn-acl-logging/0.log"
Jul 01 17:38:16.025398 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:16.025366    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-6ac71-9d4758597-gcf7j"]
Jul 01 17:38:16.026894 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:16.025592    2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-6ac71-9d4758597-gcf7j" podUID="40577f87-46c7-483d-a615-428687ecf35b" containerName="switch-graph-6ac71" containerID="cri-o://8059c76a6d9fc8040d8dbdf944c6dda57c49c6d3cc245bfdfc0f10db970afe2d" gracePeriod=30
Jul 01 17:38:16.367438 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:16.367403    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g"]
Jul 01 17:38:16.367722 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:16.367672    2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g" podUID="51c0df56-15c4-4eaa-988f-1c5b154853af" containerName="kserve-container" containerID="cri-o://44a3cc08a02720c3590e95aae55233ce77776a1f3989d3c1cae806d8c37352b1" gracePeriod=30
Jul 01 17:38:16.367842 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:16.367701    2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g" podUID="51c0df56-15c4-4eaa-988f-1c5b154853af" containerName="kube-rbac-proxy" containerID="cri-o://26d79261a6a3f208b5d25fafeacda6ee36676db2e6af5f35104a7fef951e9370" gracePeriod=30
Jul 01 17:38:16.431066 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:16.431032    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q"]
Jul 01 17:38:16.433440 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:16.433424    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q"
Jul 01 17:38:16.436634 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:16.436616    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-43ae3-predictor-serving-cert\""
Jul 01 17:38:16.437016 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:16.436998    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-43ae3-kube-rbac-proxy-sar-config\""
Jul 01 17:38:16.447846 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:16.447825    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q"]
Jul 01 17:38:16.513983 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:16.513954    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-43ae3-predictor-669cb6b886-wfs2q\" (UID: \"f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67\") " pod="kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q"
Jul 01 17:38:16.514094 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:16.513987    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67-proxy-tls\") pod \"error-404-isvc-43ae3-predictor-669cb6b886-wfs2q\" (UID: \"f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67\") " pod="kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q"
Jul 01 17:38:16.514094 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:16.514052    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqrnv\" (UniqueName: \"kubernetes.io/projected/f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67-kube-api-access-lqrnv\") pod \"error-404-isvc-43ae3-predictor-669cb6b886-wfs2q\" (UID: \"f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67\") " pod="kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q"
Jul 01 17:38:16.614502 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:16.614472    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-43ae3-predictor-669cb6b886-wfs2q\" (UID: \"f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67\") " pod="kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q"
Jul 01 17:38:16.614708 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:16.614516    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67-proxy-tls\") pod \"error-404-isvc-43ae3-predictor-669cb6b886-wfs2q\" (UID: \"f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67\") " pod="kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q"
Jul 01 17:38:16.614708 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:16.614571    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqrnv\" (UniqueName: \"kubernetes.io/projected/f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67-kube-api-access-lqrnv\") pod \"error-404-isvc-43ae3-predictor-669cb6b886-wfs2q\" (UID: \"f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67\") " pod="kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q"
Jul 01 17:38:16.614820 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:38:16.614719    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-serving-cert: secret "error-404-isvc-43ae3-predictor-serving-cert" not found
Jul 01 17:38:16.614820 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:38:16.614805    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67-proxy-tls podName:f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67 nodeName:}" failed. No retries permitted until 2026-07-01 17:38:17.114787776 +0000 UTC m=+1385.419985931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67-proxy-tls") pod "error-404-isvc-43ae3-predictor-669cb6b886-wfs2q" (UID: "f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67") : secret "error-404-isvc-43ae3-predictor-serving-cert" not found
Jul 01 17:38:16.615182 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:16.615162    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-43ae3-predictor-669cb6b886-wfs2q\" (UID: \"f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67\") " pod="kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q"
Jul 01 17:38:16.623107 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:16.623053    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqrnv\" (UniqueName: \"kubernetes.io/projected/f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67-kube-api-access-lqrnv\") pod \"error-404-isvc-43ae3-predictor-669cb6b886-wfs2q\" (UID: \"f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67\") " pod="kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q"
Jul 01 17:38:17.118317 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:17.118287    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67-proxy-tls\") pod \"error-404-isvc-43ae3-predictor-669cb6b886-wfs2q\" (UID: \"f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67\") " pod="kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q"
Jul 01 17:38:17.120795 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:17.120769    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67-proxy-tls\") pod \"error-404-isvc-43ae3-predictor-669cb6b886-wfs2q\" (UID: \"f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67\") " pod="kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q"
Jul 01 17:38:17.259321 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:17.259288    2585 generic.go:356] "Generic (PLEG): container finished" podID="51c0df56-15c4-4eaa-988f-1c5b154853af" containerID="26d79261a6a3f208b5d25fafeacda6ee36676db2e6af5f35104a7fef951e9370" exitCode=2
Jul 01 17:38:17.259321 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:17.259325    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g" event={"ID":"51c0df56-15c4-4eaa-988f-1c5b154853af","Type":"ContainerDied","Data":"26d79261a6a3f208b5d25fafeacda6ee36676db2e6af5f35104a7fef951e9370"}
Jul 01 17:38:17.343711 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:17.343673    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q"
Jul 01 17:38:17.475012 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:17.474929    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q"]
Jul 01 17:38:17.477219 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:38:17.477191    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e53a5f_7554_4eff_a0d1_b3ef8bf10b67.slice/crio-ca63db91c7964efc5d4d3e6b43efe6bfcd449c0311a2886707e8ded11f8cf49a WatchSource:0}: Error finding container ca63db91c7964efc5d4d3e6b43efe6bfcd449c0311a2886707e8ded11f8cf49a: Status 404 returned error can't find the container with id ca63db91c7964efc5d4d3e6b43efe6bfcd449c0311a2886707e8ded11f8cf49a
Jul 01 17:38:18.265958 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:18.265913    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q" event={"ID":"f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67","Type":"ContainerStarted","Data":"7519e09bb777aa377229f63e12683271f0edf3e1f81f66ee73328621cbf5ece2"}
Jul 01 17:38:18.266345 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:18.265977    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q" event={"ID":"f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67","Type":"ContainerStarted","Data":"a41b5681f04d4c5b9a7521503fc04ec5ca3f25168a25985cf397a856cde6064e"}
Jul 01 17:38:18.266345 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:18.266002    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q" event={"ID":"f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67","Type":"ContainerStarted","Data":"ca63db91c7964efc5d4d3e6b43efe6bfcd449c0311a2886707e8ded11f8cf49a"}
Jul 01 17:38:18.266345 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:18.266060    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q"
Jul 01 17:38:18.285270 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:18.285220    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q" podStartSLOduration=2.285204911 podStartE2EDuration="2.285204911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:38:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:38:18.283654271 +0000 UTC m=+1386.588852448" watchObservedRunningTime="2026-07-01 17:38:18.285204911 +0000 UTC m=+1386.590403087"
Jul 01 17:38:19.047775 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:19.047736    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g" podUID="51c0df56-15c4-4eaa-988f-1c5b154853af" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Jul 01 17:38:19.052279 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:19.052247    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g" podUID="51c0df56-15c4-4eaa-988f-1c5b154853af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 01 17:38:19.270535 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:19.270503    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q"
Jul 01 17:38:19.271931 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:19.271903    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q" podUID="f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 17:38:19.289422 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:19.289386    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-6ac71-9d4758597-gcf7j" podUID="40577f87-46c7-483d-a615-428687ecf35b" containerName="switch-graph-6ac71" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:38:19.518924 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:19.518891    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g"
Jul 01 17:38:19.536314 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:19.536267    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcn5g\" (UniqueName: \"kubernetes.io/projected/51c0df56-15c4-4eaa-988f-1c5b154853af-kube-api-access-jcn5g\") pod \"51c0df56-15c4-4eaa-988f-1c5b154853af\" (UID: \"51c0df56-15c4-4eaa-988f-1c5b154853af\") "
Jul 01 17:38:19.536505 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:19.536373    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51c0df56-15c4-4eaa-988f-1c5b154853af-proxy-tls\") pod \"51c0df56-15c4-4eaa-988f-1c5b154853af\" (UID: \"51c0df56-15c4-4eaa-988f-1c5b154853af\") "
Jul 01 17:38:19.536505 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:19.536506    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51c0df56-15c4-4eaa-988f-1c5b154853af-kube-rbac-proxy-sar-config\") pod \"51c0df56-15c4-4eaa-988f-1c5b154853af\" (UID: \"51c0df56-15c4-4eaa-988f-1c5b154853af\") "
Jul 01 17:38:19.536925 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:19.536896    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51c0df56-15c4-4eaa-988f-1c5b154853af-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "51c0df56-15c4-4eaa-988f-1c5b154853af" (UID: "51c0df56-15c4-4eaa-988f-1c5b154853af"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:38:19.538979 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:19.538947    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51c0df56-15c4-4eaa-988f-1c5b154853af-kube-api-access-jcn5g" (OuterVolumeSpecName: "kube-api-access-jcn5g") pod "51c0df56-15c4-4eaa-988f-1c5b154853af" (UID: "51c0df56-15c4-4eaa-988f-1c5b154853af"). InnerVolumeSpecName "kube-api-access-jcn5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:38:19.539088 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:19.539017    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51c0df56-15c4-4eaa-988f-1c5b154853af-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "51c0df56-15c4-4eaa-988f-1c5b154853af" (UID: "51c0df56-15c4-4eaa-988f-1c5b154853af"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:38:19.637730 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:19.637648    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcn5g\" (UniqueName: \"kubernetes.io/projected/51c0df56-15c4-4eaa-988f-1c5b154853af-kube-api-access-jcn5g\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:38:19.637730 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:19.637673    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51c0df56-15c4-4eaa-988f-1c5b154853af-proxy-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:38:19.637730 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:19.637682    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51c0df56-15c4-4eaa-988f-1c5b154853af-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:38:20.276269 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:20.276228    2585 generic.go:356] "Generic (PLEG): container finished" podID="51c0df56-15c4-4eaa-988f-1c5b154853af" containerID="44a3cc08a02720c3590e95aae55233ce77776a1f3989d3c1cae806d8c37352b1" exitCode=0
Jul 01 17:38:20.276656 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:20.276294    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g"
Jul 01 17:38:20.276656 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:20.276309    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g" event={"ID":"51c0df56-15c4-4eaa-988f-1c5b154853af","Type":"ContainerDied","Data":"44a3cc08a02720c3590e95aae55233ce77776a1f3989d3c1cae806d8c37352b1"}
Jul 01 17:38:20.276656 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:20.276370    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g" event={"ID":"51c0df56-15c4-4eaa-988f-1c5b154853af","Type":"ContainerDied","Data":"1940912da05bff3729b02213c1929adadffe6f9158da59bffe148cfb303e81a6"}
Jul 01 17:38:20.276656 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:20.276394    2585 scope.go:117] "RemoveContainer" containerID="26d79261a6a3f208b5d25fafeacda6ee36676db2e6af5f35104a7fef951e9370"
Jul 01 17:38:20.277005 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:20.276971    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q" podUID="f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 17:38:20.284390 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:20.284370    2585 scope.go:117] "RemoveContainer" containerID="44a3cc08a02720c3590e95aae55233ce77776a1f3989d3c1cae806d8c37352b1"
Jul 01 17:38:20.291246 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:20.291231    2585 scope.go:117] "RemoveContainer" containerID="26d79261a6a3f208b5d25fafeacda6ee36676db2e6af5f35104a7fef951e9370"
Jul 01 17:38:20.291538 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:38:20.291518    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26d79261a6a3f208b5d25fafeacda6ee36676db2e6af5f35104a7fef951e9370\": container with ID starting with 26d79261a6a3f208b5d25fafeacda6ee36676db2e6af5f35104a7fef951e9370 not found: ID does not exist" containerID="26d79261a6a3f208b5d25fafeacda6ee36676db2e6af5f35104a7fef951e9370"
Jul 01 17:38:20.291734 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:20.291548    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26d79261a6a3f208b5d25fafeacda6ee36676db2e6af5f35104a7fef951e9370"} err="failed to get container status \"26d79261a6a3f208b5d25fafeacda6ee36676db2e6af5f35104a7fef951e9370\": rpc error: code = NotFound desc = could not find container \"26d79261a6a3f208b5d25fafeacda6ee36676db2e6af5f35104a7fef951e9370\": container with ID starting with 26d79261a6a3f208b5d25fafeacda6ee36676db2e6af5f35104a7fef951e9370 not found: ID does not exist"
Jul 01 17:38:20.291734 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:20.291565    2585 scope.go:117] "RemoveContainer" containerID="44a3cc08a02720c3590e95aae55233ce77776a1f3989d3c1cae806d8c37352b1"
Jul 01 17:38:20.291861 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:38:20.291836    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44a3cc08a02720c3590e95aae55233ce77776a1f3989d3c1cae806d8c37352b1\": container with ID starting with 44a3cc08a02720c3590e95aae55233ce77776a1f3989d3c1cae806d8c37352b1 not found: ID does not exist" containerID="44a3cc08a02720c3590e95aae55233ce77776a1f3989d3c1cae806d8c37352b1"
Jul 01 17:38:20.291914 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:20.291871    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44a3cc08a02720c3590e95aae55233ce77776a1f3989d3c1cae806d8c37352b1"} err="failed to get container status \"44a3cc08a02720c3590e95aae55233ce77776a1f3989d3c1cae806d8c37352b1\": rpc error: code = NotFound desc = could not find container \"44a3cc08a02720c3590e95aae55233ce77776a1f3989d3c1cae806d8c37352b1\": container with ID starting with 44a3cc08a02720c3590e95aae55233ce77776a1f3989d3c1cae806d8c37352b1 not found: ID does not exist"
Jul 01 17:38:20.294222 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:20.294203    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g"]
Jul 01 17:38:20.297482 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:20.297465    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6ac71-predictor-779d48fcd6-2lz9g"]
Jul 01 17:38:22.172731 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:22.172701    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51c0df56-15c4-4eaa-988f-1c5b154853af" path="/var/lib/kubelet/pods/51c0df56-15c4-4eaa-988f-1c5b154853af/volumes"
Jul 01 17:38:24.290135 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:24.290086    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-6ac71-9d4758597-gcf7j" podUID="40577f87-46c7-483d-a615-428687ecf35b" containerName="switch-graph-6ac71" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:38:25.281735 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:25.281709    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q"
Jul 01 17:38:25.282167 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:25.282140    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q" podUID="f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 17:38:29.289395 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:29.289356    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-6ac71-9d4758597-gcf7j" podUID="40577f87-46c7-483d-a615-428687ecf35b" containerName="switch-graph-6ac71" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:38:29.289876 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:29.289450    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-6ac71-9d4758597-gcf7j"
Jul 01 17:38:34.290347 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:34.290297    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-6ac71-9d4758597-gcf7j" podUID="40577f87-46c7-483d-a615-428687ecf35b" containerName="switch-graph-6ac71" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:38:35.282213 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:35.282170    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q" podUID="f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 17:38:39.290264 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:39.290221    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-6ac71-9d4758597-gcf7j" podUID="40577f87-46c7-483d-a615-428687ecf35b" containerName="switch-graph-6ac71" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:38:44.289919 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:44.289865    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-6ac71-9d4758597-gcf7j" podUID="40577f87-46c7-483d-a615-428687ecf35b" containerName="switch-graph-6ac71" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:38:45.283085 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:45.283047    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q" podUID="f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 17:38:46.165961 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:46.165939    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-6ac71-9d4758597-gcf7j"
Jul 01 17:38:46.245391 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:46.245358    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40577f87-46c7-483d-a615-428687ecf35b-proxy-tls\") pod \"40577f87-46c7-483d-a615-428687ecf35b\" (UID: \"40577f87-46c7-483d-a615-428687ecf35b\") "
Jul 01 17:38:46.245540 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:46.245445    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40577f87-46c7-483d-a615-428687ecf35b-openshift-service-ca-bundle\") pod \"40577f87-46c7-483d-a615-428687ecf35b\" (UID: \"40577f87-46c7-483d-a615-428687ecf35b\") "
Jul 01 17:38:46.245775 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:46.245751    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40577f87-46c7-483d-a615-428687ecf35b-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "40577f87-46c7-483d-a615-428687ecf35b" (UID: "40577f87-46c7-483d-a615-428687ecf35b"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:38:46.247630 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:46.247602    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40577f87-46c7-483d-a615-428687ecf35b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "40577f87-46c7-483d-a615-428687ecf35b" (UID: "40577f87-46c7-483d-a615-428687ecf35b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:38:46.346835 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:46.346805    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40577f87-46c7-483d-a615-428687ecf35b-openshift-service-ca-bundle\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:38:46.346835 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:46.346833    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40577f87-46c7-483d-a615-428687ecf35b-proxy-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:38:46.382998 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:46.382964    2585 generic.go:356] "Generic (PLEG): container finished" podID="40577f87-46c7-483d-a615-428687ecf35b" containerID="8059c76a6d9fc8040d8dbdf944c6dda57c49c6d3cc245bfdfc0f10db970afe2d" exitCode=0
Jul 01 17:38:46.383112 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:46.383011    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-6ac71-9d4758597-gcf7j"
Jul 01 17:38:46.383112 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:46.383047    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-6ac71-9d4758597-gcf7j" event={"ID":"40577f87-46c7-483d-a615-428687ecf35b","Type":"ContainerDied","Data":"8059c76a6d9fc8040d8dbdf944c6dda57c49c6d3cc245bfdfc0f10db970afe2d"}
Jul 01 17:38:46.383112 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:46.383102    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-6ac71-9d4758597-gcf7j" event={"ID":"40577f87-46c7-483d-a615-428687ecf35b","Type":"ContainerDied","Data":"375a4ada234ac5603698daa906c54228e61979e9355eb60401a96822d3b86a16"}
Jul 01 17:38:46.383209 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:46.383126    2585 scope.go:117] "RemoveContainer" containerID="8059c76a6d9fc8040d8dbdf944c6dda57c49c6d3cc245bfdfc0f10db970afe2d"
Jul 01 17:38:46.391725 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:46.391692    2585 scope.go:117] "RemoveContainer" containerID="8059c76a6d9fc8040d8dbdf944c6dda57c49c6d3cc245bfdfc0f10db970afe2d"
Jul 01 17:38:46.391950 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:38:46.391934    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8059c76a6d9fc8040d8dbdf944c6dda57c49c6d3cc245bfdfc0f10db970afe2d\": container with ID starting with 8059c76a6d9fc8040d8dbdf944c6dda57c49c6d3cc245bfdfc0f10db970afe2d not found: ID does not exist" containerID="8059c76a6d9fc8040d8dbdf944c6dda57c49c6d3cc245bfdfc0f10db970afe2d"
Jul 01 17:38:46.391986 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:46.391972    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8059c76a6d9fc8040d8dbdf944c6dda57c49c6d3cc245bfdfc0f10db970afe2d"} err="failed to get container status \"8059c76a6d9fc8040d8dbdf944c6dda57c49c6d3cc245bfdfc0f10db970afe2d\": rpc error: code = NotFound desc = could not find container \"8059c76a6d9fc8040d8dbdf944c6dda57c49c6d3cc245bfdfc0f10db970afe2d\": container with ID starting with 8059c76a6d9fc8040d8dbdf944c6dda57c49c6d3cc245bfdfc0f10db970afe2d not found: ID does not exist"
Jul 01 17:38:46.403401 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:46.403380    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-6ac71-9d4758597-gcf7j"]
Jul 01 17:38:46.406867 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:46.406844    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-6ac71-9d4758597-gcf7j"]
Jul 01 17:38:48.171797 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:48.171763    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40577f87-46c7-483d-a615-428687ecf35b" path="/var/lib/kubelet/pods/40577f87-46c7-483d-a615-428687ecf35b/volumes"
Jul 01 17:38:55.282679 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:38:55.282635    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q" podUID="f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 17:39:01.808100 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:01.808068    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-766f3-f9786597b-xhzhn"]
Jul 01 17:39:01.808463 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:01.808284    2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-766f3-f9786597b-xhzhn" podUID="fe90fe7b-f8fd-4fcb-8962-f9a22544d016" containerName="sequence-graph-766f3" containerID="cri-o://27a4c23ced9581cf29ac473ab6a72713ac17564af4c997aa3f5325f4b6584094" gracePeriod=30
Jul 01 17:39:02.074220 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:02.074119    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7"]
Jul 01 17:39:02.074383 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:02.074361    2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7" podUID="8664e04d-5336-401b-9a09-6b159a33f1d7" containerName="kserve-container" containerID="cri-o://ae06692c478739eda6e871d6577b6c9262f8df2d593617978194df49926726c2" gracePeriod=30
Jul 01 17:39:02.074487 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:02.074384    2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7" podUID="8664e04d-5336-401b-9a09-6b159a33f1d7" containerName="kube-rbac-proxy" containerID="cri-o://8aa476ebeb7b525cac913de165161469a50ae956c84df43b7903f29c0cb7b684" gracePeriod=30
Jul 01 17:39:02.080607 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:02.078616    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp"]
Jul 01 17:39:02.080607 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:02.079261    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51c0df56-15c4-4eaa-988f-1c5b154853af" containerName="kube-rbac-proxy"
Jul 01 17:39:02.080607 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:02.079278    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="51c0df56-15c4-4eaa-988f-1c5b154853af" containerName="kube-rbac-proxy"
Jul 01 17:39:02.080607 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:02.079327    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40577f87-46c7-483d-a615-428687ecf35b" containerName="switch-graph-6ac71"
Jul 01 17:39:02.080607 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:02.079336    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="40577f87-46c7-483d-a615-428687ecf35b" containerName="switch-graph-6ac71"
Jul 01 17:39:02.080607 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:02.079369    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51c0df56-15c4-4eaa-988f-1c5b154853af" containerName="kserve-container"
Jul 01 17:39:02.080607 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:02.079378    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="51c0df56-15c4-4eaa-988f-1c5b154853af" containerName="kserve-container"
Jul 01 17:39:02.080607 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:02.079513    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="51c0df56-15c4-4eaa-988f-1c5b154853af" containerName="kube-rbac-proxy"
Jul 01 17:39:02.080607 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:02.079532    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="51c0df56-15c4-4eaa-988f-1c5b154853af" containerName="kserve-container"
Jul 01 17:39:02.080607 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:02.079542    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="40577f87-46c7-483d-a615-428687ecf35b" containerName="switch-graph-6ac71"
Jul 01 17:39:02.084520 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:02.084500    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp"
Jul 01 17:39:02.087335 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:02.087313    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-98f1c-kube-rbac-proxy-sar-config\""
Jul 01 17:39:02.087443 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:02.087383    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-98f1c-predictor-serving-cert\""
Jul 01 17:39:02.091463 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:02.091443    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp"]
Jul 01 17:39:02.165406 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:02.165379    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92a915eb-bc3c-4e50-8b6b-48f7de29d84f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp\" (UID: \"92a915eb-bc3c-4e50-8b6b-48f7de29d84f\") " pod="kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp"
Jul 01 17:39:02.165540 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:02.165412    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92a915eb-bc3c-4e50-8b6b-48f7de29d84f-proxy-tls\") pod \"error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp\" (UID: \"92a915eb-bc3c-4e50-8b6b-48f7de29d84f\") " pod="kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp"
Jul 01 17:39:02.165540 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:02.165443    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzp7k\" (UniqueName: \"kubernetes.io/projected/92a915eb-bc3c-4e50-8b6b-48f7de29d84f-kube-api-access-lzp7k\") pod \"error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp\" (UID: \"92a915eb-bc3c-4e50-8b6b-48f7de29d84f\") " pod="kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp"
Jul 01 17:39:02.266727 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:02.266696    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92a915eb-bc3c-4e50-8b6b-48f7de29d84f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp\" (UID: \"92a915eb-bc3c-4e50-8b6b-48f7de29d84f\") " pod="kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp"
Jul 01 17:39:02.266727 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:02.266729    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92a915eb-bc3c-4e50-8b6b-48f7de29d84f-proxy-tls\") pod \"error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp\" (UID: \"92a915eb-bc3c-4e50-8b6b-48f7de29d84f\") " pod="kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp"
Jul 01 17:39:02.266901 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:02.266755    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzp7k\" (UniqueName: \"kubernetes.io/projected/92a915eb-bc3c-4e50-8b6b-48f7de29d84f-kube-api-access-lzp7k\") pod \"error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp\" (UID: \"92a915eb-bc3c-4e50-8b6b-48f7de29d84f\") " pod="kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp"
Jul 01 17:39:02.266901 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:39:02.266852    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-serving-cert: secret "error-404-isvc-98f1c-predictor-serving-cert" not found
Jul 01 17:39:02.266988 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:39:02.266916    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92a915eb-bc3c-4e50-8b6b-48f7de29d84f-proxy-tls podName:92a915eb-bc3c-4e50-8b6b-48f7de29d84f nodeName:}" failed. No retries permitted until 2026-07-01 17:39:02.766897515 +0000 UTC m=+1431.072095671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/92a915eb-bc3c-4e50-8b6b-48f7de29d84f-proxy-tls") pod "error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp" (UID: "92a915eb-bc3c-4e50-8b6b-48f7de29d84f") : secret "error-404-isvc-98f1c-predictor-serving-cert" not found
Jul 01 17:39:02.267287 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:02.267269    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92a915eb-bc3c-4e50-8b6b-48f7de29d84f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp\" (UID: \"92a915eb-bc3c-4e50-8b6b-48f7de29d84f\") " pod="kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp"
Jul 01 17:39:02.275939 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:02.275907    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzp7k\" (UniqueName: \"kubernetes.io/projected/92a915eb-bc3c-4e50-8b6b-48f7de29d84f-kube-api-access-lzp7k\") pod \"error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp\" (UID: \"92a915eb-bc3c-4e50-8b6b-48f7de29d84f\") " pod="kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp"
Jul 01 17:39:02.448135 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:02.448055    2585 generic.go:356] "Generic (PLEG): container finished" podID="8664e04d-5336-401b-9a09-6b159a33f1d7" containerID="8aa476ebeb7b525cac913de165161469a50ae956c84df43b7903f29c0cb7b684" exitCode=2
Jul 01 17:39:02.448135 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:02.448090    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7" event={"ID":"8664e04d-5336-401b-9a09-6b159a33f1d7","Type":"ContainerDied","Data":"8aa476ebeb7b525cac913de165161469a50ae956c84df43b7903f29c0cb7b684"}
Jul 01 17:39:02.770669 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:02.770562    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92a915eb-bc3c-4e50-8b6b-48f7de29d84f-proxy-tls\") pod \"error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp\" (UID: \"92a915eb-bc3c-4e50-8b6b-48f7de29d84f\") " pod="kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp"
Jul 01 17:39:02.773229 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:02.773207    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92a915eb-bc3c-4e50-8b6b-48f7de29d84f-proxy-tls\") pod \"error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp\" (UID: \"92a915eb-bc3c-4e50-8b6b-48f7de29d84f\") " pod="kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp"
Jul 01 17:39:03.001355 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:03.001324    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp"
Jul 01 17:39:03.128767 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:03.128727    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp"]
Jul 01 17:39:03.132027 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:39:03.132002    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a915eb_bc3c_4e50_8b6b_48f7de29d84f.slice/crio-714f29dfd13b0d4b836d2692a4d6939e5d9ac904a9408e136cbcd8e31eb0e37c WatchSource:0}: Error finding container 714f29dfd13b0d4b836d2692a4d6939e5d9ac904a9408e136cbcd8e31eb0e37c: Status 404 returned error can't find the container with id 714f29dfd13b0d4b836d2692a4d6939e5d9ac904a9408e136cbcd8e31eb0e37c
Jul 01 17:39:03.455078 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:03.455043    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp" event={"ID":"92a915eb-bc3c-4e50-8b6b-48f7de29d84f","Type":"ContainerStarted","Data":"4afd9d8aba9a4b1282dea1e9005dee241cfbb0d0bcbe9a9313ab850ae9ab5172"}
Jul 01 17:39:03.455256 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:03.455110    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp" event={"ID":"92a915eb-bc3c-4e50-8b6b-48f7de29d84f","Type":"ContainerStarted","Data":"1021fba1e78c9ea3572cd246e48ee4a1e6001d7e6a90473c0e5927508416949f"}
Jul 01 17:39:03.455256 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:03.455146    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp"
Jul 01 17:39:03.455256 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:03.455172    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp" event={"ID":"92a915eb-bc3c-4e50-8b6b-48f7de29d84f","Type":"ContainerStarted","Data":"714f29dfd13b0d4b836d2692a4d6939e5d9ac904a9408e136cbcd8e31eb0e37c"}
Jul 01 17:39:03.475301 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:03.475255    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp" podStartSLOduration=1.4752398310000001 podStartE2EDuration="1.475239831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:39:03.473723931 +0000 UTC m=+1431.778922111" watchObservedRunningTime="2026-07-01 17:39:03.475239831 +0000 UTC m=+1431.780438007"
Jul 01 17:39:04.459895 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:04.459860    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp"
Jul 01 17:39:04.461303 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:04.461274    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp" podUID="92a915eb-bc3c-4e50-8b6b-48f7de29d84f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 17:39:04.477769 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:04.477740    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-766f3-f9786597b-xhzhn" podUID="fe90fe7b-f8fd-4fcb-8962-f9a22544d016" containerName="sequence-graph-766f3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:39:05.131337 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:05.131313    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7"
Jul 01 17:39:05.283216 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:05.283147    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q"
Jul 01 17:39:05.289811 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:05.289791    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8664e04d-5336-401b-9a09-6b159a33f1d7-kube-rbac-proxy-sar-config\") pod \"8664e04d-5336-401b-9a09-6b159a33f1d7\" (UID: \"8664e04d-5336-401b-9a09-6b159a33f1d7\") "
Jul 01 17:39:05.289903 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:05.289858    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2wx5\" (UniqueName: \"kubernetes.io/projected/8664e04d-5336-401b-9a09-6b159a33f1d7-kube-api-access-z2wx5\") pod \"8664e04d-5336-401b-9a09-6b159a33f1d7\" (UID: \"8664e04d-5336-401b-9a09-6b159a33f1d7\") "
Jul 01 17:39:05.290032 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:05.290015    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8664e04d-5336-401b-9a09-6b159a33f1d7-proxy-tls\") pod \"8664e04d-5336-401b-9a09-6b159a33f1d7\" (UID: \"8664e04d-5336-401b-9a09-6b159a33f1d7\") "
Jul 01 17:39:05.290188 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:05.290169    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8664e04d-5336-401b-9a09-6b159a33f1d7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8664e04d-5336-401b-9a09-6b159a33f1d7" (UID: "8664e04d-5336-401b-9a09-6b159a33f1d7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:39:05.290254 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:05.290243    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8664e04d-5336-401b-9a09-6b159a33f1d7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:39:05.292035 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:05.292007    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8664e04d-5336-401b-9a09-6b159a33f1d7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8664e04d-5336-401b-9a09-6b159a33f1d7" (UID: "8664e04d-5336-401b-9a09-6b159a33f1d7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:39:05.292035 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:05.292016    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8664e04d-5336-401b-9a09-6b159a33f1d7-kube-api-access-z2wx5" (OuterVolumeSpecName: "kube-api-access-z2wx5") pod "8664e04d-5336-401b-9a09-6b159a33f1d7" (UID: "8664e04d-5336-401b-9a09-6b159a33f1d7"). InnerVolumeSpecName "kube-api-access-z2wx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:39:05.391560 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:05.391522    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2wx5\" (UniqueName: \"kubernetes.io/projected/8664e04d-5336-401b-9a09-6b159a33f1d7-kube-api-access-z2wx5\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:39:05.391560 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:05.391553    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8664e04d-5336-401b-9a09-6b159a33f1d7-proxy-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:39:05.465164 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:05.465124    2585 generic.go:356] "Generic (PLEG): container finished" podID="8664e04d-5336-401b-9a09-6b159a33f1d7" containerID="ae06692c478739eda6e871d6577b6c9262f8df2d593617978194df49926726c2" exitCode=0
Jul 01 17:39:05.465524 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:05.465185    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7"
Jul 01 17:39:05.465524 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:05.465197    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7" event={"ID":"8664e04d-5336-401b-9a09-6b159a33f1d7","Type":"ContainerDied","Data":"ae06692c478739eda6e871d6577b6c9262f8df2d593617978194df49926726c2"}
Jul 01 17:39:05.465524 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:05.465295    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7" event={"ID":"8664e04d-5336-401b-9a09-6b159a33f1d7","Type":"ContainerDied","Data":"3aa5085792830fb9f4539d3bbe215a8114c638e8a456333f7f35fae5fc4977da"}
Jul 01 17:39:05.465524 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:05.465334    2585 scope.go:117] "RemoveContainer" containerID="8aa476ebeb7b525cac913de165161469a50ae956c84df43b7903f29c0cb7b684"
Jul 01 17:39:05.465819 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:05.465785    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp" podUID="92a915eb-bc3c-4e50-8b6b-48f7de29d84f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 17:39:05.473643 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:05.473628    2585 scope.go:117] "RemoveContainer" containerID="ae06692c478739eda6e871d6577b6c9262f8df2d593617978194df49926726c2"
Jul 01 17:39:05.481312 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:05.481298    2585 scope.go:117] "RemoveContainer" containerID="8aa476ebeb7b525cac913de165161469a50ae956c84df43b7903f29c0cb7b684"
Jul 01 17:39:05.481520 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:39:05.481505    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aa476ebeb7b525cac913de165161469a50ae956c84df43b7903f29c0cb7b684\": container with ID starting with 8aa476ebeb7b525cac913de165161469a50ae956c84df43b7903f29c0cb7b684 not found: ID does not exist" containerID="8aa476ebeb7b525cac913de165161469a50ae956c84df43b7903f29c0cb7b684"
Jul 01 17:39:05.481588 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:05.481527    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aa476ebeb7b525cac913de165161469a50ae956c84df43b7903f29c0cb7b684"} err="failed to get container status \"8aa476ebeb7b525cac913de165161469a50ae956c84df43b7903f29c0cb7b684\": rpc error: code = NotFound desc = could not find container \"8aa476ebeb7b525cac913de165161469a50ae956c84df43b7903f29c0cb7b684\": container with ID starting with 8aa476ebeb7b525cac913de165161469a50ae956c84df43b7903f29c0cb7b684 not found: ID does not exist"
Jul 01 17:39:05.481588 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:05.481541    2585 scope.go:117] "RemoveContainer" containerID="ae06692c478739eda6e871d6577b6c9262f8df2d593617978194df49926726c2"
Jul 01 17:39:05.481762 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:39:05.481747    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae06692c478739eda6e871d6577b6c9262f8df2d593617978194df49926726c2\": container with ID starting with ae06692c478739eda6e871d6577b6c9262f8df2d593617978194df49926726c2 not found: ID does not exist" containerID="ae06692c478739eda6e871d6577b6c9262f8df2d593617978194df49926726c2"
Jul 01 17:39:05.481800 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:05.481763    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae06692c478739eda6e871d6577b6c9262f8df2d593617978194df49926726c2"} err="failed to get container status \"ae06692c478739eda6e871d6577b6c9262f8df2d593617978194df49926726c2\": rpc error: code = NotFound desc = could not find container \"ae06692c478739eda6e871d6577b6c9262f8df2d593617978194df49926726c2\": container with ID starting with ae06692c478739eda6e871d6577b6c9262f8df2d593617978194df49926726c2 not found: ID does not exist"
Jul 01 17:39:05.490024 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:05.490005    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7"]
Jul 01 17:39:05.493301 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:05.493285    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-766f3-predictor-6b6855b58b-dmhf7"]
Jul 01 17:39:06.172286 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:06.172254    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8664e04d-5336-401b-9a09-6b159a33f1d7" path="/var/lib/kubelet/pods/8664e04d-5336-401b-9a09-6b159a33f1d7/volumes"
Jul 01 17:39:09.477944 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:09.477909    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-766f3-f9786597b-xhzhn" podUID="fe90fe7b-f8fd-4fcb-8962-f9a22544d016" containerName="sequence-graph-766f3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:39:10.471595 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:10.471554    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp"
Jul 01 17:39:10.472073 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:10.472032    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp" podUID="92a915eb-bc3c-4e50-8b6b-48f7de29d84f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 17:39:14.478345 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:14.478308    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-766f3-f9786597b-xhzhn" podUID="fe90fe7b-f8fd-4fcb-8962-f9a22544d016" containerName="sequence-graph-766f3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:39:14.478754 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:14.478403    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-766f3-f9786597b-xhzhn"
Jul 01 17:39:16.246139 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:16.246102    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-43ae3-65f97db95d-lv2vz"]
Jul 01 17:39:16.246494 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:16.246418    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8664e04d-5336-401b-9a09-6b159a33f1d7" containerName="kserve-container"
Jul 01 17:39:16.246494 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:16.246429    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8664e04d-5336-401b-9a09-6b159a33f1d7" containerName="kserve-container"
Jul 01 17:39:16.246494 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:16.246442    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8664e04d-5336-401b-9a09-6b159a33f1d7" containerName="kube-rbac-proxy"
Jul 01 17:39:16.246494 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:16.246447    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8664e04d-5336-401b-9a09-6b159a33f1d7" containerName="kube-rbac-proxy"
Jul 01 17:39:16.246700 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:16.246509    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8664e04d-5336-401b-9a09-6b159a33f1d7" containerName="kube-rbac-proxy"
Jul 01 17:39:16.246700 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:16.246519    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8664e04d-5336-401b-9a09-6b159a33f1d7" containerName="kserve-container"
Jul 01 17:39:16.250611 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:16.250594    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-43ae3-65f97db95d-lv2vz"
Jul 01 17:39:16.256804 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:16.256779    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-43ae3-serving-cert\""
Jul 01 17:39:16.256917 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:16.256778    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-43ae3-kube-rbac-proxy-sar-config\""
Jul 01 17:39:16.262269 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:16.262244    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-43ae3-65f97db95d-lv2vz"]
Jul 01 17:39:16.369681 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:16.369650    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222-openshift-service-ca-bundle\") pod \"ensemble-graph-43ae3-65f97db95d-lv2vz\" (UID: \"a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222\") " pod="kserve-ci-e2e-test/ensemble-graph-43ae3-65f97db95d-lv2vz"
Jul 01 17:39:16.369831 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:16.369706    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222-proxy-tls\") pod \"ensemble-graph-43ae3-65f97db95d-lv2vz\" (UID: \"a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222\") " pod="kserve-ci-e2e-test/ensemble-graph-43ae3-65f97db95d-lv2vz"
Jul 01 17:39:16.470347 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:16.470316    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222-proxy-tls\") pod \"ensemble-graph-43ae3-65f97db95d-lv2vz\" (UID: \"a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222\") " pod="kserve-ci-e2e-test/ensemble-graph-43ae3-65f97db95d-lv2vz"
Jul 01 17:39:16.470494 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:16.470356    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222-openshift-service-ca-bundle\") pod \"ensemble-graph-43ae3-65f97db95d-lv2vz\" (UID: \"a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222\") " pod="kserve-ci-e2e-test/ensemble-graph-43ae3-65f97db95d-lv2vz"
Jul 01 17:39:16.470494 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:39:16.470452    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-43ae3-serving-cert: secret "ensemble-graph-43ae3-serving-cert" not found
Jul 01 17:39:16.470602 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:39:16.470516    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222-proxy-tls podName:a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222 nodeName:}" failed. No retries permitted until 2026-07-01 17:39:16.970500521 +0000 UTC m=+1445.275698675 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222-proxy-tls") pod "ensemble-graph-43ae3-65f97db95d-lv2vz" (UID: "a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222") : secret "ensemble-graph-43ae3-serving-cert" not found
Jul 01 17:39:16.470947 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:16.470930    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222-openshift-service-ca-bundle\") pod \"ensemble-graph-43ae3-65f97db95d-lv2vz\" (UID: \"a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222\") " pod="kserve-ci-e2e-test/ensemble-graph-43ae3-65f97db95d-lv2vz"
Jul 01 17:39:16.973479 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:16.973447    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222-proxy-tls\") pod \"ensemble-graph-43ae3-65f97db95d-lv2vz\" (UID: \"a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222\") " pod="kserve-ci-e2e-test/ensemble-graph-43ae3-65f97db95d-lv2vz"
Jul 01 17:39:16.975954 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:16.975934    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222-proxy-tls\") pod \"ensemble-graph-43ae3-65f97db95d-lv2vz\" (UID: \"a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222\") " pod="kserve-ci-e2e-test/ensemble-graph-43ae3-65f97db95d-lv2vz"
Jul 01 17:39:17.167624 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:17.167590    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-43ae3-65f97db95d-lv2vz"
Jul 01 17:39:17.307392 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:17.307365    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-43ae3-65f97db95d-lv2vz"]
Jul 01 17:39:17.309452 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:39:17.309419    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f2fa2f_b96b_4fd2_b630_a8ef0a4e0222.slice/crio-ce8e2d7eaffd1f24d7f7e36772ce4fac72fe13035d252c8370c55b3567410905 WatchSource:0}: Error finding container ce8e2d7eaffd1f24d7f7e36772ce4fac72fe13035d252c8370c55b3567410905: Status 404 returned error can't find the container with id ce8e2d7eaffd1f24d7f7e36772ce4fac72fe13035d252c8370c55b3567410905
Jul 01 17:39:17.514436 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:17.514344    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-43ae3-65f97db95d-lv2vz" event={"ID":"a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222","Type":"ContainerStarted","Data":"6f84d9814d8f976241cf032279afc322a533d746fa7065ff0bf337d035f2d3d2"}
Jul 01 17:39:17.514436 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:17.514392    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-43ae3-65f97db95d-lv2vz" event={"ID":"a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222","Type":"ContainerStarted","Data":"ce8e2d7eaffd1f24d7f7e36772ce4fac72fe13035d252c8370c55b3567410905"}
Jul 01 17:39:17.514690 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:17.514485    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-43ae3-65f97db95d-lv2vz"
Jul 01 17:39:17.542368 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:17.542317    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-43ae3-65f97db95d-lv2vz" podStartSLOduration=1.542300656 podStartE2EDuration="1.542300656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:39:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:39:17.541993592 +0000 UTC m=+1445.847191769" watchObservedRunningTime="2026-07-01 17:39:17.542300656 +0000 UTC m=+1445.847498831"
Jul 01 17:39:19.478179 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:19.478139    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-766f3-f9786597b-xhzhn" podUID="fe90fe7b-f8fd-4fcb-8962-f9a22544d016" containerName="sequence-graph-766f3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:39:20.472806 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:20.472772    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp" podUID="92a915eb-bc3c-4e50-8b6b-48f7de29d84f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 17:39:23.524254 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:23.524226    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-43ae3-65f97db95d-lv2vz"
Jul 01 17:39:24.478270 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:24.478234    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-766f3-f9786597b-xhzhn" podUID="fe90fe7b-f8fd-4fcb-8962-f9a22544d016" containerName="sequence-graph-766f3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:39:26.301040 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:26.300969    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-43ae3-65f97db95d-lv2vz"]
Jul 01 17:39:26.301409 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:26.301169    2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-43ae3-65f97db95d-lv2vz" podUID="a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222" containerName="ensemble-graph-43ae3" containerID="cri-o://6f84d9814d8f976241cf032279afc322a533d746fa7065ff0bf337d035f2d3d2" gracePeriod=30
Jul 01 17:39:26.573686 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:26.573651    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q"]
Jul 01 17:39:26.573938 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:26.573899    2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q" podUID="f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67" containerName="kserve-container" containerID="cri-o://a41b5681f04d4c5b9a7521503fc04ec5ca3f25168a25985cf397a856cde6064e" gracePeriod=30
Jul 01 17:39:26.574024 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:26.573969    2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q" podUID="f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67" containerName="kube-rbac-proxy" containerID="cri-o://7519e09bb777aa377229f63e12683271f0edf3e1f81f66ee73328621cbf5ece2" gracePeriod=30
Jul 01 17:39:26.596601 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:26.596558    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9"]
Jul 01 17:39:26.600042 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:26.600026    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9"
Jul 01 17:39:26.602733 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:26.602712    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c4578-predictor-serving-cert\""
Jul 01 17:39:26.602827 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:26.602733    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c4578-kube-rbac-proxy-sar-config\""
Jul 01 17:39:26.609667 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:26.609644    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9"]
Jul 01 17:39:26.742775 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:26.742745    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87e604e1-c30c-47f6-bd5a-aab6e79f83da-proxy-tls\") pod \"error-404-isvc-c4578-predictor-8585647f56-4hsc9\" (UID: \"87e604e1-c30c-47f6-bd5a-aab6e79f83da\") " pod="kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9"
Jul 01 17:39:26.742926 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:26.742784    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87e604e1-c30c-47f6-bd5a-aab6e79f83da-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c4578-predictor-8585647f56-4hsc9\" (UID: \"87e604e1-c30c-47f6-bd5a-aab6e79f83da\") " pod="kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9"
Jul 01 17:39:26.742926 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:26.742885    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22fs9\" (UniqueName: \"kubernetes.io/projected/87e604e1-c30c-47f6-bd5a-aab6e79f83da-kube-api-access-22fs9\") pod \"error-404-isvc-c4578-predictor-8585647f56-4hsc9\" (UID: \"87e604e1-c30c-47f6-bd5a-aab6e79f83da\") " pod="kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9"
Jul 01 17:39:26.843261 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:26.843191    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22fs9\" (UniqueName: \"kubernetes.io/projected/87e604e1-c30c-47f6-bd5a-aab6e79f83da-kube-api-access-22fs9\") pod \"error-404-isvc-c4578-predictor-8585647f56-4hsc9\" (UID: \"87e604e1-c30c-47f6-bd5a-aab6e79f83da\") " pod="kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9"
Jul 01 17:39:26.843261 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:26.843229    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87e604e1-c30c-47f6-bd5a-aab6e79f83da-proxy-tls\") pod \"error-404-isvc-c4578-predictor-8585647f56-4hsc9\" (UID: \"87e604e1-c30c-47f6-bd5a-aab6e79f83da\") " pod="kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9"
Jul 01 17:39:26.843261 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:26.843254    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87e604e1-c30c-47f6-bd5a-aab6e79f83da-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c4578-predictor-8585647f56-4hsc9\" (UID: \"87e604e1-c30c-47f6-bd5a-aab6e79f83da\") " pod="kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9"
Jul 01 17:39:26.843846 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:26.843823    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87e604e1-c30c-47f6-bd5a-aab6e79f83da-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c4578-predictor-8585647f56-4hsc9\" (UID: \"87e604e1-c30c-47f6-bd5a-aab6e79f83da\") " pod="kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9"
Jul 01 17:39:26.845796 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:26.845765    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87e604e1-c30c-47f6-bd5a-aab6e79f83da-proxy-tls\") pod \"error-404-isvc-c4578-predictor-8585647f56-4hsc9\" (UID: \"87e604e1-c30c-47f6-bd5a-aab6e79f83da\") " pod="kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9"
Jul 01 17:39:26.853920 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:26.853895    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22fs9\" (UniqueName: \"kubernetes.io/projected/87e604e1-c30c-47f6-bd5a-aab6e79f83da-kube-api-access-22fs9\") pod \"error-404-isvc-c4578-predictor-8585647f56-4hsc9\" (UID: \"87e604e1-c30c-47f6-bd5a-aab6e79f83da\") " pod="kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9"
Jul 01 17:39:26.910749 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:26.910717    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9"
Jul 01 17:39:27.037640 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:27.037604    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9"]
Jul 01 17:39:27.040266 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:39:27.040239    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e604e1_c30c_47f6_bd5a_aab6e79f83da.slice/crio-c210e9c15ff02a25ff40ad2c6c06bd6101585c3a711910ca8548f9f347cdc3d5 WatchSource:0}: Error finding container c210e9c15ff02a25ff40ad2c6c06bd6101585c3a711910ca8548f9f347cdc3d5: Status 404 returned error can't find the container with id c210e9c15ff02a25ff40ad2c6c06bd6101585c3a711910ca8548f9f347cdc3d5
Jul 01 17:39:27.556416 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:27.556377    2585 generic.go:356] "Generic (PLEG): container finished" podID="f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67" containerID="7519e09bb777aa377229f63e12683271f0edf3e1f81f66ee73328621cbf5ece2" exitCode=2
Jul 01 17:39:27.556856 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:27.556450    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q" event={"ID":"f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67","Type":"ContainerDied","Data":"7519e09bb777aa377229f63e12683271f0edf3e1f81f66ee73328621cbf5ece2"}
Jul 01 17:39:27.558541 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:27.558514    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9" event={"ID":"87e604e1-c30c-47f6-bd5a-aab6e79f83da","Type":"ContainerStarted","Data":"c8e5b334e77e25d94c798e5023990635683dc260020df5bbce8636e69c23e748"}
Jul 01 17:39:27.558698 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:27.558599    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9" event={"ID":"87e604e1-c30c-47f6-bd5a-aab6e79f83da","Type":"ContainerStarted","Data":"cd3f5950fb891a5c63968f4efdf676064b4a696e7d20733689d8a185b52676a2"}
Jul 01 17:39:27.558698 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:27.558652    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9" event={"ID":"87e604e1-c30c-47f6-bd5a-aab6e79f83da","Type":"ContainerStarted","Data":"c210e9c15ff02a25ff40ad2c6c06bd6101585c3a711910ca8548f9f347cdc3d5"}
Jul 01 17:39:27.558787 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:27.558766    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9"
Jul 01 17:39:27.577817 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:27.577780    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9" podStartSLOduration=1.577768558 podStartE2EDuration="1.577768558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:39:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:39:27.57644286 +0000 UTC m=+1455.881641038" watchObservedRunningTime="2026-07-01 17:39:27.577768558 +0000 UTC m=+1455.882966733"
Jul 01 17:39:28.522704 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:28.522668    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-43ae3-65f97db95d-lv2vz" podUID="a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222" containerName="ensemble-graph-43ae3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:39:28.562344 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:28.562319    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9"
Jul 01 17:39:28.563375 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:28.563350    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9" podUID="87e604e1-c30c-47f6-bd5a-aab6e79f83da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 01 17:39:29.478308 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:29.478276    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-766f3-f9786597b-xhzhn" podUID="fe90fe7b-f8fd-4fcb-8962-f9a22544d016" containerName="sequence-graph-766f3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:39:29.568529 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:29.568496    2585 generic.go:356] "Generic (PLEG): container finished" podID="f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67" containerID="a41b5681f04d4c5b9a7521503fc04ec5ca3f25168a25985cf397a856cde6064e" exitCode=0
Jul 01 17:39:29.568865 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:29.568567    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q" event={"ID":"f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67","Type":"ContainerDied","Data":"a41b5681f04d4c5b9a7521503fc04ec5ca3f25168a25985cf397a856cde6064e"}
Jul 01 17:39:29.569149 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:29.569125    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9" podUID="87e604e1-c30c-47f6-bd5a-aab6e79f83da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 01 17:39:29.617454 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:29.617435    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q"
Jul 01 17:39:29.767378 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:29.767287    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqrnv\" (UniqueName: \"kubernetes.io/projected/f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67-kube-api-access-lqrnv\") pod \"f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67\" (UID: \"f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67\") "
Jul 01 17:39:29.767378 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:29.767371    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67-kube-rbac-proxy-sar-config\") pod \"f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67\" (UID: \"f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67\") "
Jul 01 17:39:29.767554 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:29.767461    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67-proxy-tls\") pod \"f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67\" (UID: \"f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67\") "
Jul 01 17:39:29.767713 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:29.767691    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67" (UID: "f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:39:29.769483 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:29.769461    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67-kube-api-access-lqrnv" (OuterVolumeSpecName: "kube-api-access-lqrnv") pod "f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67" (UID: "f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67"). InnerVolumeSpecName "kube-api-access-lqrnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:39:29.769585 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:29.769513    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67" (UID: "f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:39:29.868087 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:29.868047    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:39:29.868087 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:29.868083    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67-proxy-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:39:29.868087 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:29.868092    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqrnv\" (UniqueName: \"kubernetes.io/projected/f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67-kube-api-access-lqrnv\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:39:30.472715 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:30.472618    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp" podUID="92a915eb-bc3c-4e50-8b6b-48f7de29d84f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 17:39:30.573791 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:30.573758    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q"
Jul 01 17:39:30.573791 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:30.573782    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q" event={"ID":"f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67","Type":"ContainerDied","Data":"ca63db91c7964efc5d4d3e6b43efe6bfcd449c0311a2886707e8ded11f8cf49a"}
Jul 01 17:39:30.574254 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:30.573848    2585 scope.go:117] "RemoveContainer" containerID="7519e09bb777aa377229f63e12683271f0edf3e1f81f66ee73328621cbf5ece2"
Jul 01 17:39:30.582111 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:30.582094    2585 scope.go:117] "RemoveContainer" containerID="a41b5681f04d4c5b9a7521503fc04ec5ca3f25168a25985cf397a856cde6064e"
Jul 01 17:39:30.590964 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:30.590945    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q"]
Jul 01 17:39:30.595032 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:30.595014    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-43ae3-predictor-669cb6b886-wfs2q"]
Jul 01 17:39:32.173109 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:32.173038    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67" path="/var/lib/kubelet/pods/f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67/volumes"
Jul 01 17:39:32.457420 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:32.457390    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-766f3-f9786597b-xhzhn"
Jul 01 17:39:32.584230 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:32.584198    2585 generic.go:356] "Generic (PLEG): container finished" podID="fe90fe7b-f8fd-4fcb-8962-f9a22544d016" containerID="27a4c23ced9581cf29ac473ab6a72713ac17564af4c997aa3f5325f4b6584094" exitCode=0
Jul 01 17:39:32.584400 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:32.584255    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-766f3-f9786597b-xhzhn"
Jul 01 17:39:32.584400 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:32.584280    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-766f3-f9786597b-xhzhn" event={"ID":"fe90fe7b-f8fd-4fcb-8962-f9a22544d016","Type":"ContainerDied","Data":"27a4c23ced9581cf29ac473ab6a72713ac17564af4c997aa3f5325f4b6584094"}
Jul 01 17:39:32.584400 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:32.584338    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-766f3-f9786597b-xhzhn" event={"ID":"fe90fe7b-f8fd-4fcb-8962-f9a22544d016","Type":"ContainerDied","Data":"dfcdc4aec08b126bbc16f24360585dcb03898b6adbaa7be9c63842a3e2854d09"}
Jul 01 17:39:32.584400 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:32.584377    2585 scope.go:117] "RemoveContainer" containerID="27a4c23ced9581cf29ac473ab6a72713ac17564af4c997aa3f5325f4b6584094"
Jul 01 17:39:32.591949 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:32.591932    2585 scope.go:117] "RemoveContainer" containerID="27a4c23ced9581cf29ac473ab6a72713ac17564af4c997aa3f5325f4b6584094"
Jul 01 17:39:32.592180 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:32.592166    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe90fe7b-f8fd-4fcb-8962-f9a22544d016-openshift-service-ca-bundle\") pod \"fe90fe7b-f8fd-4fcb-8962-f9a22544d016\" (UID: \"fe90fe7b-f8fd-4fcb-8962-f9a22544d016\") "
Jul 01 17:39:32.592225 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:39:32.592201    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27a4c23ced9581cf29ac473ab6a72713ac17564af4c997aa3f5325f4b6584094\": container with ID starting with 27a4c23ced9581cf29ac473ab6a72713ac17564af4c997aa3f5325f4b6584094 not found: ID does not exist" containerID="27a4c23ced9581cf29ac473ab6a72713ac17564af4c997aa3f5325f4b6584094"
Jul 01 17:39:32.592260 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:32.592237    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27a4c23ced9581cf29ac473ab6a72713ac17564af4c997aa3f5325f4b6584094"} err="failed to get container status \"27a4c23ced9581cf29ac473ab6a72713ac17564af4c997aa3f5325f4b6584094\": rpc error: code = NotFound desc = could not find container \"27a4c23ced9581cf29ac473ab6a72713ac17564af4c997aa3f5325f4b6584094\": container with ID starting with 27a4c23ced9581cf29ac473ab6a72713ac17564af4c997aa3f5325f4b6584094 not found: ID does not exist"
Jul 01 17:39:32.592304 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:32.592263    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe90fe7b-f8fd-4fcb-8962-f9a22544d016-proxy-tls\") pod \"fe90fe7b-f8fd-4fcb-8962-f9a22544d016\" (UID: \"fe90fe7b-f8fd-4fcb-8962-f9a22544d016\") "
Jul 01 17:39:32.592539 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:32.592519    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe90fe7b-f8fd-4fcb-8962-f9a22544d016-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "fe90fe7b-f8fd-4fcb-8962-f9a22544d016" (UID: "fe90fe7b-f8fd-4fcb-8962-f9a22544d016"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:39:32.594179 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:32.594154    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe90fe7b-f8fd-4fcb-8962-f9a22544d016-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fe90fe7b-f8fd-4fcb-8962-f9a22544d016" (UID: "fe90fe7b-f8fd-4fcb-8962-f9a22544d016"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:39:32.693819 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:32.693731    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe90fe7b-f8fd-4fcb-8962-f9a22544d016-proxy-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:39:32.693819 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:32.693770    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe90fe7b-f8fd-4fcb-8962-f9a22544d016-openshift-service-ca-bundle\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:39:32.907470 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:32.907443    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-766f3-f9786597b-xhzhn"]
Jul 01 17:39:32.913068 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:32.913048    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-766f3-f9786597b-xhzhn"]
Jul 01 17:39:33.522435 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:33.522396    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-43ae3-65f97db95d-lv2vz" podUID="a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222" containerName="ensemble-graph-43ae3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:39:34.173608 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:34.173556    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe90fe7b-f8fd-4fcb-8962-f9a22544d016" path="/var/lib/kubelet/pods/fe90fe7b-f8fd-4fcb-8962-f9a22544d016/volumes"
Jul 01 17:39:34.573137 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:34.573110    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9"
Jul 01 17:39:34.573633 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:34.573607    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9" podUID="87e604e1-c30c-47f6-bd5a-aab6e79f83da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 01 17:39:38.522870 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:38.522831    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-43ae3-65f97db95d-lv2vz" podUID="a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222" containerName="ensemble-graph-43ae3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:39:38.523237 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:38.522925    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-43ae3-65f97db95d-lv2vz"
Jul 01 17:39:40.472262 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:40.472219    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp" podUID="92a915eb-bc3c-4e50-8b6b-48f7de29d84f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 17:39:43.522425 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:43.522381    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-43ae3-65f97db95d-lv2vz" podUID="a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222" containerName="ensemble-graph-43ae3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:39:44.573748 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:44.573710    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9" podUID="87e604e1-c30c-47f6-bd5a-aab6e79f83da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 01 17:39:48.522472 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:48.522435    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-43ae3-65f97db95d-lv2vz" podUID="a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222" containerName="ensemble-graph-43ae3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:39:50.472721 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:50.472695    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp"
Jul 01 17:39:53.522204 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:53.522170    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-43ae3-65f97db95d-lv2vz" podUID="a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222" containerName="ensemble-graph-43ae3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:39:54.573966 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:54.573924    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9" podUID="87e604e1-c30c-47f6-bd5a-aab6e79f83da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 01 17:39:56.444752 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:56.444726    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-43ae3-65f97db95d-lv2vz"
Jul 01 17:39:56.467623 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:56.467599    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222-proxy-tls\") pod \"a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222\" (UID: \"a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222\") "
Jul 01 17:39:56.467760 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:56.467696    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222-openshift-service-ca-bundle\") pod \"a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222\" (UID: \"a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222\") "
Jul 01 17:39:56.468047 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:56.468026    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222" (UID: "a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:39:56.469791 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:56.469771    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222" (UID: "a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:39:56.569046 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:56.568977    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222-openshift-service-ca-bundle\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:39:56.569046 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:56.569000    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222-proxy-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:39:56.680688 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:56.680654    2585 generic.go:356] "Generic (PLEG): container finished" podID="a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222" containerID="6f84d9814d8f976241cf032279afc322a533d746fa7065ff0bf337d035f2d3d2" exitCode=0
Jul 01 17:39:56.680850 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:56.680697    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-43ae3-65f97db95d-lv2vz"
Jul 01 17:39:56.680850 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:56.680733    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-43ae3-65f97db95d-lv2vz" event={"ID":"a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222","Type":"ContainerDied","Data":"6f84d9814d8f976241cf032279afc322a533d746fa7065ff0bf337d035f2d3d2"}
Jul 01 17:39:56.680850 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:56.680773    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-43ae3-65f97db95d-lv2vz" event={"ID":"a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222","Type":"ContainerDied","Data":"ce8e2d7eaffd1f24d7f7e36772ce4fac72fe13035d252c8370c55b3567410905"}
Jul 01 17:39:56.680850 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:56.680794    2585 scope.go:117] "RemoveContainer" containerID="6f84d9814d8f976241cf032279afc322a533d746fa7065ff0bf337d035f2d3d2"
Jul 01 17:39:56.689289 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:56.689266    2585 scope.go:117] "RemoveContainer" containerID="6f84d9814d8f976241cf032279afc322a533d746fa7065ff0bf337d035f2d3d2"
Jul 01 17:39:56.689587 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:39:56.689555    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f84d9814d8f976241cf032279afc322a533d746fa7065ff0bf337d035f2d3d2\": container with ID starting with 6f84d9814d8f976241cf032279afc322a533d746fa7065ff0bf337d035f2d3d2 not found: ID does not exist" containerID="6f84d9814d8f976241cf032279afc322a533d746fa7065ff0bf337d035f2d3d2"
Jul 01 17:39:56.689634 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:56.689600    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f84d9814d8f976241cf032279afc322a533d746fa7065ff0bf337d035f2d3d2"} err="failed to get container status \"6f84d9814d8f976241cf032279afc322a533d746fa7065ff0bf337d035f2d3d2\": rpc error: code = NotFound desc = could not find container \"6f84d9814d8f976241cf032279afc322a533d746fa7065ff0bf337d035f2d3d2\": container with ID starting with 6f84d9814d8f976241cf032279afc322a533d746fa7065ff0bf337d035f2d3d2 not found: ID does not exist"
Jul 01 17:39:56.701085 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:56.701067    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-43ae3-65f97db95d-lv2vz"]
Jul 01 17:39:56.706113 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:56.706095    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-43ae3-65f97db95d-lv2vz"]
Jul 01 17:39:58.172538 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:39:58.172501    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222" path="/var/lib/kubelet/pods/a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222/volumes"
Jul 01 17:40:01.972100 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:01.972070    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-98f1c-788bc79767-kkl6d"]
Jul 01 17:40:01.972458 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:01.972357    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67" containerName="kserve-container"
Jul 01 17:40:01.972458 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:01.972369    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67" containerName="kserve-container"
Jul 01 17:40:01.972458 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:01.972383    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67" containerName="kube-rbac-proxy"
Jul 01 17:40:01.972458 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:01.972388    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67" containerName="kube-rbac-proxy"
Jul 01 17:40:01.972458 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:01.972399    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222" containerName="ensemble-graph-43ae3"
Jul 01 17:40:01.972458 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:01.972404    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222" containerName="ensemble-graph-43ae3"
Jul 01 17:40:01.972458 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:01.972413    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe90fe7b-f8fd-4fcb-8962-f9a22544d016" containerName="sequence-graph-766f3"
Jul 01 17:40:01.972458 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:01.972418    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe90fe7b-f8fd-4fcb-8962-f9a22544d016" containerName="sequence-graph-766f3"
Jul 01 17:40:01.972727 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:01.972464    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6f2fa2f-b96b-4fd2-b630-a8ef0a4e0222" containerName="ensemble-graph-43ae3"
Jul 01 17:40:01.972727 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:01.972477    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67" containerName="kserve-container"
Jul 01 17:40:01.972727 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:01.972485    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8e53a5f-7554-4eff-a0d1-b3ef8bf10b67" containerName="kube-rbac-proxy"
Jul 01 17:40:01.972727 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:01.972491    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe90fe7b-f8fd-4fcb-8962-f9a22544d016" containerName="sequence-graph-766f3"
Jul 01 17:40:01.976311 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:01.976293    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-98f1c-788bc79767-kkl6d"
Jul 01 17:40:01.978935 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:01.978917    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-98f1c-serving-cert\""
Jul 01 17:40:01.979022 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:01.978923    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-98f1c-kube-rbac-proxy-sar-config\""
Jul 01 17:40:01.982894 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:01.982871    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-98f1c-788bc79767-kkl6d"]
Jul 01 17:40:02.012155 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:02.012131    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26e5d5c9-ef69-4bef-9383-df0e1e3de06a-proxy-tls\") pod \"sequence-graph-98f1c-788bc79767-kkl6d\" (UID: \"26e5d5c9-ef69-4bef-9383-df0e1e3de06a\") " pod="kserve-ci-e2e-test/sequence-graph-98f1c-788bc79767-kkl6d"
Jul 01 17:40:02.012293 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:02.012165    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26e5d5c9-ef69-4bef-9383-df0e1e3de06a-openshift-service-ca-bundle\") pod \"sequence-graph-98f1c-788bc79767-kkl6d\" (UID: \"26e5d5c9-ef69-4bef-9383-df0e1e3de06a\") " pod="kserve-ci-e2e-test/sequence-graph-98f1c-788bc79767-kkl6d"
Jul 01 17:40:02.112973 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:02.112944    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26e5d5c9-ef69-4bef-9383-df0e1e3de06a-proxy-tls\") pod \"sequence-graph-98f1c-788bc79767-kkl6d\" (UID: \"26e5d5c9-ef69-4bef-9383-df0e1e3de06a\") " pod="kserve-ci-e2e-test/sequence-graph-98f1c-788bc79767-kkl6d"
Jul 01 17:40:02.113142 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:02.112980    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26e5d5c9-ef69-4bef-9383-df0e1e3de06a-openshift-service-ca-bundle\") pod \"sequence-graph-98f1c-788bc79767-kkl6d\" (UID: \"26e5d5c9-ef69-4bef-9383-df0e1e3de06a\") " pod="kserve-ci-e2e-test/sequence-graph-98f1c-788bc79767-kkl6d"
Jul 01 17:40:02.113560 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:02.113537    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26e5d5c9-ef69-4bef-9383-df0e1e3de06a-openshift-service-ca-bundle\") pod \"sequence-graph-98f1c-788bc79767-kkl6d\" (UID: \"26e5d5c9-ef69-4bef-9383-df0e1e3de06a\") " pod="kserve-ci-e2e-test/sequence-graph-98f1c-788bc79767-kkl6d"
Jul 01 17:40:02.115428 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:02.115407    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26e5d5c9-ef69-4bef-9383-df0e1e3de06a-proxy-tls\") pod \"sequence-graph-98f1c-788bc79767-kkl6d\" (UID: \"26e5d5c9-ef69-4bef-9383-df0e1e3de06a\") " pod="kserve-ci-e2e-test/sequence-graph-98f1c-788bc79767-kkl6d"
Jul 01 17:40:02.287397 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:02.287309    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-98f1c-788bc79767-kkl6d"
Jul 01 17:40:02.414652 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:02.414625    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-98f1c-788bc79767-kkl6d"]
Jul 01 17:40:02.417219 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:40:02.417194    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26e5d5c9_ef69_4bef_9383_df0e1e3de06a.slice/crio-826a5891308e7d492e59f466a213218795550b07a4b1a7b42eb4c84df6ab94ec WatchSource:0}: Error finding container 826a5891308e7d492e59f466a213218795550b07a4b1a7b42eb4c84df6ab94ec: Status 404 returned error can't find the container with id 826a5891308e7d492e59f466a213218795550b07a4b1a7b42eb4c84df6ab94ec
Jul 01 17:40:02.705470 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:02.705429    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-98f1c-788bc79767-kkl6d" event={"ID":"26e5d5c9-ef69-4bef-9383-df0e1e3de06a","Type":"ContainerStarted","Data":"d2deccf1e53e1f37e5bd1f856abd6003a1d6ef6dfbfe9b4db937fea399be9b42"}
Jul 01 17:40:02.705663 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:02.705490    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-98f1c-788bc79767-kkl6d" event={"ID":"26e5d5c9-ef69-4bef-9383-df0e1e3de06a","Type":"ContainerStarted","Data":"826a5891308e7d492e59f466a213218795550b07a4b1a7b42eb4c84df6ab94ec"}
Jul 01 17:40:02.705663 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:02.705557    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-98f1c-788bc79767-kkl6d"
Jul 01 17:40:02.723820 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:02.723777    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-98f1c-788bc79767-kkl6d" podStartSLOduration=1.723763425 podStartE2EDuration="1.723763425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:40:02.72256659 +0000 UTC m=+1491.027764766" watchObservedRunningTime="2026-07-01 17:40:02.723763425 +0000 UTC m=+1491.028961601"
Jul 01 17:40:04.573533 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:04.573485    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9" podUID="87e604e1-c30c-47f6-bd5a-aab6e79f83da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 01 17:40:08.714381 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:08.714348    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-98f1c-788bc79767-kkl6d"
Jul 01 17:40:12.056211 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:12.056176    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-98f1c-788bc79767-kkl6d"]
Jul 01 17:40:12.056594 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:12.056393    2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-98f1c-788bc79767-kkl6d" podUID="26e5d5c9-ef69-4bef-9383-df0e1e3de06a" containerName="sequence-graph-98f1c" containerID="cri-o://d2deccf1e53e1f37e5bd1f856abd6003a1d6ef6dfbfe9b4db937fea399be9b42" gracePeriod=30
Jul 01 17:40:12.213786 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:12.213761    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8tsw_500d2a90-2d21-42b3-9584-b04c14becaec/ovn-acl-logging/0.log"
Jul 01 17:40:12.216644 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:12.216616    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8tsw_500d2a90-2d21-42b3-9584-b04c14becaec/ovn-acl-logging/0.log"
Jul 01 17:40:12.295556 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:12.295524    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd"]
Jul 01 17:40:12.299916 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:12.299894    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd"
Jul 01 17:40:12.302512 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:12.302492    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8b57f-predictor-serving-cert\""
Jul 01 17:40:12.302650 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:12.302514    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8b57f-kube-rbac-proxy-sar-config\""
Jul 01 17:40:12.310607 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:12.310508    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd"]
Jul 01 17:40:12.333808 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:12.333777    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp"]
Jul 01 17:40:12.334009 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:12.333985    2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp" podUID="92a915eb-bc3c-4e50-8b6b-48f7de29d84f" containerName="kserve-container" containerID="cri-o://1021fba1e78c9ea3572cd246e48ee4a1e6001d7e6a90473c0e5927508416949f" gracePeriod=30
Jul 01 17:40:12.334094 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:12.334046    2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp" podUID="92a915eb-bc3c-4e50-8b6b-48f7de29d84f" containerName="kube-rbac-proxy" containerID="cri-o://4afd9d8aba9a4b1282dea1e9005dee241cfbb0d0bcbe9a9313ab850ae9ab5172" gracePeriod=30
Jul 01 17:40:12.388045 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:12.388007    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e25145e5-82ef-4c78-8e97-84a50628dc1a-proxy-tls\") pod \"error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd\" (UID: \"e25145e5-82ef-4c78-8e97-84a50628dc1a\") " pod="kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd"
Jul 01 17:40:12.388206 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:12.388073    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e25145e5-82ef-4c78-8e97-84a50628dc1a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd\" (UID: \"e25145e5-82ef-4c78-8e97-84a50628dc1a\") " pod="kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd"
Jul 01 17:40:12.388206 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:12.388105    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55hlk\" (UniqueName: \"kubernetes.io/projected/e25145e5-82ef-4c78-8e97-84a50628dc1a-kube-api-access-55hlk\") pod \"error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd\" (UID: \"e25145e5-82ef-4c78-8e97-84a50628dc1a\") " pod="kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd"
Jul 01 17:40:12.489417 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:12.489383    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e25145e5-82ef-4c78-8e97-84a50628dc1a-proxy-tls\") pod \"error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd\" (UID: \"e25145e5-82ef-4c78-8e97-84a50628dc1a\") " pod="kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd"
Jul 01 17:40:12.489603 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:12.489447    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e25145e5-82ef-4c78-8e97-84a50628dc1a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd\" (UID: \"e25145e5-82ef-4c78-8e97-84a50628dc1a\") " pod="kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd"
Jul 01 17:40:12.489603 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:12.489486    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55hlk\" (UniqueName: \"kubernetes.io/projected/e25145e5-82ef-4c78-8e97-84a50628dc1a-kube-api-access-55hlk\") pod \"error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd\" (UID: \"e25145e5-82ef-4c78-8e97-84a50628dc1a\") " pod="kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd"
Jul 01 17:40:12.489603 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:40:12.489546    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-serving-cert: secret "error-404-isvc-8b57f-predictor-serving-cert" not found
Jul 01 17:40:12.489752 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:40:12.489641    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e25145e5-82ef-4c78-8e97-84a50628dc1a-proxy-tls podName:e25145e5-82ef-4c78-8e97-84a50628dc1a nodeName:}" failed. No retries permitted until 2026-07-01 17:40:12.989620697 +0000 UTC m=+1501.294818851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e25145e5-82ef-4c78-8e97-84a50628dc1a-proxy-tls") pod "error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd" (UID: "e25145e5-82ef-4c78-8e97-84a50628dc1a") : secret "error-404-isvc-8b57f-predictor-serving-cert" not found
Jul 01 17:40:12.490121 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:12.490100    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e25145e5-82ef-4c78-8e97-84a50628dc1a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd\" (UID: \"e25145e5-82ef-4c78-8e97-84a50628dc1a\") " pod="kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd"
Jul 01 17:40:12.498420 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:12.498389    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55hlk\" (UniqueName: \"kubernetes.io/projected/e25145e5-82ef-4c78-8e97-84a50628dc1a-kube-api-access-55hlk\") pod \"error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd\" (UID: \"e25145e5-82ef-4c78-8e97-84a50628dc1a\") " pod="kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd"
Jul 01 17:40:12.745608 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:12.745506    2585 generic.go:356] "Generic (PLEG): container finished" podID="92a915eb-bc3c-4e50-8b6b-48f7de29d84f" containerID="4afd9d8aba9a4b1282dea1e9005dee241cfbb0d0bcbe9a9313ab850ae9ab5172" exitCode=2
Jul 01 17:40:12.745757 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:12.745609    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp" event={"ID":"92a915eb-bc3c-4e50-8b6b-48f7de29d84f","Type":"ContainerDied","Data":"4afd9d8aba9a4b1282dea1e9005dee241cfbb0d0bcbe9a9313ab850ae9ab5172"}
Jul 01 17:40:12.993238 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:12.993201    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e25145e5-82ef-4c78-8e97-84a50628dc1a-proxy-tls\") pod \"error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd\" (UID: \"e25145e5-82ef-4c78-8e97-84a50628dc1a\") " pod="kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd"
Jul 01 17:40:12.995821 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:12.995750    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e25145e5-82ef-4c78-8e97-84a50628dc1a-proxy-tls\") pod \"error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd\" (UID: \"e25145e5-82ef-4c78-8e97-84a50628dc1a\") " pod="kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd"
Jul 01 17:40:13.210749 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:13.210705    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd"
Jul 01 17:40:13.343863 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:13.343751    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd"]
Jul 01 17:40:13.346099 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:40:13.346074    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode25145e5_82ef_4c78_8e97_84a50628dc1a.slice/crio-2feaabed9bb07f43f9ee11674aabddff9d548b4279e34f030c02c7884fb72b27 WatchSource:0}: Error finding container 2feaabed9bb07f43f9ee11674aabddff9d548b4279e34f030c02c7884fb72b27: Status 404 returned error can't find the container with id 2feaabed9bb07f43f9ee11674aabddff9d548b4279e34f030c02c7884fb72b27
Jul 01 17:40:13.713163 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:13.713123    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-98f1c-788bc79767-kkl6d" podUID="26e5d5c9-ef69-4bef-9383-df0e1e3de06a" containerName="sequence-graph-98f1c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:40:13.752218 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:13.752178    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd" event={"ID":"e25145e5-82ef-4c78-8e97-84a50628dc1a","Type":"ContainerStarted","Data":"9fcf1091a6730580c9d6a6a57b1f66864da79980f9d8cc59d0382edc92270f3c"}
Jul 01 17:40:13.752382 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:13.752296    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd" event={"ID":"e25145e5-82ef-4c78-8e97-84a50628dc1a","Type":"ContainerStarted","Data":"dc801cb4bb26d4f2f067422b94dbfb418b733d718021de4ccf99c97b475e3c12"}
Jul 01 17:40:13.752382 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:13.752328    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd" event={"ID":"e25145e5-82ef-4c78-8e97-84a50628dc1a","Type":"ContainerStarted","Data":"2feaabed9bb07f43f9ee11674aabddff9d548b4279e34f030c02c7884fb72b27"}
Jul 01 17:40:13.752382 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:13.752379    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd"
Jul 01 17:40:13.773667 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:13.773621    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd" podStartSLOduration=1.773605128 podStartE2EDuration="1.773605128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:40:13.772685442 +0000 UTC m=+1502.077883618" watchObservedRunningTime="2026-07-01 17:40:13.773605128 +0000 UTC m=+1502.078803304"
Jul 01 17:40:14.574730 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:14.574701    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9"
Jul 01 17:40:14.757637 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:14.757607    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd"
Jul 01 17:40:14.759060 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:14.759029    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd" podUID="e25145e5-82ef-4c78-8e97-84a50628dc1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 01 17:40:15.466941 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:15.466902    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp" podUID="92a915eb-bc3c-4e50-8b6b-48f7de29d84f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 01 17:40:15.674981 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:15.674958    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp"
Jul 01 17:40:15.712503 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:15.712416    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92a915eb-bc3c-4e50-8b6b-48f7de29d84f-proxy-tls\") pod \"92a915eb-bc3c-4e50-8b6b-48f7de29d84f\" (UID: \"92a915eb-bc3c-4e50-8b6b-48f7de29d84f\") "
Jul 01 17:40:15.712503 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:15.712454    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzp7k\" (UniqueName: \"kubernetes.io/projected/92a915eb-bc3c-4e50-8b6b-48f7de29d84f-kube-api-access-lzp7k\") pod \"92a915eb-bc3c-4e50-8b6b-48f7de29d84f\" (UID: \"92a915eb-bc3c-4e50-8b6b-48f7de29d84f\") "
Jul 01 17:40:15.712503 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:15.712481    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92a915eb-bc3c-4e50-8b6b-48f7de29d84f-kube-rbac-proxy-sar-config\") pod \"92a915eb-bc3c-4e50-8b6b-48f7de29d84f\" (UID: \"92a915eb-bc3c-4e50-8b6b-48f7de29d84f\") "
Jul 01 17:40:15.712974 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:15.712941    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92a915eb-bc3c-4e50-8b6b-48f7de29d84f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "92a915eb-bc3c-4e50-8b6b-48f7de29d84f" (UID: "92a915eb-bc3c-4e50-8b6b-48f7de29d84f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:40:15.714751 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:15.714721    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92a915eb-bc3c-4e50-8b6b-48f7de29d84f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "92a915eb-bc3c-4e50-8b6b-48f7de29d84f" (UID: "92a915eb-bc3c-4e50-8b6b-48f7de29d84f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:40:15.714855 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:15.714776    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a915eb-bc3c-4e50-8b6b-48f7de29d84f-kube-api-access-lzp7k" (OuterVolumeSpecName: "kube-api-access-lzp7k") pod "92a915eb-bc3c-4e50-8b6b-48f7de29d84f" (UID: "92a915eb-bc3c-4e50-8b6b-48f7de29d84f"). InnerVolumeSpecName "kube-api-access-lzp7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:40:15.762886 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:15.762852    2585 generic.go:356] "Generic (PLEG): container finished" podID="92a915eb-bc3c-4e50-8b6b-48f7de29d84f" containerID="1021fba1e78c9ea3572cd246e48ee4a1e6001d7e6a90473c0e5927508416949f" exitCode=0
Jul 01 17:40:15.763045 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:15.762905    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp"
Jul 01 17:40:15.763045 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:15.762939    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp" event={"ID":"92a915eb-bc3c-4e50-8b6b-48f7de29d84f","Type":"ContainerDied","Data":"1021fba1e78c9ea3572cd246e48ee4a1e6001d7e6a90473c0e5927508416949f"}
Jul 01 17:40:15.763045 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:15.763009    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp" event={"ID":"92a915eb-bc3c-4e50-8b6b-48f7de29d84f","Type":"ContainerDied","Data":"714f29dfd13b0d4b836d2692a4d6939e5d9ac904a9408e136cbcd8e31eb0e37c"}
Jul 01 17:40:15.763179 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:15.763045    2585 scope.go:117] "RemoveContainer" containerID="4afd9d8aba9a4b1282dea1e9005dee241cfbb0d0bcbe9a9313ab850ae9ab5172"
Jul 01 17:40:15.763677 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:15.763643    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd" podUID="e25145e5-82ef-4c78-8e97-84a50628dc1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 01 17:40:15.771859 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:15.771841    2585 scope.go:117] "RemoveContainer" containerID="1021fba1e78c9ea3572cd246e48ee4a1e6001d7e6a90473c0e5927508416949f"
Jul 01 17:40:15.779266 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:15.779248    2585 scope.go:117] "RemoveContainer" containerID="4afd9d8aba9a4b1282dea1e9005dee241cfbb0d0bcbe9a9313ab850ae9ab5172"
Jul 01 17:40:15.779517 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:40:15.779498    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4afd9d8aba9a4b1282dea1e9005dee241cfbb0d0bcbe9a9313ab850ae9ab5172\": container with ID starting with 4afd9d8aba9a4b1282dea1e9005dee241cfbb0d0bcbe9a9313ab850ae9ab5172 not found: ID does not exist" containerID="4afd9d8aba9a4b1282dea1e9005dee241cfbb0d0bcbe9a9313ab850ae9ab5172"
Jul 01 17:40:15.779558 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:15.779526    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4afd9d8aba9a4b1282dea1e9005dee241cfbb0d0bcbe9a9313ab850ae9ab5172"} err="failed to get container status \"4afd9d8aba9a4b1282dea1e9005dee241cfbb0d0bcbe9a9313ab850ae9ab5172\": rpc error: code = NotFound desc = could not find container \"4afd9d8aba9a4b1282dea1e9005dee241cfbb0d0bcbe9a9313ab850ae9ab5172\": container with ID starting with 4afd9d8aba9a4b1282dea1e9005dee241cfbb0d0bcbe9a9313ab850ae9ab5172 not found: ID does not exist"
Jul 01 17:40:15.779558 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:15.779544    2585 scope.go:117] "RemoveContainer" containerID="1021fba1e78c9ea3572cd246e48ee4a1e6001d7e6a90473c0e5927508416949f"
Jul 01 17:40:15.779797 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:40:15.779781    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1021fba1e78c9ea3572cd246e48ee4a1e6001d7e6a90473c0e5927508416949f\": container with ID starting with 1021fba1e78c9ea3572cd246e48ee4a1e6001d7e6a90473c0e5927508416949f not found: ID does not exist" containerID="1021fba1e78c9ea3572cd246e48ee4a1e6001d7e6a90473c0e5927508416949f"
Jul 01 17:40:15.779849 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:15.779802    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1021fba1e78c9ea3572cd246e48ee4a1e6001d7e6a90473c0e5927508416949f"} err="failed to get container status \"1021fba1e78c9ea3572cd246e48ee4a1e6001d7e6a90473c0e5927508416949f\": rpc error: code = NotFound desc = could not find container \"1021fba1e78c9ea3572cd246e48ee4a1e6001d7e6a90473c0e5927508416949f\": container with ID starting with 1021fba1e78c9ea3572cd246e48ee4a1e6001d7e6a90473c0e5927508416949f not found: ID does not exist"
Jul 01 17:40:15.784812 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:15.784793    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp"]
Jul 01 17:40:15.788796 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:15.788778    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-98f1c-predictor-666bcdd5dd-nxjqp"]
Jul 01 17:40:15.813984 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:15.813955    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92a915eb-bc3c-4e50-8b6b-48f7de29d84f-proxy-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:40:15.813984 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:15.813980    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzp7k\" (UniqueName: \"kubernetes.io/projected/92a915eb-bc3c-4e50-8b6b-48f7de29d84f-kube-api-access-lzp7k\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:40:15.814184 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:15.813996    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92a915eb-bc3c-4e50-8b6b-48f7de29d84f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:40:16.172371 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:16.172337    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92a915eb-bc3c-4e50-8b6b-48f7de29d84f" path="/var/lib/kubelet/pods/92a915eb-bc3c-4e50-8b6b-48f7de29d84f/volumes"
Jul 01 17:40:18.713110 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:18.713063    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-98f1c-788bc79767-kkl6d" podUID="26e5d5c9-ef69-4bef-9383-df0e1e3de06a" containerName="sequence-graph-98f1c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:40:20.768531 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:20.768489    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd"
Jul 01 17:40:20.769164 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:20.769137    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd" podUID="e25145e5-82ef-4c78-8e97-84a50628dc1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 01 17:40:23.713103 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:23.713059    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-98f1c-788bc79767-kkl6d" podUID="26e5d5c9-ef69-4bef-9383-df0e1e3de06a" containerName="sequence-graph-98f1c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:40:23.713613 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:23.713176    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-98f1c-788bc79767-kkl6d"
Jul 01 17:40:26.524951 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:26.524909    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c4578-749655d7b7-hd9bg"]
Jul 01 17:40:26.525478 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:26.525354    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92a915eb-bc3c-4e50-8b6b-48f7de29d84f" containerName="kube-rbac-proxy"
Jul 01 17:40:26.525478 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:26.525372    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="92a915eb-bc3c-4e50-8b6b-48f7de29d84f" containerName="kube-rbac-proxy"
Jul 01 17:40:26.525478 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:26.525390    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92a915eb-bc3c-4e50-8b6b-48f7de29d84f" containerName="kserve-container"
Jul 01 17:40:26.525478 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:26.525399    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="92a915eb-bc3c-4e50-8b6b-48f7de29d84f" containerName="kserve-container"
Jul 01 17:40:26.525478 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:26.525472    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="92a915eb-bc3c-4e50-8b6b-48f7de29d84f" containerName="kube-rbac-proxy"
Jul 01 17:40:26.525762 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:26.525484    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="92a915eb-bc3c-4e50-8b6b-48f7de29d84f" containerName="kserve-container"
Jul 01 17:40:26.529969 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:26.529944    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-c4578-749655d7b7-hd9bg"
Jul 01 17:40:26.535989 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:26.535964    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-c4578-kube-rbac-proxy-sar-config\""
Jul 01 17:40:26.536175 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:26.536110    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-c4578-serving-cert\""
Jul 01 17:40:26.544618 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:26.544556    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c4578-749655d7b7-hd9bg"]
Jul 01 17:40:26.599378 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:26.599333    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4de8cd46-61b5-43cd-a3bb-dd661796db92-openshift-service-ca-bundle\") pod \"ensemble-graph-c4578-749655d7b7-hd9bg\" (UID: \"4de8cd46-61b5-43cd-a3bb-dd661796db92\") " pod="kserve-ci-e2e-test/ensemble-graph-c4578-749655d7b7-hd9bg"
Jul 01 17:40:26.599378 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:26.599380    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4de8cd46-61b5-43cd-a3bb-dd661796db92-proxy-tls\") pod \"ensemble-graph-c4578-749655d7b7-hd9bg\" (UID: \"4de8cd46-61b5-43cd-a3bb-dd661796db92\") " pod="kserve-ci-e2e-test/ensemble-graph-c4578-749655d7b7-hd9bg"
Jul 01 17:40:26.699984 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:26.699939    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4de8cd46-61b5-43cd-a3bb-dd661796db92-openshift-service-ca-bundle\") pod \"ensemble-graph-c4578-749655d7b7-hd9bg\" (UID: \"4de8cd46-61b5-43cd-a3bb-dd661796db92\") " pod="kserve-ci-e2e-test/ensemble-graph-c4578-749655d7b7-hd9bg"
Jul 01 17:40:26.699984 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:26.699987    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4de8cd46-61b5-43cd-a3bb-dd661796db92-proxy-tls\") pod \"ensemble-graph-c4578-749655d7b7-hd9bg\" (UID: \"4de8cd46-61b5-43cd-a3bb-dd661796db92\") " pod="kserve-ci-e2e-test/ensemble-graph-c4578-749655d7b7-hd9bg"
Jul 01 17:40:26.700211 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:40:26.700102    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-c4578-serving-cert: secret "ensemble-graph-c4578-serving-cert" not found
Jul 01 17:40:26.700211 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:40:26.700159    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4de8cd46-61b5-43cd-a3bb-dd661796db92-proxy-tls podName:4de8cd46-61b5-43cd-a3bb-dd661796db92 nodeName:}" failed. No retries permitted until 2026-07-01 17:40:27.200143627 +0000 UTC m=+1515.505341782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4de8cd46-61b5-43cd-a3bb-dd661796db92-proxy-tls") pod "ensemble-graph-c4578-749655d7b7-hd9bg" (UID: "4de8cd46-61b5-43cd-a3bb-dd661796db92") : secret "ensemble-graph-c4578-serving-cert" not found
Jul 01 17:40:26.700625 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:26.700607    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4de8cd46-61b5-43cd-a3bb-dd661796db92-openshift-service-ca-bundle\") pod \"ensemble-graph-c4578-749655d7b7-hd9bg\" (UID: \"4de8cd46-61b5-43cd-a3bb-dd661796db92\") " pod="kserve-ci-e2e-test/ensemble-graph-c4578-749655d7b7-hd9bg"
Jul 01 17:40:27.204917 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:27.204881    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4de8cd46-61b5-43cd-a3bb-dd661796db92-proxy-tls\") pod \"ensemble-graph-c4578-749655d7b7-hd9bg\" (UID: \"4de8cd46-61b5-43cd-a3bb-dd661796db92\") " pod="kserve-ci-e2e-test/ensemble-graph-c4578-749655d7b7-hd9bg"
Jul 01 17:40:27.207396 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:27.207371    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4de8cd46-61b5-43cd-a3bb-dd661796db92-proxy-tls\") pod \"ensemble-graph-c4578-749655d7b7-hd9bg\" (UID: \"4de8cd46-61b5-43cd-a3bb-dd661796db92\") " pod="kserve-ci-e2e-test/ensemble-graph-c4578-749655d7b7-hd9bg"
Jul 01 17:40:27.440479 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:27.440445    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-c4578-749655d7b7-hd9bg"
Jul 01 17:40:27.576097 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:27.576064    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c4578-749655d7b7-hd9bg"]
Jul 01 17:40:27.579153 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:40:27.579121    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4de8cd46_61b5_43cd_a3bb_dd661796db92.slice/crio-f8344df89eca13b8639cba082060cc98f5aaafeac37054be7c0f9c9891c4ade4 WatchSource:0}: Error finding container f8344df89eca13b8639cba082060cc98f5aaafeac37054be7c0f9c9891c4ade4: Status 404 returned error can't find the container with id f8344df89eca13b8639cba082060cc98f5aaafeac37054be7c0f9c9891c4ade4
Jul 01 17:40:27.810708 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:27.810667    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-c4578-749655d7b7-hd9bg" event={"ID":"4de8cd46-61b5-43cd-a3bb-dd661796db92","Type":"ContainerStarted","Data":"7f524009537cdc6502205a5774ffe99ca7d26a63bc7d320d08ff2c2e076cc0f2"}
Jul 01 17:40:27.810886 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:27.810727    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-c4578-749655d7b7-hd9bg" event={"ID":"4de8cd46-61b5-43cd-a3bb-dd661796db92","Type":"ContainerStarted","Data":"f8344df89eca13b8639cba082060cc98f5aaafeac37054be7c0f9c9891c4ade4"}
Jul 01 17:40:27.810886 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:27.810806    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-c4578-749655d7b7-hd9bg"
Jul 01 17:40:27.828138 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:27.828085    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-c4578-749655d7b7-hd9bg" podStartSLOduration=1.828067965 podStartE2EDuration="1.828067965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:40:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:40:27.827177233 +0000 UTC m=+1516.132375412" watchObservedRunningTime="2026-07-01 17:40:27.828067965 +0000 UTC m=+1516.133266141"
Jul 01 17:40:28.713311 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:28.713268    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-98f1c-788bc79767-kkl6d" podUID="26e5d5c9-ef69-4bef-9383-df0e1e3de06a" containerName="sequence-graph-98f1c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:40:30.769435 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:30.769386    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd" podUID="e25145e5-82ef-4c78-8e97-84a50628dc1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 01 17:40:33.713049 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:33.712961    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-98f1c-788bc79767-kkl6d" podUID="26e5d5c9-ef69-4bef-9383-df0e1e3de06a" containerName="sequence-graph-98f1c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:40:33.821470 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:33.821430    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-c4578-749655d7b7-hd9bg"
Jul 01 17:40:38.713430 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:38.713382    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-98f1c-788bc79767-kkl6d" podUID="26e5d5c9-ef69-4bef-9383-df0e1e3de06a" containerName="sequence-graph-98f1c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:40:40.769328 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:40.769291    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd" podUID="e25145e5-82ef-4c78-8e97-84a50628dc1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 01 17:40:42.703309 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:42.703282    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-98f1c-788bc79767-kkl6d"
Jul 01 17:40:42.831768 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:42.831669    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26e5d5c9-ef69-4bef-9383-df0e1e3de06a-openshift-service-ca-bundle\") pod \"26e5d5c9-ef69-4bef-9383-df0e1e3de06a\" (UID: \"26e5d5c9-ef69-4bef-9383-df0e1e3de06a\") "
Jul 01 17:40:42.831768 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:42.831739    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26e5d5c9-ef69-4bef-9383-df0e1e3de06a-proxy-tls\") pod \"26e5d5c9-ef69-4bef-9383-df0e1e3de06a\" (UID: \"26e5d5c9-ef69-4bef-9383-df0e1e3de06a\") "
Jul 01 17:40:42.832053 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:42.832026    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26e5d5c9-ef69-4bef-9383-df0e1e3de06a-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "26e5d5c9-ef69-4bef-9383-df0e1e3de06a" (UID: "26e5d5c9-ef69-4bef-9383-df0e1e3de06a"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:40:42.833925 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:42.833901    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26e5d5c9-ef69-4bef-9383-df0e1e3de06a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "26e5d5c9-ef69-4bef-9383-df0e1e3de06a" (UID: "26e5d5c9-ef69-4bef-9383-df0e1e3de06a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:40:42.871326 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:42.871283    2585 generic.go:356] "Generic (PLEG): container finished" podID="26e5d5c9-ef69-4bef-9383-df0e1e3de06a" containerID="d2deccf1e53e1f37e5bd1f856abd6003a1d6ef6dfbfe9b4db937fea399be9b42" exitCode=0
Jul 01 17:40:42.871326 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:42.871328    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-98f1c-788bc79767-kkl6d"
Jul 01 17:40:42.871512 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:42.871368    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-98f1c-788bc79767-kkl6d" event={"ID":"26e5d5c9-ef69-4bef-9383-df0e1e3de06a","Type":"ContainerDied","Data":"d2deccf1e53e1f37e5bd1f856abd6003a1d6ef6dfbfe9b4db937fea399be9b42"}
Jul 01 17:40:42.871512 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:42.871427    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-98f1c-788bc79767-kkl6d" event={"ID":"26e5d5c9-ef69-4bef-9383-df0e1e3de06a","Type":"ContainerDied","Data":"826a5891308e7d492e59f466a213218795550b07a4b1a7b42eb4c84df6ab94ec"}
Jul 01 17:40:42.871512 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:42.871449    2585 scope.go:117] "RemoveContainer" containerID="d2deccf1e53e1f37e5bd1f856abd6003a1d6ef6dfbfe9b4db937fea399be9b42"
Jul 01 17:40:42.882040 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:42.882015    2585 scope.go:117] "RemoveContainer" containerID="d2deccf1e53e1f37e5bd1f856abd6003a1d6ef6dfbfe9b4db937fea399be9b42"
Jul 01 17:40:42.882398 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:40:42.882379    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2deccf1e53e1f37e5bd1f856abd6003a1d6ef6dfbfe9b4db937fea399be9b42\": container with ID starting with d2deccf1e53e1f37e5bd1f856abd6003a1d6ef6dfbfe9b4db937fea399be9b42 not found: ID does not exist" containerID="d2deccf1e53e1f37e5bd1f856abd6003a1d6ef6dfbfe9b4db937fea399be9b42"
Jul 01 17:40:42.882529 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:42.882410    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2deccf1e53e1f37e5bd1f856abd6003a1d6ef6dfbfe9b4db937fea399be9b42"} err="failed to get container status \"d2deccf1e53e1f37e5bd1f856abd6003a1d6ef6dfbfe9b4db937fea399be9b42\": rpc error: code = NotFound desc = could not find container \"d2deccf1e53e1f37e5bd1f856abd6003a1d6ef6dfbfe9b4db937fea399be9b42\": container with ID starting with d2deccf1e53e1f37e5bd1f856abd6003a1d6ef6dfbfe9b4db937fea399be9b42 not found: ID does not exist"
Jul 01 17:40:42.893180 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:42.893150    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-98f1c-788bc79767-kkl6d"]
Jul 01 17:40:42.897020 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:42.897000    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-98f1c-788bc79767-kkl6d"]
Jul 01 17:40:42.932785 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:42.932750    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26e5d5c9-ef69-4bef-9383-df0e1e3de06a-proxy-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:40:42.932785 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:42.932779    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26e5d5c9-ef69-4bef-9383-df0e1e3de06a-openshift-service-ca-bundle\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:40:44.172315 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:44.172281    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26e5d5c9-ef69-4bef-9383-df0e1e3de06a" path="/var/lib/kubelet/pods/26e5d5c9-ef69-4bef-9383-df0e1e3de06a/volumes"
Jul 01 17:40:50.769250 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:40:50.769207    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd" podUID="e25145e5-82ef-4c78-8e97-84a50628dc1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 01 17:41:00.770141 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:41:00.770112    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd"
Jul 01 17:41:22.251291 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:41:22.251257    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-8b57f-6c5d78755-f6zxd"]
Jul 01 17:41:22.251741 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:41:22.251547    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26e5d5c9-ef69-4bef-9383-df0e1e3de06a" containerName="sequence-graph-98f1c"
Jul 01 17:41:22.251741 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:41:22.251557    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="26e5d5c9-ef69-4bef-9383-df0e1e3de06a" containerName="sequence-graph-98f1c"
Jul 01 17:41:22.251741 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:41:22.251632    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="26e5d5c9-ef69-4bef-9383-df0e1e3de06a" containerName="sequence-graph-98f1c"
Jul 01 17:41:22.255468 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:41:22.255447    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-8b57f-6c5d78755-f6zxd"
Jul 01 17:41:22.258136 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:41:22.258114    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-8b57f-kube-rbac-proxy-sar-config\""
Jul 01 17:41:22.258361 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:41:22.258345    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-8b57f-serving-cert\""
Jul 01 17:41:22.266934 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:41:22.266907    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-8b57f-6c5d78755-f6zxd"]
Jul 01 17:41:22.326647 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:41:22.326568    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b50d7a8a-f0ca-464f-8956-cd51ff1dd989-openshift-service-ca-bundle\") pod \"sequence-graph-8b57f-6c5d78755-f6zxd\" (UID: \"b50d7a8a-f0ca-464f-8956-cd51ff1dd989\") " pod="kserve-ci-e2e-test/sequence-graph-8b57f-6c5d78755-f6zxd"
Jul 01 17:41:22.326820 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:41:22.326656    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b50d7a8a-f0ca-464f-8956-cd51ff1dd989-proxy-tls\") pod \"sequence-graph-8b57f-6c5d78755-f6zxd\" (UID: \"b50d7a8a-f0ca-464f-8956-cd51ff1dd989\") " pod="kserve-ci-e2e-test/sequence-graph-8b57f-6c5d78755-f6zxd"
Jul 01 17:41:22.427649 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:41:22.427616    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b50d7a8a-f0ca-464f-8956-cd51ff1dd989-openshift-service-ca-bundle\") pod \"sequence-graph-8b57f-6c5d78755-f6zxd\" (UID: \"b50d7a8a-f0ca-464f-8956-cd51ff1dd989\") " pod="kserve-ci-e2e-test/sequence-graph-8b57f-6c5d78755-f6zxd"
Jul 01 17:41:22.427774 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:41:22.427658    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b50d7a8a-f0ca-464f-8956-cd51ff1dd989-proxy-tls\") pod \"sequence-graph-8b57f-6c5d78755-f6zxd\" (UID: \"b50d7a8a-f0ca-464f-8956-cd51ff1dd989\") " pod="kserve-ci-e2e-test/sequence-graph-8b57f-6c5d78755-f6zxd"
Jul 01 17:41:22.427811 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:41:22.427786    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-8b57f-serving-cert: secret "sequence-graph-8b57f-serving-cert" not found
Jul 01 17:41:22.427860 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:41:22.427851    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b50d7a8a-f0ca-464f-8956-cd51ff1dd989-proxy-tls podName:b50d7a8a-f0ca-464f-8956-cd51ff1dd989 nodeName:}" failed. No retries permitted until 2026-07-01 17:41:22.927833118 +0000 UTC m=+1571.233031273 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b50d7a8a-f0ca-464f-8956-cd51ff1dd989-proxy-tls") pod "sequence-graph-8b57f-6c5d78755-f6zxd" (UID: "b50d7a8a-f0ca-464f-8956-cd51ff1dd989") : secret "sequence-graph-8b57f-serving-cert" not found
Jul 01 17:41:22.428260 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:41:22.428239    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b50d7a8a-f0ca-464f-8956-cd51ff1dd989-openshift-service-ca-bundle\") pod \"sequence-graph-8b57f-6c5d78755-f6zxd\" (UID: \"b50d7a8a-f0ca-464f-8956-cd51ff1dd989\") " pod="kserve-ci-e2e-test/sequence-graph-8b57f-6c5d78755-f6zxd"
Jul 01 17:41:22.931400 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:41:22.931357    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b50d7a8a-f0ca-464f-8956-cd51ff1dd989-proxy-tls\") pod \"sequence-graph-8b57f-6c5d78755-f6zxd\" (UID: \"b50d7a8a-f0ca-464f-8956-cd51ff1dd989\") " pod="kserve-ci-e2e-test/sequence-graph-8b57f-6c5d78755-f6zxd"
Jul 01 17:41:22.933905 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:41:22.933871    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b50d7a8a-f0ca-464f-8956-cd51ff1dd989-proxy-tls\") pod \"sequence-graph-8b57f-6c5d78755-f6zxd\" (UID: \"b50d7a8a-f0ca-464f-8956-cd51ff1dd989\") " pod="kserve-ci-e2e-test/sequence-graph-8b57f-6c5d78755-f6zxd"
Jul 01 17:41:23.166020 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:41:23.165977    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-8b57f-6c5d78755-f6zxd"
Jul 01 17:41:23.301364 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:41:23.301209    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-8b57f-6c5d78755-f6zxd"]
Jul 01 17:41:23.304185 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:41:23.304154    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb50d7a8a_f0ca_464f_8956_cd51ff1dd989.slice/crio-c0f6642808999dfc6474699e2d7d649ccb54ba6025d5ec219aff5dffc7498d77 WatchSource:0}: Error finding container c0f6642808999dfc6474699e2d7d649ccb54ba6025d5ec219aff5dffc7498d77: Status 404 returned error can't find the container with id c0f6642808999dfc6474699e2d7d649ccb54ba6025d5ec219aff5dffc7498d77
Jul 01 17:41:24.034634 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:41:24.034597    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-8b57f-6c5d78755-f6zxd" event={"ID":"b50d7a8a-f0ca-464f-8956-cd51ff1dd989","Type":"ContainerStarted","Data":"84ba4146cf35b814c79657035f770a91799a629b1d35a61a3b7bf1f7560f4809"}
Jul 01 17:41:24.034816 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:41:24.034655    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-8b57f-6c5d78755-f6zxd" event={"ID":"b50d7a8a-f0ca-464f-8956-cd51ff1dd989","Type":"ContainerStarted","Data":"c0f6642808999dfc6474699e2d7d649ccb54ba6025d5ec219aff5dffc7498d77"}
Jul 01 17:41:24.034816 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:41:24.034739    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-8b57f-6c5d78755-f6zxd"
Jul 01 17:41:24.052720 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:41:24.052669    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-8b57f-6c5d78755-f6zxd" podStartSLOduration=2.052650748 podStartE2EDuration="2.052650748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:41:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:41:24.051080112 +0000 UTC m=+1572.356278288" watchObservedRunningTime="2026-07-01 17:41:24.052650748 +0000 UTC m=+1572.357848929"
Jul 01 17:41:30.044848 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:41:30.044819    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-8b57f-6c5d78755-f6zxd"
Jul 01 17:45:12.236735 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:45:12.236708    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8tsw_500d2a90-2d21-42b3-9584-b04c14becaec/ovn-acl-logging/0.log"
Jul 01 17:45:12.241456 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:45:12.241437    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8tsw_500d2a90-2d21-42b3-9584-b04c14becaec/ovn-acl-logging/0.log"
Jul 01 17:48:41.238989 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:41.238956    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c4578-749655d7b7-hd9bg"]
Jul 01 17:48:41.241513 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:41.239208    2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-c4578-749655d7b7-hd9bg" podUID="4de8cd46-61b5-43cd-a3bb-dd661796db92" containerName="ensemble-graph-c4578" containerID="cri-o://7f524009537cdc6502205a5774ffe99ca7d26a63bc7d320d08ff2c2e076cc0f2" gracePeriod=30
Jul 01 17:48:41.510821 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:41.510747    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9"]
Jul 01 17:48:41.513946 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:41.513930    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9"
Jul 01 17:48:41.517139 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:41.517116    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-10517-predictor-serving-cert\""
Jul 01 17:48:41.517248 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:41.517143    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-10517-kube-rbac-proxy-sar-config\""
Jul 01 17:48:41.523822 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:41.523798    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9"]
Jul 01 17:48:41.568769 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:41.568733    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9"]
Jul 01 17:48:41.569011 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:41.568980    2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9" podUID="87e604e1-c30c-47f6-bd5a-aab6e79f83da" containerName="kserve-container" containerID="cri-o://cd3f5950fb891a5c63968f4efdf676064b4a696e7d20733689d8a185b52676a2" gracePeriod=30
Jul 01 17:48:41.569147 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:41.569036    2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9" podUID="87e604e1-c30c-47f6-bd5a-aab6e79f83da" containerName="kube-rbac-proxy" containerID="cri-o://c8e5b334e77e25d94c798e5023990635683dc260020df5bbce8636e69c23e748" gracePeriod=30
Jul 01 17:48:41.592880 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:41.592853    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21db6b05-cc24-497d-83d1-595ccc04fc7b-proxy-tls\") pod \"error-404-isvc-10517-predictor-f6447f778-924c9\" (UID: \"21db6b05-cc24-497d-83d1-595ccc04fc7b\") " pod="kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9"
Jul 01 17:48:41.592993 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:41.592893    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21db6b05-cc24-497d-83d1-595ccc04fc7b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-10517-predictor-f6447f778-924c9\" (UID: \"21db6b05-cc24-497d-83d1-595ccc04fc7b\") " pod="kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9"
Jul 01 17:48:41.592993 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:41.592946    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz7vm\" (UniqueName: \"kubernetes.io/projected/21db6b05-cc24-497d-83d1-595ccc04fc7b-kube-api-access-rz7vm\") pod \"error-404-isvc-10517-predictor-f6447f778-924c9\" (UID: \"21db6b05-cc24-497d-83d1-595ccc04fc7b\") " pod="kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9"
Jul 01 17:48:41.694302 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:41.694272    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz7vm\" (UniqueName: \"kubernetes.io/projected/21db6b05-cc24-497d-83d1-595ccc04fc7b-kube-api-access-rz7vm\") pod \"error-404-isvc-10517-predictor-f6447f778-924c9\" (UID: \"21db6b05-cc24-497d-83d1-595ccc04fc7b\") " pod="kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9"
Jul 01 17:48:41.694435 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:41.694357    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21db6b05-cc24-497d-83d1-595ccc04fc7b-proxy-tls\") pod \"error-404-isvc-10517-predictor-f6447f778-924c9\" (UID: \"21db6b05-cc24-497d-83d1-595ccc04fc7b\") " pod="kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9"
Jul 01 17:48:41.694435 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:41.694390    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21db6b05-cc24-497d-83d1-595ccc04fc7b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-10517-predictor-f6447f778-924c9\" (UID: \"21db6b05-cc24-497d-83d1-595ccc04fc7b\") " pod="kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9"
Jul 01 17:48:41.695009 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:41.694986    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21db6b05-cc24-497d-83d1-595ccc04fc7b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-10517-predictor-f6447f778-924c9\" (UID: \"21db6b05-cc24-497d-83d1-595ccc04fc7b\") " pod="kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9"
Jul 01 17:48:41.696826 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:41.696805    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21db6b05-cc24-497d-83d1-595ccc04fc7b-proxy-tls\") pod \"error-404-isvc-10517-predictor-f6447f778-924c9\" (UID: \"21db6b05-cc24-497d-83d1-595ccc04fc7b\") " pod="kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9"
Jul 01 17:48:41.702732 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:41.702710    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz7vm\" (UniqueName: \"kubernetes.io/projected/21db6b05-cc24-497d-83d1-595ccc04fc7b-kube-api-access-rz7vm\") pod \"error-404-isvc-10517-predictor-f6447f778-924c9\" (UID: \"21db6b05-cc24-497d-83d1-595ccc04fc7b\") " pod="kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9"
Jul 01 17:48:41.767296 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:41.767207    2585 generic.go:356] "Generic (PLEG): container finished" podID="87e604e1-c30c-47f6-bd5a-aab6e79f83da" containerID="c8e5b334e77e25d94c798e5023990635683dc260020df5bbce8636e69c23e748" exitCode=2
Jul 01 17:48:41.767428 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:41.767293    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9" event={"ID":"87e604e1-c30c-47f6-bd5a-aab6e79f83da","Type":"ContainerDied","Data":"c8e5b334e77e25d94c798e5023990635683dc260020df5bbce8636e69c23e748"}
Jul 01 17:48:41.824183 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:41.824150    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9"
Jul 01 17:48:41.954820 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:41.954753    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9"]
Jul 01 17:48:41.957179 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:48:41.957147    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21db6b05_cc24_497d_83d1_595ccc04fc7b.slice/crio-9c212112f9421c91cbf527d33028784ef76a25a4933c95a90315612a6c707ac3 WatchSource:0}: Error finding container 9c212112f9421c91cbf527d33028784ef76a25a4933c95a90315612a6c707ac3: Status 404 returned error can't find the container with id 9c212112f9421c91cbf527d33028784ef76a25a4933c95a90315612a6c707ac3
Jul 01 17:48:42.772595 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:42.772542    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9" event={"ID":"21db6b05-cc24-497d-83d1-595ccc04fc7b","Type":"ContainerStarted","Data":"2f3bb71499ca634e412d9e1102046ebb36e9b254d13fcc981a61eff41477c5ae"}
Jul 01 17:48:42.773055 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:42.772625    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9" event={"ID":"21db6b05-cc24-497d-83d1-595ccc04fc7b","Type":"ContainerStarted","Data":"db47f2b9e7f572c13210a2aad2f5587613f62e4c97858983de306e2abdf3a2f4"}
Jul 01 17:48:42.773055 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:42.772651    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9" event={"ID":"21db6b05-cc24-497d-83d1-595ccc04fc7b","Type":"ContainerStarted","Data":"9c212112f9421c91cbf527d33028784ef76a25a4933c95a90315612a6c707ac3"}
Jul 01 17:48:42.773055 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:42.772674    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9"
Jul 01 17:48:42.773055 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:42.772718    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9"
Jul 01 17:48:42.774197 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:42.774170    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9" podUID="21db6b05-cc24-497d-83d1-595ccc04fc7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 01 17:48:42.790320 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:42.790267    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9" podStartSLOduration=1.790253634 podStartE2EDuration="1.790253634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:48:42.788332289 +0000 UTC m=+2011.093530466" watchObservedRunningTime="2026-07-01 17:48:42.790253634 +0000 UTC m=+2011.095451810"
Jul 01 17:48:43.776972 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:43.776938    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9" podUID="21db6b05-cc24-497d-83d1-595ccc04fc7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 01 17:48:43.818619 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:43.818561    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c4578-749655d7b7-hd9bg" podUID="4de8cd46-61b5-43cd-a3bb-dd661796db92" containerName="ensemble-graph-c4578" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:48:44.569824 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:44.569788    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9" podUID="87e604e1-c30c-47f6-bd5a-aab6e79f83da" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Jul 01 17:48:44.574521 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:44.574479    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9" podUID="87e604e1-c30c-47f6-bd5a-aab6e79f83da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 01 17:48:44.718762 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:44.718732    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9"
Jul 01 17:48:44.781727 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:44.781680    2585 generic.go:356] "Generic (PLEG): container finished" podID="87e604e1-c30c-47f6-bd5a-aab6e79f83da" containerID="cd3f5950fb891a5c63968f4efdf676064b4a696e7d20733689d8a185b52676a2" exitCode=0
Jul 01 17:48:44.782177 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:44.781736    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9"
Jul 01 17:48:44.782177 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:44.781761    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9" event={"ID":"87e604e1-c30c-47f6-bd5a-aab6e79f83da","Type":"ContainerDied","Data":"cd3f5950fb891a5c63968f4efdf676064b4a696e7d20733689d8a185b52676a2"}
Jul 01 17:48:44.782177 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:44.781819    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9" event={"ID":"87e604e1-c30c-47f6-bd5a-aab6e79f83da","Type":"ContainerDied","Data":"c210e9c15ff02a25ff40ad2c6c06bd6101585c3a711910ca8548f9f347cdc3d5"}
Jul 01 17:48:44.782177 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:44.781845    2585 scope.go:117] "RemoveContainer" containerID="c8e5b334e77e25d94c798e5023990635683dc260020df5bbce8636e69c23e748"
Jul 01 17:48:44.789645 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:44.789630    2585 scope.go:117] "RemoveContainer" containerID="cd3f5950fb891a5c63968f4efdf676064b4a696e7d20733689d8a185b52676a2"
Jul 01 17:48:44.796631 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:44.796615    2585 scope.go:117] "RemoveContainer" containerID="c8e5b334e77e25d94c798e5023990635683dc260020df5bbce8636e69c23e748"
Jul 01 17:48:44.796939 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:48:44.796923    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8e5b334e77e25d94c798e5023990635683dc260020df5bbce8636e69c23e748\": container with ID starting with c8e5b334e77e25d94c798e5023990635683dc260020df5bbce8636e69c23e748 not found: ID does not exist" containerID="c8e5b334e77e25d94c798e5023990635683dc260020df5bbce8636e69c23e748"
Jul 01 17:48:44.796994 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:44.796949    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8e5b334e77e25d94c798e5023990635683dc260020df5bbce8636e69c23e748"} err="failed to get container status \"c8e5b334e77e25d94c798e5023990635683dc260020df5bbce8636e69c23e748\": rpc error: code = NotFound desc = could not find container \"c8e5b334e77e25d94c798e5023990635683dc260020df5bbce8636e69c23e748\": container with ID starting with c8e5b334e77e25d94c798e5023990635683dc260020df5bbce8636e69c23e748 not found: ID does not exist"
Jul 01 17:48:44.796994 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:44.796966    2585 scope.go:117] "RemoveContainer" containerID="cd3f5950fb891a5c63968f4efdf676064b4a696e7d20733689d8a185b52676a2"
Jul 01 17:48:44.797197 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:48:44.797182    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd3f5950fb891a5c63968f4efdf676064b4a696e7d20733689d8a185b52676a2\": container with ID starting with cd3f5950fb891a5c63968f4efdf676064b4a696e7d20733689d8a185b52676a2 not found: ID does not exist" containerID="cd3f5950fb891a5c63968f4efdf676064b4a696e7d20733689d8a185b52676a2"
Jul 01 17:48:44.797243 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:44.797200    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd3f5950fb891a5c63968f4efdf676064b4a696e7d20733689d8a185b52676a2"} err="failed to get container status \"cd3f5950fb891a5c63968f4efdf676064b4a696e7d20733689d8a185b52676a2\": rpc error: code = NotFound desc = could not find container \"cd3f5950fb891a5c63968f4efdf676064b4a696e7d20733689d8a185b52676a2\": container with ID starting with cd3f5950fb891a5c63968f4efdf676064b4a696e7d20733689d8a185b52676a2 not found: ID does not exist"
Jul 01 17:48:44.820504 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:44.820464    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87e604e1-c30c-47f6-bd5a-aab6e79f83da-kube-rbac-proxy-sar-config\") pod \"87e604e1-c30c-47f6-bd5a-aab6e79f83da\" (UID: \"87e604e1-c30c-47f6-bd5a-aab6e79f83da\") "
Jul 01 17:48:44.820628 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:44.820521    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87e604e1-c30c-47f6-bd5a-aab6e79f83da-proxy-tls\") pod \"87e604e1-c30c-47f6-bd5a-aab6e79f83da\" (UID: \"87e604e1-c30c-47f6-bd5a-aab6e79f83da\") "
Jul 01 17:48:44.820628 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:44.820595    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22fs9\" (UniqueName: \"kubernetes.io/projected/87e604e1-c30c-47f6-bd5a-aab6e79f83da-kube-api-access-22fs9\") pod \"87e604e1-c30c-47f6-bd5a-aab6e79f83da\" (UID: \"87e604e1-c30c-47f6-bd5a-aab6e79f83da\") "
Jul 01 17:48:44.820831 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:44.820800    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87e604e1-c30c-47f6-bd5a-aab6e79f83da-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "87e604e1-c30c-47f6-bd5a-aab6e79f83da" (UID: "87e604e1-c30c-47f6-bd5a-aab6e79f83da"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:48:44.822625 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:44.822605    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87e604e1-c30c-47f6-bd5a-aab6e79f83da-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "87e604e1-c30c-47f6-bd5a-aab6e79f83da" (UID: "87e604e1-c30c-47f6-bd5a-aab6e79f83da"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:48:44.822758 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:44.822740    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e604e1-c30c-47f6-bd5a-aab6e79f83da-kube-api-access-22fs9" (OuterVolumeSpecName: "kube-api-access-22fs9") pod "87e604e1-c30c-47f6-bd5a-aab6e79f83da" (UID: "87e604e1-c30c-47f6-bd5a-aab6e79f83da"). InnerVolumeSpecName "kube-api-access-22fs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:48:44.922073 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:44.922026    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22fs9\" (UniqueName: \"kubernetes.io/projected/87e604e1-c30c-47f6-bd5a-aab6e79f83da-kube-api-access-22fs9\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:48:44.922073 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:44.922068    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87e604e1-c30c-47f6-bd5a-aab6e79f83da-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:48:44.922073 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:44.922078    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87e604e1-c30c-47f6-bd5a-aab6e79f83da-proxy-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:48:45.104214 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:45.104182    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9"]
Jul 01 17:48:45.108280 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:45.108257    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c4578-predictor-8585647f56-4hsc9"]
Jul 01 17:48:46.172683 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:46.172645    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87e604e1-c30c-47f6-bd5a-aab6e79f83da" path="/var/lib/kubelet/pods/87e604e1-c30c-47f6-bd5a-aab6e79f83da/volumes"
Jul 01 17:48:48.780974 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:48.780948    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9"
Jul 01 17:48:48.781390 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:48.781333    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9" podUID="21db6b05-cc24-497d-83d1-595ccc04fc7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 01 17:48:48.818828 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:48.818791    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c4578-749655d7b7-hd9bg" podUID="4de8cd46-61b5-43cd-a3bb-dd661796db92" containerName="ensemble-graph-c4578" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:48:53.818194 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:53.818159    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c4578-749655d7b7-hd9bg" podUID="4de8cd46-61b5-43cd-a3bb-dd661796db92" containerName="ensemble-graph-c4578" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:48:53.818664 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:53.818263    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-c4578-749655d7b7-hd9bg"
Jul 01 17:48:58.782026 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:58.781988    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9" podUID="21db6b05-cc24-497d-83d1-595ccc04fc7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 01 17:48:58.818443 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:48:58.818408    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c4578-749655d7b7-hd9bg" podUID="4de8cd46-61b5-43cd-a3bb-dd661796db92" containerName="ensemble-graph-c4578" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:49:03.818165 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:03.818124    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c4578-749655d7b7-hd9bg" podUID="4de8cd46-61b5-43cd-a3bb-dd661796db92" containerName="ensemble-graph-c4578" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:49:08.781493 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:08.781450    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9" podUID="21db6b05-cc24-497d-83d1-595ccc04fc7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 01 17:49:08.818816 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:08.818777    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c4578-749655d7b7-hd9bg" podUID="4de8cd46-61b5-43cd-a3bb-dd661796db92" containerName="ensemble-graph-c4578" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:49:11.880532 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:11.880506    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-c4578-749655d7b7-hd9bg"
Jul 01 17:49:11.893323 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:11.893294    2585 generic.go:356] "Generic (PLEG): container finished" podID="4de8cd46-61b5-43cd-a3bb-dd661796db92" containerID="7f524009537cdc6502205a5774ffe99ca7d26a63bc7d320d08ff2c2e076cc0f2" exitCode=0
Jul 01 17:49:11.893437 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:11.893334    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-c4578-749655d7b7-hd9bg"
Jul 01 17:49:11.893437 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:11.893368    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-c4578-749655d7b7-hd9bg" event={"ID":"4de8cd46-61b5-43cd-a3bb-dd661796db92","Type":"ContainerDied","Data":"7f524009537cdc6502205a5774ffe99ca7d26a63bc7d320d08ff2c2e076cc0f2"}
Jul 01 17:49:11.893437 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:11.893406    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-c4578-749655d7b7-hd9bg" event={"ID":"4de8cd46-61b5-43cd-a3bb-dd661796db92","Type":"ContainerDied","Data":"f8344df89eca13b8639cba082060cc98f5aaafeac37054be7c0f9c9891c4ade4"}
Jul 01 17:49:11.893437 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:11.893426    2585 scope.go:117] "RemoveContainer" containerID="7f524009537cdc6502205a5774ffe99ca7d26a63bc7d320d08ff2c2e076cc0f2"
Jul 01 17:49:11.902163 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:11.902141    2585 scope.go:117] "RemoveContainer" containerID="7f524009537cdc6502205a5774ffe99ca7d26a63bc7d320d08ff2c2e076cc0f2"
Jul 01 17:49:11.902522 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:49:11.902500    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f524009537cdc6502205a5774ffe99ca7d26a63bc7d320d08ff2c2e076cc0f2\": container with ID starting with 7f524009537cdc6502205a5774ffe99ca7d26a63bc7d320d08ff2c2e076cc0f2 not found: ID does not exist" containerID="7f524009537cdc6502205a5774ffe99ca7d26a63bc7d320d08ff2c2e076cc0f2"
Jul 01 17:49:11.902622 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:11.902531    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f524009537cdc6502205a5774ffe99ca7d26a63bc7d320d08ff2c2e076cc0f2"} err="failed to get container status \"7f524009537cdc6502205a5774ffe99ca7d26a63bc7d320d08ff2c2e076cc0f2\": rpc error: code = NotFound desc = could not find container \"7f524009537cdc6502205a5774ffe99ca7d26a63bc7d320d08ff2c2e076cc0f2\": container with ID starting with 7f524009537cdc6502205a5774ffe99ca7d26a63bc7d320d08ff2c2e076cc0f2 not found: ID does not exist"
Jul 01 17:49:11.927846 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:11.927809    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4de8cd46-61b5-43cd-a3bb-dd661796db92-openshift-service-ca-bundle\") pod \"4de8cd46-61b5-43cd-a3bb-dd661796db92\" (UID: \"4de8cd46-61b5-43cd-a3bb-dd661796db92\") "
Jul 01 17:49:11.928016 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:11.927877    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4de8cd46-61b5-43cd-a3bb-dd661796db92-proxy-tls\") pod \"4de8cd46-61b5-43cd-a3bb-dd661796db92\" (UID: \"4de8cd46-61b5-43cd-a3bb-dd661796db92\") "
Jul 01 17:49:11.928149 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:11.928124    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4de8cd46-61b5-43cd-a3bb-dd661796db92-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "4de8cd46-61b5-43cd-a3bb-dd661796db92" (UID: "4de8cd46-61b5-43cd-a3bb-dd661796db92"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:49:11.930063 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:11.930037    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4de8cd46-61b5-43cd-a3bb-dd661796db92-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4de8cd46-61b5-43cd-a3bb-dd661796db92" (UID: "4de8cd46-61b5-43cd-a3bb-dd661796db92"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:49:12.028323 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:12.028238    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4de8cd46-61b5-43cd-a3bb-dd661796db92-proxy-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:49:12.028323 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:12.028267    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4de8cd46-61b5-43cd-a3bb-dd661796db92-openshift-service-ca-bundle\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:49:12.209308 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:12.209262    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c4578-749655d7b7-hd9bg"]
Jul 01 17:49:12.212796 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:12.212767    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c4578-749655d7b7-hd9bg"]
Jul 01 17:49:14.172746 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:14.172710    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4de8cd46-61b5-43cd-a3bb-dd661796db92" path="/var/lib/kubelet/pods/4de8cd46-61b5-43cd-a3bb-dd661796db92/volumes"
Jul 01 17:49:18.782249 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:18.782203    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9" podUID="21db6b05-cc24-497d-83d1-595ccc04fc7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 01 17:49:28.782794 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:28.782760    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9"
Jul 01 17:49:36.964161 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:36.964121    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-8b57f-6c5d78755-f6zxd"]
Jul 01 17:49:36.964784 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:36.964751    2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-8b57f-6c5d78755-f6zxd" podUID="b50d7a8a-f0ca-464f-8956-cd51ff1dd989" containerName="sequence-graph-8b57f" containerID="cri-o://84ba4146cf35b814c79657035f770a91799a629b1d35a61a3b7bf1f7560f4809" gracePeriod=30
Jul 01 17:49:37.143825 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:37.143790    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd"]
Jul 01 17:49:37.144074 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:37.144049    2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd" podUID="e25145e5-82ef-4c78-8e97-84a50628dc1a" containerName="kube-rbac-proxy" containerID="cri-o://9fcf1091a6730580c9d6a6a57b1f66864da79980f9d8cc59d0382edc92270f3c" gracePeriod=30
Jul 01 17:49:37.144189 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:37.144022    2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd" podUID="e25145e5-82ef-4c78-8e97-84a50628dc1a" containerName="kserve-container" containerID="cri-o://dc801cb4bb26d4f2f067422b94dbfb418b733d718021de4ccf99c97b475e3c12" gracePeriod=30
Jul 01 17:49:37.213326 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:37.212878    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56"]
Jul 01 17:49:37.213326 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:37.213299    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87e604e1-c30c-47f6-bd5a-aab6e79f83da" containerName="kserve-container"
Jul 01 17:49:37.213326 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:37.213318    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="87e604e1-c30c-47f6-bd5a-aab6e79f83da" containerName="kserve-container"
Jul 01 17:49:37.213326 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:37.213335    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87e604e1-c30c-47f6-bd5a-aab6e79f83da" containerName="kube-rbac-proxy"
Jul 01 17:49:37.213629 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:37.213342    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="87e604e1-c30c-47f6-bd5a-aab6e79f83da" containerName="kube-rbac-proxy"
Jul 01 17:49:37.213629 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:37.213369    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4de8cd46-61b5-43cd-a3bb-dd661796db92" containerName="ensemble-graph-c4578"
Jul 01 17:49:37.213629 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:37.213377    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="4de8cd46-61b5-43cd-a3bb-dd661796db92" containerName="ensemble-graph-c4578"
Jul 01 17:49:37.213629 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:37.213448    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="4de8cd46-61b5-43cd-a3bb-dd661796db92" containerName="ensemble-graph-c4578"
Jul 01 17:49:37.213629 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:37.213459    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="87e604e1-c30c-47f6-bd5a-aab6e79f83da" containerName="kube-rbac-proxy"
Jul 01 17:49:37.213629 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:37.213472    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="87e604e1-c30c-47f6-bd5a-aab6e79f83da" containerName="kserve-container"
Jul 01 17:49:37.216811 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:37.216757    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56"
Jul 01 17:49:37.219432 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:37.219408    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4822a-predictor-serving-cert\""
Jul 01 17:49:37.219555 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:37.219437    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4822a-kube-rbac-proxy-sar-config\""
Jul 01 17:49:37.225190 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:37.225167    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56"]
Jul 01 17:49:37.318368 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:37.318329    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86b90960-cd85-4aa7-b01e-100c95520755-proxy-tls\") pod \"error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56\" (UID: \"86b90960-cd85-4aa7-b01e-100c95520755\") " pod="kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56"
Jul 01 17:49:37.318539 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:37.318382    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86b90960-cd85-4aa7-b01e-100c95520755-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56\" (UID: \"86b90960-cd85-4aa7-b01e-100c95520755\") " pod="kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56"
Jul 01 17:49:37.318539 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:37.318418    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c57sx\" (UniqueName: \"kubernetes.io/projected/86b90960-cd85-4aa7-b01e-100c95520755-kube-api-access-c57sx\") pod \"error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56\" (UID: \"86b90960-cd85-4aa7-b01e-100c95520755\") " pod="kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56"
Jul 01 17:49:37.419244 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:37.419207    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86b90960-cd85-4aa7-b01e-100c95520755-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56\" (UID: \"86b90960-cd85-4aa7-b01e-100c95520755\") " pod="kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56"
Jul 01 17:49:37.419244 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:37.419246    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c57sx\" (UniqueName: \"kubernetes.io/projected/86b90960-cd85-4aa7-b01e-100c95520755-kube-api-access-c57sx\") pod \"error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56\" (UID: \"86b90960-cd85-4aa7-b01e-100c95520755\") " pod="kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56"
Jul 01 17:49:37.419457 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:37.419284    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86b90960-cd85-4aa7-b01e-100c95520755-proxy-tls\") pod \"error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56\" (UID: \"86b90960-cd85-4aa7-b01e-100c95520755\") " pod="kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56"
Jul 01 17:49:37.419457 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:49:37.419383    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-4822a-predictor-serving-cert: secret "error-404-isvc-4822a-predictor-serving-cert" not found
Jul 01 17:49:37.419457 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:49:37.419450    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86b90960-cd85-4aa7-b01e-100c95520755-proxy-tls podName:86b90960-cd85-4aa7-b01e-100c95520755 nodeName:}" failed. No retries permitted until 2026-07-01 17:49:37.919433187 +0000 UTC m=+2066.224631348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/86b90960-cd85-4aa7-b01e-100c95520755-proxy-tls") pod "error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56" (UID: "86b90960-cd85-4aa7-b01e-100c95520755") : secret "error-404-isvc-4822a-predictor-serving-cert" not found
Jul 01 17:49:37.419921 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:37.419903    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86b90960-cd85-4aa7-b01e-100c95520755-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56\" (UID: \"86b90960-cd85-4aa7-b01e-100c95520755\") " pod="kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56"
Jul 01 17:49:37.428333 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:37.428308    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c57sx\" (UniqueName: \"kubernetes.io/projected/86b90960-cd85-4aa7-b01e-100c95520755-kube-api-access-c57sx\") pod \"error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56\" (UID: \"86b90960-cd85-4aa7-b01e-100c95520755\") " pod="kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56"
Jul 01 17:49:37.924471 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:37.924425    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86b90960-cd85-4aa7-b01e-100c95520755-proxy-tls\") pod \"error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56\" (UID: \"86b90960-cd85-4aa7-b01e-100c95520755\") " pod="kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56"
Jul 01 17:49:37.927016 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:37.926988    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86b90960-cd85-4aa7-b01e-100c95520755-proxy-tls\") pod \"error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56\" (UID: \"86b90960-cd85-4aa7-b01e-100c95520755\") " pod="kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56"
Jul 01 17:49:37.998457 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:37.998421    2585 generic.go:356] "Generic (PLEG): container finished" podID="e25145e5-82ef-4c78-8e97-84a50628dc1a" containerID="9fcf1091a6730580c9d6a6a57b1f66864da79980f9d8cc59d0382edc92270f3c" exitCode=2
Jul 01 17:49:37.998875 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:37.998477    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd" event={"ID":"e25145e5-82ef-4c78-8e97-84a50628dc1a","Type":"ContainerDied","Data":"9fcf1091a6730580c9d6a6a57b1f66864da79980f9d8cc59d0382edc92270f3c"}
Jul 01 17:49:38.127155 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:38.127115    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56"
Jul 01 17:49:38.259236 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:38.259191    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56"]
Jul 01 17:49:38.262080 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:49:38.262055    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86b90960_cd85_4aa7_b01e_100c95520755.slice/crio-0d8b25cf15aa12d20b197c4c781d10ebc733ea8cc9d628498c76e41104f162c2 WatchSource:0}: Error finding container 0d8b25cf15aa12d20b197c4c781d10ebc733ea8cc9d628498c76e41104f162c2: Status 404 returned error can't find the container with id 0d8b25cf15aa12d20b197c4c781d10ebc733ea8cc9d628498c76e41104f162c2
Jul 01 17:49:39.004107 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:39.004066    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56" event={"ID":"86b90960-cd85-4aa7-b01e-100c95520755","Type":"ContainerStarted","Data":"5c65d0f34ecd86538451867b4f5c7c27de67806a88ffb2c968fb0ab09438820b"}
Jul 01 17:49:39.004482 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:39.004140    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56" event={"ID":"86b90960-cd85-4aa7-b01e-100c95520755","Type":"ContainerStarted","Data":"154b075b0860d4abbcccd454f111b9fe64bc1c586004ed8cb8c5850c52331f8d"}
Jul 01 17:49:39.004482 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:39.004167    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56" event={"ID":"86b90960-cd85-4aa7-b01e-100c95520755","Type":"ContainerStarted","Data":"0d8b25cf15aa12d20b197c4c781d10ebc733ea8cc9d628498c76e41104f162c2"}
Jul 01 17:49:39.004482 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:39.004197    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56"
Jul 01 17:49:39.023470 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:39.023421    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56" podStartSLOduration=2.02340531 podStartE2EDuration="2.02340531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:49:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:49:39.021750389 +0000 UTC m=+2067.326948565" watchObservedRunningTime="2026-07-01 17:49:39.02340531 +0000 UTC m=+2067.328603487"
Jul 01 17:49:40.008209 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:40.008166    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56"
Jul 01 17:49:40.009495 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:40.009472    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56" podUID="86b90960-cd85-4aa7-b01e-100c95520755" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 01 17:49:40.043171 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:40.043138    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-8b57f-6c5d78755-f6zxd" podUID="b50d7a8a-f0ca-464f-8956-cd51ff1dd989" containerName="sequence-graph-8b57f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:49:40.391298 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:40.391273    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd"
Jul 01 17:49:40.446205 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:40.446168    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e25145e5-82ef-4c78-8e97-84a50628dc1a-proxy-tls\") pod \"e25145e5-82ef-4c78-8e97-84a50628dc1a\" (UID: \"e25145e5-82ef-4c78-8e97-84a50628dc1a\") "
Jul 01 17:49:40.446205 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:40.446211    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e25145e5-82ef-4c78-8e97-84a50628dc1a-kube-rbac-proxy-sar-config\") pod \"e25145e5-82ef-4c78-8e97-84a50628dc1a\" (UID: \"e25145e5-82ef-4c78-8e97-84a50628dc1a\") "
Jul 01 17:49:40.446427 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:40.446265    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55hlk\" (UniqueName: \"kubernetes.io/projected/e25145e5-82ef-4c78-8e97-84a50628dc1a-kube-api-access-55hlk\") pod \"e25145e5-82ef-4c78-8e97-84a50628dc1a\" (UID: \"e25145e5-82ef-4c78-8e97-84a50628dc1a\") "
Jul 01 17:49:40.446687 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:40.446656    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e25145e5-82ef-4c78-8e97-84a50628dc1a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e25145e5-82ef-4c78-8e97-84a50628dc1a" (UID: "e25145e5-82ef-4c78-8e97-84a50628dc1a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:49:40.448487 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:40.448467    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e25145e5-82ef-4c78-8e97-84a50628dc1a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e25145e5-82ef-4c78-8e97-84a50628dc1a" (UID: "e25145e5-82ef-4c78-8e97-84a50628dc1a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:49:40.448599 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:40.448569    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e25145e5-82ef-4c78-8e97-84a50628dc1a-kube-api-access-55hlk" (OuterVolumeSpecName: "kube-api-access-55hlk") pod "e25145e5-82ef-4c78-8e97-84a50628dc1a" (UID: "e25145e5-82ef-4c78-8e97-84a50628dc1a"). InnerVolumeSpecName "kube-api-access-55hlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:49:40.547428 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:40.547389    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55hlk\" (UniqueName: \"kubernetes.io/projected/e25145e5-82ef-4c78-8e97-84a50628dc1a-kube-api-access-55hlk\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:49:40.547428 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:40.547420    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e25145e5-82ef-4c78-8e97-84a50628dc1a-proxy-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:49:40.547428 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:40.547430    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e25145e5-82ef-4c78-8e97-84a50628dc1a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:49:41.014109 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:41.014073    2585 generic.go:356] "Generic (PLEG): container finished" podID="e25145e5-82ef-4c78-8e97-84a50628dc1a" containerID="dc801cb4bb26d4f2f067422b94dbfb418b733d718021de4ccf99c97b475e3c12" exitCode=0
Jul 01 17:49:41.014506 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:41.014122    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd"
Jul 01 17:49:41.014506 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:41.014158    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd" event={"ID":"e25145e5-82ef-4c78-8e97-84a50628dc1a","Type":"ContainerDied","Data":"dc801cb4bb26d4f2f067422b94dbfb418b733d718021de4ccf99c97b475e3c12"}
Jul 01 17:49:41.014506 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:41.014214    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd" event={"ID":"e25145e5-82ef-4c78-8e97-84a50628dc1a","Type":"ContainerDied","Data":"2feaabed9bb07f43f9ee11674aabddff9d548b4279e34f030c02c7884fb72b27"}
Jul 01 17:49:41.014506 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:41.014244    2585 scope.go:117] "RemoveContainer" containerID="9fcf1091a6730580c9d6a6a57b1f66864da79980f9d8cc59d0382edc92270f3c"
Jul 01 17:49:41.014742 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:41.014724    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56" podUID="86b90960-cd85-4aa7-b01e-100c95520755" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 01 17:49:41.022801 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:41.022786    2585 scope.go:117] "RemoveContainer" containerID="dc801cb4bb26d4f2f067422b94dbfb418b733d718021de4ccf99c97b475e3c12"
Jul 01 17:49:41.030060 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:41.030042    2585 scope.go:117] "RemoveContainer" containerID="9fcf1091a6730580c9d6a6a57b1f66864da79980f9d8cc59d0382edc92270f3c"
Jul 01 17:49:41.030301 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:49:41.030285    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fcf1091a6730580c9d6a6a57b1f66864da79980f9d8cc59d0382edc92270f3c\": container with ID starting with 9fcf1091a6730580c9d6a6a57b1f66864da79980f9d8cc59d0382edc92270f3c not found: ID does not exist" containerID="9fcf1091a6730580c9d6a6a57b1f66864da79980f9d8cc59d0382edc92270f3c"
Jul 01 17:49:41.030348 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:41.030309    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fcf1091a6730580c9d6a6a57b1f66864da79980f9d8cc59d0382edc92270f3c"} err="failed to get container status \"9fcf1091a6730580c9d6a6a57b1f66864da79980f9d8cc59d0382edc92270f3c\": rpc error: code = NotFound desc = could not find container \"9fcf1091a6730580c9d6a6a57b1f66864da79980f9d8cc59d0382edc92270f3c\": container with ID starting with 9fcf1091a6730580c9d6a6a57b1f66864da79980f9d8cc59d0382edc92270f3c not found: ID does not exist"
Jul 01 17:49:41.030348 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:41.030324    2585 scope.go:117] "RemoveContainer" containerID="dc801cb4bb26d4f2f067422b94dbfb418b733d718021de4ccf99c97b475e3c12"
Jul 01 17:49:41.030643 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:49:41.030607    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc801cb4bb26d4f2f067422b94dbfb418b733d718021de4ccf99c97b475e3c12\": container with ID starting with dc801cb4bb26d4f2f067422b94dbfb418b733d718021de4ccf99c97b475e3c12 not found: ID does not exist" containerID="dc801cb4bb26d4f2f067422b94dbfb418b733d718021de4ccf99c97b475e3c12"
Jul 01 17:49:41.030703 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:41.030649    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc801cb4bb26d4f2f067422b94dbfb418b733d718021de4ccf99c97b475e3c12"} err="failed to get container status \"dc801cb4bb26d4f2f067422b94dbfb418b733d718021de4ccf99c97b475e3c12\": rpc error: code = NotFound desc = could not find container \"dc801cb4bb26d4f2f067422b94dbfb418b733d718021de4ccf99c97b475e3c12\": container with ID starting with dc801cb4bb26d4f2f067422b94dbfb418b733d718021de4ccf99c97b475e3c12 not found: ID does not exist"
Jul 01 17:49:41.034646 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:41.034608    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd"]
Jul 01 17:49:41.038031 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:41.038009    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8b57f-predictor-5c84bc7466-fn2rd"]
Jul 01 17:49:41.462456 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:41.462427    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-10517-86976cfcdc-6xlxr"]
Jul 01 17:49:41.462765 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:41.462750    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e25145e5-82ef-4c78-8e97-84a50628dc1a" containerName="kube-rbac-proxy"
Jul 01 17:49:41.462812 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:41.462766    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e25145e5-82ef-4c78-8e97-84a50628dc1a" containerName="kube-rbac-proxy"
Jul 01 17:49:41.462812 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:41.462780    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e25145e5-82ef-4c78-8e97-84a50628dc1a" containerName="kserve-container"
Jul 01 17:49:41.462812 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:41.462786    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e25145e5-82ef-4c78-8e97-84a50628dc1a" containerName="kserve-container"
Jul 01 17:49:41.462905 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:41.462840    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e25145e5-82ef-4c78-8e97-84a50628dc1a" containerName="kube-rbac-proxy"
Jul 01 17:49:41.462905 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:41.462849    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e25145e5-82ef-4c78-8e97-84a50628dc1a" containerName="kserve-container"
Jul 01 17:49:41.467006 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:41.466989    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-10517-86976cfcdc-6xlxr"
Jul 01 17:49:41.469612 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:41.469590    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-10517-serving-cert\""
Jul 01 17:49:41.469731 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:41.469568    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-10517-kube-rbac-proxy-sar-config\""
Jul 01 17:49:41.474350 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:41.474327    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-10517-86976cfcdc-6xlxr"]
Jul 01 17:49:41.554735 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:41.554702    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7abc0508-1bab-449c-985f-d87bae73b814-proxy-tls\") pod \"splitter-graph-10517-86976cfcdc-6xlxr\" (UID: \"7abc0508-1bab-449c-985f-d87bae73b814\") " pod="kserve-ci-e2e-test/splitter-graph-10517-86976cfcdc-6xlxr"
Jul 01 17:49:41.554945 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:41.554761    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7abc0508-1bab-449c-985f-d87bae73b814-openshift-service-ca-bundle\") pod \"splitter-graph-10517-86976cfcdc-6xlxr\" (UID: \"7abc0508-1bab-449c-985f-d87bae73b814\") " pod="kserve-ci-e2e-test/splitter-graph-10517-86976cfcdc-6xlxr"
Jul 01 17:49:41.655456 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:41.655416    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7abc0508-1bab-449c-985f-d87bae73b814-openshift-service-ca-bundle\") pod \"splitter-graph-10517-86976cfcdc-6xlxr\" (UID: \"7abc0508-1bab-449c-985f-d87bae73b814\") " pod="kserve-ci-e2e-test/splitter-graph-10517-86976cfcdc-6xlxr"
Jul 01 17:49:41.655690 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:41.655482    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7abc0508-1bab-449c-985f-d87bae73b814-proxy-tls\") pod \"splitter-graph-10517-86976cfcdc-6xlxr\" (UID: \"7abc0508-1bab-449c-985f-d87bae73b814\") " pod="kserve-ci-e2e-test/splitter-graph-10517-86976cfcdc-6xlxr"
Jul 01 17:49:41.656199 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:41.656169    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7abc0508-1bab-449c-985f-d87bae73b814-openshift-service-ca-bundle\") pod \"splitter-graph-10517-86976cfcdc-6xlxr\" (UID: \"7abc0508-1bab-449c-985f-d87bae73b814\") " pod="kserve-ci-e2e-test/splitter-graph-10517-86976cfcdc-6xlxr"
Jul 01 17:49:41.658040 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:41.658020    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7abc0508-1bab-449c-985f-d87bae73b814-proxy-tls\") pod \"splitter-graph-10517-86976cfcdc-6xlxr\" (UID: \"7abc0508-1bab-449c-985f-d87bae73b814\") " pod="kserve-ci-e2e-test/splitter-graph-10517-86976cfcdc-6xlxr"
Jul 01 17:49:41.777901 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:41.777809    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-10517-86976cfcdc-6xlxr"
Jul 01 17:49:41.906760 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:41.906725    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-10517-86976cfcdc-6xlxr"]
Jul 01 17:49:41.909567 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:49:41.909546    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7abc0508_1bab_449c_985f_d87bae73b814.slice/crio-f2e6123d2a67a200a0faebf427b2f0403730e7a714d7f8193bc3dc2d970a909d WatchSource:0}: Error finding container f2e6123d2a67a200a0faebf427b2f0403730e7a714d7f8193bc3dc2d970a909d: Status 404 returned error can't find the container with id f2e6123d2a67a200a0faebf427b2f0403730e7a714d7f8193bc3dc2d970a909d
Jul 01 17:49:42.019888 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:42.019850    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-10517-86976cfcdc-6xlxr" event={"ID":"7abc0508-1bab-449c-985f-d87bae73b814","Type":"ContainerStarted","Data":"58b008e0a77f53f46ee7e51fe5f3e5c0214e03c0d2de1037fe5feb94ac76fc3a"}
Jul 01 17:49:42.020346 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:42.020113    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-10517-86976cfcdc-6xlxr" event={"ID":"7abc0508-1bab-449c-985f-d87bae73b814","Type":"ContainerStarted","Data":"f2e6123d2a67a200a0faebf427b2f0403730e7a714d7f8193bc3dc2d970a909d"}
Jul 01 17:49:42.024849 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:42.020504    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-10517-86976cfcdc-6xlxr"
Jul 01 17:49:42.037869 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:42.037770    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-10517-86976cfcdc-6xlxr" podStartSLOduration=1.03775582 podStartE2EDuration="1.03775582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:49:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:49:42.037214788 +0000 UTC m=+2070.342412963" watchObservedRunningTime="2026-07-01 17:49:42.03775582 +0000 UTC m=+2070.342954024"
Jul 01 17:49:42.173382 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:42.173345    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e25145e5-82ef-4c78-8e97-84a50628dc1a" path="/var/lib/kubelet/pods/e25145e5-82ef-4c78-8e97-84a50628dc1a/volumes"
Jul 01 17:49:45.043550 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:45.043515    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-8b57f-6c5d78755-f6zxd" podUID="b50d7a8a-f0ca-464f-8956-cd51ff1dd989" containerName="sequence-graph-8b57f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:49:46.018863 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:46.018835    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56"
Jul 01 17:49:46.019303 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:46.019278    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56" podUID="86b90960-cd85-4aa7-b01e-100c95520755" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 01 17:49:48.035926 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:48.035894    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-10517-86976cfcdc-6xlxr"
Jul 01 17:49:50.043490 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:50.043446    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-8b57f-6c5d78755-f6zxd" podUID="b50d7a8a-f0ca-464f-8956-cd51ff1dd989" containerName="sequence-graph-8b57f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:49:50.043892 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:50.043560    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-8b57f-6c5d78755-f6zxd"
Jul 01 17:49:51.539351 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:51.539251    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-10517-86976cfcdc-6xlxr"]
Jul 01 17:49:51.539810 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:51.539493    2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-10517-86976cfcdc-6xlxr" podUID="7abc0508-1bab-449c-985f-d87bae73b814" containerName="splitter-graph-10517" containerID="cri-o://58b008e0a77f53f46ee7e51fe5f3e5c0214e03c0d2de1037fe5feb94ac76fc3a" gracePeriod=30
Jul 01 17:49:51.743348 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:51.743314    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9"]
Jul 01 17:49:51.743711 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:51.743548    2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9" podUID="21db6b05-cc24-497d-83d1-595ccc04fc7b" containerName="kserve-container" containerID="cri-o://db47f2b9e7f572c13210a2aad2f5587613f62e4c97858983de306e2abdf3a2f4" gracePeriod=30
Jul 01 17:49:51.743711 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:51.743616    2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9" podUID="21db6b05-cc24-497d-83d1-595ccc04fc7b" containerName="kube-rbac-proxy" containerID="cri-o://2f3bb71499ca634e412d9e1102046ebb36e9b254d13fcc981a61eff41477c5ae" gracePeriod=30
Jul 01 17:49:51.784072 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:51.784041    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m"]
Jul 01 17:49:51.787251 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:51.787231    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m"
Jul 01 17:49:51.789796 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:51.789743    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-42b28-predictor-serving-cert\""
Jul 01 17:49:51.789796 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:51.789772    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-42b28-kube-rbac-proxy-sar-config\""
Jul 01 17:49:51.796944 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:51.796921    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m"]
Jul 01 17:49:51.833244 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:51.833211    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c0e28fa-dd0b-455f-8be4-d9bb36f8833b-proxy-tls\") pod \"error-404-isvc-42b28-predictor-544cbc8654-8458m\" (UID: \"8c0e28fa-dd0b-455f-8be4-d9bb36f8833b\") " pod="kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m"
Jul 01 17:49:51.833393 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:51.833266    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts2cx\" (UniqueName: \"kubernetes.io/projected/8c0e28fa-dd0b-455f-8be4-d9bb36f8833b-kube-api-access-ts2cx\") pod \"error-404-isvc-42b28-predictor-544cbc8654-8458m\" (UID: \"8c0e28fa-dd0b-455f-8be4-d9bb36f8833b\") " pod="kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m"
Jul 01 17:49:51.833393 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:51.833294    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c0e28fa-dd0b-455f-8be4-d9bb36f8833b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-42b28-predictor-544cbc8654-8458m\" (UID: \"8c0e28fa-dd0b-455f-8be4-d9bb36f8833b\") " pod="kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m"
Jul 01 17:49:51.934073 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:51.934031    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts2cx\" (UniqueName: \"kubernetes.io/projected/8c0e28fa-dd0b-455f-8be4-d9bb36f8833b-kube-api-access-ts2cx\") pod \"error-404-isvc-42b28-predictor-544cbc8654-8458m\" (UID: \"8c0e28fa-dd0b-455f-8be4-d9bb36f8833b\") " pod="kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m"
Jul 01 17:49:51.934073 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:51.934072    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c0e28fa-dd0b-455f-8be4-d9bb36f8833b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-42b28-predictor-544cbc8654-8458m\" (UID: \"8c0e28fa-dd0b-455f-8be4-d9bb36f8833b\") " pod="kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m"
Jul 01 17:49:51.934301 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:51.934117    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c0e28fa-dd0b-455f-8be4-d9bb36f8833b-proxy-tls\") pod \"error-404-isvc-42b28-predictor-544cbc8654-8458m\" (UID: \"8c0e28fa-dd0b-455f-8be4-d9bb36f8833b\") " pod="kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m"
Jul 01 17:49:51.934875 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:51.934854    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c0e28fa-dd0b-455f-8be4-d9bb36f8833b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-42b28-predictor-544cbc8654-8458m\" (UID: \"8c0e28fa-dd0b-455f-8be4-d9bb36f8833b\") " pod="kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m"
Jul 01 17:49:51.936592 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:51.936562    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c0e28fa-dd0b-455f-8be4-d9bb36f8833b-proxy-tls\") pod \"error-404-isvc-42b28-predictor-544cbc8654-8458m\" (UID: \"8c0e28fa-dd0b-455f-8be4-d9bb36f8833b\") " pod="kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m"
Jul 01 17:49:51.942479 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:51.942457    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts2cx\" (UniqueName: \"kubernetes.io/projected/8c0e28fa-dd0b-455f-8be4-d9bb36f8833b-kube-api-access-ts2cx\") pod \"error-404-isvc-42b28-predictor-544cbc8654-8458m\" (UID: \"8c0e28fa-dd0b-455f-8be4-d9bb36f8833b\") " pod="kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m"
Jul 01 17:49:52.068509 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:52.068477    2585 generic.go:356] "Generic (PLEG): container finished" podID="21db6b05-cc24-497d-83d1-595ccc04fc7b" containerID="2f3bb71499ca634e412d9e1102046ebb36e9b254d13fcc981a61eff41477c5ae" exitCode=2
Jul 01 17:49:52.068701 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:52.068516    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9" event={"ID":"21db6b05-cc24-497d-83d1-595ccc04fc7b","Type":"ContainerDied","Data":"2f3bb71499ca634e412d9e1102046ebb36e9b254d13fcc981a61eff41477c5ae"}
Jul 01 17:49:52.097980 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:52.097944    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m"
Jul 01 17:49:52.234510 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:52.234478    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m"]
Jul 01 17:49:52.237537 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:49:52.237510    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c0e28fa_dd0b_455f_8be4_d9bb36f8833b.slice/crio-3ca82d2b8f427f214c9ede7005f6a91921de6770ca9c9b4799f557b57713381e WatchSource:0}: Error finding container 3ca82d2b8f427f214c9ede7005f6a91921de6770ca9c9b4799f557b57713381e: Status 404 returned error can't find the container with id 3ca82d2b8f427f214c9ede7005f6a91921de6770ca9c9b4799f557b57713381e
Jul 01 17:49:53.034288 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:53.034248    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-10517-86976cfcdc-6xlxr" podUID="7abc0508-1bab-449c-985f-d87bae73b814" containerName="splitter-graph-10517" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:49:53.074818 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:53.074784    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m" event={"ID":"8c0e28fa-dd0b-455f-8be4-d9bb36f8833b","Type":"ContainerStarted","Data":"2aad47135c4296636232b4e4b896fdd07998cc365b56693fb3ce4ca7c2a5c6e1"}
Jul 01 17:49:53.075019 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:53.074842    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m" event={"ID":"8c0e28fa-dd0b-455f-8be4-d9bb36f8833b","Type":"ContainerStarted","Data":"66f8ff32a5c3b98c63180087f4105d2df6eb988fd35d446a30b10fc729ae0dad"}
Jul 01 17:49:53.075019 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:53.074873    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m" event={"ID":"8c0e28fa-dd0b-455f-8be4-d9bb36f8833b","Type":"ContainerStarted","Data":"3ca82d2b8f427f214c9ede7005f6a91921de6770ca9c9b4799f557b57713381e"}
Jul 01 17:49:53.075019 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:53.074900    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m"
Jul 01 17:49:53.095285 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:53.095216    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m" podStartSLOduration=2.095193778 podStartE2EDuration="2.095193778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:49:53.093207857 +0000 UTC m=+2081.398406045" watchObservedRunningTime="2026-07-01 17:49:53.095193778 +0000 UTC m=+2081.400391957"
Jul 01 17:49:53.777781 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:53.777737    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9" podUID="21db6b05-cc24-497d-83d1-595ccc04fc7b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Jul 01 17:49:54.079189 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:54.079152    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m"
Jul 01 17:49:54.080490 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:54.080461    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m" podUID="8c0e28fa-dd0b-455f-8be4-d9bb36f8833b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 01 17:49:55.042913 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:55.042871    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-8b57f-6c5d78755-f6zxd" podUID="b50d7a8a-f0ca-464f-8956-cd51ff1dd989" containerName="sequence-graph-8b57f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:49:55.085641 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:55.085601    2585 generic.go:356] "Generic (PLEG): container finished" podID="21db6b05-cc24-497d-83d1-595ccc04fc7b" containerID="db47f2b9e7f572c13210a2aad2f5587613f62e4c97858983de306e2abdf3a2f4" exitCode=0
Jul 01 17:49:55.086027 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:55.085616    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9" event={"ID":"21db6b05-cc24-497d-83d1-595ccc04fc7b","Type":"ContainerDied","Data":"db47f2b9e7f572c13210a2aad2f5587613f62e4c97858983de306e2abdf3a2f4"}
Jul 01 17:49:55.086027 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:55.085988    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m" podUID="8c0e28fa-dd0b-455f-8be4-d9bb36f8833b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 01 17:49:55.189437 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:55.189409    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9"
Jul 01 17:49:55.260943 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:55.260857    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21db6b05-cc24-497d-83d1-595ccc04fc7b-kube-rbac-proxy-sar-config\") pod \"21db6b05-cc24-497d-83d1-595ccc04fc7b\" (UID: \"21db6b05-cc24-497d-83d1-595ccc04fc7b\") "
Jul 01 17:49:55.260943 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:55.260914    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rz7vm\" (UniqueName: \"kubernetes.io/projected/21db6b05-cc24-497d-83d1-595ccc04fc7b-kube-api-access-rz7vm\") pod \"21db6b05-cc24-497d-83d1-595ccc04fc7b\" (UID: \"21db6b05-cc24-497d-83d1-595ccc04fc7b\") "
Jul 01 17:49:55.261167 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:55.260953    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21db6b05-cc24-497d-83d1-595ccc04fc7b-proxy-tls\") pod \"21db6b05-cc24-497d-83d1-595ccc04fc7b\" (UID: \"21db6b05-cc24-497d-83d1-595ccc04fc7b\") "
Jul 01 17:49:55.261257 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:55.261237    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21db6b05-cc24-497d-83d1-595ccc04fc7b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "21db6b05-cc24-497d-83d1-595ccc04fc7b" (UID: "21db6b05-cc24-497d-83d1-595ccc04fc7b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:49:55.263209 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:55.263179    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21db6b05-cc24-497d-83d1-595ccc04fc7b-kube-api-access-rz7vm" (OuterVolumeSpecName: "kube-api-access-rz7vm") pod "21db6b05-cc24-497d-83d1-595ccc04fc7b" (UID: "21db6b05-cc24-497d-83d1-595ccc04fc7b"). InnerVolumeSpecName "kube-api-access-rz7vm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:49:55.263209 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:55.263179    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21db6b05-cc24-497d-83d1-595ccc04fc7b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "21db6b05-cc24-497d-83d1-595ccc04fc7b" (UID: "21db6b05-cc24-497d-83d1-595ccc04fc7b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:49:55.362325 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:55.362285    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21db6b05-cc24-497d-83d1-595ccc04fc7b-proxy-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:49:55.362325 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:55.362317    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21db6b05-cc24-497d-83d1-595ccc04fc7b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:49:55.362325 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:55.362327    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz7vm\" (UniqueName: \"kubernetes.io/projected/21db6b05-cc24-497d-83d1-595ccc04fc7b-kube-api-access-rz7vm\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:49:56.019838 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:56.019800    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56" podUID="86b90960-cd85-4aa7-b01e-100c95520755" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 01 17:49:56.091508 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:56.091474    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9"
Jul 01 17:49:56.092015 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:56.091479    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9" event={"ID":"21db6b05-cc24-497d-83d1-595ccc04fc7b","Type":"ContainerDied","Data":"9c212112f9421c91cbf527d33028784ef76a25a4933c95a90315612a6c707ac3"}
Jul 01 17:49:56.092015 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:56.091654    2585 scope.go:117] "RemoveContainer" containerID="2f3bb71499ca634e412d9e1102046ebb36e9b254d13fcc981a61eff41477c5ae"
Jul 01 17:49:56.100222 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:56.100203    2585 scope.go:117] "RemoveContainer" containerID="db47f2b9e7f572c13210a2aad2f5587613f62e4c97858983de306e2abdf3a2f4"
Jul 01 17:49:56.112385 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:56.112360    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9"]
Jul 01 17:49:56.117751 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:56.117730    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-10517-predictor-f6447f778-924c9"]
Jul 01 17:49:56.172628 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:56.172599    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21db6b05-cc24-497d-83d1-595ccc04fc7b" path="/var/lib/kubelet/pods/21db6b05-cc24-497d-83d1-595ccc04fc7b/volumes"
Jul 01 17:49:58.034374 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:49:58.034325    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-10517-86976cfcdc-6xlxr" podUID="7abc0508-1bab-449c-985f-d87bae73b814" containerName="splitter-graph-10517" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:50:00.042699 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:00.042654    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-8b57f-6c5d78755-f6zxd" podUID="b50d7a8a-f0ca-464f-8956-cd51ff1dd989" containerName="sequence-graph-8b57f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:50:00.091196 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:00.091168    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m"
Jul 01 17:50:00.091639 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:00.091609    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m" podUID="8c0e28fa-dd0b-455f-8be4-d9bb36f8833b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 01 17:50:03.034087 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:03.034048    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-10517-86976cfcdc-6xlxr" podUID="7abc0508-1bab-449c-985f-d87bae73b814" containerName="splitter-graph-10517" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:50:03.034493 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:03.034157    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-10517-86976cfcdc-6xlxr"
Jul 01 17:50:05.043344 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:05.043306    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-8b57f-6c5d78755-f6zxd" podUID="b50d7a8a-f0ca-464f-8956-cd51ff1dd989" containerName="sequence-graph-8b57f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:50:06.019425 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:06.019384    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56" podUID="86b90960-cd85-4aa7-b01e-100c95520755" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 01 17:50:07.136342 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:07.136302    2585 generic.go:356] "Generic (PLEG): container finished" podID="b50d7a8a-f0ca-464f-8956-cd51ff1dd989" containerID="84ba4146cf35b814c79657035f770a91799a629b1d35a61a3b7bf1f7560f4809" exitCode=0
Jul 01 17:50:07.136342 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:07.136343    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-8b57f-6c5d78755-f6zxd" event={"ID":"b50d7a8a-f0ca-464f-8956-cd51ff1dd989","Type":"ContainerDied","Data":"84ba4146cf35b814c79657035f770a91799a629b1d35a61a3b7bf1f7560f4809"}
Jul 01 17:50:07.611666 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:07.611635    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-8b57f-6c5d78755-f6zxd"
Jul 01 17:50:07.766546 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:07.766448    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b50d7a8a-f0ca-464f-8956-cd51ff1dd989-proxy-tls\") pod \"b50d7a8a-f0ca-464f-8956-cd51ff1dd989\" (UID: \"b50d7a8a-f0ca-464f-8956-cd51ff1dd989\") "
Jul 01 17:50:07.766546 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:07.766493    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b50d7a8a-f0ca-464f-8956-cd51ff1dd989-openshift-service-ca-bundle\") pod \"b50d7a8a-f0ca-464f-8956-cd51ff1dd989\" (UID: \"b50d7a8a-f0ca-464f-8956-cd51ff1dd989\") "
Jul 01 17:50:07.766913 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:07.766887    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b50d7a8a-f0ca-464f-8956-cd51ff1dd989-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "b50d7a8a-f0ca-464f-8956-cd51ff1dd989" (UID: "b50d7a8a-f0ca-464f-8956-cd51ff1dd989"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:50:07.768696 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:07.768673    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b50d7a8a-f0ca-464f-8956-cd51ff1dd989-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b50d7a8a-f0ca-464f-8956-cd51ff1dd989" (UID: "b50d7a8a-f0ca-464f-8956-cd51ff1dd989"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:50:07.867895 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:07.867859    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b50d7a8a-f0ca-464f-8956-cd51ff1dd989-proxy-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:50:07.867895 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:07.867894    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b50d7a8a-f0ca-464f-8956-cd51ff1dd989-openshift-service-ca-bundle\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:50:08.034431 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:08.034328    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-10517-86976cfcdc-6xlxr" podUID="7abc0508-1bab-449c-985f-d87bae73b814" containerName="splitter-graph-10517" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:50:08.141935 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:08.141903    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-8b57f-6c5d78755-f6zxd" event={"ID":"b50d7a8a-f0ca-464f-8956-cd51ff1dd989","Type":"ContainerDied","Data":"c0f6642808999dfc6474699e2d7d649ccb54ba6025d5ec219aff5dffc7498d77"}
Jul 01 17:50:08.142437 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:08.141967    2585 scope.go:117] "RemoveContainer" containerID="84ba4146cf35b814c79657035f770a91799a629b1d35a61a3b7bf1f7560f4809"
Jul 01 17:50:08.142437 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:08.141918    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-8b57f-6c5d78755-f6zxd"
Jul 01 17:50:08.162802 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:08.162773    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-8b57f-6c5d78755-f6zxd"]
Jul 01 17:50:08.165900 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:08.165873    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-8b57f-6c5d78755-f6zxd"]
Jul 01 17:50:08.172477 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:08.172453    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b50d7a8a-f0ca-464f-8956-cd51ff1dd989" path="/var/lib/kubelet/pods/b50d7a8a-f0ca-464f-8956-cd51ff1dd989/volumes"
Jul 01 17:50:10.092255 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:10.092215    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m" podUID="8c0e28fa-dd0b-455f-8be4-d9bb36f8833b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 01 17:50:12.266341 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:12.266313    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8tsw_500d2a90-2d21-42b3-9584-b04c14becaec/ovn-acl-logging/0.log"
Jul 01 17:50:12.271728 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:12.271708    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8tsw_500d2a90-2d21-42b3-9584-b04c14becaec/ovn-acl-logging/0.log"
Jul 01 17:50:13.034344 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:13.034296    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-10517-86976cfcdc-6xlxr" podUID="7abc0508-1bab-449c-985f-d87bae73b814" containerName="splitter-graph-10517" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:50:16.019638 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:16.019601    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56" podUID="86b90960-cd85-4aa7-b01e-100c95520755" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 01 17:50:18.034555 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:18.034505    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-10517-86976cfcdc-6xlxr" podUID="7abc0508-1bab-449c-985f-d87bae73b814" containerName="splitter-graph-10517" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:50:20.092158 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:20.092117    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m" podUID="8c0e28fa-dd0b-455f-8be4-d9bb36f8833b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 01 17:50:22.187018 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:22.186989    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-10517-86976cfcdc-6xlxr"
Jul 01 17:50:22.195090 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:22.195064    2585 generic.go:356] "Generic (PLEG): container finished" podID="7abc0508-1bab-449c-985f-d87bae73b814" containerID="58b008e0a77f53f46ee7e51fe5f3e5c0214e03c0d2de1037fe5feb94ac76fc3a" exitCode=0
Jul 01 17:50:22.195251 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:22.195103    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-10517-86976cfcdc-6xlxr"
Jul 01 17:50:22.195251 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:22.195123    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-10517-86976cfcdc-6xlxr" event={"ID":"7abc0508-1bab-449c-985f-d87bae73b814","Type":"ContainerDied","Data":"58b008e0a77f53f46ee7e51fe5f3e5c0214e03c0d2de1037fe5feb94ac76fc3a"}
Jul 01 17:50:22.195251 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:22.195165    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-10517-86976cfcdc-6xlxr" event={"ID":"7abc0508-1bab-449c-985f-d87bae73b814","Type":"ContainerDied","Data":"f2e6123d2a67a200a0faebf427b2f0403730e7a714d7f8193bc3dc2d970a909d"}
Jul 01 17:50:22.195251 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:22.195187    2585 scope.go:117] "RemoveContainer" containerID="58b008e0a77f53f46ee7e51fe5f3e5c0214e03c0d2de1037fe5feb94ac76fc3a"
Jul 01 17:50:22.203441 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:22.203420    2585 scope.go:117] "RemoveContainer" containerID="58b008e0a77f53f46ee7e51fe5f3e5c0214e03c0d2de1037fe5feb94ac76fc3a"
Jul 01 17:50:22.203775 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:50:22.203754    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58b008e0a77f53f46ee7e51fe5f3e5c0214e03c0d2de1037fe5feb94ac76fc3a\": container with ID starting with 58b008e0a77f53f46ee7e51fe5f3e5c0214e03c0d2de1037fe5feb94ac76fc3a not found: ID does not exist" containerID="58b008e0a77f53f46ee7e51fe5f3e5c0214e03c0d2de1037fe5feb94ac76fc3a"
Jul 01 17:50:22.203852 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:22.203789    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58b008e0a77f53f46ee7e51fe5f3e5c0214e03c0d2de1037fe5feb94ac76fc3a"} err="failed to get container status \"58b008e0a77f53f46ee7e51fe5f3e5c0214e03c0d2de1037fe5feb94ac76fc3a\": rpc error: code = NotFound desc = could not find container \"58b008e0a77f53f46ee7e51fe5f3e5c0214e03c0d2de1037fe5feb94ac76fc3a\": container with ID starting with 58b008e0a77f53f46ee7e51fe5f3e5c0214e03c0d2de1037fe5feb94ac76fc3a not found: ID does not exist"
Jul 01 17:50:22.278333 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:22.278297    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7abc0508-1bab-449c-985f-d87bae73b814-openshift-service-ca-bundle\") pod \"7abc0508-1bab-449c-985f-d87bae73b814\" (UID: \"7abc0508-1bab-449c-985f-d87bae73b814\") "
Jul 01 17:50:22.278546 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:22.278394    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7abc0508-1bab-449c-985f-d87bae73b814-proxy-tls\") pod \"7abc0508-1bab-449c-985f-d87bae73b814\" (UID: \"7abc0508-1bab-449c-985f-d87bae73b814\") "
Jul 01 17:50:22.278799 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:22.278776    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7abc0508-1bab-449c-985f-d87bae73b814-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "7abc0508-1bab-449c-985f-d87bae73b814" (UID: "7abc0508-1bab-449c-985f-d87bae73b814"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:50:22.280810 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:22.280787    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7abc0508-1bab-449c-985f-d87bae73b814-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7abc0508-1bab-449c-985f-d87bae73b814" (UID: "7abc0508-1bab-449c-985f-d87bae73b814"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:50:22.379524 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:22.379430    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7abc0508-1bab-449c-985f-d87bae73b814-proxy-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:50:22.379524 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:22.379463    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7abc0508-1bab-449c-985f-d87bae73b814-openshift-service-ca-bundle\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:50:22.516971 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:22.516934    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-10517-86976cfcdc-6xlxr"]
Jul 01 17:50:22.521440 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:22.521409    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-10517-86976cfcdc-6xlxr"]
Jul 01 17:50:24.172246 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:24.172212    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7abc0508-1bab-449c-985f-d87bae73b814" path="/var/lib/kubelet/pods/7abc0508-1bab-449c-985f-d87bae73b814/volumes"
Jul 01 17:50:26.020231 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:26.020196    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56"
Jul 01 17:50:30.092484 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:30.092443    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m" podUID="8c0e28fa-dd0b-455f-8be4-d9bb36f8833b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 01 17:50:37.238008 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:37.237970    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-4822a-76849cbcd4-cg6kj"]
Jul 01 17:50:37.238375 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:37.238242    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b50d7a8a-f0ca-464f-8956-cd51ff1dd989" containerName="sequence-graph-8b57f"
Jul 01 17:50:37.238375 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:37.238253    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b50d7a8a-f0ca-464f-8956-cd51ff1dd989" containerName="sequence-graph-8b57f"
Jul 01 17:50:37.238375 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:37.238264    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21db6b05-cc24-497d-83d1-595ccc04fc7b" containerName="kube-rbac-proxy"
Jul 01 17:50:37.238375 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:37.238269    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="21db6b05-cc24-497d-83d1-595ccc04fc7b" containerName="kube-rbac-proxy"
Jul 01 17:50:37.238375 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:37.238277    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21db6b05-cc24-497d-83d1-595ccc04fc7b" containerName="kserve-container"
Jul 01 17:50:37.238375 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:37.238282    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="21db6b05-cc24-497d-83d1-595ccc04fc7b" containerName="kserve-container"
Jul 01 17:50:37.238375 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:37.238291    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7abc0508-1bab-449c-985f-d87bae73b814" containerName="splitter-graph-10517"
Jul 01 17:50:37.238375 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:37.238297    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="7abc0508-1bab-449c-985f-d87bae73b814" containerName="splitter-graph-10517"
Jul 01 17:50:37.238375 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:37.238345    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="21db6b05-cc24-497d-83d1-595ccc04fc7b" containerName="kserve-container"
Jul 01 17:50:37.238375 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:37.238354    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="7abc0508-1bab-449c-985f-d87bae73b814" containerName="splitter-graph-10517"
Jul 01 17:50:37.238375 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:37.238362    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b50d7a8a-f0ca-464f-8956-cd51ff1dd989" containerName="sequence-graph-8b57f"
Jul 01 17:50:37.238375 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:37.238368    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="21db6b05-cc24-497d-83d1-595ccc04fc7b" containerName="kube-rbac-proxy"
Jul 01 17:50:37.242420 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:37.242404    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-4822a-76849cbcd4-cg6kj"
Jul 01 17:50:37.245721 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:37.245698    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-4822a-serving-cert\""
Jul 01 17:50:37.245958 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:37.245939    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-4822a-kube-rbac-proxy-sar-config\""
Jul 01 17:50:37.254730 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:37.254705    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-4822a-76849cbcd4-cg6kj"]
Jul 01 17:50:37.293262 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:37.293220    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f-openshift-service-ca-bundle\") pod \"switch-graph-4822a-76849cbcd4-cg6kj\" (UID: \"70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f\") " pod="kserve-ci-e2e-test/switch-graph-4822a-76849cbcd4-cg6kj"
Jul 01 17:50:37.293262 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:37.293265    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f-proxy-tls\") pod \"switch-graph-4822a-76849cbcd4-cg6kj\" (UID: \"70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f\") " pod="kserve-ci-e2e-test/switch-graph-4822a-76849cbcd4-cg6kj"
Jul 01 17:50:37.394017 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:37.393977    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f-openshift-service-ca-bundle\") pod \"switch-graph-4822a-76849cbcd4-cg6kj\" (UID: \"70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f\") " pod="kserve-ci-e2e-test/switch-graph-4822a-76849cbcd4-cg6kj"
Jul 01 17:50:37.394017 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:37.394021    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f-proxy-tls\") pod \"switch-graph-4822a-76849cbcd4-cg6kj\" (UID: \"70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f\") " pod="kserve-ci-e2e-test/switch-graph-4822a-76849cbcd4-cg6kj"
Jul 01 17:50:37.394229 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:50:37.394133    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-4822a-serving-cert: secret "switch-graph-4822a-serving-cert" not found
Jul 01 17:50:37.394266 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:50:37.394234    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f-proxy-tls podName:70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f nodeName:}" failed. No retries permitted until 2026-07-01 17:50:37.894210807 +0000 UTC m=+2126.199408964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f-proxy-tls") pod "switch-graph-4822a-76849cbcd4-cg6kj" (UID: "70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f") : secret "switch-graph-4822a-serving-cert" not found
Jul 01 17:50:37.394772 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:37.394753    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f-openshift-service-ca-bundle\") pod \"switch-graph-4822a-76849cbcd4-cg6kj\" (UID: \"70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f\") " pod="kserve-ci-e2e-test/switch-graph-4822a-76849cbcd4-cg6kj"
Jul 01 17:50:37.897297 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:37.897256    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f-proxy-tls\") pod \"switch-graph-4822a-76849cbcd4-cg6kj\" (UID: \"70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f\") " pod="kserve-ci-e2e-test/switch-graph-4822a-76849cbcd4-cg6kj"
Jul 01 17:50:37.899802 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:37.899781    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f-proxy-tls\") pod \"switch-graph-4822a-76849cbcd4-cg6kj\" (UID: \"70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f\") " pod="kserve-ci-e2e-test/switch-graph-4822a-76849cbcd4-cg6kj"
Jul 01 17:50:38.152446 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:38.152352    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-4822a-76849cbcd4-cg6kj"
Jul 01 17:50:38.289090 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:38.289055    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-4822a-76849cbcd4-cg6kj"]
Jul 01 17:50:38.293899 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:50:38.293861    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70eef7eb_09d3_4f6d_98a1_3eeb00e09d7f.slice/crio-96057af699c9bd8217d310f8665659f7e3dc341d9110201137fbc197d855bf5a WatchSource:0}: Error finding container 96057af699c9bd8217d310f8665659f7e3dc341d9110201137fbc197d855bf5a: Status 404 returned error can't find the container with id 96057af699c9bd8217d310f8665659f7e3dc341d9110201137fbc197d855bf5a
Jul 01 17:50:39.260902 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:39.260861    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-4822a-76849cbcd4-cg6kj" event={"ID":"70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f","Type":"ContainerStarted","Data":"b8ea285619d2e540381041295566f92bbba7721477b9703fa24f1373a9d73810"}
Jul 01 17:50:39.261148 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:39.260915    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-4822a-76849cbcd4-cg6kj" event={"ID":"70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f","Type":"ContainerStarted","Data":"96057af699c9bd8217d310f8665659f7e3dc341d9110201137fbc197d855bf5a"}
Jul 01 17:50:39.261148 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:39.260952    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-4822a-76849cbcd4-cg6kj"
Jul 01 17:50:39.277420 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:39.277366    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-4822a-76849cbcd4-cg6kj" podStartSLOduration=2.277351307 podStartE2EDuration="2.277351307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:50:39.275866091 +0000 UTC m=+2127.581064268" watchObservedRunningTime="2026-07-01 17:50:39.277351307 +0000 UTC m=+2127.582549520"
Jul 01 17:50:40.093223 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:40.093188    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m"
Jul 01 17:50:45.271425 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:45.271392    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-4822a-76849cbcd4-cg6kj"
Jul 01 17:50:51.743021 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:51.742984    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-42b28-5f7d665cfb-9t2zj"]
Jul 01 17:50:51.745868 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:51.745852    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-42b28-5f7d665cfb-9t2zj"
Jul 01 17:50:51.748591 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:51.748555    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-42b28-kube-rbac-proxy-sar-config\""
Jul 01 17:50:51.748719 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:51.748558    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-42b28-serving-cert\""
Jul 01 17:50:51.754823 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:51.754798    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-42b28-5f7d665cfb-9t2zj"]
Jul 01 17:50:51.806793 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:51.806749    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9b45c3b-ae08-4355-8930-ecdfd03190d4-openshift-service-ca-bundle\") pod \"splitter-graph-42b28-5f7d665cfb-9t2zj\" (UID: \"f9b45c3b-ae08-4355-8930-ecdfd03190d4\") " pod="kserve-ci-e2e-test/splitter-graph-42b28-5f7d665cfb-9t2zj"
Jul 01 17:50:51.806793 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:51.806791    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9b45c3b-ae08-4355-8930-ecdfd03190d4-proxy-tls\") pod \"splitter-graph-42b28-5f7d665cfb-9t2zj\" (UID: \"f9b45c3b-ae08-4355-8930-ecdfd03190d4\") " pod="kserve-ci-e2e-test/splitter-graph-42b28-5f7d665cfb-9t2zj"
Jul 01 17:50:51.908094 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:51.908041    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9b45c3b-ae08-4355-8930-ecdfd03190d4-openshift-service-ca-bundle\") pod \"splitter-graph-42b28-5f7d665cfb-9t2zj\" (UID: \"f9b45c3b-ae08-4355-8930-ecdfd03190d4\") " pod="kserve-ci-e2e-test/splitter-graph-42b28-5f7d665cfb-9t2zj"
Jul 01 17:50:51.908094 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:51.908109    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9b45c3b-ae08-4355-8930-ecdfd03190d4-proxy-tls\") pod \"splitter-graph-42b28-5f7d665cfb-9t2zj\" (UID: \"f9b45c3b-ae08-4355-8930-ecdfd03190d4\") " pod="kserve-ci-e2e-test/splitter-graph-42b28-5f7d665cfb-9t2zj"
Jul 01 17:50:51.908770 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:51.908741    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9b45c3b-ae08-4355-8930-ecdfd03190d4-openshift-service-ca-bundle\") pod \"splitter-graph-42b28-5f7d665cfb-9t2zj\" (UID: \"f9b45c3b-ae08-4355-8930-ecdfd03190d4\") " pod="kserve-ci-e2e-test/splitter-graph-42b28-5f7d665cfb-9t2zj"
Jul 01 17:50:51.910708 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:51.910684    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9b45c3b-ae08-4355-8930-ecdfd03190d4-proxy-tls\") pod \"splitter-graph-42b28-5f7d665cfb-9t2zj\" (UID: \"f9b45c3b-ae08-4355-8930-ecdfd03190d4\") " pod="kserve-ci-e2e-test/splitter-graph-42b28-5f7d665cfb-9t2zj"
Jul 01 17:50:52.056429 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:52.056388    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-42b28-5f7d665cfb-9t2zj"
Jul 01 17:50:52.185654 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:52.185622    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-42b28-5f7d665cfb-9t2zj"]
Jul 01 17:50:52.190040 ip-10-0-141-213 kubenswrapper[2585]: W0701 17:50:52.190009    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b45c3b_ae08_4355_8930_ecdfd03190d4.slice/crio-6bca3aed174cc74db250a7da7fc7bc12249f481a9c1c1cb6901867777807778d WatchSource:0}: Error finding container 6bca3aed174cc74db250a7da7fc7bc12249f481a9c1c1cb6901867777807778d: Status 404 returned error can't find the container with id 6bca3aed174cc74db250a7da7fc7bc12249f481a9c1c1cb6901867777807778d
Jul 01 17:50:52.313962 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:52.313861    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-42b28-5f7d665cfb-9t2zj" event={"ID":"f9b45c3b-ae08-4355-8930-ecdfd03190d4","Type":"ContainerStarted","Data":"57cbfdb09b8bc556861dec03750d53a4d5b22c01cff3da8e115a4a477456e366"}
Jul 01 17:50:52.313962 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:52.313924    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-42b28-5f7d665cfb-9t2zj" event={"ID":"f9b45c3b-ae08-4355-8930-ecdfd03190d4","Type":"ContainerStarted","Data":"6bca3aed174cc74db250a7da7fc7bc12249f481a9c1c1cb6901867777807778d"}
Jul 01 17:50:52.314181 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:52.314030    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-42b28-5f7d665cfb-9t2zj"
Jul 01 17:50:52.331964 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:52.331912    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-42b28-5f7d665cfb-9t2zj" podStartSLOduration=1.33189546 podStartE2EDuration="1.33189546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:50:52.330736132 +0000 UTC m=+2140.635934310" watchObservedRunningTime="2026-07-01 17:50:52.33189546 +0000 UTC m=+2140.637093667"
Jul 01 17:50:58.323459 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:50:58.323431    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-42b28-5f7d665cfb-9t2zj"
Jul 01 17:55:12.287770 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:55:12.287744    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8tsw_500d2a90-2d21-42b3-9584-b04c14becaec/ovn-acl-logging/0.log"
Jul 01 17:55:12.294016 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:55:12.293997    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8tsw_500d2a90-2d21-42b3-9584-b04c14becaec/ovn-acl-logging/0.log"
Jul 01 17:59:06.224624 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:06.224592    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-42b28-5f7d665cfb-9t2zj"]
Jul 01 17:59:06.227072 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:06.224770    2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-42b28-5f7d665cfb-9t2zj" podUID="f9b45c3b-ae08-4355-8930-ecdfd03190d4" containerName="splitter-graph-42b28" containerID="cri-o://57cbfdb09b8bc556861dec03750d53a4d5b22c01cff3da8e115a4a477456e366" gracePeriod=30
Jul 01 17:59:06.404545 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:06.404511    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m"]
Jul 01 17:59:06.404838 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:06.404812    2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m" podUID="8c0e28fa-dd0b-455f-8be4-d9bb36f8833b" containerName="kserve-container" containerID="cri-o://66f8ff32a5c3b98c63180087f4105d2df6eb988fd35d446a30b10fc729ae0dad" gracePeriod=30
Jul 01 17:59:06.405030 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:06.404958    2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m" podUID="8c0e28fa-dd0b-455f-8be4-d9bb36f8833b" containerName="kube-rbac-proxy" containerID="cri-o://2aad47135c4296636232b4e4b896fdd07998cc365b56693fb3ce4ca7c2a5c6e1" gracePeriod=30
Jul 01 17:59:07.273127 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:07.273094    2585 generic.go:356] "Generic (PLEG): container finished" podID="8c0e28fa-dd0b-455f-8be4-d9bb36f8833b" containerID="2aad47135c4296636232b4e4b896fdd07998cc365b56693fb3ce4ca7c2a5c6e1" exitCode=2
Jul 01 17:59:07.273504 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:07.273136    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m" event={"ID":"8c0e28fa-dd0b-455f-8be4-d9bb36f8833b","Type":"ContainerDied","Data":"2aad47135c4296636232b4e4b896fdd07998cc365b56693fb3ce4ca7c2a5c6e1"}
Jul 01 17:59:08.322479 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:08.322442    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-42b28-5f7d665cfb-9t2zj" podUID="f9b45c3b-ae08-4355-8930-ecdfd03190d4" containerName="splitter-graph-42b28" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:59:10.086397 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:10.086354    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m" podUID="8c0e28fa-dd0b-455f-8be4-d9bb36f8833b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Jul 01 17:59:10.091715 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:10.091690    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m" podUID="8c0e28fa-dd0b-455f-8be4-d9bb36f8833b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 01 17:59:13.300311 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:13.300281    2585 generic.go:356] "Generic (PLEG): container finished" podID="8c0e28fa-dd0b-455f-8be4-d9bb36f8833b" containerID="66f8ff32a5c3b98c63180087f4105d2df6eb988fd35d446a30b10fc729ae0dad" exitCode=0
Jul 01 17:59:13.300680 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:13.300325    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m" event={"ID":"8c0e28fa-dd0b-455f-8be4-d9bb36f8833b","Type":"ContainerDied","Data":"66f8ff32a5c3b98c63180087f4105d2df6eb988fd35d446a30b10fc729ae0dad"}
Jul 01 17:59:13.323072 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:13.323038    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-42b28-5f7d665cfb-9t2zj" podUID="f9b45c3b-ae08-4355-8930-ecdfd03190d4" containerName="splitter-graph-42b28" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:59:13.352408 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:13.352385    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m"
Jul 01 17:59:13.452126 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:13.452041    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ts2cx\" (UniqueName: \"kubernetes.io/projected/8c0e28fa-dd0b-455f-8be4-d9bb36f8833b-kube-api-access-ts2cx\") pod \"8c0e28fa-dd0b-455f-8be4-d9bb36f8833b\" (UID: \"8c0e28fa-dd0b-455f-8be4-d9bb36f8833b\") "
Jul 01 17:59:13.452126 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:13.452086    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c0e28fa-dd0b-455f-8be4-d9bb36f8833b-kube-rbac-proxy-sar-config\") pod \"8c0e28fa-dd0b-455f-8be4-d9bb36f8833b\" (UID: \"8c0e28fa-dd0b-455f-8be4-d9bb36f8833b\") "
Jul 01 17:59:13.452126 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:13.452121    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c0e28fa-dd0b-455f-8be4-d9bb36f8833b-proxy-tls\") pod \"8c0e28fa-dd0b-455f-8be4-d9bb36f8833b\" (UID: \"8c0e28fa-dd0b-455f-8be4-d9bb36f8833b\") "
Jul 01 17:59:13.452437 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:13.452417    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c0e28fa-dd0b-455f-8be4-d9bb36f8833b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8c0e28fa-dd0b-455f-8be4-d9bb36f8833b" (UID: "8c0e28fa-dd0b-455f-8be4-d9bb36f8833b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:59:13.454254 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:13.454226    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c0e28fa-dd0b-455f-8be4-d9bb36f8833b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8c0e28fa-dd0b-455f-8be4-d9bb36f8833b" (UID: "8c0e28fa-dd0b-455f-8be4-d9bb36f8833b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:59:13.454370 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:13.454318    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c0e28fa-dd0b-455f-8be4-d9bb36f8833b-kube-api-access-ts2cx" (OuterVolumeSpecName: "kube-api-access-ts2cx") pod "8c0e28fa-dd0b-455f-8be4-d9bb36f8833b" (UID: "8c0e28fa-dd0b-455f-8be4-d9bb36f8833b"). InnerVolumeSpecName "kube-api-access-ts2cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:59:13.553600 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:13.553520    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c0e28fa-dd0b-455f-8be4-d9bb36f8833b-proxy-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:59:13.553600 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:13.553597    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts2cx\" (UniqueName: \"kubernetes.io/projected/8c0e28fa-dd0b-455f-8be4-d9bb36f8833b-kube-api-access-ts2cx\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:59:13.553789 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:13.553615    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c0e28fa-dd0b-455f-8be4-d9bb36f8833b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:59:14.306146 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:14.306117    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m"
Jul 01 17:59:14.306146 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:14.306133    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m" event={"ID":"8c0e28fa-dd0b-455f-8be4-d9bb36f8833b","Type":"ContainerDied","Data":"3ca82d2b8f427f214c9ede7005f6a91921de6770ca9c9b4799f557b57713381e"}
Jul 01 17:59:14.306634 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:14.306201    2585 scope.go:117] "RemoveContainer" containerID="2aad47135c4296636232b4e4b896fdd07998cc365b56693fb3ce4ca7c2a5c6e1"
Jul 01 17:59:14.314690 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:14.314672    2585 scope.go:117] "RemoveContainer" containerID="66f8ff32a5c3b98c63180087f4105d2df6eb988fd35d446a30b10fc729ae0dad"
Jul 01 17:59:14.329832 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:14.329812    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m"]
Jul 01 17:59:14.333634 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:14.333615    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-42b28-predictor-544cbc8654-8458m"]
Jul 01 17:59:16.175798 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:16.175765    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c0e28fa-dd0b-455f-8be4-d9bb36f8833b" path="/var/lib/kubelet/pods/8c0e28fa-dd0b-455f-8be4-d9bb36f8833b/volumes"
Jul 01 17:59:18.322844 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:18.322797    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-42b28-5f7d665cfb-9t2zj" podUID="f9b45c3b-ae08-4355-8930-ecdfd03190d4" containerName="splitter-graph-42b28" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:59:18.323230 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:18.322956    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-42b28-5f7d665cfb-9t2zj"
Jul 01 17:59:23.323161 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:23.323117    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-42b28-5f7d665cfb-9t2zj" podUID="f9b45c3b-ae08-4355-8930-ecdfd03190d4" containerName="splitter-graph-42b28" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:59:28.322242 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:28.322202    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-42b28-5f7d665cfb-9t2zj" podUID="f9b45c3b-ae08-4355-8930-ecdfd03190d4" containerName="splitter-graph-42b28" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:59:33.322828 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:33.322781    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-42b28-5f7d665cfb-9t2zj" podUID="f9b45c3b-ae08-4355-8930-ecdfd03190d4" containerName="splitter-graph-42b28" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:59:36.373680 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:36.373653    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-42b28-5f7d665cfb-9t2zj"
Jul 01 17:59:36.402147 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:36.402121    2585 generic.go:356] "Generic (PLEG): container finished" podID="f9b45c3b-ae08-4355-8930-ecdfd03190d4" containerID="57cbfdb09b8bc556861dec03750d53a4d5b22c01cff3da8e115a4a477456e366" exitCode=0
Jul 01 17:59:36.402303 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:36.402160    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-42b28-5f7d665cfb-9t2zj"
Jul 01 17:59:36.402303 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:36.402167    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-42b28-5f7d665cfb-9t2zj" event={"ID":"f9b45c3b-ae08-4355-8930-ecdfd03190d4","Type":"ContainerDied","Data":"57cbfdb09b8bc556861dec03750d53a4d5b22c01cff3da8e115a4a477456e366"}
Jul 01 17:59:36.402303 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:36.402209    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-42b28-5f7d665cfb-9t2zj" event={"ID":"f9b45c3b-ae08-4355-8930-ecdfd03190d4","Type":"ContainerDied","Data":"6bca3aed174cc74db250a7da7fc7bc12249f481a9c1c1cb6901867777807778d"}
Jul 01 17:59:36.402303 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:36.402232    2585 scope.go:117] "RemoveContainer" containerID="57cbfdb09b8bc556861dec03750d53a4d5b22c01cff3da8e115a4a477456e366"
Jul 01 17:59:36.410214 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:36.410194    2585 scope.go:117] "RemoveContainer" containerID="57cbfdb09b8bc556861dec03750d53a4d5b22c01cff3da8e115a4a477456e366"
Jul 01 17:59:36.410476 ip-10-0-141-213 kubenswrapper[2585]: E0701 17:59:36.410456    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57cbfdb09b8bc556861dec03750d53a4d5b22c01cff3da8e115a4a477456e366\": container with ID starting with 57cbfdb09b8bc556861dec03750d53a4d5b22c01cff3da8e115a4a477456e366 not found: ID does not exist" containerID="57cbfdb09b8bc556861dec03750d53a4d5b22c01cff3da8e115a4a477456e366"
Jul 01 17:59:36.410538 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:36.410488    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57cbfdb09b8bc556861dec03750d53a4d5b22c01cff3da8e115a4a477456e366"} err="failed to get container status \"57cbfdb09b8bc556861dec03750d53a4d5b22c01cff3da8e115a4a477456e366\": rpc error: code = NotFound desc = could not find container \"57cbfdb09b8bc556861dec03750d53a4d5b22c01cff3da8e115a4a477456e366\": container with ID starting with 57cbfdb09b8bc556861dec03750d53a4d5b22c01cff3da8e115a4a477456e366 not found: ID does not exist"
Jul 01 17:59:36.423247 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:36.423225    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9b45c3b-ae08-4355-8930-ecdfd03190d4-proxy-tls\") pod \"f9b45c3b-ae08-4355-8930-ecdfd03190d4\" (UID: \"f9b45c3b-ae08-4355-8930-ecdfd03190d4\") "
Jul 01 17:59:36.423354 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:36.423316    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9b45c3b-ae08-4355-8930-ecdfd03190d4-openshift-service-ca-bundle\") pod \"f9b45c3b-ae08-4355-8930-ecdfd03190d4\" (UID: \"f9b45c3b-ae08-4355-8930-ecdfd03190d4\") "
Jul 01 17:59:36.423685 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:36.423661    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9b45c3b-ae08-4355-8930-ecdfd03190d4-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "f9b45c3b-ae08-4355-8930-ecdfd03190d4" (UID: "f9b45c3b-ae08-4355-8930-ecdfd03190d4"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:59:36.425435 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:36.425417    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9b45c3b-ae08-4355-8930-ecdfd03190d4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f9b45c3b-ae08-4355-8930-ecdfd03190d4" (UID: "f9b45c3b-ae08-4355-8930-ecdfd03190d4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:59:36.524697 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:36.524613    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9b45c3b-ae08-4355-8930-ecdfd03190d4-proxy-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:59:36.524697 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:36.524643    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9b45c3b-ae08-4355-8930-ecdfd03190d4-openshift-service-ca-bundle\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 17:59:36.723685 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:36.723653    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-42b28-5f7d665cfb-9t2zj"]
Jul 01 17:59:36.727276 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:36.727246    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-42b28-5f7d665cfb-9t2zj"]
Jul 01 17:59:38.172121 ip-10-0-141-213 kubenswrapper[2585]: I0701 17:59:38.172091    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9b45c3b-ae08-4355-8930-ecdfd03190d4" path="/var/lib/kubelet/pods/f9b45c3b-ae08-4355-8930-ecdfd03190d4/volumes"
Jul 01 18:00:12.313588 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:00:12.313493    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8tsw_500d2a90-2d21-42b3-9584-b04c14becaec/ovn-acl-logging/0.log"
Jul 01 18:00:12.321037 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:00:12.321015    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8tsw_500d2a90-2d21-42b3-9584-b04c14becaec/ovn-acl-logging/0.log"
Jul 01 18:05:12.334830 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:05:12.334799    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8tsw_500d2a90-2d21-42b3-9584-b04c14becaec/ovn-acl-logging/0.log"
Jul 01 18:05:12.342786 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:05:12.342763    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8tsw_500d2a90-2d21-42b3-9584-b04c14becaec/ovn-acl-logging/0.log"
Jul 01 18:06:56.466741 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:06:56.466694    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-4822a-76849cbcd4-cg6kj"]
Jul 01 18:06:56.467247 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:06:56.466964    2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-4822a-76849cbcd4-cg6kj" podUID="70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f" containerName="switch-graph-4822a" containerID="cri-o://b8ea285619d2e540381041295566f92bbba7721477b9703fa24f1373a9d73810" gracePeriod=30
Jul 01 18:06:56.652480 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:06:56.652445    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56"]
Jul 01 18:06:56.652804 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:06:56.652749    2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56" podUID="86b90960-cd85-4aa7-b01e-100c95520755" containerName="kserve-container" containerID="cri-o://154b075b0860d4abbcccd454f111b9fe64bc1c586004ed8cb8c5850c52331f8d" gracePeriod=30
Jul 01 18:06:56.652948 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:06:56.652835    2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56" podUID="86b90960-cd85-4aa7-b01e-100c95520755" containerName="kube-rbac-proxy" containerID="cri-o://5c65d0f34ecd86538451867b4f5c7c27de67806a88ffb2c968fb0ab09438820b" gracePeriod=30
Jul 01 18:06:56.995712 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:06:56.995678    2585 generic.go:356] "Generic (PLEG): container finished" podID="86b90960-cd85-4aa7-b01e-100c95520755" containerID="5c65d0f34ecd86538451867b4f5c7c27de67806a88ffb2c968fb0ab09438820b" exitCode=2
Jul 01 18:06:56.995877 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:06:56.995741    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56" event={"ID":"86b90960-cd85-4aa7-b01e-100c95520755","Type":"ContainerDied","Data":"5c65d0f34ecd86538451867b4f5c7c27de67806a88ffb2c968fb0ab09438820b"}
Jul 01 18:06:59.691125 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:06:59.691098    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56"
Jul 01 18:06:59.754502 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:06:59.754473    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86b90960-cd85-4aa7-b01e-100c95520755-proxy-tls\") pod \"86b90960-cd85-4aa7-b01e-100c95520755\" (UID: \"86b90960-cd85-4aa7-b01e-100c95520755\") "
Jul 01 18:06:59.754687 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:06:59.754518    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c57sx\" (UniqueName: \"kubernetes.io/projected/86b90960-cd85-4aa7-b01e-100c95520755-kube-api-access-c57sx\") pod \"86b90960-cd85-4aa7-b01e-100c95520755\" (UID: \"86b90960-cd85-4aa7-b01e-100c95520755\") "
Jul 01 18:06:59.754687 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:06:59.754539    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86b90960-cd85-4aa7-b01e-100c95520755-kube-rbac-proxy-sar-config\") pod \"86b90960-cd85-4aa7-b01e-100c95520755\" (UID: \"86b90960-cd85-4aa7-b01e-100c95520755\") "
Jul 01 18:06:59.754971 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:06:59.754936    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86b90960-cd85-4aa7-b01e-100c95520755-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "86b90960-cd85-4aa7-b01e-100c95520755" (UID: "86b90960-cd85-4aa7-b01e-100c95520755"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:06:59.756807 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:06:59.756783    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86b90960-cd85-4aa7-b01e-100c95520755-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "86b90960-cd85-4aa7-b01e-100c95520755" (UID: "86b90960-cd85-4aa7-b01e-100c95520755"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:06:59.756907 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:06:59.756815    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86b90960-cd85-4aa7-b01e-100c95520755-kube-api-access-c57sx" (OuterVolumeSpecName: "kube-api-access-c57sx") pod "86b90960-cd85-4aa7-b01e-100c95520755" (UID: "86b90960-cd85-4aa7-b01e-100c95520755"). InnerVolumeSpecName "kube-api-access-c57sx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:06:59.855359 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:06:59.855327    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86b90960-cd85-4aa7-b01e-100c95520755-proxy-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 18:06:59.855359 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:06:59.855354    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c57sx\" (UniqueName: \"kubernetes.io/projected/86b90960-cd85-4aa7-b01e-100c95520755-kube-api-access-c57sx\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 18:06:59.855359 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:06:59.855365    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86b90960-cd85-4aa7-b01e-100c95520755-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 18:07:00.007238 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:00.007206    2585 generic.go:356] "Generic (PLEG): container finished" podID="86b90960-cd85-4aa7-b01e-100c95520755" containerID="154b075b0860d4abbcccd454f111b9fe64bc1c586004ed8cb8c5850c52331f8d" exitCode=0
Jul 01 18:07:00.007238 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:00.007242    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56" event={"ID":"86b90960-cd85-4aa7-b01e-100c95520755","Type":"ContainerDied","Data":"154b075b0860d4abbcccd454f111b9fe64bc1c586004ed8cb8c5850c52331f8d"}
Jul 01 18:07:00.007442 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:00.007253    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56"
Jul 01 18:07:00.007442 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:00.007291    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56" event={"ID":"86b90960-cd85-4aa7-b01e-100c95520755","Type":"ContainerDied","Data":"0d8b25cf15aa12d20b197c4c781d10ebc733ea8cc9d628498c76e41104f162c2"}
Jul 01 18:07:00.007442 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:00.007315    2585 scope.go:117] "RemoveContainer" containerID="5c65d0f34ecd86538451867b4f5c7c27de67806a88ffb2c968fb0ab09438820b"
Jul 01 18:07:00.017428 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:00.017342    2585 scope.go:117] "RemoveContainer" containerID="154b075b0860d4abbcccd454f111b9fe64bc1c586004ed8cb8c5850c52331f8d"
Jul 01 18:07:00.024918 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:00.024899    2585 scope.go:117] "RemoveContainer" containerID="5c65d0f34ecd86538451867b4f5c7c27de67806a88ffb2c968fb0ab09438820b"
Jul 01 18:07:00.025157 ip-10-0-141-213 kubenswrapper[2585]: E0701 18:07:00.025138    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c65d0f34ecd86538451867b4f5c7c27de67806a88ffb2c968fb0ab09438820b\": container with ID starting with 5c65d0f34ecd86538451867b4f5c7c27de67806a88ffb2c968fb0ab09438820b not found: ID does not exist" containerID="5c65d0f34ecd86538451867b4f5c7c27de67806a88ffb2c968fb0ab09438820b"
Jul 01 18:07:00.025212 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:00.025166    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c65d0f34ecd86538451867b4f5c7c27de67806a88ffb2c968fb0ab09438820b"} err="failed to get container status \"5c65d0f34ecd86538451867b4f5c7c27de67806a88ffb2c968fb0ab09438820b\": rpc error: code = NotFound desc = could not find container \"5c65d0f34ecd86538451867b4f5c7c27de67806a88ffb2c968fb0ab09438820b\": container with ID starting with 5c65d0f34ecd86538451867b4f5c7c27de67806a88ffb2c968fb0ab09438820b not found: ID does not exist"
Jul 01 18:07:00.025212 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:00.025184    2585 scope.go:117] "RemoveContainer" containerID="154b075b0860d4abbcccd454f111b9fe64bc1c586004ed8cb8c5850c52331f8d"
Jul 01 18:07:00.025462 ip-10-0-141-213 kubenswrapper[2585]: E0701 18:07:00.025442    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"154b075b0860d4abbcccd454f111b9fe64bc1c586004ed8cb8c5850c52331f8d\": container with ID starting with 154b075b0860d4abbcccd454f111b9fe64bc1c586004ed8cb8c5850c52331f8d not found: ID does not exist" containerID="154b075b0860d4abbcccd454f111b9fe64bc1c586004ed8cb8c5850c52331f8d"
Jul 01 18:07:00.025524 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:00.025474    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"154b075b0860d4abbcccd454f111b9fe64bc1c586004ed8cb8c5850c52331f8d"} err="failed to get container status \"154b075b0860d4abbcccd454f111b9fe64bc1c586004ed8cb8c5850c52331f8d\": rpc error: code = NotFound desc = could not find container \"154b075b0860d4abbcccd454f111b9fe64bc1c586004ed8cb8c5850c52331f8d\": container with ID starting with 154b075b0860d4abbcccd454f111b9fe64bc1c586004ed8cb8c5850c52331f8d not found: ID does not exist"
Jul 01 18:07:00.029423 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:00.029403    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56"]
Jul 01 18:07:00.033679 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:00.033661    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4822a-predictor-6fb5b8b67b-jnl56"]
Jul 01 18:07:00.171803 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:00.171719    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86b90960-cd85-4aa7-b01e-100c95520755" path="/var/lib/kubelet/pods/86b90960-cd85-4aa7-b01e-100c95520755/volumes"
Jul 01 18:07:00.268821 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:00.268786    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-4822a-76849cbcd4-cg6kj" podUID="70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f" containerName="switch-graph-4822a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:07:05.268216 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:05.268171    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-4822a-76849cbcd4-cg6kj" podUID="70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f" containerName="switch-graph-4822a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:07:10.268369 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:10.268324    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-4822a-76849cbcd4-cg6kj" podUID="70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f" containerName="switch-graph-4822a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:07:10.268782 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:10.268440    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-4822a-76849cbcd4-cg6kj"
Jul 01 18:07:13.179737 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:13.179707    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-4822a-76849cbcd4-cg6kj_70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f/switch-graph-4822a/0.log"
Jul 01 18:07:13.980125 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:13.980094    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-4822a-76849cbcd4-cg6kj_70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f/switch-graph-4822a/0.log"
Jul 01 18:07:14.788992 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:14.788959    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-4822a-76849cbcd4-cg6kj_70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f/switch-graph-4822a/0.log"
Jul 01 18:07:15.268353 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:15.268305    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-4822a-76849cbcd4-cg6kj" podUID="70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f" containerName="switch-graph-4822a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:07:15.582750 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:15.582719    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-4822a-76849cbcd4-cg6kj_70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f/switch-graph-4822a/0.log"
Jul 01 18:07:16.366818 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:16.366787    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-4822a-76849cbcd4-cg6kj_70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f/switch-graph-4822a/0.log"
Jul 01 18:07:17.166537 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:17.166507    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-4822a-76849cbcd4-cg6kj_70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f/switch-graph-4822a/0.log"
Jul 01 18:07:17.977570 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:17.977539    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-4822a-76849cbcd4-cg6kj_70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f/switch-graph-4822a/0.log"
Jul 01 18:07:18.763943 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:18.763916    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-4822a-76849cbcd4-cg6kj_70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f/switch-graph-4822a/0.log"
Jul 01 18:07:19.543285 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:19.543253    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-4822a-76849cbcd4-cg6kj_70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f/switch-graph-4822a/0.log"
Jul 01 18:07:20.269173 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:20.269136    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-4822a-76849cbcd4-cg6kj" podUID="70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f" containerName="switch-graph-4822a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:07:20.324395 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:20.324370    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-4822a-76849cbcd4-cg6kj_70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f/switch-graph-4822a/0.log"
Jul 01 18:07:21.105758 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:21.105727    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-4822a-76849cbcd4-cg6kj_70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f/switch-graph-4822a/0.log"
Jul 01 18:07:21.876965 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:21.876930    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-4822a-76849cbcd4-cg6kj_70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f/switch-graph-4822a/0.log"
Jul 01 18:07:22.668889 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:22.668859    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-4822a-76849cbcd4-cg6kj_70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f/switch-graph-4822a/0.log"
Jul 01 18:07:23.453393 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:23.453367    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-4822a-76849cbcd4-cg6kj_70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f/switch-graph-4822a/0.log"
Jul 01 18:07:25.268887 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:25.268852    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-4822a-76849cbcd4-cg6kj" podUID="70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f" containerName="switch-graph-4822a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:07:26.619323 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:26.619289    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-4822a-76849cbcd4-cg6kj"
Jul 01 18:07:26.650225 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:26.650197    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f-proxy-tls\") pod \"70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f\" (UID: \"70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f\") "
Jul 01 18:07:26.650383 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:26.650256    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f-openshift-service-ca-bundle\") pod \"70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f\" (UID: \"70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f\") "
Jul 01 18:07:26.650607 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:26.650564    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f" (UID: "70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:07:26.652431 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:26.652409    2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f" (UID: "70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:07:26.751263 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:26.751180    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f-proxy-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 18:07:26.751263 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:26.751211    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f-openshift-service-ca-bundle\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\""
Jul 01 18:07:27.108914 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:27.108880    2585 generic.go:356] "Generic (PLEG): container finished" podID="70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f" containerID="b8ea285619d2e540381041295566f92bbba7721477b9703fa24f1373a9d73810" exitCode=0
Jul 01 18:07:27.109069 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:27.108919    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-4822a-76849cbcd4-cg6kj"
Jul 01 18:07:27.109069 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:27.108961    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-4822a-76849cbcd4-cg6kj" event={"ID":"70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f","Type":"ContainerDied","Data":"b8ea285619d2e540381041295566f92bbba7721477b9703fa24f1373a9d73810"}
Jul 01 18:07:27.109069 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:27.109008    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-4822a-76849cbcd4-cg6kj" event={"ID":"70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f","Type":"ContainerDied","Data":"96057af699c9bd8217d310f8665659f7e3dc341d9110201137fbc197d855bf5a"}
Jul 01 18:07:27.109069 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:27.109029    2585 scope.go:117] "RemoveContainer" containerID="b8ea285619d2e540381041295566f92bbba7721477b9703fa24f1373a9d73810"
Jul 01 18:07:27.117547 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:27.117528    2585 scope.go:117] "RemoveContainer" containerID="b8ea285619d2e540381041295566f92bbba7721477b9703fa24f1373a9d73810"
Jul 01 18:07:27.117805 ip-10-0-141-213 kubenswrapper[2585]: E0701 18:07:27.117789    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8ea285619d2e540381041295566f92bbba7721477b9703fa24f1373a9d73810\": container with ID starting with b8ea285619d2e540381041295566f92bbba7721477b9703fa24f1373a9d73810 not found: ID does not exist" containerID="b8ea285619d2e540381041295566f92bbba7721477b9703fa24f1373a9d73810"
Jul 01 18:07:27.117859 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:27.117812    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8ea285619d2e540381041295566f92bbba7721477b9703fa24f1373a9d73810"} err="failed to get container status \"b8ea285619d2e540381041295566f92bbba7721477b9703fa24f1373a9d73810\": rpc error: code = NotFound desc = could not find container \"b8ea285619d2e540381041295566f92bbba7721477b9703fa24f1373a9d73810\": container with ID starting with b8ea285619d2e540381041295566f92bbba7721477b9703fa24f1373a9d73810 not found: ID does not exist"
Jul 01 18:07:27.129522 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:27.129496    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-4822a-76849cbcd4-cg6kj"]
Jul 01 18:07:27.133434 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:27.133415    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-4822a-76849cbcd4-cg6kj"]
Jul 01 18:07:28.171943 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:28.171909    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f" path="/var/lib/kubelet/pods/70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f/volumes"
Jul 01 18:07:28.944814 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:28.944784    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-49wz8/must-gather-z2dx2"]
Jul 01 18:07:28.945035 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:28.945022    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c0e28fa-dd0b-455f-8be4-d9bb36f8833b" containerName="kserve-container"
Jul 01 18:07:28.945080 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:28.945036    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c0e28fa-dd0b-455f-8be4-d9bb36f8833b" containerName="kserve-container"
Jul 01 18:07:28.945080 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:28.945049    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86b90960-cd85-4aa7-b01e-100c95520755" containerName="kube-rbac-proxy"
Jul 01 18:07:28.945080 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:28.945054    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="86b90960-cd85-4aa7-b01e-100c95520755" containerName="kube-rbac-proxy"
Jul 01 18:07:28.945080 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:28.945063    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f" containerName="switch-graph-4822a"
Jul 01 18:07:28.945080 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:28.945068    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f" containerName="switch-graph-4822a"
Jul 01 18:07:28.945080 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:28.945074    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9b45c3b-ae08-4355-8930-ecdfd03190d4" containerName="splitter-graph-42b28"
Jul 01 18:07:28.945080 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:28.945079    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9b45c3b-ae08-4355-8930-ecdfd03190d4" containerName="splitter-graph-42b28"
Jul 01 18:07:28.945301 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:28.945091    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86b90960-cd85-4aa7-b01e-100c95520755" containerName="kserve-container"
Jul 01 18:07:28.945301 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:28.945097    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="86b90960-cd85-4aa7-b01e-100c95520755" containerName="kserve-container"
Jul 01 18:07:28.945301 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:28.945103    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c0e28fa-dd0b-455f-8be4-d9bb36f8833b" containerName="kube-rbac-proxy"
Jul 01 18:07:28.945301 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:28.945108    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c0e28fa-dd0b-455f-8be4-d9bb36f8833b" containerName="kube-rbac-proxy"
Jul 01 18:07:28.945301 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:28.945146    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9b45c3b-ae08-4355-8930-ecdfd03190d4" containerName="splitter-graph-42b28"
Jul 01 18:07:28.945301 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:28.945154    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="86b90960-cd85-4aa7-b01e-100c95520755" containerName="kube-rbac-proxy"
Jul 01 18:07:28.945301 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:28.945159    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c0e28fa-dd0b-455f-8be4-d9bb36f8833b" containerName="kube-rbac-proxy"
Jul 01 18:07:28.945301 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:28.945165    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="86b90960-cd85-4aa7-b01e-100c95520755" containerName="kserve-container"
Jul 01 18:07:28.945301 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:28.945171    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="70eef7eb-09d3-4f6d-98a1-3eeb00e09d7f" containerName="switch-graph-4822a"
Jul 01 18:07:28.945301 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:28.945177    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c0e28fa-dd0b-455f-8be4-d9bb36f8833b" containerName="kserve-container"
Jul 01 18:07:28.949272 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:28.949254    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-49wz8/must-gather-z2dx2"
Jul 01 18:07:28.951696 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:28.951663    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-49wz8\"/\"default-dockercfg-9fgpq\""
Jul 01 18:07:28.951810 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:28.951723    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-49wz8\"/\"openshift-service-ca.crt\""
Jul 01 18:07:28.951863 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:28.951831    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-49wz8\"/\"kube-root-ca.crt\""
Jul 01 18:07:28.956936 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:28.956914    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-49wz8/must-gather-z2dx2"]
Jul 01 18:07:29.068277 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:29.068246    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hrgg\" (UniqueName: \"kubernetes.io/projected/b412a2de-676c-4101-bfb2-97ef2013b493-kube-api-access-5hrgg\") pod \"must-gather-z2dx2\" (UID: \"b412a2de-676c-4101-bfb2-97ef2013b493\") " pod="openshift-must-gather-49wz8/must-gather-z2dx2"
Jul 01 18:07:29.068479 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:29.068301    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b412a2de-676c-4101-bfb2-97ef2013b493-must-gather-output\") pod \"must-gather-z2dx2\" (UID: \"b412a2de-676c-4101-bfb2-97ef2013b493\") " pod="openshift-must-gather-49wz8/must-gather-z2dx2"
Jul 01 18:07:29.169308 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:29.169253    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b412a2de-676c-4101-bfb2-97ef2013b493-must-gather-output\") pod \"must-gather-z2dx2\" (UID: \"b412a2de-676c-4101-bfb2-97ef2013b493\") " pod="openshift-must-gather-49wz8/must-gather-z2dx2"
Jul 01 18:07:29.169308 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:29.169323    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hrgg\" (UniqueName: \"kubernetes.io/projected/b412a2de-676c-4101-bfb2-97ef2013b493-kube-api-access-5hrgg\") pod \"must-gather-z2dx2\" (UID: \"b412a2de-676c-4101-bfb2-97ef2013b493\") " pod="openshift-must-gather-49wz8/must-gather-z2dx2"
Jul 01 18:07:29.169643 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:29.169621    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b412a2de-676c-4101-bfb2-97ef2013b493-must-gather-output\") pod \"must-gather-z2dx2\" (UID: \"b412a2de-676c-4101-bfb2-97ef2013b493\") " pod="openshift-must-gather-49wz8/must-gather-z2dx2"
Jul 01 18:07:29.177694 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:29.177669    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hrgg\" (UniqueName: \"kubernetes.io/projected/b412a2de-676c-4101-bfb2-97ef2013b493-kube-api-access-5hrgg\") pod \"must-gather-z2dx2\" (UID: \"b412a2de-676c-4101-bfb2-97ef2013b493\") " pod="openshift-must-gather-49wz8/must-gather-z2dx2"
Jul 01 18:07:29.258710 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:29.258609    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-49wz8/must-gather-z2dx2"
Jul 01 18:07:29.386487 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:29.386450    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-49wz8/must-gather-z2dx2"]
Jul 01 18:07:29.389272 ip-10-0-141-213 kubenswrapper[2585]: W0701 18:07:29.389238    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb412a2de_676c_4101_bfb2_97ef2013b493.slice/crio-74d1593c6a2eb3a55c157252f918f4f99e08638b942fe6550a6e006699730551 WatchSource:0}: Error finding container 74d1593c6a2eb3a55c157252f918f4f99e08638b942fe6550a6e006699730551: Status 404 returned error can't find the container with id 74d1593c6a2eb3a55c157252f918f4f99e08638b942fe6550a6e006699730551
Jul 01 18:07:29.391408 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:29.391390    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 18:07:30.122861 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:30.122824    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-49wz8/must-gather-z2dx2" event={"ID":"b412a2de-676c-4101-bfb2-97ef2013b493","Type":"ContainerStarted","Data":"74d1593c6a2eb3a55c157252f918f4f99e08638b942fe6550a6e006699730551"}
Jul 01 18:07:31.130269 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:31.129452    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-49wz8/must-gather-z2dx2" event={"ID":"b412a2de-676c-4101-bfb2-97ef2013b493","Type":"ContainerStarted","Data":"cb27323aaedcc52c9dbbb26139c032abf460bc6b3f8e3bf01d0330f38ecfe8aa"}
Jul 01 18:07:31.130269 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:31.129504    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-49wz8/must-gather-z2dx2" event={"ID":"b412a2de-676c-4101-bfb2-97ef2013b493","Type":"ContainerStarted","Data":"448185c951331cbfca7fd6b1e59df0b47b8741263e6e52a2c15a7cb818c878f8"}
Jul 01 18:07:31.146280 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:31.146233    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-49wz8/must-gather-z2dx2" podStartSLOduration=1.971346165 podStartE2EDuration="3.146219782s" totalImagesPullingTime="1.174873617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:07:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:07:31.146142946 +0000 UTC m=+3139.451341125" watchObservedRunningTime="2026-07-01 18:07:31.146219782 +0000 UTC m=+3139.451417960"
Jul 01 18:07:31.872843 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:31.872812    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-knvdq_3defaf9e-f5ef-45ea-a02e-4729f15db90c/global-pull-secret-syncer/0.log"
Jul 01 18:07:32.011750 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:32.011678    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mgc4x_5db90618-fcdb-4ea6-ac95-a2d535908e0e/konnectivity-agent/0.log"
Jul 01 18:07:32.079336 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:32.079309    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-213.ec2.internal_608b014fc4a8976e8619472b298b39c8/haproxy/0.log"
Jul 01 18:07:36.770334 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:36.770300    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2mb48_b5d1f6d5-d5b2-456c-8a97-74fb30efc048/node-exporter/0.log"
Jul 01 18:07:36.793456 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:36.793417    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2mb48_b5d1f6d5-d5b2-456c-8a97-74fb30efc048/kube-rbac-proxy/0.log"
Jul 01 18:07:36.817993 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:36.817968    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2mb48_b5d1f6d5-d5b2-456c-8a97-74fb30efc048/init-textfile/0.log"
Jul 01 18:07:38.762745 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:38.762717    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cdb5968b9-phc4p_dece537c-3a74-44b7-b4bf-ef9fbd0f1d01/networking-console-plugin/0.log"
Jul 01 18:07:40.347325 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:40.347289    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-49wz8/perf-node-gather-daemonset-zbppc"]
Jul 01 18:07:40.353137 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:40.353109    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-49wz8/perf-node-gather-daemonset-zbppc"
Jul 01 18:07:40.363422 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:40.363392    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-49wz8/perf-node-gather-daemonset-zbppc"]
Jul 01 18:07:40.472398 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:40.472366    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn8n9\" (UniqueName: \"kubernetes.io/projected/52fdbb45-a27a-40db-8a29-1212be66a142-kube-api-access-rn8n9\") pod \"perf-node-gather-daemonset-zbppc\" (UID: \"52fdbb45-a27a-40db-8a29-1212be66a142\") " pod="openshift-must-gather-49wz8/perf-node-gather-daemonset-zbppc"
Jul 01 18:07:40.472600 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:40.472418    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/52fdbb45-a27a-40db-8a29-1212be66a142-podres\") pod \"perf-node-gather-daemonset-zbppc\" (UID: \"52fdbb45-a27a-40db-8a29-1212be66a142\") " pod="openshift-must-gather-49wz8/perf-node-gather-daemonset-zbppc"
Jul 01 18:07:40.472600 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:40.472482    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/52fdbb45-a27a-40db-8a29-1212be66a142-proc\") pod \"perf-node-gather-daemonset-zbppc\" (UID: \"52fdbb45-a27a-40db-8a29-1212be66a142\") " pod="openshift-must-gather-49wz8/perf-node-gather-daemonset-zbppc"
Jul 01 18:07:40.472600 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:40.472522    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52fdbb45-a27a-40db-8a29-1212be66a142-sys\") pod \"perf-node-gather-daemonset-zbppc\" (UID: \"52fdbb45-a27a-40db-8a29-1212be66a142\") " pod="openshift-must-gather-49wz8/perf-node-gather-daemonset-zbppc"
Jul 01 18:07:40.472600 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:40.472550    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/52fdbb45-a27a-40db-8a29-1212be66a142-lib-modules\") pod \"perf-node-gather-daemonset-zbppc\" (UID: \"52fdbb45-a27a-40db-8a29-1212be66a142\") " pod="openshift-must-gather-49wz8/perf-node-gather-daemonset-zbppc"
Jul 01 18:07:40.573018 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:40.572977    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/52fdbb45-a27a-40db-8a29-1212be66a142-podres\") pod \"perf-node-gather-daemonset-zbppc\" (UID: \"52fdbb45-a27a-40db-8a29-1212be66a142\") " pod="openshift-must-gather-49wz8/perf-node-gather-daemonset-zbppc"
Jul 01 18:07:40.573325 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:40.573279    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/52fdbb45-a27a-40db-8a29-1212be66a142-podres\") pod \"perf-node-gather-daemonset-zbppc\" (UID: \"52fdbb45-a27a-40db-8a29-1212be66a142\") " pod="openshift-must-gather-49wz8/perf-node-gather-daemonset-zbppc"
Jul 01 18:07:40.573506 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:40.573391    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/52fdbb45-a27a-40db-8a29-1212be66a142-proc\") pod \"perf-node-gather-daemonset-zbppc\" (UID: \"52fdbb45-a27a-40db-8a29-1212be66a142\") " pod="openshift-must-gather-49wz8/perf-node-gather-daemonset-zbppc"
Jul 01 18:07:40.573678 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:40.573663    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52fdbb45-a27a-40db-8a29-1212be66a142-sys\") pod \"perf-node-gather-daemonset-zbppc\" (UID: \"52fdbb45-a27a-40db-8a29-1212be66a142\") " pod="openshift-must-gather-49wz8/perf-node-gather-daemonset-zbppc"
Jul 01 18:07:40.573959 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:40.573873    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52fdbb45-a27a-40db-8a29-1212be66a142-sys\") pod \"perf-node-gather-daemonset-zbppc\" (UID: \"52fdbb45-a27a-40db-8a29-1212be66a142\") " pod="openshift-must-gather-49wz8/perf-node-gather-daemonset-zbppc"
Jul 01 18:07:40.574257 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:40.573430    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/52fdbb45-a27a-40db-8a29-1212be66a142-proc\") pod \"perf-node-gather-daemonset-zbppc\" (UID: \"52fdbb45-a27a-40db-8a29-1212be66a142\") " pod="openshift-must-gather-49wz8/perf-node-gather-daemonset-zbppc"
Jul 01 18:07:40.574395 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:40.574196    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/52fdbb45-a27a-40db-8a29-1212be66a142-lib-modules\") pod \"perf-node-gather-daemonset-zbppc\" (UID: \"52fdbb45-a27a-40db-8a29-1212be66a142\") " pod="openshift-must-gather-49wz8/perf-node-gather-daemonset-zbppc"
Jul 01 18:07:40.574530 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:40.573922    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/52fdbb45-a27a-40db-8a29-1212be66a142-lib-modules\") pod \"perf-node-gather-daemonset-zbppc\" (UID: \"52fdbb45-a27a-40db-8a29-1212be66a142\") " pod="openshift-must-gather-49wz8/perf-node-gather-daemonset-zbppc"
Jul 01 18:07:40.574765 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:40.574746    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn8n9\" (UniqueName: \"kubernetes.io/projected/52fdbb45-a27a-40db-8a29-1212be66a142-kube-api-access-rn8n9\") pod \"perf-node-gather-daemonset-zbppc\" (UID: \"52fdbb45-a27a-40db-8a29-1212be66a142\") " pod="openshift-must-gather-49wz8/perf-node-gather-daemonset-zbppc"
Jul 01 18:07:40.583969 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:40.583931    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn8n9\" (UniqueName: \"kubernetes.io/projected/52fdbb45-a27a-40db-8a29-1212be66a142-kube-api-access-rn8n9\") pod \"perf-node-gather-daemonset-zbppc\" (UID: \"52fdbb45-a27a-40db-8a29-1212be66a142\") " pod="openshift-must-gather-49wz8/perf-node-gather-daemonset-zbppc"
Jul 01 18:07:40.668121 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:40.668019    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-49wz8/perf-node-gather-daemonset-zbppc"
Jul 01 18:07:40.833311 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:40.833243    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-49wz8/perf-node-gather-daemonset-zbppc"]
Jul 01 18:07:41.014604 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:41.012524    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n2fz9_67bb5c8b-7164-4be3-a11a-9d6755b5a6be/dns/0.log"
Jul 01 18:07:41.034382 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:41.034339    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n2fz9_67bb5c8b-7164-4be3-a11a-9d6755b5a6be/kube-rbac-proxy/0.log"
Jul 01 18:07:41.131835 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:41.131793    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jwvgk_3ff59413-27fa-4837-8549-f2a6898826a9/dns-node-resolver/0.log"
Jul 01 18:07:41.180635 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:41.180593    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-49wz8/perf-node-gather-daemonset-zbppc" event={"ID":"52fdbb45-a27a-40db-8a29-1212be66a142","Type":"ContainerStarted","Data":"5a19abfbe27ddf7be474057888caa5befae3b279ba294052200f018209869e11"}
Jul 01 18:07:41.180827 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:41.180691    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-49wz8/perf-node-gather-daemonset-zbppc" event={"ID":"52fdbb45-a27a-40db-8a29-1212be66a142","Type":"ContainerStarted","Data":"5ad3d7a9665d917b48ed1df1cbfd0e3dd0db5a6e903396f71bc3d37c321b3191"}
Jul 01 18:07:41.181427 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:41.181398    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-49wz8/perf-node-gather-daemonset-zbppc"
Jul 01 18:07:41.197151 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:41.197101    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-49wz8/perf-node-gather-daemonset-zbppc" podStartSLOduration=1.197087202 podStartE2EDuration="1.197087202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:07:41.196892718 +0000 UTC m=+3149.502090897" watchObservedRunningTime="2026-07-01 18:07:41.197087202 +0000 UTC m=+3149.502285379"
Jul 01 18:07:41.650064 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:41.650021    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r54bn_d6614097-0891-470c-b830-26bf595ff80e/node-ca/0.log"
Jul 01 18:07:42.780642 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:42.780610    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jq2fj_6ec33cc2-606d-4445-9f9b-4fe556dfbc6e/serve-healthcheck-canary/0.log"
Jul 01 18:07:43.249334 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:43.249298    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bzvrj_0b513528-3295-478a-92da-b184d6708e46/kube-rbac-proxy/0.log"
Jul 01 18:07:43.266775 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:43.266746    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bzvrj_0b513528-3295-478a-92da-b184d6708e46/exporter/0.log"
Jul 01 18:07:43.284455 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:43.284424    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bzvrj_0b513528-3295-478a-92da-b184d6708e46/extractor/0.log"
Jul 01 18:07:45.507434 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:45.507400    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-lxr8b_21b36ae9-4319-4dee-8bb1-c10f6d0b64a0/manager/0.log"
Jul 01 18:07:45.571564 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:45.571532    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-ls7gm_e39523e9-5d87-43ae-b7bb-63b0a5f8c9aa/seaweedfs/0.log"
Jul 01 18:07:48.199044 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:48.199015    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-49wz8/perf-node-gather-daemonset-zbppc"
Jul 01 18:07:50.581335 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:50.581305    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-89p2n_ebd82416-a8de-490c-a980-74946452dc40/kube-multus/0.log"
Jul 01 18:07:50.864464 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:50.864351    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hjngx_7bab8af3-8766-4098-95ab-720089120d08/kube-multus-additional-cni-plugins/0.log"
Jul 01 18:07:50.884122 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:50.884096    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hjngx_7bab8af3-8766-4098-95ab-720089120d08/egress-router-binary-copy/0.log"
Jul 01 18:07:50.901724 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:50.901700    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hjngx_7bab8af3-8766-4098-95ab-720089120d08/cni-plugins/0.log"
Jul 01 18:07:50.919087 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:50.919058    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hjngx_7bab8af3-8766-4098-95ab-720089120d08/bond-cni-plugin/0.log"
Jul 01 18:07:50.937830 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:50.937804    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hjngx_7bab8af3-8766-4098-95ab-720089120d08/routeoverride-cni/0.log"
Jul 01 18:07:50.957547 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:50.957502    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hjngx_7bab8af3-8766-4098-95ab-720089120d08/whereabouts-cni-bincopy/0.log"
Jul 01 18:07:50.976912 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:50.976878    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hjngx_7bab8af3-8766-4098-95ab-720089120d08/whereabouts-cni/0.log"
Jul 01 18:07:51.096142 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:51.096113    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4rrth_5026d3ed-589a-4390-a2b7-d5f64c06e687/network-metrics-daemon/0.log"
Jul 01 18:07:51.113235 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:51.113201    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4rrth_5026d3ed-589a-4390-a2b7-d5f64c06e687/kube-rbac-proxy/0.log"
Jul 01 18:07:52.231763 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:52.231692    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8tsw_500d2a90-2d21-42b3-9584-b04c14becaec/ovn-controller/0.log"
Jul 01 18:07:52.247128 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:52.247101    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8tsw_500d2a90-2d21-42b3-9584-b04c14becaec/ovn-acl-logging/1.log"
Jul 01 18:07:52.247128 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:52.247108    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8tsw_500d2a90-2d21-42b3-9584-b04c14becaec/ovn-acl-logging/0.log"
Jul 01 18:07:52.268285 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:52.268238    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8tsw_500d2a90-2d21-42b3-9584-b04c14becaec/kube-rbac-proxy-node/0.log"
Jul 01 18:07:52.287127 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:52.287099    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8tsw_500d2a90-2d21-42b3-9584-b04c14becaec/kube-rbac-proxy-ovn-metrics/0.log"
Jul 01 18:07:52.303346 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:52.303320    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8tsw_500d2a90-2d21-42b3-9584-b04c14becaec/northd/0.log"
Jul 01 18:07:52.321312 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:52.321268    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8tsw_500d2a90-2d21-42b3-9584-b04c14becaec/nbdb/0.log"
Jul 01 18:07:52.340274 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:52.340250    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8tsw_500d2a90-2d21-42b3-9584-b04c14becaec/sbdb/0.log"
Jul 01 18:07:52.392383 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:52.392348    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8tsw_500d2a90-2d21-42b3-9584-b04c14becaec/ovnkube-controller/0.log"
Jul 01 18:07:53.368461 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:53.368432    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-2x2qg_415f427a-adac-47bb-9cad-3befc6a711b6/network-check-target-container/0.log"
Jul 01 18:07:54.240816 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:54.240792    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-kb8gh_3af4c82a-7a73-4de0-8f7b-25768775f79e/iptables-alerter/0.log"
Jul 01 18:07:54.895316 ip-10-0-141-213 kubenswrapper[2585]: I0701 18:07:54.895281    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-v8wlm_4bccceba-0d8e-402a-a82f-accee4a1be82/tuned/0.log"