May 26 11:24:27.675099 ip-10-0-129-97 systemd[1]: Starting Kubernetes Kubelet...
May 26 11:24:28.183486 ip-10-0-129-97 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 11:24:28.183486 ip-10-0-129-97 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 26 11:24:28.183486 ip-10-0-129-97 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 11:24:28.183486 ip-10-0-129-97 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 26 11:24:28.183486 ip-10-0-129-97 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 11:24:28.187190 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.187094    2563 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 26 11:24:28.191132 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191108    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 11:24:28.191132 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191127    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 11:24:28.191132 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191130    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 11:24:28.191132 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191134    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 11:24:28.191132 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191136    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 11:24:28.191132 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191140    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 11:24:28.191343 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191143    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 11:24:28.191343 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191146    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 11:24:28.191343 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191150    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 11:24:28.191343 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191153    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 11:24:28.191343 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191155    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 11:24:28.191343 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191158    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 11:24:28.191343 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191160    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 11:24:28.191343 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191163    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 11:24:28.191343 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191165    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 11:24:28.191343 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191168    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 11:24:28.191343 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191171    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 11:24:28.191343 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191174    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 11:24:28.191343 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191176    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 11:24:28.191343 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191178    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 11:24:28.191343 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191181    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 11:24:28.191343 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191184    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 11:24:28.191343 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191186    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 11:24:28.191343 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191189    2563 feature_gate.go:328] unrecognized feature gate: Example
May 26 11:24:28.191343 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191191    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 11:24:28.191343 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191198    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 11:24:28.191829 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191201    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 11:24:28.191829 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191204    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 11:24:28.191829 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191208    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 11:24:28.191829 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191211    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 11:24:28.191829 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191215    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 11:24:28.191829 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191217    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 11:24:28.191829 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191220    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 11:24:28.191829 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191223    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 11:24:28.191829 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191225    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 11:24:28.191829 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191228    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 11:24:28.191829 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191230    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 11:24:28.191829 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191233    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 11:24:28.191829 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191235    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 11:24:28.191829 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191238    2563 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 11:24:28.191829 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191241    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 11:24:28.191829 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191243    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 11:24:28.191829 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191246    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 11:24:28.191829 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191249    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 11:24:28.191829 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191251    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 11:24:28.192299 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191254    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 11:24:28.192299 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191256    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 11:24:28.192299 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191259    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 11:24:28.192299 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191261    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 11:24:28.192299 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191264    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 11:24:28.192299 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191266    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 11:24:28.192299 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191268    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 11:24:28.192299 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191271    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 11:24:28.192299 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191273    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 11:24:28.192299 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191275    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 11:24:28.192299 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191279    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 11:24:28.192299 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191281    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 11:24:28.192299 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191283    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 11:24:28.192299 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191286    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 11:24:28.192299 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191288    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 11:24:28.192299 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191291    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 11:24:28.192299 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191293    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 11:24:28.192299 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191296    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 11:24:28.192299 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191299    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 11:24:28.192299 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191302    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 11:24:28.192831 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191305    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 11:24:28.192831 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191307    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 11:24:28.192831 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191310    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 11:24:28.192831 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191313    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 11:24:28.192831 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191315    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 11:24:28.192831 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191317    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 11:24:28.192831 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191321    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 11:24:28.192831 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191323    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 11:24:28.192831 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191326    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 11:24:28.192831 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191328    2563 feature_gate.go:328] unrecognized feature gate: Example2
May 26 11:24:28.192831 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191331    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 11:24:28.192831 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191333    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 11:24:28.192831 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191335    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 11:24:28.192831 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191339    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 11:24:28.192831 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191342    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 11:24:28.192831 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191345    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 11:24:28.192831 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191347    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 11:24:28.192831 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191350    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 11:24:28.192831 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191353    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 11:24:28.193286 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191357    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 11:24:28.193286 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191360    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 11:24:28.193286 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191760    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 11:24:28.193286 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191765    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 11:24:28.193286 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191769    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 11:24:28.193286 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191787    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 11:24:28.193286 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191790    2563 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 11:24:28.193286 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191793    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 11:24:28.193286 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191795    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 11:24:28.193286 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191798    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 11:24:28.193286 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191800    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 11:24:28.193286 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191803    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 11:24:28.193286 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191806    2563 feature_gate.go:328] unrecognized feature gate: Example2
May 26 11:24:28.193286 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191808    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 11:24:28.193286 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191811    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 11:24:28.193286 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191813    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 11:24:28.193286 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191816    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 11:24:28.193286 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191818    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 11:24:28.193286 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191821    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 11:24:28.193286 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191824    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 11:24:28.193751 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191826    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 11:24:28.193751 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191831    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 11:24:28.193751 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191835    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 11:24:28.193751 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191838    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 11:24:28.193751 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191841    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 11:24:28.193751 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191844    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 11:24:28.193751 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191847    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 11:24:28.193751 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191850    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 11:24:28.193751 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191853    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 11:24:28.193751 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191855    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 11:24:28.193751 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191858    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 11:24:28.193751 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191860    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 11:24:28.193751 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191863    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 11:24:28.193751 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191865    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 11:24:28.193751 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191868    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 11:24:28.193751 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191870    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 11:24:28.193751 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191873    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 11:24:28.193751 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191875    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 11:24:28.193751 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191877    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 11:24:28.194238 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191880    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 11:24:28.194238 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191882    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 11:24:28.194238 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191885    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 11:24:28.194238 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191888    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 11:24:28.194238 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191892    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 11:24:28.194238 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191896    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 11:24:28.194238 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191900    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 11:24:28.194238 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191902    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 11:24:28.194238 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191905    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 11:24:28.194238 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191907    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 11:24:28.194238 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191910    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 11:24:28.194238 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191913    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 11:24:28.194238 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191915    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 11:24:28.194238 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191918    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 11:24:28.194238 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191920    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 11:24:28.194238 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191922    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 11:24:28.194238 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191925    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 11:24:28.194238 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191927    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 11:24:28.194238 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191930    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 11:24:28.194238 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191933    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 11:24:28.194720 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191936    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 11:24:28.194720 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191939    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 11:24:28.194720 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191941    2563 feature_gate.go:328] unrecognized feature gate: Example
May 26 11:24:28.194720 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191944    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 11:24:28.194720 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191947    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 11:24:28.194720 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191949    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 11:24:28.194720 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191951    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 11:24:28.194720 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191954    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 11:24:28.194720 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191956    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 11:24:28.194720 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191958    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 11:24:28.194720 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191961    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 11:24:28.194720 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191963    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 11:24:28.194720 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191965    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 11:24:28.194720 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191968    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 11:24:28.194720 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191970    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 11:24:28.194720 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191972    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 11:24:28.194720 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191975    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 11:24:28.194720 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191977    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 11:24:28.194720 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191980    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 11:24:28.194720 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191982    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 11:24:28.195221 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191985    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 11:24:28.195221 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191987    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 11:24:28.195221 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191989    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 11:24:28.195221 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191992    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 11:24:28.195221 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191995    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 11:24:28.195221 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.191997    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 11:24:28.195221 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.192000    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 11:24:28.195221 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.192002    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 11:24:28.195221 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.192005    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 11:24:28.195221 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193375    2563 flags.go:64] FLAG: --address="0.0.0.0"
May 26 11:24:28.195221 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193384    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 26 11:24:28.195221 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193392    2563 flags.go:64] FLAG: --anonymous-auth="true"
May 26 11:24:28.195221 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193396    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 26 11:24:28.195221 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193401    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
May 26 11:24:28.195221 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193404    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 26 11:24:28.195221 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193409    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 26 11:24:28.195221 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193413    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 26 11:24:28.195221 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193416    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 26 11:24:28.195221 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193419    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 26 11:24:28.195221 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193423    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 26 11:24:28.195221 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193426    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 26 11:24:28.195737 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193429    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 26 11:24:28.195737 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193432    2563 flags.go:64] FLAG: --cgroup-root=""
May 26 11:24:28.195737 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193436    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
May 26 11:24:28.195737 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193439    2563 flags.go:64] FLAG: --client-ca-file=""
May 26 11:24:28.195737 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193442    2563 flags.go:64] FLAG: --cloud-config=""
May 26 11:24:28.195737 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193444    2563 flags.go:64] FLAG: --cloud-provider="external"
May 26 11:24:28.195737 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193447    2563 flags.go:64] FLAG: --cluster-dns="[]"
May 26 11:24:28.195737 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193452    2563 flags.go:64] FLAG: --cluster-domain=""
May 26 11:24:28.195737 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193455    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 26 11:24:28.195737 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193458    2563 flags.go:64] FLAG: --config-dir=""
May 26 11:24:28.195737 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193461    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 26 11:24:28.195737 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193464    2563 flags.go:64] FLAG: --container-log-max-files="5"
May 26 11:24:28.195737 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193468    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 26 11:24:28.195737 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193471    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 26 11:24:28.195737 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193474    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 26 11:24:28.195737 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193477    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 26 11:24:28.195737 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193480    2563 flags.go:64] FLAG: --contention-profiling="false"
May 26 11:24:28.195737 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193483    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 26 11:24:28.195737 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193485    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 26 11:24:28.195737 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193488    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
May 26 11:24:28.195737 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193491    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 26 11:24:28.195737 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193495    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 26 11:24:28.195737 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193498    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 26 11:24:28.195737 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193501    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 26 11:24:28.195737 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193504    2563 flags.go:64] FLAG: --enable-load-reader="false"
May 26 11:24:28.196344 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193507    2563 flags.go:64] FLAG: --enable-server="true"
May 26 11:24:28.196344 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193510    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 26 11:24:28.196344 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193515    2563 flags.go:64] FLAG: --event-burst="100"
May 26 11:24:28.196344 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193518    2563 flags.go:64] FLAG: --event-qps="50"
May 26 11:24:28.196344 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193521    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 26 11:24:28.196344 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193524    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 26 11:24:28.196344 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193527    2563 flags.go:64] FLAG: --eviction-hard=""
May 26 11:24:28.196344 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193530    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 26 11:24:28.196344 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193533    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 26 11:24:28.196344 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193536    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 26 11:24:28.196344 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193541    2563 flags.go:64] FLAG: --eviction-soft=""
May 26 11:24:28.196344 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193544    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 26 11:24:28.196344 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193546    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 26 11:24:28.196344 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193549    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 26 11:24:28.196344 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193552    2563 flags.go:64] FLAG: --experimental-mounter-path=""
May 26 11:24:28.196344 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193555    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
May 26 11:24:28.196344 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193558    2563 flags.go:64] FLAG: --fail-swap-on="true"
May 26 11:24:28.196344 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193561    2563 flags.go:64] FLAG: --feature-gates=""
May 26 11:24:28.196344 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193565    2563 flags.go:64] FLAG: --file-check-frequency="20s"
May 26 11:24:28.196344 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193568    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 26 11:24:28.196344 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193571    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 26 11:24:28.196344 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193574    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 26 11:24:28.196344 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193577    2563 flags.go:64] FLAG: --healthz-port="10248"
May 26 11:24:28.196344 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193580    2563 flags.go:64] FLAG: --help="false"
May 26 11:24:28.196344 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193582    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-129-97.ec2.internal"
May 26 11:24:28.196964 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193585    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
May 26 11:24:28.196964 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193588    2563 flags.go:64] FLAG: --http-check-frequency="20s"
May 26 11:24:28.196964 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193591    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 26 11:24:28.196964 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193595    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 26 11:24:28.196964 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193598    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 26 11:24:28.196964 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193601    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 26 11:24:28.196964 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193603    2563 flags.go:64] FLAG: --image-service-endpoint=""
May 26 11:24:28.196964 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193606    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 26 11:24:28.196964 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193609    2563 flags.go:64] FLAG: --kube-api-burst="100"
May 26 11:24:28.196964 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193612    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 26 11:24:28.196964 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193615    2563 flags.go:64] FLAG: --kube-api-qps="50"
May 26 11:24:28.196964 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193618    2563 flags.go:64] FLAG: --kube-reserved=""
May 26 11:24:28.196964 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193621    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 26 11:24:28.196964 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193623    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 26 11:24:28.196964 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193626    2563 flags.go:64] FLAG: --kubelet-cgroups=""
May 26 11:24:28.196964 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193629    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 26 11:24:28.196964 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193631    2563 flags.go:64] FLAG: --lock-file=""
May 26 11:24:28.196964 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193635    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 26 11:24:28.196964 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193639    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
May 26 11:24:28.196964 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193641    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 26 11:24:28.196964 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193647    2563 flags.go:64] FLAG: --log-json-split-stream="false"
May 26 11:24:28.196964 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193649    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 26 11:24:28.196964 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193652    2563 flags.go:64] FLAG: --log-text-split-stream="false"
May 26 11:24:28.196964 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193655    2563 flags.go:64] FLAG: --logging-format="text"
May 26 11:24:28.197562 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193658    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 26 11:24:28.197562 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193661    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 26 11:24:28.197562 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193664    2563 flags.go:64] FLAG: --manifest-url=""
May 26 11:24:28.197562 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193667    2563 flags.go:64] FLAG: --manifest-url-header=""
May 26 11:24:28.197562 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193671    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 26 11:24:28.197562 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193673    2563 flags.go:64] FLAG: --max-open-files="1000000"
May 26 11:24:28.197562 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193677    2563 flags.go:64] FLAG: --max-pods="110"
May 26 11:24:28.197562 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193680    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 26 11:24:28.197562 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193683    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 26 11:24:28.197562 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193685    2563 flags.go:64] FLAG: --memory-manager-policy="None"
May 26 11:24:28.197562 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193688    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 26 11:24:28.197562 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193691    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 26 11:24:28.197562 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193694    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 26 11:24:28.197562 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193697    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 26 11:24:28.197562 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193704    2563 flags.go:64] FLAG: --node-status-max-images="50"
May 26 11:24:28.197562 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193707    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 26 11:24:28.197562 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193710    2563 flags.go:64] FLAG: --oom-score-adj="-999"
May 26 11:24:28.197562 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193713    2563 flags.go:64] FLAG: --pod-cidr=""
May 26 11:24:28.197562 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193716    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 26 11:24:28.197562 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193722    2563 flags.go:64] FLAG: --pod-manifest-path=""
May 26 11:24:28.197562 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193725    2563 flags.go:64] FLAG: --pod-max-pids="-1"
May 26 11:24:28.197562 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193728    2563 flags.go:64] FLAG: --pods-per-core="0"
May 26 11:24:28.197562 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193731    2563 flags.go:64] FLAG: --port="10250"
May 26 11:24:28.197562 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193733    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 26 11:24:28.198161 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193736    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01c9c7f5c1b886fee"
May 26 11:24:28.198161 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193741    2563 flags.go:64] FLAG: --qos-reserved=""
May 26 11:24:28.198161 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193743    2563 flags.go:64] FLAG: --read-only-port="10255"
May 26 11:24:28.198161 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193746    2563 flags.go:64] FLAG: --register-node="true"
May 26 11:24:28.198161 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193749    2563 flags.go:64] FLAG: --register-schedulable="true"
May 26 11:24:28.198161 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193752    2563 flags.go:64] FLAG: --register-with-taints=""
May 26 11:24:28.198161 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193755    2563 flags.go:64] FLAG: --registry-burst="10"
May 26 11:24:28.198161 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193759    2563 flags.go:64] FLAG: --registry-qps="5"
May 26 11:24:28.198161 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193761    2563 flags.go:64] FLAG: --reserved-cpus=""
May 26 11:24:28.198161 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193765    2563 flags.go:64] FLAG: --reserved-memory=""
May 26 11:24:28.198161 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193768    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 26 11:24:28.198161 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193785    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 26 11:24:28.198161 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193788    2563 flags.go:64] FLAG: --rotate-certificates="false"
May 26 11:24:28.198161 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193791    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
May 26 11:24:28.198161 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193794    2563 flags.go:64] FLAG: --runonce="false"
May 26 11:24:28.198161 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193796    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 26 11:24:28.198161 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193799    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 26 11:24:28.198161 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193802    2563 flags.go:64] FLAG: --seccomp-default="false"
May 26 11:24:28.198161 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193805    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
May 26 11:24:28.198161 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193808    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 26 11:24:28.198161 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193810    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 26 11:24:28.198161 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193813    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 26 11:24:28.198161 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193816    2563 flags.go:64] FLAG: --storage-driver-password="root"
May 26 11:24:28.198161 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193819    2563 flags.go:64] FLAG: --storage-driver-secure="false"
May 26 11:24:28.198161 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193821    2563 flags.go:64] FLAG: --storage-driver-table="stats"
May 26 11:24:28.198161 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193824    2563 flags.go:64] FLAG: --storage-driver-user="root"
May 26 11:24:28.198796 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193828    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 26 11:24:28.198796 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193831    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
May 26 11:24:28.198796 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193834    2563 flags.go:64] FLAG: --system-cgroups=""
May 26 11:24:28.198796 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193837    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 26 11:24:28.198796 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193842    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
May 26 11:24:28.198796 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193844    2563 flags.go:64] FLAG: --tls-cert-file=""
May 26 11:24:28.198796 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193851    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 26 11:24:28.198796 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193857    2563 flags.go:64] FLAG: --tls-min-version=""
May 26 11:24:28.198796 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193859    2563 flags.go:64] FLAG: --tls-private-key-file=""
May 26 11:24:28.198796 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193862    2563 flags.go:64] FLAG: --topology-manager-policy="none"
May 26 11:24:28.198796 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193865    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
May 26 11:24:28.198796 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193868    2563 flags.go:64] FLAG: --topology-manager-scope="container"
May 26 11:24:28.198796 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193870    2563 flags.go:64] FLAG: --v="2"
May 26 11:24:28.198796 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193875    2563 flags.go:64] FLAG: --version="false"
May 26 11:24:28.198796 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193878    2563 flags.go:64] FLAG: --vmodule=""
May 26 11:24:28.198796 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193883    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 26 11:24:28.198796 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.193886    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 26 11:24:28.198796 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.193996    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 11:24:28.198796 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194000    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 11:24:28.198796 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194003    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 11:24:28.198796 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194008    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 11:24:28.198796 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194012    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 11:24:28.198796 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194015    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 11:24:28.199353 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194018    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 11:24:28.199353 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194021    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 11:24:28.199353 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194023    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 11:24:28.199353 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194026    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 11:24:28.199353 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194029    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 11:24:28.199353 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194031    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 11:24:28.199353 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194034    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 11:24:28.199353 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194036    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 11:24:28.199353 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194039    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 11:24:28.199353 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194041    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 11:24:28.199353 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194044    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 11:24:28.199353 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194047    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 11:24:28.199353 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194049    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 11:24:28.199353 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194053    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 11:24:28.199353 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194057    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 11:24:28.199353 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194063    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 11:24:28.199353 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194067    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 11:24:28.199353 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194069    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 11:24:28.199353 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194072    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 11:24:28.199854 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194074    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 11:24:28.199854 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194077    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 11:24:28.199854 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194079    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 11:24:28.199854 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194081    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 11:24:28.199854 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194084    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 11:24:28.199854 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194086    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 11:24:28.199854 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194089    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 11:24:28.199854 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194091    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 11:24:28.199854 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194093    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 11:24:28.199854 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194096    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 11:24:28.199854 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194098    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 11:24:28.199854 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194100    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 11:24:28.199854 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194103    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 11:24:28.199854 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194105    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 11:24:28.199854 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194107    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 11:24:28.199854 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194110    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 11:24:28.199854 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194112    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 11:24:28.199854 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194114    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 11:24:28.199854 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194117    2563 feature_gate.go:328] unrecognized feature gate: Example
May 26 11:24:28.199854 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194120    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 11:24:28.200344 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194122    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 11:24:28.200344 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194124    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 11:24:28.200344 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194127    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 11:24:28.200344 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194129    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 11:24:28.200344 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194132    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 11:24:28.200344 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194135    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 11:24:28.200344 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194137    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 11:24:28.200344 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194140    2563 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 11:24:28.200344 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194143    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 11:24:28.200344 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194149    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 11:24:28.200344 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194152    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 11:24:28.200344 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194155    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 11:24:28.200344 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194157    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 11:24:28.200344 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194160    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 11:24:28.200344 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194162    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 11:24:28.200344 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194164    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 11:24:28.200344 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194167    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 11:24:28.200344 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194169    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 11:24:28.200344 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194172    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 11:24:28.200344 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194174    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 11:24:28.200856 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194176    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 11:24:28.200856 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194179    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 11:24:28.200856 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194181    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 11:24:28.200856 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194183    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 11:24:28.200856 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194186    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 11:24:28.200856 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194188    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 11:24:28.200856 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194191    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 11:24:28.200856 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194193    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 11:24:28.200856 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194195    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 11:24:28.200856 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194198    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 11:24:28.200856 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194200    2563 feature_gate.go:328] unrecognized feature gate: Example2
May 26 11:24:28.200856 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194203    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 11:24:28.200856 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194205    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 11:24:28.200856 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194208    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 11:24:28.200856 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194210    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 11:24:28.200856 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194213    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 11:24:28.200856 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194215    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 11:24:28.200856 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194218    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 11:24:28.200856 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194221    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 11:24:28.200856 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194223    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 11:24:28.201386 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.194227    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 11:24:28.201386 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.194232    2563 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 11:24:28.202102 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.202083    2563 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 26 11:24:28.202133 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.202103    2563 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 26 11:24:28.202167 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202155    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 11:24:28.202167 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202160    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 11:24:28.202167 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202163    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 11:24:28.202167 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202167    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 11:24:28.202274 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202170    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 11:24:28.202274 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202173    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 11:24:28.202274 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202175    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 11:24:28.202274 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202178    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 11:24:28.202274 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202180    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 11:24:28.202274 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202184    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 11:24:28.202274 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202187    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 11:24:28.202274 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202189    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 11:24:28.202274 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202191    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 11:24:28.202274 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202194    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 11:24:28.202274 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202196    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 11:24:28.202274 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202199    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 11:24:28.202274 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202202    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 11:24:28.202274 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202204    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 11:24:28.202274 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202207    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 11:24:28.202274 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202215    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 11:24:28.202274 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202218    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 11:24:28.202274 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202221    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 11:24:28.202274 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202223    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 11:24:28.202274 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202226    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 11:24:28.202766 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202228    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 11:24:28.202766 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202230    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 11:24:28.202766 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202233    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 11:24:28.202766 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202235    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 11:24:28.202766 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202238    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 11:24:28.202766 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202240    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 11:24:28.202766 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202243    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 11:24:28.202766 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202246    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 11:24:28.202766 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202249    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 11:24:28.202766 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202251    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 11:24:28.202766 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202253    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 11:24:28.202766 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202256    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 11:24:28.202766 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202259    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 11:24:28.202766 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202262    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 11:24:28.202766 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202265    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 11:24:28.202766 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202267    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 11:24:28.202766 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202271    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 11:24:28.202766 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202276    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 11:24:28.202766 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202281    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 11:24:28.203239 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202284    2563 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 11:24:28.203239 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202287    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 11:24:28.203239 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202290    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 11:24:28.203239 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202292    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 11:24:28.203239 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202295    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 11:24:28.203239 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202298    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 11:24:28.203239 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202300    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 11:24:28.203239 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202303    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 11:24:28.203239 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202306    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 11:24:28.203239 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202308    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 11:24:28.203239 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202311    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 11:24:28.203239 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202313    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 11:24:28.203239 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202315    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 11:24:28.203239 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202318    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 11:24:28.203239 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202320    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 11:24:28.203239 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202322    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 11:24:28.203239 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202325    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 11:24:28.203239 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202328    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 11:24:28.203239 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202331    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 11:24:28.203711 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202334    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 11:24:28.203711 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202336    2563 feature_gate.go:328] unrecognized feature gate: Example
May 26 11:24:28.203711 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202339    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 11:24:28.203711 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202341    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 11:24:28.203711 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202344    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 11:24:28.203711 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202346    2563 feature_gate.go:328] unrecognized feature gate: Example2
May 26 11:24:28.203711 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202349    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 11:24:28.203711 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202352    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 11:24:28.203711 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202354    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 11:24:28.203711 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202357    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 11:24:28.203711 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202359    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 11:24:28.203711 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202362    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 11:24:28.203711 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202364    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 11:24:28.203711 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202367    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 11:24:28.203711 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202369    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 11:24:28.203711 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202372    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 11:24:28.203711 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202374    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 11:24:28.203711 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202377    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 11:24:28.203711 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202379    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 11:24:28.203711 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202382    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 11:24:28.204216 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202384    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 11:24:28.204216 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202387    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 11:24:28.204216 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202389    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 11:24:28.204216 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202392    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 11:24:28.204216 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.202398    2563 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 11:24:28.204216 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202500    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 11:24:28.204216 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202505    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 11:24:28.204216 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202508    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 11:24:28.204216 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202510    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 11:24:28.204216 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202513    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 11:24:28.204216 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202516    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 11:24:28.204216 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202518    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 11:24:28.204216 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202521    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 11:24:28.204216 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202523    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 11:24:28.204216 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202526    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 11:24:28.204573 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202529    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 11:24:28.204573 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202531    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 11:24:28.204573 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202533    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 11:24:28.204573 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202536    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 11:24:28.204573 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202539    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 11:24:28.204573 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202542    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 11:24:28.204573 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202545    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 11:24:28.204573 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202547    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 11:24:28.204573 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202550    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 11:24:28.204573 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202552    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 11:24:28.204573 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202555    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 11:24:28.204573 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202557    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 11:24:28.204573 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202559    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 11:24:28.204573 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202562    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 11:24:28.204573 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202564    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 11:24:28.204573 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202567    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 11:24:28.204573 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202570    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 11:24:28.204573 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202572    2563 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 11:24:28.204573 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202575    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 11:24:28.205142 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202577    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 11:24:28.205142 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202579    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 11:24:28.205142 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202582    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 11:24:28.205142 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202584    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 11:24:28.205142 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202586    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 11:24:28.205142 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202589    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 11:24:28.205142 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202592    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 11:24:28.205142 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202595    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 11:24:28.205142 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202598    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 11:24:28.205142 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202600    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 11:24:28.205142 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202603    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 11:24:28.205142 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202605    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 11:24:28.205142 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202608    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 11:24:28.205142 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202610    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 11:24:28.205142 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202613    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 11:24:28.205142 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202616    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 11:24:28.205142 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202618    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 11:24:28.205142 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202621    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 11:24:28.205142 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202624    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 11:24:28.205142 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202626    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 11:24:28.205617 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202629    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 11:24:28.205617 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202631    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 11:24:28.205617 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202634    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 11:24:28.205617 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202636    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 11:24:28.205617 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202639    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 11:24:28.205617 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202641    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 11:24:28.205617 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202643    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 11:24:28.205617 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202646    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 11:24:28.205617 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202648    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 11:24:28.205617 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202651    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 11:24:28.205617 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202653    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 11:24:28.205617 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202655    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 11:24:28.205617 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202658    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 11:24:28.205617 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202660    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 11:24:28.205617 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202662    2563 feature_gate.go:328] unrecognized feature gate: Example
May 26 11:24:28.205617 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202665    2563 feature_gate.go:328] unrecognized feature gate: Example2
May 26 11:24:28.205617 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202667    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 11:24:28.205617 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202671    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 11:24:28.205617 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202674    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 11:24:28.205617 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202676    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 11:24:28.206125 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202679    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 11:24:28.206125 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202681    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 11:24:28.206125 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202684    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 11:24:28.206125 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202687    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 11:24:28.206125 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202689    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 11:24:28.206125 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202692    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 11:24:28.206125 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202694    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 11:24:28.206125 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202697    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 11:24:28.206125 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202700    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 11:24:28.206125 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202702    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 11:24:28.206125 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202705    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 11:24:28.206125 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202707    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 11:24:28.206125 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202710    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 11:24:28.206125 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202712    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 11:24:28.206125 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202715    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 11:24:28.206125 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202717    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 11:24:28.206125 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:28.202720    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 11:24:28.206563 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.202724    2563 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 11:24:28.206563 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.203506    2563 server.go:962] "Client rotation is on, will bootstrap in background"
May 26 11:24:28.206563 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.205660    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 26 11:24:28.206641 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.206619    2563 server.go:1019] "Starting client certificate rotation"
May 26 11:24:28.206736 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.206717    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 26 11:24:28.206797 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.206766    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 26 11:24:28.236535 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.236514    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 26 11:24:28.238494 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.238476    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 26 11:24:28.258830 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.258808    2563 log.go:25] "Validated CRI v1 runtime API"
May 26 11:24:28.265003 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.264985    2563 log.go:25] "Validated CRI v1 image API"
May 26 11:24:28.269789 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.269747    2563 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 26 11:24:28.276051 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.276031    2563 fs.go:135] Filesystem UUIDs: map[0504ff24-c60f-44e1-a126-2442aa7c1a64:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8f9d45f7-97aa-4737-ad5b-e4aee86bc4f2:/dev/nvme0n1p4]
May 26 11:24:28.276112 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.276051    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 26 11:24:28.279819 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.279802    2563 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 26 11:24:28.280608 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.280496    2563 manager.go:217] Machine: {Timestamp:2026-05-26 11:24:28.279941148 +0000 UTC m=+0.471383327 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100494 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2723dfb0bb9d60aefd80e46dce9a84 SystemUUID:ec2723df-b0bb-9d60-aefd-80e46dce9a84 BootID:76c654c1-3a85-4b84-b80e-ef9bf45d581d Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:50:10:2d:49:99 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:50:10:2d:49:99 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fe:aa:02:4d:7f:59 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 26 11:24:28.280608 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.280608    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 26 11:24:28.280735 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.280705    2563 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 26 11:24:28.282944 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.282918    2563 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 26 11:24:28.283081 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.282947    2563 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-97.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 26 11:24:28.283123 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.283090    2563 topology_manager.go:138] "Creating topology manager with none policy"
May 26 11:24:28.283123 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.283098    2563 container_manager_linux.go:306] "Creating device plugin manager"
May 26 11:24:28.283123 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.283105    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 26 11:24:28.283123 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.283121    2563 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 26 11:24:28.285160 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.285149    2563 state_mem.go:36] "Initialized new in-memory state store"
May 26 11:24:28.285267 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.285258    2563 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 26 11:24:28.287984 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.287974    2563 kubelet.go:491] "Attempting to sync node with API server"
May 26 11:24:28.288018 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.287988    2563 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 26 11:24:28.288018 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.288001    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 26 11:24:28.288018 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.288010    2563 kubelet.go:397] "Adding apiserver pod source"
May 26 11:24:28.288018 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.288018    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 26 11:24:28.289313 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.289301    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 26 11:24:28.289349 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.289321    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 26 11:24:28.293177 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.293160    2563 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 26 11:24:28.295545 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.295531    2563 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 26 11:24:28.297702 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.297690    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 26 11:24:28.297743 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.297711    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 26 11:24:28.297743 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.297722    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 26 11:24:28.297743 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.297730    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 26 11:24:28.297743 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.297739    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 26 11:24:28.297866 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.297746    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 26 11:24:28.297866 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.297753    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 26 11:24:28.297866 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.297758    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 26 11:24:28.297866 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.297765    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 26 11:24:28.297866 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.297788    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 26 11:24:28.297866 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.297805    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 26 11:24:28.297866 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.297810    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 26 11:24:28.297866 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.297834    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 26 11:24:28.297866 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.297840    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 26 11:24:28.299170 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.299151    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4z8fg"
May 26 11:24:28.300711 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:28.300690    2563 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 26 11:24:28.300853 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:28.300836    2563 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-97.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 26 11:24:28.300894 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.300848    2563 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-97.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 26 11:24:28.301162 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.301150    2563 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 26 11:24:28.301195 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.301187    2563 server.go:1295] "Started kubelet"
May 26 11:24:28.301304 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.301249    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 26 11:24:28.301394 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.301332    2563 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 26 11:24:28.301445 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.301402    2563 server_v1.go:47] "podresources" method="list" useActivePods=true
May 26 11:24:28.302075 ip-10-0-129-97 systemd[1]: Started Kubernetes Kubelet.
May 26 11:24:28.304029 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.304008    2563 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 26 11:24:28.305964 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.305937    2563 server.go:317] "Adding debug handlers to kubelet server"
May 26 11:24:28.307388 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.307368    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4z8fg"
May 26 11:24:28.314997 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:28.314971    2563 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 26 11:24:28.315347 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.315327    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 26 11:24:28.315347 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.315337    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 26 11:24:28.315493 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.315421    2563 volume_manager.go:295] "The desired_state_of_world populator starts"
May 26 11:24:28.315493 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.315436    2563 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 26 11:24:28.315589 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.315500    2563 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 26 11:24:28.315636 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.315612    2563 reconstruct.go:97] "Volume reconstruction finished"
May 26 11:24:28.315636 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.315620    2563 reconciler.go:26] "Reconciler: start to sync state"
May 26 11:24:28.316274 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.316258    2563 factory.go:55] Registering systemd factory
May 26 11:24:28.316356 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.316287    2563 factory.go:223] Registration of the systemd container factory successfully
May 26 11:24:28.316689 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:28.316663    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-97.ec2.internal\" not found"
May 26 11:24:28.316762 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.316722    2563 factory.go:153] Registering CRI-O factory
May 26 11:24:28.316762 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.316745    2563 factory.go:223] Registration of the crio container factory successfully
May 26 11:24:28.316873 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.316815    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 26 11:24:28.316873 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.316835    2563 factory.go:103] Registering Raw factory
May 26 11:24:28.316873 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.316847    2563 manager.go:1196] Started watching for new ooms in manager
May 26 11:24:28.317179 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.317168    2563 manager.go:319] Starting recovery of all containers
May 26 11:24:28.324715 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.324539    2563 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 11:24:28.325962 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:28.325938    2563 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-97.ec2.internal\" not found" node="ip-10-0-129-97.ec2.internal"
May 26 11:24:28.328312 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.328295    2563 manager.go:324] Recovery completed
May 26 11:24:28.334747 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.334731    2563 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 11:24:28.336651 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.336634    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-97.ec2.internal" event="NodeHasSufficientMemory"
May 26 11:24:28.336713 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.336665    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-97.ec2.internal" event="NodeHasNoDiskPressure"
May 26 11:24:28.336713 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.336676    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-97.ec2.internal" event="NodeHasSufficientPID"
May 26 11:24:28.337208 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.337191    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 26 11:24:28.337208 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.337206    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 26 11:24:28.337302 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.337226    2563 state_mem.go:36] "Initialized new in-memory state store"
May 26 11:24:28.340679 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.340667    2563 policy_none.go:49] "None policy: Start"
May 26 11:24:28.340716 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.340684    2563 memory_manager.go:186] "Starting memorymanager" policy="None"
May 26 11:24:28.340716 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.340695    2563 state_mem.go:35] "Initializing new in-memory state store"
May 26 11:24:28.375452 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.375433    2563 manager.go:341] "Starting Device Plugin manager"
May 26 11:24:28.375572 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:28.375473    2563 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 26 11:24:28.375572 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.375487    2563 server.go:85] "Starting device plugin registration server"
May 26 11:24:28.375758 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.375745    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
May 26 11:24:28.375847 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.375761    2563 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 26 11:24:28.375895 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.375876    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 26 11:24:28.375996 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.375960    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 26 11:24:28.375996 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.375976    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 26 11:24:28.377706 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:28.377684    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 26 11:24:28.377842 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:28.377728    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-97.ec2.internal\" not found"
May 26 11:24:28.422675 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.422635    2563 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 26 11:24:28.423970 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.423956    2563 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 26 11:24:28.424039 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.423976    2563 status_manager.go:230] "Starting to sync pod status with apiserver"
May 26 11:24:28.424039 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.424000    2563 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 26 11:24:28.424039 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.424009    2563 kubelet.go:2451] "Starting kubelet main sync loop"
May 26 11:24:28.424135 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:28.424053    2563 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 26 11:24:28.427322 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.427302    2563 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 11:24:28.476276 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.476196    2563 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 11:24:28.477403 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.477385    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-97.ec2.internal" event="NodeHasSufficientMemory"
May 26 11:24:28.477512 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.477419    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-97.ec2.internal" event="NodeHasNoDiskPressure"
May 26 11:24:28.477512 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.477434    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-97.ec2.internal" event="NodeHasSufficientPID"
May 26 11:24:28.477512 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.477470    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-97.ec2.internal"
May 26 11:24:28.486179 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.486153    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-97.ec2.internal"
May 26 11:24:28.486297 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:28.486184    2563 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-97.ec2.internal\": node \"ip-10-0-129-97.ec2.internal\" not found"
May 26 11:24:28.502629 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:28.502597    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-97.ec2.internal\" not found"
May 26 11:24:28.524269 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.524237    2563 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-97.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-97.ec2.internal"]
May 26 11:24:28.524363 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.524339    2563 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 11:24:28.525305 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.525287    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-97.ec2.internal" event="NodeHasSufficientMemory"
May 26 11:24:28.525411 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.525314    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-97.ec2.internal" event="NodeHasNoDiskPressure"
May 26 11:24:28.525411 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.525325    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-97.ec2.internal" event="NodeHasSufficientPID"
May 26 11:24:28.526675 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.526657    2563 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 11:24:28.526858 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.526844    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-97.ec2.internal"
May 26 11:24:28.526907 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.526874    2563 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 11:24:28.527348 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.527333    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-97.ec2.internal" event="NodeHasSufficientMemory"
May 26 11:24:28.527413 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.527356    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-97.ec2.internal" event="NodeHasNoDiskPressure"
May 26 11:24:28.527413 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.527370    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-97.ec2.internal" event="NodeHasSufficientPID"
May 26 11:24:28.527413 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.527403    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-97.ec2.internal" event="NodeHasSufficientMemory"
May 26 11:24:28.527546 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.527429    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-97.ec2.internal" event="NodeHasNoDiskPressure"
May 26 11:24:28.527546 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.527444    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-97.ec2.internal" event="NodeHasSufficientPID"
May 26 11:24:28.528448 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.528432    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-97.ec2.internal"
May 26 11:24:28.528535 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.528459    2563 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 11:24:28.530110 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.530092    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-97.ec2.internal" event="NodeHasSufficientMemory"
May 26 11:24:28.530203 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.530118    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-97.ec2.internal" event="NodeHasNoDiskPressure"
May 26 11:24:28.530203 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.530128    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-97.ec2.internal" event="NodeHasSufficientPID"
May 26 11:24:28.552663 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:28.552637    2563 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-97.ec2.internal\" not found" node="ip-10-0-129-97.ec2.internal"
May 26 11:24:28.557190 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:28.557171    2563 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-97.ec2.internal\" not found" node="ip-10-0-129-97.ec2.internal"
May 26 11:24:28.602874 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:28.602837    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-97.ec2.internal\" not found"
May 26 11:24:28.617927 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.617892    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0802379863c3c768639fbc45cfce5c9e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-97.ec2.internal\" (UID: \"0802379863c3c768639fbc45cfce5c9e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-97.ec2.internal"
May 26 11:24:28.617927 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.617929    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0802379863c3c768639fbc45cfce5c9e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-97.ec2.internal\" (UID: \"0802379863c3c768639fbc45cfce5c9e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-97.ec2.internal"
May 26 11:24:28.618133 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.617955    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7b05a01f2c158d2611520e5fe1f839c3-config\") pod \"kube-apiserver-proxy-ip-10-0-129-97.ec2.internal\" (UID: \"7b05a01f2c158d2611520e5fe1f839c3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-97.ec2.internal"
May 26 11:24:28.703679 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:28.703629    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-97.ec2.internal\" not found"
May 26 11:24:28.719054 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.719017    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0802379863c3c768639fbc45cfce5c9e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-97.ec2.internal\" (UID: \"0802379863c3c768639fbc45cfce5c9e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-97.ec2.internal"
May 26 11:24:28.719054 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.719028    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0802379863c3c768639fbc45cfce5c9e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-97.ec2.internal\" (UID: \"0802379863c3c768639fbc45cfce5c9e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-97.ec2.internal"
May 26 11:24:28.719155 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.719088    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0802379863c3c768639fbc45cfce5c9e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-97.ec2.internal\" (UID: \"0802379863c3c768639fbc45cfce5c9e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-97.ec2.internal"
May 26 11:24:28.719155 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.719110    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7b05a01f2c158d2611520e5fe1f839c3-config\") pod \"kube-apiserver-proxy-ip-10-0-129-97.ec2.internal\" (UID: \"7b05a01f2c158d2611520e5fe1f839c3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-97.ec2.internal"
May 26 11:24:28.719219 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.719159    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0802379863c3c768639fbc45cfce5c9e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-97.ec2.internal\" (UID: \"0802379863c3c768639fbc45cfce5c9e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-97.ec2.internal"
May 26 11:24:28.719219 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.719159    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7b05a01f2c158d2611520e5fe1f839c3-config\") pod \"kube-apiserver-proxy-ip-10-0-129-97.ec2.internal\" (UID: \"7b05a01f2c158d2611520e5fe1f839c3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-97.ec2.internal"
May 26 11:24:28.804439 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:28.804368    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-97.ec2.internal\" not found"
May 26 11:24:28.856856 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.856827    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-97.ec2.internal"
May 26 11:24:28.860400 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:28.860382    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-97.ec2.internal"
May 26 11:24:28.904824 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:28.904796    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-97.ec2.internal\" not found"
May 26 11:24:29.005284 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:29.005257    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-97.ec2.internal\" not found"
May 26 11:24:29.105811 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:29.105730    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-97.ec2.internal\" not found"
May 26 11:24:29.206241 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:29.206211    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-97.ec2.internal\" not found"
May 26 11:24:29.207310 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:29.207291    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 26 11:24:29.207426 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:29.207411    2563 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 26 11:24:29.207465 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:29.207442    2563 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 26 11:24:29.306464 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:29.306431    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-97.ec2.internal\" not found"
May 26 11:24:29.311910 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:29.311881    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-25 11:19:28 +0000 UTC" deadline="2027-12-09 22:15:38.828046729 +0000 UTC"
May 26 11:24:29.311910 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:29.311907    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13498h51m9.516142541s"
May 26 11:24:29.315476 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:29.315462    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 26 11:24:29.329450 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:29.329431    2563 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 26 11:24:29.360134 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:29.360087    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xlnpl"
May 26 11:24:29.367442 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:29.367428    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xlnpl"
May 26 11:24:29.407140 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:29.407114    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-97.ec2.internal\" not found"
May 26 11:24:29.424427 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:29.424402    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b05a01f2c158d2611520e5fe1f839c3.slice/crio-22fde98053ac27e3d14b6456b2ddaad004e237255ecac0dee4c9c5ef01f562d7 WatchSource:0}: Error finding container 22fde98053ac27e3d14b6456b2ddaad004e237255ecac0dee4c9c5ef01f562d7: Status 404 returned error can't find the container with id 22fde98053ac27e3d14b6456b2ddaad004e237255ecac0dee4c9c5ef01f562d7
May 26 11:24:29.424635 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:29.424613    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0802379863c3c768639fbc45cfce5c9e.slice/crio-978aba50b95c6be232718e96dab6b05229c77184a5d5a178da922e1f24f25462 WatchSource:0}: Error finding container 978aba50b95c6be232718e96dab6b05229c77184a5d5a178da922e1f24f25462: Status 404 returned error can't find the container with id 978aba50b95c6be232718e96dab6b05229c77184a5d5a178da922e1f24f25462
May 26 11:24:29.427789 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:29.427678    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-97.ec2.internal" event={"ID":"7b05a01f2c158d2611520e5fe1f839c3","Type":"ContainerStarted","Data":"22fde98053ac27e3d14b6456b2ddaad004e237255ecac0dee4c9c5ef01f562d7"}
May 26 11:24:29.427944 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:29.427927    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 11:24:29.428676 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:29.428610    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-97.ec2.internal" event={"ID":"0802379863c3c768639fbc45cfce5c9e","Type":"ContainerStarted","Data":"978aba50b95c6be232718e96dab6b05229c77184a5d5a178da922e1f24f25462"}
May 26 11:24:29.508097 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:29.508058    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-97.ec2.internal\" not found"
May 26 11:24:29.595050 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:29.595013    2563 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 11:24:29.609209 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:29.609179    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-97.ec2.internal\" not found"
May 26 11:24:29.709792 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:29.709695    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-97.ec2.internal\" not found"
May 26 11:24:29.763788 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:29.763746    2563 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 11:24:29.816680 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:29.816652    2563 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-97.ec2.internal"
May 26 11:24:29.828786 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:29.828742    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 26 11:24:29.830037 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:29.830011    2563 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-97.ec2.internal"
May 26 11:24:29.839428 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:29.839400    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 26 11:24:30.289004 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.288956    2563 apiserver.go:52] "Watching apiserver"
May 26 11:24:30.297805 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.297759    2563 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 26 11:24:30.298137 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.298099    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-xtkmg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmrml","openshift-cluster-node-tuning-operator/tuned-njvjv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-97.ec2.internal","openshift-multus/multus-pfkj5","openshift-multus/network-metrics-daemon-pp9l8","openshift-network-diagnostics/network-check-target-cbxbl","kube-system/kube-apiserver-proxy-ip-10-0-129-97.ec2.internal","openshift-image-registry/node-ca-58x22","openshift-multus/multus-additional-cni-plugins-kz76h","openshift-network-operator/iptables-alerter-ldk8g","openshift-ovn-kubernetes/ovnkube-node-wxzlw"]
May 26 11:24:30.300849 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.300823    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xtkmg"
May 26 11:24:30.302902 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.302875    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmrml"
May 26 11:24:30.305047 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.305026    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.306387 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.306366    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 26 11:24:30.306387 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.306378    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 26 11:24:30.306543 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.306408    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 26 11:24:30.306543 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.306411    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 26 11:24:30.306856 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.306833    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-f28xs\""
May 26 11:24:30.307206 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.307185    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.307645 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.307624    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 26 11:24:30.307733 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.307693    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 26 11:24:30.307873 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.307693    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 26 11:24:30.308216 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.307901    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-b7x8p\""
May 26 11:24:30.309270 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.308741    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5pwbx\""
May 26 11:24:30.310917 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.310260    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp9l8"
May 26 11:24:30.310917 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:30.310359    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pp9l8" podUID="9d573a77-5520-489f-99cb-86419b23cd19"
May 26 11:24:30.310917 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.310442    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 26 11:24:30.310917 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.310655    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 26 11:24:30.311157 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.311132    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 26 11:24:30.312798 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.311181    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 26 11:24:30.312798 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.311392    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wr99k\""
May 26 11:24:30.314041 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.314023    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbxbl"
May 26 11:24:30.314130 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:30.314093    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbxbl" podUID="8528f6a0-0b8d-47f8-a20c-715dd15d1076"
May 26 11:24:30.317405 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.317384    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-58x22"
May 26 11:24:30.320071 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.320055    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 26 11:24:30.320162 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.320079    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 26 11:24:30.320162 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.320128    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 26 11:24:30.320283 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.320189    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xg9c2\""
May 26 11:24:30.321942 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.321909    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kz76h"
May 26 11:24:30.322021 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.321941    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ldk8g"
May 26 11:24:30.324097 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.324078    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nkhjs\""
May 26 11:24:30.324185 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.324079    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.324547 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.324525    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 26 11:24:30.324547 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.324544    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-czw2q\""
May 26 11:24:30.324681 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.324621    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 26 11:24:30.324681 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.324631    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 26 11:24:30.325360 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.325341    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 26 11:24:30.325433 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.325392    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 26 11:24:30.326365 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.326349    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 26 11:24:30.326667 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.326651    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 26 11:24:30.327714 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.327692    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 26 11:24:30.327714 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.327711    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6b4d4\""
May 26 11:24:30.327907 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.327739    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 26 11:24:30.327907 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.327767    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 26 11:24:30.327907 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.327696    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 26 11:24:30.327907 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.327818    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/be852edc-c490-4de5-99b0-a73297a80fa0-etc-systemd\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.327907 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.327853    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-host-run-k8s-cni-cncf-io\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.327907 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.327879    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/be852edc-c490-4de5-99b0-a73297a80fa0-etc-modprobe-d\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.327907 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.327903    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be852edc-c490-4de5-99b0-a73297a80fa0-etc-kubernetes\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.328199 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.327926    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be852edc-c490-4de5-99b0-a73297a80fa0-var-lib-kubelet\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.328199 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.327955    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgg7z\" (UniqueName: \"kubernetes.io/projected/e991c96a-3b5f-486a-b2a9-6bfaf4130356-kube-api-access-fgg7z\") pod \"aws-ebs-csi-driver-node-zmrml\" (UID: \"e991c96a-3b5f-486a-b2a9-6bfaf4130356\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmrml"
May 26 11:24:30.328199 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.327981    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-cnibin\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.328199 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.328024    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-host-run-multus-certs\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.328199 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.328075    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/be852edc-c490-4de5-99b0-a73297a80fa0-etc-sysconfig\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.328199 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.328104    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xrms\" (UniqueName: \"kubernetes.io/projected/f1ab37a2-3fa3-4941-8cc3-0149f5478b36-kube-api-access-9xrms\") pod \"node-ca-58x22\" (UID: \"f1ab37a2-3fa3-4941-8cc3-0149f5478b36\") " pod="openshift-image-registry/node-ca-58x22"
May 26 11:24:30.328199 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.328130    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-host-var-lib-cni-bin\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.328199 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.328156    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bfdd44fe-70d3-4e4e-864b-f007de978ffe-multus-daemon-config\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.328199 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.328181    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt6x9\" (UniqueName: \"kubernetes.io/projected/8528f6a0-0b8d-47f8-a20c-715dd15d1076-kube-api-access-bt6x9\") pod \"network-check-target-cbxbl\" (UID: \"8528f6a0-0b8d-47f8-a20c-715dd15d1076\") " pod="openshift-network-diagnostics/network-check-target-cbxbl"
May 26 11:24:30.328599 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.328205    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/be852edc-c490-4de5-99b0-a73297a80fa0-run\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.328599 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.328246    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/be852edc-c490-4de5-99b0-a73297a80fa0-etc-tuned\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.328599 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.328280    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1ab37a2-3fa3-4941-8cc3-0149f5478b36-host\") pod \"node-ca-58x22\" (UID: \"f1ab37a2-3fa3-4941-8cc3-0149f5478b36\") " pod="openshift-image-registry/node-ca-58x22"
May 26 11:24:30.328599 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.328310    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e991c96a-3b5f-486a-b2a9-6bfaf4130356-socket-dir\") pod \"aws-ebs-csi-driver-node-zmrml\" (UID: \"e991c96a-3b5f-486a-b2a9-6bfaf4130356\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmrml"
May 26 11:24:30.328599 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.328336    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e991c96a-3b5f-486a-b2a9-6bfaf4130356-device-dir\") pod \"aws-ebs-csi-driver-node-zmrml\" (UID: \"e991c96a-3b5f-486a-b2a9-6bfaf4130356\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmrml"
May 26 11:24:30.328599 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.328360    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e991c96a-3b5f-486a-b2a9-6bfaf4130356-sys-fs\") pod \"aws-ebs-csi-driver-node-zmrml\" (UID: \"e991c96a-3b5f-486a-b2a9-6bfaf4130356\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmrml"
May 26 11:24:30.328599 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.328383    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bfdd44fe-70d3-4e4e-864b-f007de978ffe-cni-binary-copy\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.328599 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.328405    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-etc-kubernetes\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.328599 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.328428    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m487h\" (UniqueName: \"kubernetes.io/projected/bfdd44fe-70d3-4e4e-864b-f007de978ffe-kube-api-access-m487h\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.328599 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.328452    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be852edc-c490-4de5-99b0-a73297a80fa0-lib-modules\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.328599 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.328478    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/be852edc-c490-4de5-99b0-a73297a80fa0-host\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.328599 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.328503    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e991c96a-3b5f-486a-b2a9-6bfaf4130356-registration-dir\") pod \"aws-ebs-csi-driver-node-zmrml\" (UID: \"e991c96a-3b5f-486a-b2a9-6bfaf4130356\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmrml"
May 26 11:24:30.328599 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.328525    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-os-release\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.328599 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.328565    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be852edc-c490-4de5-99b0-a73297a80fa0-tmp\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.328599 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.328595    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjpxj\" (UniqueName: \"kubernetes.io/projected/be852edc-c490-4de5-99b0-a73297a80fa0-kube-api-access-kjpxj\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.329269 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.328621    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpxzc\" (UniqueName: \"kubernetes.io/projected/9d573a77-5520-489f-99cb-86419b23cd19-kube-api-access-fpxzc\") pod \"network-metrics-daemon-pp9l8\" (UID: \"9d573a77-5520-489f-99cb-86419b23cd19\") " pod="openshift-multus/network-metrics-daemon-pp9l8"
May 26 11:24:30.329269 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.328655    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e991c96a-3b5f-486a-b2a9-6bfaf4130356-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zmrml\" (UID: \"e991c96a-3b5f-486a-b2a9-6bfaf4130356\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmrml"
May 26 11:24:30.329269 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.328684    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e991c96a-3b5f-486a-b2a9-6bfaf4130356-etc-selinux\") pod \"aws-ebs-csi-driver-node-zmrml\" (UID: \"e991c96a-3b5f-486a-b2a9-6bfaf4130356\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmrml"
May 26 11:24:30.329269 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.328756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-system-cni-dir\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.329269 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.328820    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-multus-cni-dir\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.329269 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.328862    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-host-var-lib-kubelet\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.329269 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.328897    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be852edc-c490-4de5-99b0-a73297a80fa0-sys\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.329269 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.328929    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f1ab37a2-3fa3-4941-8cc3-0149f5478b36-serviceca\") pod \"node-ca-58x22\" (UID: \"f1ab37a2-3fa3-4941-8cc3-0149f5478b36\") " pod="openshift-image-registry/node-ca-58x22"
May 26 11:24:30.329269 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.328955    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-host-run-netns\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.329269 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.328981    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-multus-conf-dir\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.329269 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.329004    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/be852edc-c490-4de5-99b0-a73297a80fa0-etc-sysctl-d\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.329269 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.329027    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/be852edc-c490-4de5-99b0-a73297a80fa0-etc-sysctl-conf\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.329269 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.329056    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d573a77-5520-489f-99cb-86419b23cd19-metrics-certs\") pod \"network-metrics-daemon-pp9l8\" (UID: \"9d573a77-5520-489f-99cb-86419b23cd19\") " pod="openshift-multus/network-metrics-daemon-pp9l8"
May 26 11:24:30.329269 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.329080    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-multus-socket-dir-parent\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.329269 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.329104    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-host-var-lib-cni-multus\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.329269 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.329126    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-hostroot\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.329269 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.329150    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b4769eef-c61f-4722-96a1-8461e3cc4801-agent-certs\") pod \"konnectivity-agent-xtkmg\" (UID: \"b4769eef-c61f-4722-96a1-8461e3cc4801\") " pod="kube-system/konnectivity-agent-xtkmg"
May 26 11:24:30.329821 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.329192    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b4769eef-c61f-4722-96a1-8461e3cc4801-konnectivity-ca\") pod \"konnectivity-agent-xtkmg\" (UID: \"b4769eef-c61f-4722-96a1-8461e3cc4801\") " pod="kube-system/konnectivity-agent-xtkmg"
May 26 11:24:30.368295 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.368257    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-25 11:19:29 +0000 UTC" deadline="2028-03-04 06:43:49.881437962 +0000 UTC"
May 26 11:24:30.368295 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.368293    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15547h19m19.513148083s"
May 26 11:24:30.382025 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.381996    2563 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 11:24:30.416434 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.416403    2563 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 26 11:24:30.429405 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.429374    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b642b5f8-5576-42de-807d-6da907ae897d-system-cni-dir\") pod \"multus-additional-cni-plugins-kz76h\" (UID: \"b642b5f8-5576-42de-807d-6da907ae897d\") " pod="openshift-multus/multus-additional-cni-plugins-kz76h"
May 26 11:24:30.429582 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.429409    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f529n\" (UniqueName: \"kubernetes.io/projected/b642b5f8-5576-42de-807d-6da907ae897d-kube-api-access-f529n\") pod \"multus-additional-cni-plugins-kz76h\" (UID: \"b642b5f8-5576-42de-807d-6da907ae897d\") " pod="openshift-multus/multus-additional-cni-plugins-kz76h"
May 26 11:24:30.429582 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.429435    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e991c96a-3b5f-486a-b2a9-6bfaf4130356-device-dir\") pod \"aws-ebs-csi-driver-node-zmrml\" (UID: \"e991c96a-3b5f-486a-b2a9-6bfaf4130356\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmrml"
May 26 11:24:30.429582 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.429509    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e991c96a-3b5f-486a-b2a9-6bfaf4130356-device-dir\") pod \"aws-ebs-csi-driver-node-zmrml\" (UID: \"e991c96a-3b5f-486a-b2a9-6bfaf4130356\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmrml"
May 26 11:24:30.429582 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.429553    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-m487h\" (UniqueName: \"kubernetes.io/projected/bfdd44fe-70d3-4e4e-864b-f007de978ffe-kube-api-access-m487h\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.429757 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.429594    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/be852edc-c490-4de5-99b0-a73297a80fa0-host\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.429757 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.429627    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-ovnkube-script-lib\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.429757 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.429657    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e991c96a-3b5f-486a-b2a9-6bfaf4130356-registration-dir\") pod \"aws-ebs-csi-driver-node-zmrml\" (UID: \"e991c96a-3b5f-486a-b2a9-6bfaf4130356\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmrml"
May 26 11:24:30.429757 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.429681    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be852edc-c490-4de5-99b0-a73297a80fa0-tmp\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.429757 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.429692    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/be852edc-c490-4de5-99b0-a73297a80fa0-host\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.429757 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.429708    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kjpxj\" (UniqueName: \"kubernetes.io/projected/be852edc-c490-4de5-99b0-a73297a80fa0-kube-api-access-kjpxj\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.429757 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.429736    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d316eade-d057-4616-b829-4c09f6ea6d16-host-slash\") pod \"iptables-alerter-ldk8g\" (UID: \"d316eade-d057-4616-b829-4c09f6ea6d16\") " pod="openshift-network-operator/iptables-alerter-ldk8g"
May 26 11:24:30.429757 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.429743    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e991c96a-3b5f-486a-b2a9-6bfaf4130356-registration-dir\") pod \"aws-ebs-csi-driver-node-zmrml\" (UID: \"e991c96a-3b5f-486a-b2a9-6bfaf4130356\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmrml"
May 26 11:24:30.430049 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.429764    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-systemd-units\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.430049 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.429807    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-run-openvswitch\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.430049 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.429830    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-node-log\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.430049 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.429853    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b642b5f8-5576-42de-807d-6da907ae897d-os-release\") pod \"multus-additional-cni-plugins-kz76h\" (UID: \"b642b5f8-5576-42de-807d-6da907ae897d\") " pod="openshift-multus/multus-additional-cni-plugins-kz76h"
May 26 11:24:30.430049 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.429880    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e991c96a-3b5f-486a-b2a9-6bfaf4130356-etc-selinux\") pod \"aws-ebs-csi-driver-node-zmrml\" (UID: \"e991c96a-3b5f-486a-b2a9-6bfaf4130356\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmrml"
May 26 11:24:30.430049 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.429904    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-system-cni-dir\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.430049 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.429926    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be852edc-c490-4de5-99b0-a73297a80fa0-sys\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.430049 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.429951    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f1ab37a2-3fa3-4941-8cc3-0149f5478b36-serviceca\") pod \"node-ca-58x22\" (UID: \"f1ab37a2-3fa3-4941-8cc3-0149f5478b36\") " pod="openshift-image-registry/node-ca-58x22"
May 26 11:24:30.430049 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.429978    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bllzp\" (UniqueName: \"kubernetes.io/projected/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-kube-api-access-bllzp\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.430049 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.430006    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/be852edc-c490-4de5-99b0-a73297a80fa0-etc-sysctl-conf\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.430049 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.430032    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-env-overrides\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.430049 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.430021    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 26 11:24:30.430386 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.430050    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-system-cni-dir\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.430386 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.430060    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b4769eef-c61f-4722-96a1-8461e3cc4801-konnectivity-ca\") pod \"konnectivity-agent-xtkmg\" (UID: \"b4769eef-c61f-4722-96a1-8461e3cc4801\") " pod="kube-system/konnectivity-agent-xtkmg"
May 26 11:24:30.430386 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.430098    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/be852edc-c490-4de5-99b0-a73297a80fa0-etc-systemd\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.430386 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.430100    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be852edc-c490-4de5-99b0-a73297a80fa0-sys\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.430386 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.430130    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-run-systemd\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.430386 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.430055    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e991c96a-3b5f-486a-b2a9-6bfaf4130356-etc-selinux\") pod \"aws-ebs-csi-driver-node-zmrml\" (UID: \"e991c96a-3b5f-486a-b2a9-6bfaf4130356\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmrml"
May 26 11:24:30.430386 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.430157    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-host-cni-netd\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.430386 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.430357    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/be852edc-c490-4de5-99b0-a73297a80fa0-etc-sysctl-conf\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.430620 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.430400    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/be852edc-c490-4de5-99b0-a73297a80fa0-etc-systemd\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.430620 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.430440    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-host-run-k8s-cni-cncf-io\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.430620 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.430467    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/be852edc-c490-4de5-99b0-a73297a80fa0-etc-modprobe-d\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.430620 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.430505    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be852edc-c490-4de5-99b0-a73297a80fa0-etc-kubernetes\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.430620 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.430528    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be852edc-c490-4de5-99b0-a73297a80fa0-var-lib-kubelet\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.430620 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.430555    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fgg7z\" (UniqueName: \"kubernetes.io/projected/e991c96a-3b5f-486a-b2a9-6bfaf4130356-kube-api-access-fgg7z\") pod \"aws-ebs-csi-driver-node-zmrml\" (UID: \"e991c96a-3b5f-486a-b2a9-6bfaf4130356\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmrml"
May 26 11:24:30.430620 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.430579    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f1ab37a2-3fa3-4941-8cc3-0149f5478b36-serviceca\") pod \"node-ca-58x22\" (UID: \"f1ab37a2-3fa3-4941-8cc3-0149f5478b36\") " pod="openshift-image-registry/node-ca-58x22"
May 26 11:24:30.430620 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.430615    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-host-run-multus-certs\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.430971 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.430616    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b4769eef-c61f-4722-96a1-8461e3cc4801-konnectivity-ca\") pod \"konnectivity-agent-xtkmg\" (UID: \"b4769eef-c61f-4722-96a1-8461e3cc4801\") " pod="kube-system/konnectivity-agent-xtkmg"
May 26 11:24:30.430971 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.430582    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-host-run-multus-certs\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.430971 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.430672    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-host-run-k8s-cni-cncf-io\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.430971 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.430719    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be852edc-c490-4de5-99b0-a73297a80fa0-etc-kubernetes\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.430971 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.430734    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/be852edc-c490-4de5-99b0-a73297a80fa0-etc-sysconfig\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.430971 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.430751    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be852edc-c490-4de5-99b0-a73297a80fa0-var-lib-kubelet\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.430971 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.430765    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d316eade-d057-4616-b829-4c09f6ea6d16-iptables-alerter-script\") pod \"iptables-alerter-ldk8g\" (UID: \"d316eade-d057-4616-b829-4c09f6ea6d16\") " pod="openshift-network-operator/iptables-alerter-ldk8g"
May 26 11:24:30.430971 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.430763    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/be852edc-c490-4de5-99b0-a73297a80fa0-etc-modprobe-d\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.430971 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.430806    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/be852edc-c490-4de5-99b0-a73297a80fa0-etc-sysconfig\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.430971 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.430883    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b642b5f8-5576-42de-807d-6da907ae897d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kz76h\" (UID: \"b642b5f8-5576-42de-807d-6da907ae897d\") " pod="openshift-multus/multus-additional-cni-plugins-kz76h"
May 26 11:24:30.430971 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.430926    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/be852edc-c490-4de5-99b0-a73297a80fa0-run\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.430971 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.430950    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/be852edc-c490-4de5-99b0-a73297a80fa0-etc-tuned\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.431492 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.430981    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-host-kubelet\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.431492 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.431010    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-host-run-netns\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.431492 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.431036    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e991c96a-3b5f-486a-b2a9-6bfaf4130356-socket-dir\") pod \"aws-ebs-csi-driver-node-zmrml\" (UID: \"e991c96a-3b5f-486a-b2a9-6bfaf4130356\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmrml"
May 26 11:24:30.431492 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.431074    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/be852edc-c490-4de5-99b0-a73297a80fa0-run\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.431492 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.431114    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e991c96a-3b5f-486a-b2a9-6bfaf4130356-sys-fs\") pod \"aws-ebs-csi-driver-node-zmrml\" (UID: \"e991c96a-3b5f-486a-b2a9-6bfaf4130356\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmrml"
May 26 11:24:30.431492 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.431152    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bfdd44fe-70d3-4e4e-864b-f007de978ffe-cni-binary-copy\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.431492 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.431190    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-etc-kubernetes\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.431492 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.431195    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e991c96a-3b5f-486a-b2a9-6bfaf4130356-sys-fs\") pod \"aws-ebs-csi-driver-node-zmrml\" (UID: \"e991c96a-3b5f-486a-b2a9-6bfaf4130356\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmrml"
May 26 11:24:30.431492 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.431207    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e991c96a-3b5f-486a-b2a9-6bfaf4130356-socket-dir\") pod \"aws-ebs-csi-driver-node-zmrml\" (UID: \"e991c96a-3b5f-486a-b2a9-6bfaf4130356\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmrml"
May 26 11:24:30.431492 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.431327    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be852edc-c490-4de5-99b0-a73297a80fa0-lib-modules\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.431492 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.431365    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-os-release\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.431492 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.431441    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-os-release\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.432124 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.431505    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-etc-kubernetes\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.432124 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.431474    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be852edc-c490-4de5-99b0-a73297a80fa0-lib-modules\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.432124 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.431553    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fpxzc\" (UniqueName: \"kubernetes.io/projected/9d573a77-5520-489f-99cb-86419b23cd19-kube-api-access-fpxzc\") pod \"network-metrics-daemon-pp9l8\" (UID: \"9d573a77-5520-489f-99cb-86419b23cd19\") " pod="openshift-multus/network-metrics-daemon-pp9l8"
May 26 11:24:30.432124 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.431739    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-host-slash\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.432124 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.431832    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e991c96a-3b5f-486a-b2a9-6bfaf4130356-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zmrml\" (UID: \"e991c96a-3b5f-486a-b2a9-6bfaf4130356\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmrml"
May 26 11:24:30.432124 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.431848    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bfdd44fe-70d3-4e4e-864b-f007de978ffe-cni-binary-copy\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.432124 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.431862    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-multus-cni-dir\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.432124 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.431891    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-host-var-lib-kubelet\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.432124 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.431898    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e991c96a-3b5f-486a-b2a9-6bfaf4130356-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zmrml\" (UID: \"e991c96a-3b5f-486a-b2a9-6bfaf4130356\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmrml"
May 26 11:24:30.432124 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.431921    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxbfr\" (UniqueName: \"kubernetes.io/projected/d316eade-d057-4616-b829-4c09f6ea6d16-kube-api-access-hxbfr\") pod \"iptables-alerter-ldk8g\" (UID: \"d316eade-d057-4616-b829-4c09f6ea6d16\") " pod="openshift-network-operator/iptables-alerter-ldk8g"
May 26 11:24:30.432124 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.431937    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-host-var-lib-kubelet\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.432124 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.431951    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-etc-openvswitch\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.432124 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.431956    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-multus-cni-dir\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.432124 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.431989    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-ovnkube-config\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.432124 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.432017    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b642b5f8-5576-42de-807d-6da907ae897d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kz76h\" (UID: \"b642b5f8-5576-42de-807d-6da907ae897d\") " pod="openshift-multus/multus-additional-cni-plugins-kz76h"
May 26 11:24:30.432124 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.432046    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-host-run-netns\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.432124 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.432090    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-multus-conf-dir\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.432920 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.432102    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-host-run-netns\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.432920 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.432143    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/be852edc-c490-4de5-99b0-a73297a80fa0-etc-sysctl-d\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.432920 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.432170    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d573a77-5520-489f-99cb-86419b23cd19-metrics-certs\") pod \"network-metrics-daemon-pp9l8\" (UID: \"9d573a77-5520-489f-99cb-86419b23cd19\") " pod="openshift-multus/network-metrics-daemon-pp9l8"
May 26 11:24:30.432920 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.432213    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-log-socket\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.432920 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.432241    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b642b5f8-5576-42de-807d-6da907ae897d-cni-binary-copy\") pod \"multus-additional-cni-plugins-kz76h\" (UID: \"b642b5f8-5576-42de-807d-6da907ae897d\") " pod="openshift-multus/multus-additional-cni-plugins-kz76h"
May 26 11:24:30.432920 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.432269    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b642b5f8-5576-42de-807d-6da907ae897d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kz76h\" (UID: \"b642b5f8-5576-42de-807d-6da907ae897d\") " pod="openshift-multus/multus-additional-cni-plugins-kz76h"
May 26 11:24:30.432920 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.432279    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/be852edc-c490-4de5-99b0-a73297a80fa0-etc-sysctl-d\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.432920 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.432299    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-multus-socket-dir-parent\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.432920 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.432325    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-host-var-lib-cni-multus\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.432920 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.432351    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-hostroot\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.432920 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:30.432371    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:30.432920 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.432418    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-host-var-lib-cni-multus\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.432920 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.432449    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b4769eef-c61f-4722-96a1-8461e3cc4801-agent-certs\") pod \"konnectivity-agent-xtkmg\" (UID: \"b4769eef-c61f-4722-96a1-8461e3cc4801\") " pod="kube-system/konnectivity-agent-xtkmg"
May 26 11:24:30.432920 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.432478    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-var-lib-openvswitch\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.432920 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.432505    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-run-ovn\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.432920 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:30.432543    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d573a77-5520-489f-99cb-86419b23cd19-metrics-certs podName:9d573a77-5520-489f-99cb-86419b23cd19 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:30.932503144 +0000 UTC m=+3.123945326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d573a77-5520-489f-99cb-86419b23cd19-metrics-certs") pod "network-metrics-daemon-pp9l8" (UID: "9d573a77-5520-489f-99cb-86419b23cd19") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:30.432920 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.432581    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-multus-conf-dir\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.433564 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.432582    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.433564 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.432596    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-hostroot\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.433564 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.432614    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-ovn-node-metrics-cert\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.433564 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.432628    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-multus-socket-dir-parent\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.433564 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.432747    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-cnibin\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.433564 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.432800    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9xrms\" (UniqueName: \"kubernetes.io/projected/f1ab37a2-3fa3-4941-8cc3-0149f5478b36-kube-api-access-9xrms\") pod \"node-ca-58x22\" (UID: \"f1ab37a2-3fa3-4941-8cc3-0149f5478b36\") " pod="openshift-image-registry/node-ca-58x22"
May 26 11:24:30.433564 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.432830    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-host-run-ovn-kubernetes\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.433564 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.432849    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-cnibin\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.433564 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.432852    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-host-cni-bin\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.433564 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.432892    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b642b5f8-5576-42de-807d-6da907ae897d-cnibin\") pod \"multus-additional-cni-plugins-kz76h\" (UID: \"b642b5f8-5576-42de-807d-6da907ae897d\") " pod="openshift-multus/multus-additional-cni-plugins-kz76h"
May 26 11:24:30.433564 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.432950    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-host-var-lib-cni-bin\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.433564 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.432981    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bfdd44fe-70d3-4e4e-864b-f007de978ffe-multus-daemon-config\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.433564 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.433009    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bfdd44fe-70d3-4e4e-864b-f007de978ffe-host-var-lib-cni-bin\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.433564 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.433054    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bt6x9\" (UniqueName: \"kubernetes.io/projected/8528f6a0-0b8d-47f8-a20c-715dd15d1076-kube-api-access-bt6x9\") pod \"network-check-target-cbxbl\" (UID: \"8528f6a0-0b8d-47f8-a20c-715dd15d1076\") " pod="openshift-network-diagnostics/network-check-target-cbxbl"
May 26 11:24:30.433564 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.433102    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1ab37a2-3fa3-4941-8cc3-0149f5478b36-host\") pod \"node-ca-58x22\" (UID: \"f1ab37a2-3fa3-4941-8cc3-0149f5478b36\") " pod="openshift-image-registry/node-ca-58x22"
May 26 11:24:30.433564 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.433203    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1ab37a2-3fa3-4941-8cc3-0149f5478b36-host\") pod \"node-ca-58x22\" (UID: \"f1ab37a2-3fa3-4941-8cc3-0149f5478b36\") " pod="openshift-image-registry/node-ca-58x22"
May 26 11:24:30.433564 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.433452    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bfdd44fe-70d3-4e4e-864b-f007de978ffe-multus-daemon-config\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.434354 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.433707    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/be852edc-c490-4de5-99b0-a73297a80fa0-etc-tuned\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.434354 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.433714    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be852edc-c490-4de5-99b0-a73297a80fa0-tmp\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.434970 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.434951    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b4769eef-c61f-4722-96a1-8461e3cc4801-agent-certs\") pod \"konnectivity-agent-xtkmg\" (UID: \"b4769eef-c61f-4722-96a1-8461e3cc4801\") " pod="kube-system/konnectivity-agent-xtkmg"
May 26 11:24:30.438403 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.438374    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjpxj\" (UniqueName: \"kubernetes.io/projected/be852edc-c490-4de5-99b0-a73297a80fa0-kube-api-access-kjpxj\") pod \"tuned-njvjv\" (UID: \"be852edc-c490-4de5-99b0-a73297a80fa0\") " pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.438521 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.438468    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m487h\" (UniqueName: \"kubernetes.io/projected/bfdd44fe-70d3-4e4e-864b-f007de978ffe-kube-api-access-m487h\") pod \"multus-pfkj5\" (UID: \"bfdd44fe-70d3-4e4e-864b-f007de978ffe\") " pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.438576 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.438561    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgg7z\" (UniqueName: \"kubernetes.io/projected/e991c96a-3b5f-486a-b2a9-6bfaf4130356-kube-api-access-fgg7z\") pod \"aws-ebs-csi-driver-node-zmrml\" (UID: \"e991c96a-3b5f-486a-b2a9-6bfaf4130356\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmrml"
May 26 11:24:30.439032 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:30.439013    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:24:30.439129 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:30.439036    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:24:30.439129 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:30.439049    2563 projected.go:194] Error preparing data for projected volume kube-api-access-bt6x9 for pod openshift-network-diagnostics/network-check-target-cbxbl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:30.439236 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:30.439138    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8528f6a0-0b8d-47f8-a20c-715dd15d1076-kube-api-access-bt6x9 podName:8528f6a0-0b8d-47f8-a20c-715dd15d1076 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:30.939120941 +0000 UTC m=+3.130563124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bt6x9" (UniqueName: "kubernetes.io/projected/8528f6a0-0b8d-47f8-a20c-715dd15d1076-kube-api-access-bt6x9") pod "network-check-target-cbxbl" (UID: "8528f6a0-0b8d-47f8-a20c-715dd15d1076") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:30.440141 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.440119    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpxzc\" (UniqueName: \"kubernetes.io/projected/9d573a77-5520-489f-99cb-86419b23cd19-kube-api-access-fpxzc\") pod \"network-metrics-daemon-pp9l8\" (UID: \"9d573a77-5520-489f-99cb-86419b23cd19\") " pod="openshift-multus/network-metrics-daemon-pp9l8"
May 26 11:24:30.440671 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.440653    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xrms\" (UniqueName: \"kubernetes.io/projected/f1ab37a2-3fa3-4941-8cc3-0149f5478b36-kube-api-access-9xrms\") pod \"node-ca-58x22\" (UID: \"f1ab37a2-3fa3-4941-8cc3-0149f5478b36\") " pod="openshift-image-registry/node-ca-58x22"
May 26 11:24:30.533897 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.533858    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-host-run-ovn-kubernetes\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.533897 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.533906    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-host-cni-bin\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.534156 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.533961    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-host-cni-bin\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.534156 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.533994    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b642b5f8-5576-42de-807d-6da907ae897d-cnibin\") pod \"multus-additional-cni-plugins-kz76h\" (UID: \"b642b5f8-5576-42de-807d-6da907ae897d\") " pod="openshift-multus/multus-additional-cni-plugins-kz76h"
May 26 11:24:30.534156 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534029    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b642b5f8-5576-42de-807d-6da907ae897d-system-cni-dir\") pod \"multus-additional-cni-plugins-kz76h\" (UID: \"b642b5f8-5576-42de-807d-6da907ae897d\") " pod="openshift-multus/multus-additional-cni-plugins-kz76h"
May 26 11:24:30.534156 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534045    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f529n\" (UniqueName: \"kubernetes.io/projected/b642b5f8-5576-42de-807d-6da907ae897d-kube-api-access-f529n\") pod \"multus-additional-cni-plugins-kz76h\" (UID: \"b642b5f8-5576-42de-807d-6da907ae897d\") " pod="openshift-multus/multus-additional-cni-plugins-kz76h"
May 26 11:24:30.534156 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.533988    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-host-run-ovn-kubernetes\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.534156 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534064    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-ovnkube-script-lib\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.534156 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534110    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b642b5f8-5576-42de-807d-6da907ae897d-system-cni-dir\") pod \"multus-additional-cni-plugins-kz76h\" (UID: \"b642b5f8-5576-42de-807d-6da907ae897d\") " pod="openshift-multus/multus-additional-cni-plugins-kz76h"
May 26 11:24:30.534156 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534129    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b642b5f8-5576-42de-807d-6da907ae897d-cnibin\") pod \"multus-additional-cni-plugins-kz76h\" (UID: \"b642b5f8-5576-42de-807d-6da907ae897d\") " pod="openshift-multus/multus-additional-cni-plugins-kz76h"
May 26 11:24:30.534156 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534143    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d316eade-d057-4616-b829-4c09f6ea6d16-host-slash\") pod \"iptables-alerter-ldk8g\" (UID: \"d316eade-d057-4616-b829-4c09f6ea6d16\") " pod="openshift-network-operator/iptables-alerter-ldk8g"
May 26 11:24:30.534548 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534299    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d316eade-d057-4616-b829-4c09f6ea6d16-host-slash\") pod \"iptables-alerter-ldk8g\" (UID: \"d316eade-d057-4616-b829-4c09f6ea6d16\") " pod="openshift-network-operator/iptables-alerter-ldk8g"
May 26 11:24:30.534548 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534312    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-systemd-units\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.534548 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534349    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-run-openvswitch\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.534548 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534375    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-node-log\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.534548 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534404    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b642b5f8-5576-42de-807d-6da907ae897d-os-release\") pod \"multus-additional-cni-plugins-kz76h\" (UID: \"b642b5f8-5576-42de-807d-6da907ae897d\") " pod="openshift-multus/multus-additional-cni-plugins-kz76h"
May 26 11:24:30.534548 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534435    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bllzp\" (UniqueName: \"kubernetes.io/projected/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-kube-api-access-bllzp\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.534548 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534437    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-node-log\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.534548 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534448    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-run-openvswitch\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.534548 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534401    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-systemd-units\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.534548 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534489    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-env-overrides\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.534548 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534499    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b642b5f8-5576-42de-807d-6da907ae897d-os-release\") pod \"multus-additional-cni-plugins-kz76h\" (UID: \"b642b5f8-5576-42de-807d-6da907ae897d\") " pod="openshift-multus/multus-additional-cni-plugins-kz76h"
May 26 11:24:30.534548 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534516    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-run-systemd\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.534548 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534541    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-host-cni-netd\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.535164 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534574    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d316eade-d057-4616-b829-4c09f6ea6d16-iptables-alerter-script\") pod \"iptables-alerter-ldk8g\" (UID: \"d316eade-d057-4616-b829-4c09f6ea6d16\") " pod="openshift-network-operator/iptables-alerter-ldk8g"
May 26 11:24:30.535164 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534601    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-ovnkube-script-lib\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.535164 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534600    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-run-systemd\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.535164 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534602    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b642b5f8-5576-42de-807d-6da907ae897d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kz76h\" (UID: \"b642b5f8-5576-42de-807d-6da907ae897d\") " pod="openshift-multus/multus-additional-cni-plugins-kz76h"
May 26 11:24:30.535164 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534662    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-host-cni-netd\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.535164 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534674    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-host-kubelet\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.535164 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534706    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-host-run-netns\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.535164 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534742    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-host-slash\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.535164 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534769    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hxbfr\" (UniqueName: \"kubernetes.io/projected/d316eade-d057-4616-b829-4c09f6ea6d16-kube-api-access-hxbfr\") pod \"iptables-alerter-ldk8g\" (UID: \"d316eade-d057-4616-b829-4c09f6ea6d16\") " pod="openshift-network-operator/iptables-alerter-ldk8g"
May 26 11:24:30.535164 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534814    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-etc-openvswitch\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.535164 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534818    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-host-run-netns\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.535164 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534829    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-ovnkube-config\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.535164 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534847    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b642b5f8-5576-42de-807d-6da907ae897d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kz76h\" (UID: \"b642b5f8-5576-42de-807d-6da907ae897d\") " pod="openshift-multus/multus-additional-cni-plugins-kz76h"
May 26 11:24:30.535164 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534877    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-log-socket\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.535164 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534893    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b642b5f8-5576-42de-807d-6da907ae897d-cni-binary-copy\") pod \"multus-additional-cni-plugins-kz76h\" (UID: \"b642b5f8-5576-42de-807d-6da907ae897d\") " pod="openshift-multus/multus-additional-cni-plugins-kz76h"
May 26 11:24:30.535164 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534909    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b642b5f8-5576-42de-807d-6da907ae897d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kz76h\" (UID: \"b642b5f8-5576-42de-807d-6da907ae897d\") " pod="openshift-multus/multus-additional-cni-plugins-kz76h"
May 26 11:24:30.535164 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534928    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-var-lib-openvswitch\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.535794 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534942    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-run-ovn\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.535794 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534959    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.535794 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534959    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-env-overrides\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.535794 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534974    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-ovn-node-metrics-cert\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.535794 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.535018    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-host-slash\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.535794 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.535048    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-log-socket\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.535794 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.535064    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-etc-openvswitch\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.535794 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.534767    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-host-kubelet\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.535794 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.535228    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b642b5f8-5576-42de-807d-6da907ae897d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kz76h\" (UID: \"b642b5f8-5576-42de-807d-6da907ae897d\") " pod="openshift-multus/multus-additional-cni-plugins-kz76h"
May 26 11:24:30.535794 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.535258    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-var-lib-openvswitch\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.535794 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.535295    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-run-ovn\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.535794 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.535354    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.535794 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.535439    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b642b5f8-5576-42de-807d-6da907ae897d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kz76h\" (UID: \"b642b5f8-5576-42de-807d-6da907ae897d\") " pod="openshift-multus/multus-additional-cni-plugins-kz76h"
May 26 11:24:30.535794 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.535532    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b642b5f8-5576-42de-807d-6da907ae897d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kz76h\" (UID: \"b642b5f8-5576-42de-807d-6da907ae897d\") " pod="openshift-multus/multus-additional-cni-plugins-kz76h"
May 26 11:24:30.535794 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.535689    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b642b5f8-5576-42de-807d-6da907ae897d-cni-binary-copy\") pod \"multus-additional-cni-plugins-kz76h\" (UID: \"b642b5f8-5576-42de-807d-6da907ae897d\") " pod="openshift-multus/multus-additional-cni-plugins-kz76h"
May 26 11:24:30.535794 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.535713    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-ovnkube-config\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.536416 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.535836    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d316eade-d057-4616-b829-4c09f6ea6d16-iptables-alerter-script\") pod \"iptables-alerter-ldk8g\" (UID: \"d316eade-d057-4616-b829-4c09f6ea6d16\") " pod="openshift-network-operator/iptables-alerter-ldk8g"
May 26 11:24:30.537123 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.537101    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-ovn-node-metrics-cert\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.542908 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.542856    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxbfr\" (UniqueName: \"kubernetes.io/projected/d316eade-d057-4616-b829-4c09f6ea6d16-kube-api-access-hxbfr\") pod \"iptables-alerter-ldk8g\" (UID: \"d316eade-d057-4616-b829-4c09f6ea6d16\") " pod="openshift-network-operator/iptables-alerter-ldk8g"
May 26 11:24:30.543319 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.543296    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f529n\" (UniqueName: \"kubernetes.io/projected/b642b5f8-5576-42de-807d-6da907ae897d-kube-api-access-f529n\") pod \"multus-additional-cni-plugins-kz76h\" (UID: \"b642b5f8-5576-42de-807d-6da907ae897d\") " pod="openshift-multus/multus-additional-cni-plugins-kz76h"
May 26 11:24:30.543571 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.543555    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bllzp\" (UniqueName: \"kubernetes.io/projected/8526b1f6-8087-4c6e-8cd0-a0188c5db3a2-kube-api-access-bllzp\") pod \"ovnkube-node-wxzlw\" (UID: \"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.612600 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.612553    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xtkmg"
May 26 11:24:30.619641 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.619615    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmrml"
May 26 11:24:30.631279 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.631258    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-njvjv"
May 26 11:24:30.635875 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.635856    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pfkj5"
May 26 11:24:30.641956 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.641936    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-58x22"
May 26 11:24:30.642055 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.641993    2563 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 11:24:30.648665 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.648646    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kz76h"
May 26 11:24:30.656203 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.656184    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ldk8g"
May 26 11:24:30.661823 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.661803    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:30.937544 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:30.937456    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d573a77-5520-489f-99cb-86419b23cd19-metrics-certs\") pod \"network-metrics-daemon-pp9l8\" (UID: \"9d573a77-5520-489f-99cb-86419b23cd19\") " pod="openshift-multus/network-metrics-daemon-pp9l8"
May 26 11:24:30.937695 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:30.937622    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:30.937749 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:30.937699    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d573a77-5520-489f-99cb-86419b23cd19-metrics-certs podName:9d573a77-5520-489f-99cb-86419b23cd19 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:31.937673457 +0000 UTC m=+4.129115622 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d573a77-5520-489f-99cb-86419b23cd19-metrics-certs") pod "network-metrics-daemon-pp9l8" (UID: "9d573a77-5520-489f-99cb-86419b23cd19") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:31.013914 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:31.013887    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4769eef_c61f_4722_96a1_8461e3cc4801.slice/crio-465d7a095debe2511dfb54b114db4433065f2d437b14741d63186b15cecd0cc6 WatchSource:0}: Error finding container 465d7a095debe2511dfb54b114db4433065f2d437b14741d63186b15cecd0cc6: Status 404 returned error can't find the container with id 465d7a095debe2511dfb54b114db4433065f2d437b14741d63186b15cecd0cc6
May 26 11:24:31.015989 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:31.015959    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe852edc_c490_4de5_99b0_a73297a80fa0.slice/crio-e40a1b7dd98f6ca50bab49869c10cdc4ef642699ab7b655e56cb88e662eac1cd WatchSource:0}: Error finding container e40a1b7dd98f6ca50bab49869c10cdc4ef642699ab7b655e56cb88e662eac1cd: Status 404 returned error can't find the container with id e40a1b7dd98f6ca50bab49869c10cdc4ef642699ab7b655e56cb88e662eac1cd
May 26 11:24:31.038507 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:31.038480    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bt6x9\" (UniqueName: \"kubernetes.io/projected/8528f6a0-0b8d-47f8-a20c-715dd15d1076-kube-api-access-bt6x9\") pod \"network-check-target-cbxbl\" (UID: \"8528f6a0-0b8d-47f8-a20c-715dd15d1076\") " pod="openshift-network-diagnostics/network-check-target-cbxbl"
May 26 11:24:31.038635 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:31.038601    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:24:31.038635 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:31.038614    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:24:31.038635 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:31.038622    2563 projected.go:194] Error preparing data for projected volume kube-api-access-bt6x9 for pod openshift-network-diagnostics/network-check-target-cbxbl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:31.038727 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:31.038669    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8528f6a0-0b8d-47f8-a20c-715dd15d1076-kube-api-access-bt6x9 podName:8528f6a0-0b8d-47f8-a20c-715dd15d1076 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:32.038657202 +0000 UTC m=+4.230099367 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bt6x9" (UniqueName: "kubernetes.io/projected/8528f6a0-0b8d-47f8-a20c-715dd15d1076-kube-api-access-bt6x9") pod "network-check-target-cbxbl" (UID: "8528f6a0-0b8d-47f8-a20c-715dd15d1076") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:31.044858 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:31.044828    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8526b1f6_8087_4c6e_8cd0_a0188c5db3a2.slice/crio-d1d05323d17d9e8c1e099b6270f26e7d4163ca9e22f75427bf03e53aee3da9bb WatchSource:0}: Error finding container d1d05323d17d9e8c1e099b6270f26e7d4163ca9e22f75427bf03e53aee3da9bb: Status 404 returned error can't find the container with id d1d05323d17d9e8c1e099b6270f26e7d4163ca9e22f75427bf03e53aee3da9bb
May 26 11:24:31.369448 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:31.369234    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-25 11:19:29 +0000 UTC" deadline="2027-11-19 06:53:15.663814633 +0000 UTC"
May 26 11:24:31.369448 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:31.369446    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13003h28m44.29437301s"
May 26 11:24:31.424248 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:31.424203    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbxbl"
May 26 11:24:31.424403 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:31.424327    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbxbl" podUID="8528f6a0-0b8d-47f8-a20c-715dd15d1076"
May 26 11:24:31.435928 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:31.435889    2563 generic.go:358] "Generic (PLEG): container finished" podID="0802379863c3c768639fbc45cfce5c9e" containerID="2ed375fde9268c0133628e8346c05aa345a3c69578d1b1a9c3519cd3c25eb4b5" exitCode=0
May 26 11:24:31.436161 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:31.435994    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-97.ec2.internal" event={"ID":"0802379863c3c768639fbc45cfce5c9e","Type":"ContainerDied","Data":"2ed375fde9268c0133628e8346c05aa345a3c69578d1b1a9c3519cd3c25eb4b5"}
May 26 11:24:31.441524 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:31.441210    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw" event={"ID":"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2","Type":"ContainerStarted","Data":"d1d05323d17d9e8c1e099b6270f26e7d4163ca9e22f75427bf03e53aee3da9bb"}
May 26 11:24:31.443585 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:31.443482    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-58x22" event={"ID":"f1ab37a2-3fa3-4941-8cc3-0149f5478b36","Type":"ContainerStarted","Data":"3bc9f1ffe2d0cffcf1030be3e02b2277c99f96a09074a5543f80dd76980470bf"}
May 26 11:24:31.447092 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:31.447063    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ldk8g" event={"ID":"d316eade-d057-4616-b829-4c09f6ea6d16","Type":"ContainerStarted","Data":"8932f4a3434eb95bf41e584908748f4c5f5a54e34fec8492848a8e986e7f97c1"}
May 26 11:24:31.448820 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:31.448768    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kz76h" event={"ID":"b642b5f8-5576-42de-807d-6da907ae897d","Type":"ContainerStarted","Data":"e4c0f2c62f6096df8e92ed7aa5722b64250f081b9f21b7ca3f85562fefa0b250"}
May 26 11:24:31.452637 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:31.452606    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmrml" event={"ID":"e991c96a-3b5f-486a-b2a9-6bfaf4130356","Type":"ContainerStarted","Data":"165ba5c2eccc4f179721cf9bfa7a1168346ba0f25f97f578631b769691d2f166"}
May 26 11:24:31.456459 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:31.456428    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-njvjv" event={"ID":"be852edc-c490-4de5-99b0-a73297a80fa0","Type":"ContainerStarted","Data":"e40a1b7dd98f6ca50bab49869c10cdc4ef642699ab7b655e56cb88e662eac1cd"}
May 26 11:24:31.460905 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:31.460033    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-97.ec2.internal" event={"ID":"7b05a01f2c158d2611520e5fe1f839c3","Type":"ContainerStarted","Data":"4ef071403305a2c0ff9b69f4ec9a6d26966009f73c3ae23d89197893ccf05259"}
May 26 11:24:31.464835 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:31.463989    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pfkj5" event={"ID":"bfdd44fe-70d3-4e4e-864b-f007de978ffe","Type":"ContainerStarted","Data":"481417a9676ee82a12262449ee961b8c1835bd03c413ee5272aefc57da6a6047"}
May 26 11:24:31.466722 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:31.466656    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xtkmg" event={"ID":"b4769eef-c61f-4722-96a1-8461e3cc4801","Type":"ContainerStarted","Data":"465d7a095debe2511dfb54b114db4433065f2d437b14741d63186b15cecd0cc6"}
May 26 11:24:31.478444 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:31.478390    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-97.ec2.internal" podStartSLOduration=2.478373845 podStartE2EDuration="2.478373845s" podCreationTimestamp="2026-05-26 11:24:29 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:24:31.478112556 +0000 UTC m=+3.669554743" watchObservedRunningTime="2026-05-26 11:24:31.478373845 +0000 UTC m=+3.669816034"
May 26 11:24:31.946987 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:31.946890    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d573a77-5520-489f-99cb-86419b23cd19-metrics-certs\") pod \"network-metrics-daemon-pp9l8\" (UID: \"9d573a77-5520-489f-99cb-86419b23cd19\") " pod="openshift-multus/network-metrics-daemon-pp9l8"
May 26 11:24:31.947154 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:31.947073    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:31.947154 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:31.947136    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d573a77-5520-489f-99cb-86419b23cd19-metrics-certs podName:9d573a77-5520-489f-99cb-86419b23cd19 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:33.947116042 +0000 UTC m=+6.138558211 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d573a77-5520-489f-99cb-86419b23cd19-metrics-certs") pod "network-metrics-daemon-pp9l8" (UID: "9d573a77-5520-489f-99cb-86419b23cd19") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:32.047976 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:32.047318    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bt6x9\" (UniqueName: \"kubernetes.io/projected/8528f6a0-0b8d-47f8-a20c-715dd15d1076-kube-api-access-bt6x9\") pod \"network-check-target-cbxbl\" (UID: \"8528f6a0-0b8d-47f8-a20c-715dd15d1076\") " pod="openshift-network-diagnostics/network-check-target-cbxbl"
May 26 11:24:32.047976 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:32.047482    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:24:32.047976 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:32.047500    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:24:32.047976 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:32.047513    2563 projected.go:194] Error preparing data for projected volume kube-api-access-bt6x9 for pod openshift-network-diagnostics/network-check-target-cbxbl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:32.047976 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:32.047571    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8528f6a0-0b8d-47f8-a20c-715dd15d1076-kube-api-access-bt6x9 podName:8528f6a0-0b8d-47f8-a20c-715dd15d1076 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:34.047553515 +0000 UTC m=+6.238995683 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bt6x9" (UniqueName: "kubernetes.io/projected/8528f6a0-0b8d-47f8-a20c-715dd15d1076-kube-api-access-bt6x9") pod "network-check-target-cbxbl" (UID: "8528f6a0-0b8d-47f8-a20c-715dd15d1076") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:32.426520 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:32.424924    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp9l8"
May 26 11:24:32.426520 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:32.425083    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pp9l8" podUID="9d573a77-5520-489f-99cb-86419b23cd19"
May 26 11:24:32.482476 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:32.482417    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-97.ec2.internal" event={"ID":"0802379863c3c768639fbc45cfce5c9e","Type":"ContainerStarted","Data":"0318a03c1bdf01201411cff495d2577914a471b82e142f4460bde1780171f42d"}
May 26 11:24:33.424743 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:33.424242    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbxbl"
May 26 11:24:33.424743 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:33.424374    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbxbl" podUID="8528f6a0-0b8d-47f8-a20c-715dd15d1076"
May 26 11:24:33.964334 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:33.964287    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d573a77-5520-489f-99cb-86419b23cd19-metrics-certs\") pod \"network-metrics-daemon-pp9l8\" (UID: \"9d573a77-5520-489f-99cb-86419b23cd19\") " pod="openshift-multus/network-metrics-daemon-pp9l8"
May 26 11:24:33.964766 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:33.964475    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:33.964766 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:33.964533    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d573a77-5520-489f-99cb-86419b23cd19-metrics-certs podName:9d573a77-5520-489f-99cb-86419b23cd19 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:37.964514911 +0000 UTC m=+10.155957081 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d573a77-5520-489f-99cb-86419b23cd19-metrics-certs") pod "network-metrics-daemon-pp9l8" (UID: "9d573a77-5520-489f-99cb-86419b23cd19") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:34.065461 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:34.065422    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bt6x9\" (UniqueName: \"kubernetes.io/projected/8528f6a0-0b8d-47f8-a20c-715dd15d1076-kube-api-access-bt6x9\") pod \"network-check-target-cbxbl\" (UID: \"8528f6a0-0b8d-47f8-a20c-715dd15d1076\") " pod="openshift-network-diagnostics/network-check-target-cbxbl"
May 26 11:24:34.065656 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:34.065603    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:24:34.065656 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:34.065623    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:24:34.065656 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:34.065636    2563 projected.go:194] Error preparing data for projected volume kube-api-access-bt6x9 for pod openshift-network-diagnostics/network-check-target-cbxbl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:34.065890 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:34.065691    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8528f6a0-0b8d-47f8-a20c-715dd15d1076-kube-api-access-bt6x9 podName:8528f6a0-0b8d-47f8-a20c-715dd15d1076 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:38.065674763 +0000 UTC m=+10.257116945 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bt6x9" (UniqueName: "kubernetes.io/projected/8528f6a0-0b8d-47f8-a20c-715dd15d1076-kube-api-access-bt6x9") pod "network-check-target-cbxbl" (UID: "8528f6a0-0b8d-47f8-a20c-715dd15d1076") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:34.424790 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:34.424686    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp9l8"
May 26 11:24:34.424954 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:34.424872    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pp9l8" podUID="9d573a77-5520-489f-99cb-86419b23cd19"
May 26 11:24:35.424374 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:35.424331    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbxbl"
May 26 11:24:35.424853 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:35.424477    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbxbl" podUID="8528f6a0-0b8d-47f8-a20c-715dd15d1076"
May 26 11:24:36.424751 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:36.424717    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp9l8"
May 26 11:24:36.425203 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:36.424882    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pp9l8" podUID="9d573a77-5520-489f-99cb-86419b23cd19"
May 26 11:24:37.424339 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:37.424291    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbxbl"
May 26 11:24:37.424526 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:37.424430    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbxbl" podUID="8528f6a0-0b8d-47f8-a20c-715dd15d1076"
May 26 11:24:38.000035 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:37.999991    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d573a77-5520-489f-99cb-86419b23cd19-metrics-certs\") pod \"network-metrics-daemon-pp9l8\" (UID: \"9d573a77-5520-489f-99cb-86419b23cd19\") " pod="openshift-multus/network-metrics-daemon-pp9l8"
May 26 11:24:38.000484 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:38.000136    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:38.000484 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:38.000220    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d573a77-5520-489f-99cb-86419b23cd19-metrics-certs podName:9d573a77-5520-489f-99cb-86419b23cd19 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:46.000199141 +0000 UTC m=+18.191641324 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d573a77-5520-489f-99cb-86419b23cd19-metrics-certs") pod "network-metrics-daemon-pp9l8" (UID: "9d573a77-5520-489f-99cb-86419b23cd19") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:38.101069 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:38.101027    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bt6x9\" (UniqueName: \"kubernetes.io/projected/8528f6a0-0b8d-47f8-a20c-715dd15d1076-kube-api-access-bt6x9\") pod \"network-check-target-cbxbl\" (UID: \"8528f6a0-0b8d-47f8-a20c-715dd15d1076\") " pod="openshift-network-diagnostics/network-check-target-cbxbl"
May 26 11:24:38.101244 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:38.101217    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:24:38.101244 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:38.101235    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:24:38.101356 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:38.101247    2563 projected.go:194] Error preparing data for projected volume kube-api-access-bt6x9 for pod openshift-network-diagnostics/network-check-target-cbxbl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:38.101356 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:38.101305    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8528f6a0-0b8d-47f8-a20c-715dd15d1076-kube-api-access-bt6x9 podName:8528f6a0-0b8d-47f8-a20c-715dd15d1076 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:46.10128781 +0000 UTC m=+18.292729980 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bt6x9" (UniqueName: "kubernetes.io/projected/8528f6a0-0b8d-47f8-a20c-715dd15d1076-kube-api-access-bt6x9") pod "network-check-target-cbxbl" (UID: "8528f6a0-0b8d-47f8-a20c-715dd15d1076") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:38.425472 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:38.425389    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp9l8"
May 26 11:24:38.425627 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:38.425493    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pp9l8" podUID="9d573a77-5520-489f-99cb-86419b23cd19"
May 26 11:24:39.424504 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:39.424469    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbxbl"
May 26 11:24:39.425004 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:39.424595    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbxbl" podUID="8528f6a0-0b8d-47f8-a20c-715dd15d1076"
May 26 11:24:40.424767 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:40.424723    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp9l8"
May 26 11:24:40.425216 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:40.424862    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pp9l8" podUID="9d573a77-5520-489f-99cb-86419b23cd19"
May 26 11:24:41.425151 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:41.425066    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbxbl"
May 26 11:24:41.425566 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:41.425202    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbxbl" podUID="8528f6a0-0b8d-47f8-a20c-715dd15d1076"
May 26 11:24:42.427785 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:42.427755    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp9l8"
May 26 11:24:42.428312 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:42.427876    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pp9l8" podUID="9d573a77-5520-489f-99cb-86419b23cd19"
May 26 11:24:43.424828 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:43.424794    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbxbl"
May 26 11:24:43.425016 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:43.424897    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbxbl" podUID="8528f6a0-0b8d-47f8-a20c-715dd15d1076"
May 26 11:24:44.424522 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:44.424475    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp9l8"
May 26 11:24:44.425012 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:44.424603    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pp9l8" podUID="9d573a77-5520-489f-99cb-86419b23cd19"
May 26 11:24:45.424225 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:45.424186    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbxbl"
May 26 11:24:45.424531 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:45.424307    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbxbl" podUID="8528f6a0-0b8d-47f8-a20c-715dd15d1076"
May 26 11:24:46.054855 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:46.054814    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d573a77-5520-489f-99cb-86419b23cd19-metrics-certs\") pod \"network-metrics-daemon-pp9l8\" (UID: \"9d573a77-5520-489f-99cb-86419b23cd19\") " pod="openshift-multus/network-metrics-daemon-pp9l8"
May 26 11:24:46.055285 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:46.055014    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:46.055285 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:46.055098    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d573a77-5520-489f-99cb-86419b23cd19-metrics-certs podName:9d573a77-5520-489f-99cb-86419b23cd19 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:02.05507649 +0000 UTC m=+34.246518669 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d573a77-5520-489f-99cb-86419b23cd19-metrics-certs") pod "network-metrics-daemon-pp9l8" (UID: "9d573a77-5520-489f-99cb-86419b23cd19") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:46.155655 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:46.155604    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bt6x9\" (UniqueName: \"kubernetes.io/projected/8528f6a0-0b8d-47f8-a20c-715dd15d1076-kube-api-access-bt6x9\") pod \"network-check-target-cbxbl\" (UID: \"8528f6a0-0b8d-47f8-a20c-715dd15d1076\") " pod="openshift-network-diagnostics/network-check-target-cbxbl"
May 26 11:24:46.155831 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:46.155751    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:24:46.155831 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:46.155770    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:24:46.155831 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:46.155801    2563 projected.go:194] Error preparing data for projected volume kube-api-access-bt6x9 for pod openshift-network-diagnostics/network-check-target-cbxbl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:46.155995 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:46.155877    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8528f6a0-0b8d-47f8-a20c-715dd15d1076-kube-api-access-bt6x9 podName:8528f6a0-0b8d-47f8-a20c-715dd15d1076 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:02.155857736 +0000 UTC m=+34.347299903 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bt6x9" (UniqueName: "kubernetes.io/projected/8528f6a0-0b8d-47f8-a20c-715dd15d1076-kube-api-access-bt6x9") pod "network-check-target-cbxbl" (UID: "8528f6a0-0b8d-47f8-a20c-715dd15d1076") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:46.424540 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:46.424503    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp9l8"
May 26 11:24:46.424714 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:46.424644    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pp9l8" podUID="9d573a77-5520-489f-99cb-86419b23cd19"
May 26 11:24:47.425221 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:47.425182    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbxbl"
May 26 11:24:47.425682 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:47.425310    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbxbl" podUID="8528f6a0-0b8d-47f8-a20c-715dd15d1076"
May 26 11:24:48.427840 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:48.427810    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp9l8"
May 26 11:24:48.428228 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:48.427946    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pp9l8" podUID="9d573a77-5520-489f-99cb-86419b23cd19"
May 26 11:24:49.425124 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:49.425089    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbxbl"
May 26 11:24:49.425279 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:49.425215    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbxbl" podUID="8528f6a0-0b8d-47f8-a20c-715dd15d1076"
May 26 11:24:50.427402 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:50.427083    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp9l8"
May 26 11:24:50.428176 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:50.427497    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pp9l8" podUID="9d573a77-5520-489f-99cb-86419b23cd19"
May 26 11:24:50.514638 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:50.514606    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pfkj5" event={"ID":"bfdd44fe-70d3-4e4e-864b-f007de978ffe","Type":"ContainerStarted","Data":"7305a97d79b6a07e8afffbb7642a1470e3fc8bb40b0f2a7b635f266e22d86e4b"}
May 26 11:24:50.515955 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:50.515926    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xtkmg" event={"ID":"b4769eef-c61f-4722-96a1-8461e3cc4801","Type":"ContainerStarted","Data":"83fa36f09f1ed2bcf98da423055d20409f8f17d151626e2f6e81593f4aac030e"}
May 26 11:24:50.521203 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:50.521174    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw" event={"ID":"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2","Type":"ContainerStarted","Data":"90911f6e6e23b34588468a1cdf808cc6115cdcffed25b27760df47d14056feda"}
May 26 11:24:50.521203 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:50.521200    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw" event={"ID":"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2","Type":"ContainerStarted","Data":"064a1b2205c6437790a962682cf3b768d18a3c900e0c3bea26b87848e083487b"}
May 26 11:24:50.521203 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:50.521209    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw" event={"ID":"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2","Type":"ContainerStarted","Data":"42c967bc5823b613fdbee421c05188a67e9d7549bdf3675427d17a7da5057e9d"}
May 26 11:24:50.521443 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:50.521217    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw" event={"ID":"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2","Type":"ContainerStarted","Data":"bce2139461d4212e7dfc0b7d55b37f0cd169559b0f3a30b142dfbbb3a9480f94"}
May 26 11:24:50.521443 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:50.521225    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw" event={"ID":"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2","Type":"ContainerStarted","Data":"45e3990cd26b8198c66a212554ae0344f5e2f8f3b546aa230028f79339e105b1"}
May 26 11:24:50.521443 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:50.521234    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw" event={"ID":"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2","Type":"ContainerStarted","Data":"abdc2b19ba97d99c51257e64a77c34aaf1e3e59058f79bbe56c6db0e52a7f28d"}
May 26 11:24:50.522644 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:50.522572    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-58x22" event={"ID":"f1ab37a2-3fa3-4941-8cc3-0149f5478b36","Type":"ContainerStarted","Data":"bcdf8a53a6075fc2c44550096d8bc484255812180a919609d05e68c1c05b5257"}
May 26 11:24:50.523830 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:50.523804    2563 generic.go:358] "Generic (PLEG): container finished" podID="b642b5f8-5576-42de-807d-6da907ae897d" containerID="d6b3e23612d478ef22d003b2009535b94ff8c734d8d21c0bb2b0215b1b8c7997" exitCode=0
May 26 11:24:50.523924 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:50.523868    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kz76h" event={"ID":"b642b5f8-5576-42de-807d-6da907ae897d","Type":"ContainerDied","Data":"d6b3e23612d478ef22d003b2009535b94ff8c734d8d21c0bb2b0215b1b8c7997"}
May 26 11:24:50.525145 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:50.525098    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmrml" event={"ID":"e991c96a-3b5f-486a-b2a9-6bfaf4130356","Type":"ContainerStarted","Data":"5bae6b948388a299a32178085498555c6f47039256a535263202b611cea57e11"}
May 26 11:24:50.526276 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:50.526257    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-njvjv" event={"ID":"be852edc-c490-4de5-99b0-a73297a80fa0","Type":"ContainerStarted","Data":"19000fc965d188fe6720bcedcb8711efb5c1a5411caa327e09135f3cde9f1a9f"}
May 26 11:24:50.537204 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:50.535404    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-97.ec2.internal" podStartSLOduration=21.535387997 podStartE2EDuration="21.535387997s" podCreationTimestamp="2026-05-26 11:24:29 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:24:32.496338052 +0000 UTC m=+4.687780243" watchObservedRunningTime="2026-05-26 11:24:50.535387997 +0000 UTC m=+22.726830186"
May 26 11:24:50.537204 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:50.535531    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-pfkj5" podStartSLOduration=4.087916363 podStartE2EDuration="22.535523348s" podCreationTimestamp="2026-05-26 11:24:28 +0000 UTC" firstStartedPulling="2026-05-26 11:24:31.041459527 +0000 UTC m=+3.232901692" lastFinishedPulling="2026-05-26 11:24:49.489066512 +0000 UTC m=+21.680508677" observedRunningTime="2026-05-26 11:24:50.534080107 +0000 UTC m=+22.725522295" watchObservedRunningTime="2026-05-26 11:24:50.535523348 +0000 UTC m=+22.726965536"
May 26 11:24:50.570660 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:50.570611    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xtkmg" podStartSLOduration=4.114013473 podStartE2EDuration="22.570567806s" podCreationTimestamp="2026-05-26 11:24:28 +0000 UTC" firstStartedPulling="2026-05-26 11:24:31.0166287 +0000 UTC m=+3.208070865" lastFinishedPulling="2026-05-26 11:24:49.473183032 +0000 UTC m=+21.664625198" observedRunningTime="2026-05-26 11:24:50.549425878 +0000 UTC m=+22.740868065" watchObservedRunningTime="2026-05-26 11:24:50.570567806 +0000 UTC m=+22.762009989"
May 26 11:24:50.570868 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:50.570794    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-njvjv" podStartSLOduration=4.112628226 podStartE2EDuration="22.570769652s" podCreationTimestamp="2026-05-26 11:24:28 +0000 UTC" firstStartedPulling="2026-05-26 11:24:31.017987815 +0000 UTC m=+3.209429980" lastFinishedPulling="2026-05-26 11:24:49.476129242 +0000 UTC m=+21.667571406" observedRunningTime="2026-05-26 11:24:50.570705933 +0000 UTC m=+22.762148122" watchObservedRunningTime="2026-05-26 11:24:50.570769652 +0000 UTC m=+22.762211838"
May 26 11:24:50.586078 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:50.586041    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-58x22" podStartSLOduration=8.901498675 podStartE2EDuration="22.586028545s" podCreationTimestamp="2026-05-26 11:24:28 +0000 UTC" firstStartedPulling="2026-05-26 11:24:31.041426768 +0000 UTC m=+3.232868932" lastFinishedPulling="2026-05-26 11:24:44.72595662 +0000 UTC m=+16.917398802" observedRunningTime="2026-05-26 11:24:50.586019808 +0000 UTC m=+22.777461995" watchObservedRunningTime="2026-05-26 11:24:50.586028545 +0000 UTC m=+22.777470732"
May 26 11:24:50.789894 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:50.789868    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 26 11:24:51.369894 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:51.369861    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-tlx86"]
May 26 11:24:51.383840 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:51.383811    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tlx86"
May 26 11:24:51.386951 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:51.386922    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 26 11:24:51.386951 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:51.386948    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vzp6h\""
May 26 11:24:51.387144 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:51.386948    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 26 11:24:51.388589 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:51.388454    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-26T11:24:50.789886686Z","UUID":"b96a30c3-8f62-4826-901e-63b364f0abbf","Handler":null,"Name":"","Endpoint":""}
May 26 11:24:51.391642 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:51.391626    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 26 11:24:51.391746 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:51.391652    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 26 11:24:51.424166 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:51.424137    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbxbl"
May 26 11:24:51.424342 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:51.424252    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbxbl" podUID="8528f6a0-0b8d-47f8-a20c-715dd15d1076"
May 26 11:24:51.492288 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:51.492253    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/463206a2-8ca0-4458-ba0b-6c3e6f59b837-tmp-dir\") pod \"node-resolver-tlx86\" (UID: \"463206a2-8ca0-4458-ba0b-6c3e6f59b837\") " pod="openshift-dns/node-resolver-tlx86"
May 26 11:24:51.492288 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:51.492300    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldkrk\" (UniqueName: \"kubernetes.io/projected/463206a2-8ca0-4458-ba0b-6c3e6f59b837-kube-api-access-ldkrk\") pod \"node-resolver-tlx86\" (UID: \"463206a2-8ca0-4458-ba0b-6c3e6f59b837\") " pod="openshift-dns/node-resolver-tlx86"
May 26 11:24:51.492883 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:51.492375    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/463206a2-8ca0-4458-ba0b-6c3e6f59b837-hosts-file\") pod \"node-resolver-tlx86\" (UID: \"463206a2-8ca0-4458-ba0b-6c3e6f59b837\") " pod="openshift-dns/node-resolver-tlx86"
May 26 11:24:51.531142 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:51.530526    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ldk8g" event={"ID":"d316eade-d057-4616-b829-4c09f6ea6d16","Type":"ContainerStarted","Data":"5c1ce0a93305e29df7a9842386c073e0f754edb769cbaa1bf1147676cc85498b"}
May 26 11:24:51.533484 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:51.532866    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmrml" event={"ID":"e991c96a-3b5f-486a-b2a9-6bfaf4130356","Type":"ContainerStarted","Data":"3c6c5ed5078b9af1240a4ee2f0010d12343a7b3829f862e03725215e5298018d"}
May 26 11:24:51.593545 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:51.593507    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/463206a2-8ca0-4458-ba0b-6c3e6f59b837-tmp-dir\") pod \"node-resolver-tlx86\" (UID: \"463206a2-8ca0-4458-ba0b-6c3e6f59b837\") " pod="openshift-dns/node-resolver-tlx86"
May 26 11:24:51.593545 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:51.593551    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ldkrk\" (UniqueName: \"kubernetes.io/projected/463206a2-8ca0-4458-ba0b-6c3e6f59b837-kube-api-access-ldkrk\") pod \"node-resolver-tlx86\" (UID: \"463206a2-8ca0-4458-ba0b-6c3e6f59b837\") " pod="openshift-dns/node-resolver-tlx86"
May 26 11:24:51.593734 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:51.593630    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/463206a2-8ca0-4458-ba0b-6c3e6f59b837-hosts-file\") pod \"node-resolver-tlx86\" (UID: \"463206a2-8ca0-4458-ba0b-6c3e6f59b837\") " pod="openshift-dns/node-resolver-tlx86"
May 26 11:24:51.593734 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:51.593701    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/463206a2-8ca0-4458-ba0b-6c3e6f59b837-hosts-file\") pod \"node-resolver-tlx86\" (UID: \"463206a2-8ca0-4458-ba0b-6c3e6f59b837\") " pod="openshift-dns/node-resolver-tlx86"
May 26 11:24:51.594426 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:51.594219    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/463206a2-8ca0-4458-ba0b-6c3e6f59b837-tmp-dir\") pod \"node-resolver-tlx86\" (UID: \"463206a2-8ca0-4458-ba0b-6c3e6f59b837\") " pod="openshift-dns/node-resolver-tlx86"
May 26 11:24:51.605724 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:51.605699    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldkrk\" (UniqueName: \"kubernetes.io/projected/463206a2-8ca0-4458-ba0b-6c3e6f59b837-kube-api-access-ldkrk\") pod \"node-resolver-tlx86\" (UID: \"463206a2-8ca0-4458-ba0b-6c3e6f59b837\") " pod="openshift-dns/node-resolver-tlx86"
May 26 11:24:51.619427 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:51.619397    2563 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xtkmg"
May 26 11:24:51.620178 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:51.620125    2563 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xtkmg"
May 26 11:24:51.639911 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:51.639861    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ldk8g" podStartSLOduration=5.205354955 podStartE2EDuration="23.63984097s" podCreationTimestamp="2026-05-26 11:24:28 +0000 UTC" firstStartedPulling="2026-05-26 11:24:31.041427904 +0000 UTC m=+3.232870088" lastFinishedPulling="2026-05-26 11:24:49.475913937 +0000 UTC m=+21.667356103" observedRunningTime="2026-05-26 11:24:51.548016456 +0000 UTC m=+23.739458647" watchObservedRunningTime="2026-05-26 11:24:51.63984097 +0000 UTC m=+23.831283157"
May 26 11:24:51.694976 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:51.694946    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tlx86"
May 26 11:24:51.764162 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:24:51.764132    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod463206a2_8ca0_4458_ba0b_6c3e6f59b837.slice/crio-0777a320b2d480a033d2f1b7f140333c649491a16581ad6d98bd849019956076 WatchSource:0}: Error finding container 0777a320b2d480a033d2f1b7f140333c649491a16581ad6d98bd849019956076: Status 404 returned error can't find the container with id 0777a320b2d480a033d2f1b7f140333c649491a16581ad6d98bd849019956076
May 26 11:24:52.425096 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:52.424793    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp9l8"
May 26 11:24:52.425282 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:52.425166    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pp9l8" podUID="9d573a77-5520-489f-99cb-86419b23cd19"
May 26 11:24:52.537536 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:52.537479    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmrml" event={"ID":"e991c96a-3b5f-486a-b2a9-6bfaf4130356","Type":"ContainerStarted","Data":"4ab35a6ffe56f1906e346e31e0bb07d7abab7a1077456f1f490fbdf99081de65"}
May 26 11:24:52.538946 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:52.538923    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tlx86" event={"ID":"463206a2-8ca0-4458-ba0b-6c3e6f59b837","Type":"ContainerStarted","Data":"c349a5e446d3c7f61ba2a648d74399b16268889246ba69eec836f9dec322050a"}
May 26 11:24:52.539082 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:52.538975    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tlx86" event={"ID":"463206a2-8ca0-4458-ba0b-6c3e6f59b837","Type":"ContainerStarted","Data":"0777a320b2d480a033d2f1b7f140333c649491a16581ad6d98bd849019956076"}
May 26 11:24:52.542485 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:52.542453    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw" event={"ID":"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2","Type":"ContainerStarted","Data":"902bb4731b26dc0aebc72e99c0e96fa20b1bfc8ed065f3633b7098b63be5677f"}
May 26 11:24:52.542848 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:52.542824    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xtkmg"
May 26 11:24:52.543212 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:52.543190    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xtkmg"
May 26 11:24:52.571478 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:52.571431    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-tlx86" podStartSLOduration=1.5714176370000001 podStartE2EDuration="1.571417637s" podCreationTimestamp="2026-05-26 11:24:51 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:24:52.571041037 +0000 UTC m=+24.762483225" watchObservedRunningTime="2026-05-26 11:24:52.571417637 +0000 UTC m=+24.762859823"
May 26 11:24:52.571618 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:52.571546    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmrml" podStartSLOduration=3.83236212 podStartE2EDuration="24.571539226s" podCreationTimestamp="2026-05-26 11:24:28 +0000 UTC" firstStartedPulling="2026-05-26 11:24:31.020644026 +0000 UTC m=+3.212086194" lastFinishedPulling="2026-05-26 11:24:51.759821131 +0000 UTC m=+23.951263300" observedRunningTime="2026-05-26 11:24:52.556553385 +0000 UTC m=+24.747995572" watchObservedRunningTime="2026-05-26 11:24:52.571539226 +0000 UTC m=+24.762981413"
May 26 11:24:53.425202 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:53.425164    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbxbl"
May 26 11:24:53.425385 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:53.425294    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbxbl" podUID="8528f6a0-0b8d-47f8-a20c-715dd15d1076"
May 26 11:24:54.424929 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:54.424891    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp9l8"
May 26 11:24:54.425663 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:54.425037    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pp9l8" podUID="9d573a77-5520-489f-99cb-86419b23cd19"
May 26 11:24:55.424662 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:55.424457    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbxbl"
May 26 11:24:55.424853 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:55.424746    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbxbl" podUID="8528f6a0-0b8d-47f8-a20c-715dd15d1076"
May 26 11:24:55.550993 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:55.550951    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw" event={"ID":"8526b1f6-8087-4c6e-8cd0-a0188c5db3a2","Type":"ContainerStarted","Data":"8bf5abe995765410afaaba68dbb3ec0ed880e7dd6359e73a60ea0cd4086fa33d"}
May 26 11:24:55.551527 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:55.551300    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:55.551527 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:55.551330    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:55.552493 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:55.552470    2563 generic.go:358] "Generic (PLEG): container finished" podID="b642b5f8-5576-42de-807d-6da907ae897d" containerID="4b4fb2fd20c239d611bb8d4cbdc7ba258855382353c6be1d27d63b489f646a01" exitCode=0
May 26 11:24:55.552622 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:55.552508    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kz76h" event={"ID":"b642b5f8-5576-42de-807d-6da907ae897d","Type":"ContainerDied","Data":"4b4fb2fd20c239d611bb8d4cbdc7ba258855382353c6be1d27d63b489f646a01"}
May 26 11:24:55.567558 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:55.567532    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:55.582118 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:55.582077    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw" podStartSLOduration=9.017285255 podStartE2EDuration="27.582066224s" podCreationTimestamp="2026-05-26 11:24:28 +0000 UTC" firstStartedPulling="2026-05-26 11:24:31.046167377 +0000 UTC m=+3.237609543" lastFinishedPulling="2026-05-26 11:24:49.610948346 +0000 UTC m=+21.802390512" observedRunningTime="2026-05-26 11:24:55.576714416 +0000 UTC m=+27.768156603" watchObservedRunningTime="2026-05-26 11:24:55.582066224 +0000 UTC m=+27.773508410"
May 26 11:24:56.425241 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:56.425210    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp9l8"
May 26 11:24:56.425407 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:56.425357    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pp9l8" podUID="9d573a77-5520-489f-99cb-86419b23cd19"
May 26 11:24:56.554701 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:56.554673    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:56.573166 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:56.573127    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:24:56.574394 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:56.574371    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pp9l8"]
May 26 11:24:56.574520 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:56.574507    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp9l8"
May 26 11:24:56.574659 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:56.574633    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pp9l8" podUID="9d573a77-5520-489f-99cb-86419b23cd19"
May 26 11:24:56.577332 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:56.577311    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cbxbl"]
May 26 11:24:56.577419 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:56.577378    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbxbl"
May 26 11:24:56.577476 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:56.577445    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbxbl" podUID="8528f6a0-0b8d-47f8-a20c-715dd15d1076"
May 26 11:24:57.558261 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:57.558039    2563 generic.go:358] "Generic (PLEG): container finished" podID="b642b5f8-5576-42de-807d-6da907ae897d" containerID="72d52b6c89ed803cda972aa1ded02fcba53b7ff04b9640b4e2ecf0b794045544" exitCode=0
May 26 11:24:57.558635 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:57.558119    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kz76h" event={"ID":"b642b5f8-5576-42de-807d-6da907ae897d","Type":"ContainerDied","Data":"72d52b6c89ed803cda972aa1ded02fcba53b7ff04b9640b4e2ecf0b794045544"}
May 26 11:24:58.425038 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:58.425011    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp9l8"
May 26 11:24:58.425178 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:58.425118    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pp9l8" podUID="9d573a77-5520-489f-99cb-86419b23cd19"
May 26 11:24:58.425178 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:58.425168    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbxbl"
May 26 11:24:58.425267 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:24:58.425250    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbxbl" podUID="8528f6a0-0b8d-47f8-a20c-715dd15d1076"
May 26 11:24:59.563667 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:59.563634    2563 generic.go:358] "Generic (PLEG): container finished" podID="b642b5f8-5576-42de-807d-6da907ae897d" containerID="63b7263c93d8f5348e45401011a4286ed8545c633a28abe3e668d02e32e55b2e" exitCode=0
May 26 11:24:59.564317 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:24:59.563678    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kz76h" event={"ID":"b642b5f8-5576-42de-807d-6da907ae897d","Type":"ContainerDied","Data":"63b7263c93d8f5348e45401011a4286ed8545c633a28abe3e668d02e32e55b2e"}
May 26 11:25:00.424851 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:00.424821    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp9l8"
May 26 11:25:00.425016 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:25:00.424936    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pp9l8" podUID="9d573a77-5520-489f-99cb-86419b23cd19"
May 26 11:25:00.425319 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:00.425303    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbxbl"
May 26 11:25:00.425396 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:25:00.425377    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbxbl" podUID="8528f6a0-0b8d-47f8-a20c-715dd15d1076"
May 26 11:25:02.072717 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.072658    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d573a77-5520-489f-99cb-86419b23cd19-metrics-certs\") pod \"network-metrics-daemon-pp9l8\" (UID: \"9d573a77-5520-489f-99cb-86419b23cd19\") " pod="openshift-multus/network-metrics-daemon-pp9l8"
May 26 11:25:02.073294 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:25:02.072830    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:25:02.073294 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:25:02.072897    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d573a77-5520-489f-99cb-86419b23cd19-metrics-certs podName:9d573a77-5520-489f-99cb-86419b23cd19 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:34.072876839 +0000 UTC m=+66.264319019 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d573a77-5520-489f-99cb-86419b23cd19-metrics-certs") pod "network-metrics-daemon-pp9l8" (UID: "9d573a77-5520-489f-99cb-86419b23cd19") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:25:02.121698 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.121668    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-97.ec2.internal" event="NodeReady"
May 26 11:25:02.121900 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.121842    2563 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 26 11:25:02.173428 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.173390    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bt6x9\" (UniqueName: \"kubernetes.io/projected/8528f6a0-0b8d-47f8-a20c-715dd15d1076-kube-api-access-bt6x9\") pod \"network-check-target-cbxbl\" (UID: \"8528f6a0-0b8d-47f8-a20c-715dd15d1076\") " pod="openshift-network-diagnostics/network-check-target-cbxbl"
May 26 11:25:02.173614 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:25:02.173592    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:25:02.173659 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:25:02.173622    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:25:02.173659 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:25:02.173632    2563 projected.go:194] Error preparing data for projected volume kube-api-access-bt6x9 for pod openshift-network-diagnostics/network-check-target-cbxbl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:25:02.173720 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:25:02.173707    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8528f6a0-0b8d-47f8-a20c-715dd15d1076-kube-api-access-bt6x9 podName:8528f6a0-0b8d-47f8-a20c-715dd15d1076 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:34.173687478 +0000 UTC m=+66.365129645 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bt6x9" (UniqueName: "kubernetes.io/projected/8528f6a0-0b8d-47f8-a20c-715dd15d1076-kube-api-access-bt6x9") pod "network-check-target-cbxbl" (UID: "8528f6a0-0b8d-47f8-a20c-715dd15d1076") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:25:02.203913 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.203879    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fn6r5"]
May 26 11:25:02.207184 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.207159    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fn6r5"
May 26 11:25:02.213589 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.212984    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-46vjq\""
May 26 11:25:02.213589 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.213220    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 26 11:25:02.213589 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.213422    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 26 11:25:02.214068 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.214042    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-d72ts"]
May 26 11:25:02.217639 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.217622    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d72ts"
May 26 11:25:02.221540 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.221518    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 26 11:25:02.221887 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.221872    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qmmns\""
May 26 11:25:02.222118 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.222096    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 26 11:25:02.225307 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.225278    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 26 11:25:02.231952 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.231932    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fn6r5"]
May 26 11:25:02.265520 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.265435    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d72ts"]
May 26 11:25:02.273974 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.273948    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0704d5eb-7568-4943-ad52-70c0f8814792-config-volume\") pod \"dns-default-fn6r5\" (UID: \"0704d5eb-7568-4943-ad52-70c0f8814792\") " pod="openshift-dns/dns-default-fn6r5"
May 26 11:25:02.274127 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.273986    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9r7n\" (UniqueName: \"kubernetes.io/projected/0704d5eb-7568-4943-ad52-70c0f8814792-kube-api-access-k9r7n\") pod \"dns-default-fn6r5\" (UID: \"0704d5eb-7568-4943-ad52-70c0f8814792\") " pod="openshift-dns/dns-default-fn6r5"
May 26 11:25:02.274127 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.274017    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0704d5eb-7568-4943-ad52-70c0f8814792-tmp-dir\") pod \"dns-default-fn6r5\" (UID: \"0704d5eb-7568-4943-ad52-70c0f8814792\") " pod="openshift-dns/dns-default-fn6r5"
May 26 11:25:02.274127 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.274055    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0704d5eb-7568-4943-ad52-70c0f8814792-metrics-tls\") pod \"dns-default-fn6r5\" (UID: \"0704d5eb-7568-4943-ad52-70c0f8814792\") " pod="openshift-dns/dns-default-fn6r5"
May 26 11:25:02.375136 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.375102    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0704d5eb-7568-4943-ad52-70c0f8814792-config-volume\") pod \"dns-default-fn6r5\" (UID: \"0704d5eb-7568-4943-ad52-70c0f8814792\") " pod="openshift-dns/dns-default-fn6r5"
May 26 11:25:02.375326 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.375146    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-k9r7n\" (UniqueName: \"kubernetes.io/projected/0704d5eb-7568-4943-ad52-70c0f8814792-kube-api-access-k9r7n\") pod \"dns-default-fn6r5\" (UID: \"0704d5eb-7568-4943-ad52-70c0f8814792\") " pod="openshift-dns/dns-default-fn6r5"
May 26 11:25:02.375326 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.375179    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0704d5eb-7568-4943-ad52-70c0f8814792-tmp-dir\") pod \"dns-default-fn6r5\" (UID: \"0704d5eb-7568-4943-ad52-70c0f8814792\") " pod="openshift-dns/dns-default-fn6r5"
May 26 11:25:02.375420 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.375368    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/13698e7d-67cc-4e22-afdd-70f4b08b2819-cert\") pod \"ingress-canary-d72ts\" (UID: \"13698e7d-67cc-4e22-afdd-70f4b08b2819\") " pod="openshift-ingress-canary/ingress-canary-d72ts"
May 26 11:25:02.375476 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.375442    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjk8x\" (UniqueName: \"kubernetes.io/projected/13698e7d-67cc-4e22-afdd-70f4b08b2819-kube-api-access-bjk8x\") pod \"ingress-canary-d72ts\" (UID: \"13698e7d-67cc-4e22-afdd-70f4b08b2819\") " pod="openshift-ingress-canary/ingress-canary-d72ts"
May 26 11:25:02.375476 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.375472    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0704d5eb-7568-4943-ad52-70c0f8814792-metrics-tls\") pod \"dns-default-fn6r5\" (UID: \"0704d5eb-7568-4943-ad52-70c0f8814792\") " pod="openshift-dns/dns-default-fn6r5"
May 26 11:25:02.375571 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.375518    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0704d5eb-7568-4943-ad52-70c0f8814792-tmp-dir\") pod \"dns-default-fn6r5\" (UID: \"0704d5eb-7568-4943-ad52-70c0f8814792\") " pod="openshift-dns/dns-default-fn6r5"
May 26 11:25:02.375620 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:25:02.375569    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:25:02.375662 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:25:02.375626    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0704d5eb-7568-4943-ad52-70c0f8814792-metrics-tls podName:0704d5eb-7568-4943-ad52-70c0f8814792 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:02.87561226 +0000 UTC m=+35.067054439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0704d5eb-7568-4943-ad52-70c0f8814792-metrics-tls") pod "dns-default-fn6r5" (UID: "0704d5eb-7568-4943-ad52-70c0f8814792") : secret "dns-default-metrics-tls" not found
May 26 11:25:02.375910 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.375892    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0704d5eb-7568-4943-ad52-70c0f8814792-config-volume\") pod \"dns-default-fn6r5\" (UID: \"0704d5eb-7568-4943-ad52-70c0f8814792\") " pod="openshift-dns/dns-default-fn6r5"
May 26 11:25:02.391224 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.391199    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9r7n\" (UniqueName: \"kubernetes.io/projected/0704d5eb-7568-4943-ad52-70c0f8814792-kube-api-access-k9r7n\") pod \"dns-default-fn6r5\" (UID: \"0704d5eb-7568-4943-ad52-70c0f8814792\") " pod="openshift-dns/dns-default-fn6r5"
May 26 11:25:02.424448 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.424417    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp9l8"
May 26 11:25:02.424619 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.424417    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbxbl"
May 26 11:25:02.428013 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.427988    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xvpvn\""
May 26 11:25:02.428013 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.428001    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zzwv5\""
May 26 11:25:02.428215 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.428102    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 26 11:25:02.428391 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.428373    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 26 11:25:02.428487 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.428377    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 26 11:25:02.476359 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.476326    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/13698e7d-67cc-4e22-afdd-70f4b08b2819-cert\") pod \"ingress-canary-d72ts\" (UID: \"13698e7d-67cc-4e22-afdd-70f4b08b2819\") " pod="openshift-ingress-canary/ingress-canary-d72ts"
May 26 11:25:02.476539 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.476396    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bjk8x\" (UniqueName: \"kubernetes.io/projected/13698e7d-67cc-4e22-afdd-70f4b08b2819-kube-api-access-bjk8x\") pod \"ingress-canary-d72ts\" (UID: \"13698e7d-67cc-4e22-afdd-70f4b08b2819\") " pod="openshift-ingress-canary/ingress-canary-d72ts"
May 26 11:25:02.476539 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:25:02.476501    2563 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:25:02.476620 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:25:02.476575    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13698e7d-67cc-4e22-afdd-70f4b08b2819-cert podName:13698e7d-67cc-4e22-afdd-70f4b08b2819 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:02.976550733 +0000 UTC m=+35.167992904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/13698e7d-67cc-4e22-afdd-70f4b08b2819-cert") pod "ingress-canary-d72ts" (UID: "13698e7d-67cc-4e22-afdd-70f4b08b2819") : secret "canary-serving-cert" not found
May 26 11:25:02.486767 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.486596    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjk8x\" (UniqueName: \"kubernetes.io/projected/13698e7d-67cc-4e22-afdd-70f4b08b2819-kube-api-access-bjk8x\") pod \"ingress-canary-d72ts\" (UID: \"13698e7d-67cc-4e22-afdd-70f4b08b2819\") " pod="openshift-ingress-canary/ingress-canary-d72ts"
May 26 11:25:02.880134 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.880098    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0704d5eb-7568-4943-ad52-70c0f8814792-metrics-tls\") pod \"dns-default-fn6r5\" (UID: \"0704d5eb-7568-4943-ad52-70c0f8814792\") " pod="openshift-dns/dns-default-fn6r5"
May 26 11:25:02.880345 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:25:02.880270    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:25:02.880405 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:25:02.880348    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0704d5eb-7568-4943-ad52-70c0f8814792-metrics-tls podName:0704d5eb-7568-4943-ad52-70c0f8814792 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:03.880327039 +0000 UTC m=+36.071769205 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0704d5eb-7568-4943-ad52-70c0f8814792-metrics-tls") pod "dns-default-fn6r5" (UID: "0704d5eb-7568-4943-ad52-70c0f8814792") : secret "dns-default-metrics-tls" not found
May 26 11:25:02.980762 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:02.980712    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/13698e7d-67cc-4e22-afdd-70f4b08b2819-cert\") pod \"ingress-canary-d72ts\" (UID: \"13698e7d-67cc-4e22-afdd-70f4b08b2819\") " pod="openshift-ingress-canary/ingress-canary-d72ts"
May 26 11:25:02.980939 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:25:02.980895    2563 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:25:02.980991 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:25:02.980976    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13698e7d-67cc-4e22-afdd-70f4b08b2819-cert podName:13698e7d-67cc-4e22-afdd-70f4b08b2819 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:03.980955822 +0000 UTC m=+36.172398004 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/13698e7d-67cc-4e22-afdd-70f4b08b2819-cert") pod "ingress-canary-d72ts" (UID: "13698e7d-67cc-4e22-afdd-70f4b08b2819") : secret "canary-serving-cert" not found
May 26 11:25:03.300190 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:03.300068    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-6k5q8"]
May 26 11:25:03.304529 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:03.304509    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-6k5q8"
May 26 11:25:03.307932 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:03.307907    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
May 26 11:25:03.308048 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:03.308001    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tl626\""
May 26 11:25:03.309479 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:03.309462    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
May 26 11:25:03.313439 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:03.313416    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-6k5q8"]
May 26 11:25:03.384107 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:03.384075    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4kdt\" (UniqueName: \"kubernetes.io/projected/47a7e428-2e75-4d78-b7f0-cf4edd466758-kube-api-access-f4kdt\") pod \"migrator-b985f5d89-6k5q8\" (UID: \"47a7e428-2e75-4d78-b7f0-cf4edd466758\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-6k5q8"
May 26 11:25:03.484658 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:03.484620    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f4kdt\" (UniqueName: \"kubernetes.io/projected/47a7e428-2e75-4d78-b7f0-cf4edd466758-kube-api-access-f4kdt\") pod \"migrator-b985f5d89-6k5q8\" (UID: \"47a7e428-2e75-4d78-b7f0-cf4edd466758\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-6k5q8"
May 26 11:25:03.496552 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:03.496501    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4kdt\" (UniqueName: \"kubernetes.io/projected/47a7e428-2e75-4d78-b7f0-cf4edd466758-kube-api-access-f4kdt\") pod \"migrator-b985f5d89-6k5q8\" (UID: \"47a7e428-2e75-4d78-b7f0-cf4edd466758\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-6k5q8"
May 26 11:25:03.615990 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:03.615952    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-6k5q8"
May 26 11:25:03.888078 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:03.887988    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0704d5eb-7568-4943-ad52-70c0f8814792-metrics-tls\") pod \"dns-default-fn6r5\" (UID: \"0704d5eb-7568-4943-ad52-70c0f8814792\") " pod="openshift-dns/dns-default-fn6r5"
May 26 11:25:03.888209 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:25:03.888155    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:25:03.888255 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:25:03.888230    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0704d5eb-7568-4943-ad52-70c0f8814792-metrics-tls podName:0704d5eb-7568-4943-ad52-70c0f8814792 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:05.888207353 +0000 UTC m=+38.079649531 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0704d5eb-7568-4943-ad52-70c0f8814792-metrics-tls") pod "dns-default-fn6r5" (UID: "0704d5eb-7568-4943-ad52-70c0f8814792") : secret "dns-default-metrics-tls" not found
May 26 11:25:03.988609 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:03.988561    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/13698e7d-67cc-4e22-afdd-70f4b08b2819-cert\") pod \"ingress-canary-d72ts\" (UID: \"13698e7d-67cc-4e22-afdd-70f4b08b2819\") " pod="openshift-ingress-canary/ingress-canary-d72ts"
May 26 11:25:03.988824 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:25:03.988741    2563 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:25:03.988894 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:25:03.988833    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13698e7d-67cc-4e22-afdd-70f4b08b2819-cert podName:13698e7d-67cc-4e22-afdd-70f4b08b2819 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:05.988811326 +0000 UTC m=+38.180253507 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/13698e7d-67cc-4e22-afdd-70f4b08b2819-cert") pod "ingress-canary-d72ts" (UID: "13698e7d-67cc-4e22-afdd-70f4b08b2819") : secret "canary-serving-cert" not found
May 26 11:25:05.191558 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:05.191524    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-6k5q8"]
May 26 11:25:05.198341 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:25:05.198308    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47a7e428_2e75_4d78_b7f0_cf4edd466758.slice/crio-09b53e0e7bb5ff1ef02879865271bdea3c33d5635c1c620aaeb588cca06f051d WatchSource:0}: Error finding container 09b53e0e7bb5ff1ef02879865271bdea3c33d5635c1c620aaeb588cca06f051d: Status 404 returned error can't find the container with id 09b53e0e7bb5ff1ef02879865271bdea3c33d5635c1c620aaeb588cca06f051d
May 26 11:25:05.577267 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:05.577221    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-6k5q8" event={"ID":"47a7e428-2e75-4d78-b7f0-cf4edd466758","Type":"ContainerStarted","Data":"09b53e0e7bb5ff1ef02879865271bdea3c33d5635c1c620aaeb588cca06f051d"}
May 26 11:25:05.583101 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:05.583065    2563 generic.go:358] "Generic (PLEG): container finished" podID="b642b5f8-5576-42de-807d-6da907ae897d" containerID="f0f09ce05156a425dfee69c81652dd7393ca9a77c51ed879f016e7f554413a11" exitCode=0
May 26 11:25:05.583439 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:05.583118    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kz76h" event={"ID":"b642b5f8-5576-42de-807d-6da907ae897d","Type":"ContainerDied","Data":"f0f09ce05156a425dfee69c81652dd7393ca9a77c51ed879f016e7f554413a11"}
May 26 11:25:05.904988 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:05.904709    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0704d5eb-7568-4943-ad52-70c0f8814792-metrics-tls\") pod \"dns-default-fn6r5\" (UID: \"0704d5eb-7568-4943-ad52-70c0f8814792\") " pod="openshift-dns/dns-default-fn6r5"
May 26 11:25:05.905130 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:25:05.904876    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:25:05.905130 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:25:05.905074    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0704d5eb-7568-4943-ad52-70c0f8814792-metrics-tls podName:0704d5eb-7568-4943-ad52-70c0f8814792 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:09.905052893 +0000 UTC m=+42.096495058 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0704d5eb-7568-4943-ad52-70c0f8814792-metrics-tls") pod "dns-default-fn6r5" (UID: "0704d5eb-7568-4943-ad52-70c0f8814792") : secret "dns-default-metrics-tls" not found
May 26 11:25:06.006134 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:06.006095    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/13698e7d-67cc-4e22-afdd-70f4b08b2819-cert\") pod \"ingress-canary-d72ts\" (UID: \"13698e7d-67cc-4e22-afdd-70f4b08b2819\") " pod="openshift-ingress-canary/ingress-canary-d72ts"
May 26 11:25:06.006309 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:25:06.006220    2563 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:25:06.006309 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:25:06.006273    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13698e7d-67cc-4e22-afdd-70f4b08b2819-cert podName:13698e7d-67cc-4e22-afdd-70f4b08b2819 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:10.006258618 +0000 UTC m=+42.197700796 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/13698e7d-67cc-4e22-afdd-70f4b08b2819-cert") pod "ingress-canary-d72ts" (UID: "13698e7d-67cc-4e22-afdd-70f4b08b2819") : secret "canary-serving-cert" not found
May 26 11:25:06.442546 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:06.442506    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tlx86_463206a2-8ca0-4458-ba0b-6c3e6f59b837/dns-node-resolver/0.log"
May 26 11:25:06.588014 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:06.587986    2563 generic.go:358] "Generic (PLEG): container finished" podID="b642b5f8-5576-42de-807d-6da907ae897d" containerID="fb6de7de31e23527e6283b5481b6e9505f33a97119801766066e3d068cf2919e" exitCode=0
May 26 11:25:06.588167 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:06.588049    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kz76h" event={"ID":"b642b5f8-5576-42de-807d-6da907ae897d","Type":"ContainerDied","Data":"fb6de7de31e23527e6283b5481b6e9505f33a97119801766066e3d068cf2919e"}
May 26 11:25:07.043362 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:07.043328    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-58x22_f1ab37a2-3fa3-4941-8cc3-0149f5478b36/node-ca/0.log"
May 26 11:25:07.592841 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:07.592804    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kz76h" event={"ID":"b642b5f8-5576-42de-807d-6da907ae897d","Type":"ContainerStarted","Data":"82483cf75497154b9158553031c5e7bae1bda4a06e70d02cc4ea8330800ad08f"}
May 26 11:25:07.594270 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:07.594243    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-6k5q8" event={"ID":"47a7e428-2e75-4d78-b7f0-cf4edd466758","Type":"ContainerStarted","Data":"2e2f838f3d8f7979e586166add36fcabb5dbc3d891630cf4459e7198a2f2ac12"}
May 26 11:25:07.594355 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:07.594280    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-6k5q8" event={"ID":"47a7e428-2e75-4d78-b7f0-cf4edd466758","Type":"ContainerStarted","Data":"4cf43b008553ee4df49633cc3e65b68fe2df7feaa32f7e35fe3e4f06211ebbfd"}
May 26 11:25:07.644829 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:07.644757    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kz76h" podStartSLOduration=5.545869096 podStartE2EDuration="39.64474134s" podCreationTimestamp="2026-05-26 11:24:28 +0000 UTC" firstStartedPulling="2026-05-26 11:24:31.022104924 +0000 UTC m=+3.213547101" lastFinishedPulling="2026-05-26 11:25:05.120977177 +0000 UTC m=+37.312419345" observedRunningTime="2026-05-26 11:25:07.625668153 +0000 UTC m=+39.817110339" watchObservedRunningTime="2026-05-26 11:25:07.64474134 +0000 UTC m=+39.836183527"
May 26 11:25:07.645013 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:07.644948    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-6k5q8" podStartSLOduration=3.06495839 podStartE2EDuration="4.644943472s" podCreationTimestamp="2026-05-26 11:25:03 +0000 UTC" firstStartedPulling="2026-05-26 11:25:05.200202707 +0000 UTC m=+37.391644878" lastFinishedPulling="2026-05-26 11:25:06.780187781 +0000 UTC m=+38.971629960" observedRunningTime="2026-05-26 11:25:07.644730181 +0000 UTC m=+39.836172369" watchObservedRunningTime="2026-05-26 11:25:07.644943472 +0000 UTC m=+39.836385659"
May 26 11:25:08.442898 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:08.442872    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-6k5q8_47a7e428-2e75-4d78-b7f0-cf4edd466758/migrator/0.log"
May 26 11:25:08.641952 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:08.641924    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-6k5q8_47a7e428-2e75-4d78-b7f0-cf4edd466758/graceful-termination/0.log"
May 26 11:25:09.933840 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:09.933789    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0704d5eb-7568-4943-ad52-70c0f8814792-metrics-tls\") pod \"dns-default-fn6r5\" (UID: \"0704d5eb-7568-4943-ad52-70c0f8814792\") " pod="openshift-dns/dns-default-fn6r5"
May 26 11:25:09.934223 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:25:09.933920    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:25:09.934223 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:25:09.933987    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0704d5eb-7568-4943-ad52-70c0f8814792-metrics-tls podName:0704d5eb-7568-4943-ad52-70c0f8814792 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:17.933969661 +0000 UTC m=+50.125411831 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0704d5eb-7568-4943-ad52-70c0f8814792-metrics-tls") pod "dns-default-fn6r5" (UID: "0704d5eb-7568-4943-ad52-70c0f8814792") : secret "dns-default-metrics-tls" not found
May 26 11:25:10.034651 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:10.034609    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/13698e7d-67cc-4e22-afdd-70f4b08b2819-cert\") pod \"ingress-canary-d72ts\" (UID: \"13698e7d-67cc-4e22-afdd-70f4b08b2819\") " pod="openshift-ingress-canary/ingress-canary-d72ts"
May 26 11:25:10.034853 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:25:10.034811    2563 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:25:10.034923 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:25:10.034890    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13698e7d-67cc-4e22-afdd-70f4b08b2819-cert podName:13698e7d-67cc-4e22-afdd-70f4b08b2819 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:18.034868808 +0000 UTC m=+50.226310980 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/13698e7d-67cc-4e22-afdd-70f4b08b2819-cert") pod "ingress-canary-d72ts" (UID: "13698e7d-67cc-4e22-afdd-70f4b08b2819") : secret "canary-serving-cert" not found
May 26 11:25:17.994035 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:17.993996    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0704d5eb-7568-4943-ad52-70c0f8814792-metrics-tls\") pod \"dns-default-fn6r5\" (UID: \"0704d5eb-7568-4943-ad52-70c0f8814792\") " pod="openshift-dns/dns-default-fn6r5"
May 26 11:25:17.998039 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:17.998003    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0704d5eb-7568-4943-ad52-70c0f8814792-metrics-tls\") pod \"dns-default-fn6r5\" (UID: \"0704d5eb-7568-4943-ad52-70c0f8814792\") " pod="openshift-dns/dns-default-fn6r5"
May 26 11:25:18.094927 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:18.094895    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/13698e7d-67cc-4e22-afdd-70f4b08b2819-cert\") pod \"ingress-canary-d72ts\" (UID: \"13698e7d-67cc-4e22-afdd-70f4b08b2819\") " pod="openshift-ingress-canary/ingress-canary-d72ts"
May 26 11:25:18.097419 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:18.097393    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/13698e7d-67cc-4e22-afdd-70f4b08b2819-cert\") pod \"ingress-canary-d72ts\" (UID: \"13698e7d-67cc-4e22-afdd-70f4b08b2819\") " pod="openshift-ingress-canary/ingress-canary-d72ts"
May 26 11:25:18.119504 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:18.119455    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fn6r5"
May 26 11:25:18.130241 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:18.130215    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d72ts"
May 26 11:25:18.279880 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:18.279849    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fn6r5"]
May 26 11:25:18.283858 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:25:18.283830    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0704d5eb_7568_4943_ad52_70c0f8814792.slice/crio-461cf8bb4b0cea6ee0d3e71250d27a6b8ecb601dcd8612e41974a0c3214ebcea WatchSource:0}: Error finding container 461cf8bb4b0cea6ee0d3e71250d27a6b8ecb601dcd8612e41974a0c3214ebcea: Status 404 returned error can't find the container with id 461cf8bb4b0cea6ee0d3e71250d27a6b8ecb601dcd8612e41974a0c3214ebcea
May 26 11:25:18.296501 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:18.296478    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d72ts"]
May 26 11:25:18.299580 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:25:18.299558    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13698e7d_67cc_4e22_afdd_70f4b08b2819.slice/crio-372ab122a19e0d77b6ef9d51bbc69860af6305781164a271179da68e00c8a88f WatchSource:0}: Error finding container 372ab122a19e0d77b6ef9d51bbc69860af6305781164a271179da68e00c8a88f: Status 404 returned error can't find the container with id 372ab122a19e0d77b6ef9d51bbc69860af6305781164a271179da68e00c8a88f
May 26 11:25:18.615894 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:18.615857    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d72ts" event={"ID":"13698e7d-67cc-4e22-afdd-70f4b08b2819","Type":"ContainerStarted","Data":"372ab122a19e0d77b6ef9d51bbc69860af6305781164a271179da68e00c8a88f"}
May 26 11:25:18.616837 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:18.616809    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fn6r5" event={"ID":"0704d5eb-7568-4943-ad52-70c0f8814792","Type":"ContainerStarted","Data":"461cf8bb4b0cea6ee0d3e71250d27a6b8ecb601dcd8612e41974a0c3214ebcea"}
May 26 11:25:21.623821 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:21.623767    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d72ts" event={"ID":"13698e7d-67cc-4e22-afdd-70f4b08b2819","Type":"ContainerStarted","Data":"fa3ec259d631c5aee38714923918f84429aef9087e4ff982ad25acb9575cd077"}
May 26 11:25:21.625368 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:21.625346    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fn6r5" event={"ID":"0704d5eb-7568-4943-ad52-70c0f8814792","Type":"ContainerStarted","Data":"fd5173fb1b5aca693057d9972ead6011ae04ebb0cd8b961258a1d4f7bd8b9789"}
May 26 11:25:21.625368 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:21.625371    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fn6r5" event={"ID":"0704d5eb-7568-4943-ad52-70c0f8814792","Type":"ContainerStarted","Data":"5db1eca543abaa64c9d5722dc6281db2c6fb189fe811f0348ee91e5bbcf908c1"}
May 26 11:25:21.625503 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:21.625487    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fn6r5"
May 26 11:25:21.643277 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:21.643237    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-d72ts" podStartSLOduration=17.203795493 podStartE2EDuration="19.643225s" podCreationTimestamp="2026-05-26 11:25:02 +0000 UTC" firstStartedPulling="2026-05-26 11:25:18.301377863 +0000 UTC m=+50.492820031" lastFinishedPulling="2026-05-26 11:25:20.740807374 +0000 UTC m=+52.932249538" observedRunningTime="2026-05-26 11:25:21.642813318 +0000 UTC m=+53.834255506" watchObservedRunningTime="2026-05-26 11:25:21.643225 +0000 UTC m=+53.834667197"
May 26 11:25:21.661457 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:21.661419    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-fn6r5" podStartSLOduration=17.208421729 podStartE2EDuration="19.661405739s" podCreationTimestamp="2026-05-26 11:25:02 +0000 UTC" firstStartedPulling="2026-05-26 11:25:18.285636143 +0000 UTC m=+50.477078312" lastFinishedPulling="2026-05-26 11:25:20.738620151 +0000 UTC m=+52.930062322" observedRunningTime="2026-05-26 11:25:21.660530261 +0000 UTC m=+53.851972449" watchObservedRunningTime="2026-05-26 11:25:21.661405739 +0000 UTC m=+53.852847927"
May 26 11:25:27.507089 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.507058    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7phq6"]
May 26 11:25:27.537183 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.537142    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7phq6"]
May 26 11:25:27.537337 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.537296    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7phq6"
May 26 11:25:27.541280 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.541253    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 26 11:25:27.541437 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.541342    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 26 11:25:27.542445 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.542397    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 26 11:25:27.542644 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.542577    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2mb7j\""
May 26 11:25:27.542700 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.542641    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 26 11:25:27.609439 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.609406    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c89b4489d-2rf2s"]
May 26 11:25:27.625833 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.625802    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c89b4489d-2rf2s"
May 26 11:25:27.629003 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.628977    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 26 11:25:27.629150 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.629066    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 26 11:25:27.629393 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.629363    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9tkg9\""
May 26 11:25:27.629514 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.629490    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c89b4489d-2rf2s"]
May 26 11:25:27.630420 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.630401    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 26 11:25:27.645976 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.645937    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 26 11:25:27.661796 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.661749    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7phq6\" (UID: \"e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4\") " pod="openshift-insights/insights-runtime-extractor-7phq6"
May 26 11:25:27.661970 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.661835    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j7vf\" (UniqueName: \"kubernetes.io/projected/e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4-kube-api-access-6j7vf\") pod \"insights-runtime-extractor-7phq6\" (UID: \"e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4\") " pod="openshift-insights/insights-runtime-extractor-7phq6"
May 26 11:25:27.661970 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.661870    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4-crio-socket\") pod \"insights-runtime-extractor-7phq6\" (UID: \"e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4\") " pod="openshift-insights/insights-runtime-extractor-7phq6"
May 26 11:25:27.661970 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.661898    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7phq6\" (UID: \"e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4\") " pod="openshift-insights/insights-runtime-extractor-7phq6"
May 26 11:25:27.662080 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.661992    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4-data-volume\") pod \"insights-runtime-extractor-7phq6\" (UID: \"e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4\") " pod="openshift-insights/insights-runtime-extractor-7phq6"
May 26 11:25:27.762697 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.762622    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3571472a-95af-4617-9cb4-ac2782af9f74-image-registry-private-configuration\") pod \"image-registry-7c89b4489d-2rf2s\" (UID: \"3571472a-95af-4617-9cb4-ac2782af9f74\") " pod="openshift-image-registry/image-registry-7c89b4489d-2rf2s"
May 26 11:25:27.762697 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.762658    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3571472a-95af-4617-9cb4-ac2782af9f74-trusted-ca\") pod \"image-registry-7c89b4489d-2rf2s\" (UID: \"3571472a-95af-4617-9cb4-ac2782af9f74\") " pod="openshift-image-registry/image-registry-7c89b4489d-2rf2s"
May 26 11:25:27.762697 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.762686    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6j7vf\" (UniqueName: \"kubernetes.io/projected/e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4-kube-api-access-6j7vf\") pod \"insights-runtime-extractor-7phq6\" (UID: \"e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4\") " pod="openshift-insights/insights-runtime-extractor-7phq6"
May 26 11:25:27.762977 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.762706    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3571472a-95af-4617-9cb4-ac2782af9f74-registry-certificates\") pod \"image-registry-7c89b4489d-2rf2s\" (UID: \"3571472a-95af-4617-9cb4-ac2782af9f74\") " pod="openshift-image-registry/image-registry-7c89b4489d-2rf2s"
May 26 11:25:27.762977 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.762722    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfqwt\" (UniqueName: \"kubernetes.io/projected/3571472a-95af-4617-9cb4-ac2782af9f74-kube-api-access-kfqwt\") pod \"image-registry-7c89b4489d-2rf2s\" (UID: \"3571472a-95af-4617-9cb4-ac2782af9f74\") " pod="openshift-image-registry/image-registry-7c89b4489d-2rf2s"
May 26 11:25:27.762977 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.762744    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4-crio-socket\") pod \"insights-runtime-extractor-7phq6\" (UID: \"e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4\") " pod="openshift-insights/insights-runtime-extractor-7phq6"
May 26 11:25:27.762977 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.762764    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7phq6\" (UID: \"e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4\") " pod="openshift-insights/insights-runtime-extractor-7phq6"
May 26 11:25:27.762977 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.762810    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3571472a-95af-4617-9cb4-ac2782af9f74-bound-sa-token\") pod \"image-registry-7c89b4489d-2rf2s\" (UID: \"3571472a-95af-4617-9cb4-ac2782af9f74\") " pod="openshift-image-registry/image-registry-7c89b4489d-2rf2s"
May 26 11:25:27.762977 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.762873    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4-data-volume\") pod \"insights-runtime-extractor-7phq6\" (UID: \"e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4\") " pod="openshift-insights/insights-runtime-extractor-7phq6"
May 26 11:25:27.762977 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.762920    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3571472a-95af-4617-9cb4-ac2782af9f74-installation-pull-secrets\") pod \"image-registry-7c89b4489d-2rf2s\" (UID: \"3571472a-95af-4617-9cb4-ac2782af9f74\") " pod="openshift-image-registry/image-registry-7c89b4489d-2rf2s"
May 26 11:25:27.762977 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.762935    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4-crio-socket\") pod \"insights-runtime-extractor-7phq6\" (UID: \"e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4\") " pod="openshift-insights/insights-runtime-extractor-7phq6"
May 26 11:25:27.762977 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.762973    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7phq6\" (UID: \"e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4\") " pod="openshift-insights/insights-runtime-extractor-7phq6"
May 26 11:25:27.762977 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.762999    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3571472a-95af-4617-9cb4-ac2782af9f74-registry-tls\") pod \"image-registry-7c89b4489d-2rf2s\" (UID: \"3571472a-95af-4617-9cb4-ac2782af9f74\") " pod="openshift-image-registry/image-registry-7c89b4489d-2rf2s"
May 26 11:25:27.763308 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.763035    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3571472a-95af-4617-9cb4-ac2782af9f74-ca-trust-extracted\") pod \"image-registry-7c89b4489d-2rf2s\" (UID: \"3571472a-95af-4617-9cb4-ac2782af9f74\") " pod="openshift-image-registry/image-registry-7c89b4489d-2rf2s"
May 26 11:25:27.769859 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.769840    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4-data-volume\") pod \"insights-runtime-extractor-7phq6\" (UID: \"e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4\") " pod="openshift-insights/insights-runtime-extractor-7phq6"
May 26 11:25:27.770053 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.770036    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7phq6\" (UID: \"e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4\") " pod="openshift-insights/insights-runtime-extractor-7phq6"
May 26 11:25:27.772101 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.772078    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7phq6\" (UID: \"e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4\") " pod="openshift-insights/insights-runtime-extractor-7phq6"
May 26 11:25:27.788680 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.788648    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j7vf\" (UniqueName: \"kubernetes.io/projected/e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4-kube-api-access-6j7vf\") pod \"insights-runtime-extractor-7phq6\" (UID: \"e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4\") " pod="openshift-insights/insights-runtime-extractor-7phq6"
May 26 11:25:27.846543 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.846513    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7phq6"
May 26 11:25:27.863428 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.863397    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3571472a-95af-4617-9cb4-ac2782af9f74-bound-sa-token\") pod \"image-registry-7c89b4489d-2rf2s\" (UID: \"3571472a-95af-4617-9cb4-ac2782af9f74\") " pod="openshift-image-registry/image-registry-7c89b4489d-2rf2s"
May 26 11:25:27.863535 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.863476    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3571472a-95af-4617-9cb4-ac2782af9f74-installation-pull-secrets\") pod \"image-registry-7c89b4489d-2rf2s\" (UID: \"3571472a-95af-4617-9cb4-ac2782af9f74\") " pod="openshift-image-registry/image-registry-7c89b4489d-2rf2s"
May 26 11:25:27.863535 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.863517    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3571472a-95af-4617-9cb4-ac2782af9f74-registry-tls\") pod \"image-registry-7c89b4489d-2rf2s\" (UID: \"3571472a-95af-4617-9cb4-ac2782af9f74\") " pod="openshift-image-registry/image-registry-7c89b4489d-2rf2s"
May 26 11:25:27.863650 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.863545    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3571472a-95af-4617-9cb4-ac2782af9f74-ca-trust-extracted\") pod \"image-registry-7c89b4489d-2rf2s\" (UID: \"3571472a-95af-4617-9cb4-ac2782af9f74\") " pod="openshift-image-registry/image-registry-7c89b4489d-2rf2s"
May 26 11:25:27.863650 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.863568    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3571472a-95af-4617-9cb4-ac2782af9f74-image-registry-private-configuration\") pod \"image-registry-7c89b4489d-2rf2s\" (UID: \"3571472a-95af-4617-9cb4-ac2782af9f74\") " pod="openshift-image-registry/image-registry-7c89b4489d-2rf2s"
May 26 11:25:27.863650 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.863588    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3571472a-95af-4617-9cb4-ac2782af9f74-trusted-ca\") pod \"image-registry-7c89b4489d-2rf2s\" (UID: \"3571472a-95af-4617-9cb4-ac2782af9f74\") " pod="openshift-image-registry/image-registry-7c89b4489d-2rf2s"
May 26 11:25:27.863799 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.863753    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3571472a-95af-4617-9cb4-ac2782af9f74-registry-certificates\") pod \"image-registry-7c89b4489d-2rf2s\" (UID: \"3571472a-95af-4617-9cb4-ac2782af9f74\") " pod="openshift-image-registry/image-registry-7c89b4489d-2rf2s"
May 26 11:25:27.863855 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.863816    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kfqwt\" (UniqueName: \"kubernetes.io/projected/3571472a-95af-4617-9cb4-ac2782af9f74-kube-api-access-kfqwt\") pod \"image-registry-7c89b4489d-2rf2s\" (UID: \"3571472a-95af-4617-9cb4-ac2782af9f74\") " pod="openshift-image-registry/image-registry-7c89b4489d-2rf2s"
May 26 11:25:27.864020 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.863994    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3571472a-95af-4617-9cb4-ac2782af9f74-ca-trust-extracted\") pod \"image-registry-7c89b4489d-2rf2s\" (UID: \"3571472a-95af-4617-9cb4-ac2782af9f74\") " pod="openshift-image-registry/image-registry-7c89b4489d-2rf2s"
May 26 11:25:27.865072 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.865048    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3571472a-95af-4617-9cb4-ac2782af9f74-registry-certificates\") pod \"image-registry-7c89b4489d-2rf2s\" (UID: \"3571472a-95af-4617-9cb4-ac2782af9f74\") " pod="openshift-image-registry/image-registry-7c89b4489d-2rf2s"
May 26 11:25:27.865521 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.865497    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3571472a-95af-4617-9cb4-ac2782af9f74-trusted-ca\") pod \"image-registry-7c89b4489d-2rf2s\" (UID: \"3571472a-95af-4617-9cb4-ac2782af9f74\") " pod="openshift-image-registry/image-registry-7c89b4489d-2rf2s"
May 26 11:25:27.866547 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.866513    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3571472a-95af-4617-9cb4-ac2782af9f74-installation-pull-secrets\") pod \"image-registry-7c89b4489d-2rf2s\" (UID: \"3571472a-95af-4617-9cb4-ac2782af9f74\") " pod="openshift-image-registry/image-registry-7c89b4489d-2rf2s"
May 26 11:25:27.866767 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.866747    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3571472a-95af-4617-9cb4-ac2782af9f74-image-registry-private-configuration\") pod \"image-registry-7c89b4489d-2rf2s\" (UID: \"3571472a-95af-4617-9cb4-ac2782af9f74\") " pod="openshift-image-registry/image-registry-7c89b4489d-2rf2s"
May 26 11:25:27.867032 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.867012    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3571472a-95af-4617-9cb4-ac2782af9f74-registry-tls\") pod \"image-registry-7c89b4489d-2rf2s\" (UID: \"3571472a-95af-4617-9cb4-ac2782af9f74\") " pod="openshift-image-registry/image-registry-7c89b4489d-2rf2s"
May 26 11:25:27.875218 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.875191    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3571472a-95af-4617-9cb4-ac2782af9f74-bound-sa-token\") pod \"image-registry-7c89b4489d-2rf2s\" (UID: \"3571472a-95af-4617-9cb4-ac2782af9f74\") " pod="openshift-image-registry/image-registry-7c89b4489d-2rf2s"
May 26 11:25:27.875848 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.875832    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfqwt\" (UniqueName: \"kubernetes.io/projected/3571472a-95af-4617-9cb4-ac2782af9f74-kube-api-access-kfqwt\") pod \"image-registry-7c89b4489d-2rf2s\" (UID: \"3571472a-95af-4617-9cb4-ac2782af9f74\") " pod="openshift-image-registry/image-registry-7c89b4489d-2rf2s"
May 26 11:25:27.936085 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.936054    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c89b4489d-2rf2s"
May 26 11:25:27.979085 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:27.979022    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7phq6"]
May 26 11:25:27.984365 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:25:27.984326    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0357dfc_ab2b_4b79_8f21_c1ee4e8a66e4.slice/crio-ddec14579547c4af78409cc233fb6f681ebb817341fba284cf8b3e1d800cdc8d WatchSource:0}: Error finding container ddec14579547c4af78409cc233fb6f681ebb817341fba284cf8b3e1d800cdc8d: Status 404 returned error can't find the container with id ddec14579547c4af78409cc233fb6f681ebb817341fba284cf8b3e1d800cdc8d
May 26 11:25:28.072873 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:28.072835    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c89b4489d-2rf2s"]
May 26 11:25:28.076462 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:25:28.076430    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3571472a_95af_4617_9cb4_ac2782af9f74.slice/crio-75f39d91a1a4d80eab7395e8254b20acd268c23777f52ca3deb9e72abed72d9b WatchSource:0}: Error finding container 75f39d91a1a4d80eab7395e8254b20acd268c23777f52ca3deb9e72abed72d9b: Status 404 returned error can't find the container with id 75f39d91a1a4d80eab7395e8254b20acd268c23777f52ca3deb9e72abed72d9b
May 26 11:25:28.571356 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:28.571326    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wxzlw"
May 26 11:25:28.646409 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:28.646377    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c89b4489d-2rf2s" event={"ID":"3571472a-95af-4617-9cb4-ac2782af9f74","Type":"ContainerStarted","Data":"dcf1b357360b1c6542e39805eb145e43366c73e5b5129ab99d084dfeb2bfafbd"}
May 26 11:25:28.646668 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:28.646629    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c89b4489d-2rf2s"
May 26 11:25:28.646834 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:28.646815    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c89b4489d-2rf2s" event={"ID":"3571472a-95af-4617-9cb4-ac2782af9f74","Type":"ContainerStarted","Data":"75f39d91a1a4d80eab7395e8254b20acd268c23777f52ca3deb9e72abed72d9b"}
May 26 11:25:28.648485 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:28.648419    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7phq6" event={"ID":"e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4","Type":"ContainerStarted","Data":"7867d0c990788e9c966d3dd11589fc197a09aa4fab8fdfd7c9c6d3ef55d2e811"}
May 26 11:25:28.648485 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:28.648448    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7phq6" event={"ID":"e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4","Type":"ContainerStarted","Data":"ddec14579547c4af78409cc233fb6f681ebb817341fba284cf8b3e1d800cdc8d"}
May 26 11:25:28.668554 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:28.668504    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7c89b4489d-2rf2s" podStartSLOduration=1.668488821 podStartE2EDuration="1.668488821s" podCreationTimestamp="2026-05-26 11:25:27 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:25:28.667560384 +0000 UTC m=+60.859002571" watchObservedRunningTime="2026-05-26 11:25:28.668488821 +0000 UTC m=+60.859931013"
May 26 11:25:29.056554 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:29.056523    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-57bd96b888-fhfpk"]
May 26 11:25:29.058449 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:29.058433    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-fhfpk"
May 26 11:25:29.061347 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:29.061309    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-64shf\""
May 26 11:25:29.061347 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:29.061343    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 26 11:25:29.061541 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:29.061342    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 26 11:25:29.069661 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:29.069635    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-fhfpk"]
May 26 11:25:29.176965 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:29.176921    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rdbp\" (UniqueName: \"kubernetes.io/projected/e0c32514-7fd6-461c-bf92-092ad4ed740e-kube-api-access-4rdbp\") pod \"downloads-57bd96b888-fhfpk\" (UID: \"e0c32514-7fd6-461c-bf92-092ad4ed740e\") " pod="openshift-console/downloads-57bd96b888-fhfpk"
May 26 11:25:29.277546 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:29.277500    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4rdbp\" (UniqueName: \"kubernetes.io/projected/e0c32514-7fd6-461c-bf92-092ad4ed740e-kube-api-access-4rdbp\") pod \"downloads-57bd96b888-fhfpk\" (UID: \"e0c32514-7fd6-461c-bf92-092ad4ed740e\") " pod="openshift-console/downloads-57bd96b888-fhfpk"
May 26 11:25:29.287123 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:29.287087    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rdbp\" (UniqueName: \"kubernetes.io/projected/e0c32514-7fd6-461c-bf92-092ad4ed740e-kube-api-access-4rdbp\") pod \"downloads-57bd96b888-fhfpk\" (UID: \"e0c32514-7fd6-461c-bf92-092ad4ed740e\") " pod="openshift-console/downloads-57bd96b888-fhfpk"
May 26 11:25:29.367036 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:29.367001    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-fhfpk"
May 26 11:25:29.498942 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:29.498910    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-fhfpk"]
May 26 11:25:29.502420 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:25:29.502390    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c32514_7fd6_461c_bf92_092ad4ed740e.slice/crio-dc8de6ab6cd86dbb99db2886b7b005d6f027f4c0259f73245a554123ae0b308d WatchSource:0}: Error finding container dc8de6ab6cd86dbb99db2886b7b005d6f027f4c0259f73245a554123ae0b308d: Status 404 returned error can't find the container with id dc8de6ab6cd86dbb99db2886b7b005d6f027f4c0259f73245a554123ae0b308d
May 26 11:25:29.653037 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:29.652941    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-fhfpk" event={"ID":"e0c32514-7fd6-461c-bf92-092ad4ed740e","Type":"ContainerStarted","Data":"dc8de6ab6cd86dbb99db2886b7b005d6f027f4c0259f73245a554123ae0b308d"}
May 26 11:25:29.654637 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:29.654610    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7phq6" event={"ID":"e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4","Type":"ContainerStarted","Data":"8a607ab3c4c83f3819539b78eec6c51098b74fbdaef890f00db396109a5d5a42"}
May 26 11:25:30.659094 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:30.659049    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7phq6" event={"ID":"e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4","Type":"ContainerStarted","Data":"afc7419fb47f54d3354d4763b019dddbc48d8a29e4214791a78a44e880f6cdfd"}
May 26 11:25:30.679985 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:30.679912    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7phq6" podStartSLOduration=1.470588939 podStartE2EDuration="3.679893135s" podCreationTimestamp="2026-05-26 11:25:27 +0000 UTC" firstStartedPulling="2026-05-26 11:25:28.043099486 +0000 UTC m=+60.234541657" lastFinishedPulling="2026-05-26 11:25:30.252403679 +0000 UTC m=+62.443845853" observedRunningTime="2026-05-26 11:25:30.679087311 +0000 UTC m=+62.870529499" watchObservedRunningTime="2026-05-26 11:25:30.679893135 +0000 UTC m=+62.871335323"
May 26 11:25:31.630355 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:31.630318    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fn6r5"
May 26 11:25:32.778494 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:32.778453    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-pcq8j"]
May 26 11:25:32.780649 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:32.780628    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-pcq8j"
May 26 11:25:32.783866 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:32.783844    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-rjmbm\""
May 26 11:25:32.783984 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:32.783849    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
May 26 11:25:32.784868 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:32.784837    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 26 11:25:32.785079 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:32.785062    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 26 11:25:32.785166 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:32.785128    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
May 26 11:25:32.785228 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:32.785180    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 26 11:25:32.791467 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:32.791421    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-pcq8j"]
May 26 11:25:32.807521 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:32.807486    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/126ae8b2-7015-4fa0-b662-6a4b3289b160-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-pcq8j\" (UID: \"126ae8b2-7015-4fa0-b662-6a4b3289b160\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-pcq8j"
May 26 11:25:32.807702 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:32.807636    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/126ae8b2-7015-4fa0-b662-6a4b3289b160-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-pcq8j\" (UID: \"126ae8b2-7015-4fa0-b662-6a4b3289b160\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-pcq8j"
May 26 11:25:32.807702 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:32.807689    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h9xl\" (UniqueName: \"kubernetes.io/projected/126ae8b2-7015-4fa0-b662-6a4b3289b160-kube-api-access-5h9xl\") pod \"prometheus-operator-7779bbdd78-pcq8j\" (UID: \"126ae8b2-7015-4fa0-b662-6a4b3289b160\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-pcq8j"
May 26 11:25:32.807961 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:32.807717    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/126ae8b2-7015-4fa0-b662-6a4b3289b160-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-pcq8j\" (UID: \"126ae8b2-7015-4fa0-b662-6a4b3289b160\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-pcq8j"
May 26 11:25:32.908798 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:32.908727    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/126ae8b2-7015-4fa0-b662-6a4b3289b160-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-pcq8j\" (UID: \"126ae8b2-7015-4fa0-b662-6a4b3289b160\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-pcq8j"
May 26 11:25:32.908955 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:32.908872    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/126ae8b2-7015-4fa0-b662-6a4b3289b160-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-pcq8j\" (UID: \"126ae8b2-7015-4fa0-b662-6a4b3289b160\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-pcq8j"
May 26 11:25:32.908955 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:32.908906    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5h9xl\" (UniqueName: \"kubernetes.io/projected/126ae8b2-7015-4fa0-b662-6a4b3289b160-kube-api-access-5h9xl\") pod \"prometheus-operator-7779bbdd78-pcq8j\" (UID: \"126ae8b2-7015-4fa0-b662-6a4b3289b160\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-pcq8j"
May 26 11:25:32.908955 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:32.908939    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/126ae8b2-7015-4fa0-b662-6a4b3289b160-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-pcq8j\" (UID: \"126ae8b2-7015-4fa0-b662-6a4b3289b160\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-pcq8j"
May 26 11:25:32.909134 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:25:32.909039    2563 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
May 26 11:25:32.909134 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:25:32.909121    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/126ae8b2-7015-4fa0-b662-6a4b3289b160-prometheus-operator-tls podName:126ae8b2-7015-4fa0-b662-6a4b3289b160 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:33.409096801 +0000 UTC m=+65.600538974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/126ae8b2-7015-4fa0-b662-6a4b3289b160-prometheus-operator-tls") pod "prometheus-operator-7779bbdd78-pcq8j" (UID: "126ae8b2-7015-4fa0-b662-6a4b3289b160") : secret "prometheus-operator-tls" not found
May 26 11:25:32.909537 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:32.909519    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/126ae8b2-7015-4fa0-b662-6a4b3289b160-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-pcq8j\" (UID: \"126ae8b2-7015-4fa0-b662-6a4b3289b160\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-pcq8j"
May 26 11:25:32.911376 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:32.911352    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/126ae8b2-7015-4fa0-b662-6a4b3289b160-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-pcq8j\" (UID: \"126ae8b2-7015-4fa0-b662-6a4b3289b160\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-pcq8j"
May 26 11:25:32.921862 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:32.921833    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h9xl\" (UniqueName: \"kubernetes.io/projected/126ae8b2-7015-4fa0-b662-6a4b3289b160-kube-api-access-5h9xl\") pod \"prometheus-operator-7779bbdd78-pcq8j\" (UID: \"126ae8b2-7015-4fa0-b662-6a4b3289b160\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-pcq8j"
May 26 11:25:33.413288 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:33.413251    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/126ae8b2-7015-4fa0-b662-6a4b3289b160-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-pcq8j\" (UID: \"126ae8b2-7015-4fa0-b662-6a4b3289b160\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-pcq8j"
May 26 11:25:33.413470 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:25:33.413408    2563 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
May 26 11:25:33.413546 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:25:33.413495    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/126ae8b2-7015-4fa0-b662-6a4b3289b160-prometheus-operator-tls podName:126ae8b2-7015-4fa0-b662-6a4b3289b160 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:34.41347702 +0000 UTC m=+66.604919190 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/126ae8b2-7015-4fa0-b662-6a4b3289b160-prometheus-operator-tls") pod "prometheus-operator-7779bbdd78-pcq8j" (UID: "126ae8b2-7015-4fa0-b662-6a4b3289b160") : secret "prometheus-operator-tls" not found
May 26 11:25:34.118457 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:34.118421    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d573a77-5520-489f-99cb-86419b23cd19-metrics-certs\") pod \"network-metrics-daemon-pp9l8\" (UID: \"9d573a77-5520-489f-99cb-86419b23cd19\") " pod="openshift-multus/network-metrics-daemon-pp9l8"
May 26 11:25:34.121519 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:34.121488    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 26 11:25:34.131201 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:34.131174    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d573a77-5520-489f-99cb-86419b23cd19-metrics-certs\") pod \"network-metrics-daemon-pp9l8\" (UID: \"9d573a77-5520-489f-99cb-86419b23cd19\") " pod="openshift-multus/network-metrics-daemon-pp9l8"
May 26 11:25:34.219546 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:34.219510    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bt6x9\" (UniqueName: \"kubernetes.io/projected/8528f6a0-0b8d-47f8-a20c-715dd15d1076-kube-api-access-bt6x9\") pod \"network-check-target-cbxbl\" (UID: \"8528f6a0-0b8d-47f8-a20c-715dd15d1076\") " pod="openshift-network-diagnostics/network-check-target-cbxbl"
May 26 11:25:34.222572 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:34.222539    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 26 11:25:34.232576 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:34.232550    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 26 11:25:34.238429 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:34.238407    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xvpvn\""
May 26 11:25:34.244063 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:34.244035    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt6x9\" (UniqueName: \"kubernetes.io/projected/8528f6a0-0b8d-47f8-a20c-715dd15d1076-kube-api-access-bt6x9\") pod \"network-check-target-cbxbl\" (UID: \"8528f6a0-0b8d-47f8-a20c-715dd15d1076\") " pod="openshift-network-diagnostics/network-check-target-cbxbl"
May 26 11:25:34.246623 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:34.246605    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp9l8"
May 26 11:25:34.392165 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:34.392073    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pp9l8"]
May 26 11:25:34.395888 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:25:34.395854    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d573a77_5520_489f_99cb_86419b23cd19.slice/crio-e3922193e1f31c8168e8f868251db6f78b93736ef4fdb96e37d11e30aa7482b1 WatchSource:0}: Error finding container e3922193e1f31c8168e8f868251db6f78b93736ef4fdb96e37d11e30aa7482b1: Status 404 returned error can't find the container with id e3922193e1f31c8168e8f868251db6f78b93736ef4fdb96e37d11e30aa7482b1
May 26 11:25:34.421377 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:34.421348    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/126ae8b2-7015-4fa0-b662-6a4b3289b160-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-pcq8j\" (UID: \"126ae8b2-7015-4fa0-b662-6a4b3289b160\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-pcq8j"
May 26 11:25:34.423716 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:34.423690    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/126ae8b2-7015-4fa0-b662-6a4b3289b160-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-pcq8j\" (UID: \"126ae8b2-7015-4fa0-b662-6a4b3289b160\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-pcq8j"
May 26 11:25:34.544013 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:34.543978    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zzwv5\""
May 26 11:25:34.552262 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:34.552242    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbxbl"
May 26 11:25:34.592028 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:34.591959    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-pcq8j"
May 26 11:25:34.673297 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:34.673257    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pp9l8" event={"ID":"9d573a77-5520-489f-99cb-86419b23cd19","Type":"ContainerStarted","Data":"e3922193e1f31c8168e8f868251db6f78b93736ef4fdb96e37d11e30aa7482b1"}
May 26 11:25:34.689317 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:34.689284    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cbxbl"]
May 26 11:25:34.747202 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:34.747164    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-pcq8j"]
May 26 11:25:34.750745 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:25:34.750706    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod126ae8b2_7015_4fa0_b662_6a4b3289b160.slice/crio-df1e16edee06876c3de5988c4c59a2639e743deffb3a1db49a524c1a77b42cee WatchSource:0}: Error finding container df1e16edee06876c3de5988c4c59a2639e743deffb3a1db49a524c1a77b42cee: Status 404 returned error can't find the container with id df1e16edee06876c3de5988c4c59a2639e743deffb3a1db49a524c1a77b42cee
May 26 11:25:35.680797 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:35.680085    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pp9l8" event={"ID":"9d573a77-5520-489f-99cb-86419b23cd19","Type":"ContainerStarted","Data":"1dfda5116c092d82106f879dade233b4462aa0fa53e1b29b3c9a0d50b8a6ece0"}
May 26 11:25:35.680797 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:35.680129    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pp9l8" event={"ID":"9d573a77-5520-489f-99cb-86419b23cd19","Type":"ContainerStarted","Data":"c2c2ca0541f759bfadd8b7c4b87cee3ecba4eff3d11781e70ca7f6d1a5852aa2"}
May 26 11:25:35.682560 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:35.682464    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-pcq8j" event={"ID":"126ae8b2-7015-4fa0-b662-6a4b3289b160","Type":"ContainerStarted","Data":"df1e16edee06876c3de5988c4c59a2639e743deffb3a1db49a524c1a77b42cee"}
May 26 11:25:35.686968 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:35.686126    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cbxbl" event={"ID":"8528f6a0-0b8d-47f8-a20c-715dd15d1076","Type":"ContainerStarted","Data":"6ce97daebca172606b8041976add2525f6d71b1fa4df5be6727a5b510618108e"}
May 26 11:25:35.703296 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:35.701921    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pp9l8" podStartSLOduration=66.666114043 podStartE2EDuration="1m7.701907549s" podCreationTimestamp="2026-05-26 11:24:28 +0000 UTC" firstStartedPulling="2026-05-26 11:25:34.397633964 +0000 UTC m=+66.589076129" lastFinishedPulling="2026-05-26 11:25:35.433427468 +0000 UTC m=+67.624869635" observedRunningTime="2026-05-26 11:25:35.701557127 +0000 UTC m=+67.892999314" watchObservedRunningTime="2026-05-26 11:25:35.701907549 +0000 UTC m=+67.893349735"
May 26 11:25:36.691763 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:36.691717    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-pcq8j" event={"ID":"126ae8b2-7015-4fa0-b662-6a4b3289b160","Type":"ContainerStarted","Data":"a550eb0c9806b5400195dc2a30610899300479bcd0b2fea3a3def25a9dcc494f"}
May 26 11:25:37.697015 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:37.696960    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-pcq8j" event={"ID":"126ae8b2-7015-4fa0-b662-6a4b3289b160","Type":"ContainerStarted","Data":"67ad2e4e1aed4e13aa4bf5df65516a09d769e7f89cac86fd58eeb438029ace1f"}
May 26 11:25:37.715190 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:37.715134    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7779bbdd78-pcq8j" podStartSLOduration=3.895734107 podStartE2EDuration="5.715114773s" podCreationTimestamp="2026-05-26 11:25:32 +0000 UTC" firstStartedPulling="2026-05-26 11:25:34.75315032 +0000 UTC m=+66.944592486" lastFinishedPulling="2026-05-26 11:25:36.572530981 +0000 UTC m=+68.763973152" observedRunningTime="2026-05-26 11:25:37.714184185 +0000 UTC m=+69.905626371" watchObservedRunningTime="2026-05-26 11:25:37.715114773 +0000 UTC m=+69.906556954"
May 26 11:25:38.701981 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:38.701927    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cbxbl" event={"ID":"8528f6a0-0b8d-47f8-a20c-715dd15d1076","Type":"ContainerStarted","Data":"d497a26161e992ea15318c4f4dcf12be8ab9dab0809abfe0d86fa7e3a5443a75"}
May 26 11:25:38.702426 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:38.702067    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cbxbl"
May 26 11:25:38.720862 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:38.720769    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cbxbl" podStartSLOduration=67.514276711 podStartE2EDuration="1m10.720750672s" podCreationTimestamp="2026-05-26 11:24:28 +0000 UTC" firstStartedPulling="2026-05-26 11:25:34.695639203 +0000 UTC m=+66.887081368" lastFinishedPulling="2026-05-26 11:25:37.90211316 +0000 UTC m=+70.093555329" observedRunningTime="2026-05-26 11:25:38.719909191 +0000 UTC m=+70.911351379" watchObservedRunningTime="2026-05-26 11:25:38.720750672 +0000 UTC m=+70.912192855"
May 26 11:25:39.170644 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:39.170609    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-c5zsn"]
May 26 11:25:39.187523 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:39.187489    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c5zsn"
May 26 11:25:39.190431 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:39.190138    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 26 11:25:39.190431 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:39.190272    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 26 11:25:39.190431 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:39.190342    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 26 11:25:39.190639 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:39.190435    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-klwwf\""
May 26 11:25:39.260755 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:39.260711    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cf8f06cf-5a6e-45cd-b44d-2817b037ed24-node-exporter-accelerators-collector-config\") pod \"node-exporter-c5zsn\" (UID: \"cf8f06cf-5a6e-45cd-b44d-2817b037ed24\") " pod="openshift-monitoring/node-exporter-c5zsn"
May 26 11:25:39.260755 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:39.260755    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cf8f06cf-5a6e-45cd-b44d-2817b037ed24-node-exporter-tls\") pod \"node-exporter-c5zsn\" (UID: \"cf8f06cf-5a6e-45cd-b44d-2817b037ed24\") " pod="openshift-monitoring/node-exporter-c5zsn"
May 26 11:25:39.260986 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:39.260808    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cf8f06cf-5a6e-45cd-b44d-2817b037ed24-node-exporter-textfile\") pod \"node-exporter-c5zsn\" (UID: \"cf8f06cf-5a6e-45cd-b44d-2817b037ed24\") " pod="openshift-monitoring/node-exporter-c5zsn"
May 26 11:25:39.260986 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:39.260826    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cf8f06cf-5a6e-45cd-b44d-2817b037ed24-node-exporter-wtmp\") pod \"node-exporter-c5zsn\" (UID: \"cf8f06cf-5a6e-45cd-b44d-2817b037ed24\") " pod="openshift-monitoring/node-exporter-c5zsn"
May 26 11:25:39.260986 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:39.260845    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf8f06cf-5a6e-45cd-b44d-2817b037ed24-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c5zsn\" (UID: \"cf8f06cf-5a6e-45cd-b44d-2817b037ed24\") " pod="openshift-monitoring/node-exporter-c5zsn"
May 26 11:25:39.260986 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:39.260878    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj922\" (UniqueName: \"kubernetes.io/projected/cf8f06cf-5a6e-45cd-b44d-2817b037ed24-kube-api-access-nj922\") pod \"node-exporter-c5zsn\" (UID: \"cf8f06cf-5a6e-45cd-b44d-2817b037ed24\") " pod="openshift-monitoring/node-exporter-c5zsn"
May 26 11:25:39.260986 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:39.260940    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cf8f06cf-5a6e-45cd-b44d-2817b037ed24-root\") pod \"node-exporter-c5zsn\" (UID: \"cf8f06cf-5a6e-45cd-b44d-2817b037ed24\") " pod="openshift-monitoring/node-exporter-c5zsn"
May 26 11:25:39.260986 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:39.260980    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf8f06cf-5a6e-45cd-b44d-2817b037ed24-sys\") pod \"node-exporter-c5zsn\" (UID: \"cf8f06cf-5a6e-45cd-b44d-2817b037ed24\") " pod="openshift-monitoring/node-exporter-c5zsn"
May 26 11:25:39.261274 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:39.261010    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf8f06cf-5a6e-45cd-b44d-2817b037ed24-metrics-client-ca\") pod \"node-exporter-c5zsn\" (UID: \"cf8f06cf-5a6e-45cd-b44d-2817b037ed24\") " pod="openshift-monitoring/node-exporter-c5zsn"
May 26 11:25:39.361980 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:39.361934    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cf8f06cf-5a6e-45cd-b44d-2817b037ed24-node-exporter-accelerators-collector-config\") pod \"node-exporter-c5zsn\" (UID: \"cf8f06cf-5a6e-45cd-b44d-2817b037ed24\") " pod="openshift-monitoring/node-exporter-c5zsn"
May 26 11:25:39.362172 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:39.361987    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cf8f06cf-5a6e-45cd-b44d-2817b037ed24-node-exporter-tls\") pod \"node-exporter-c5zsn\" (UID: \"cf8f06cf-5a6e-45cd-b44d-2817b037ed24\") " pod="openshift-monitoring/node-exporter-c5zsn"
May 26 11:25:39.362172 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:39.362029    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cf8f06cf-5a6e-45cd-b44d-2817b037ed24-node-exporter-textfile\") pod \"node-exporter-c5zsn\" (UID: \"cf8f06cf-5a6e-45cd-b44d-2817b037ed24\") " pod="openshift-monitoring/node-exporter-c5zsn"
May 26 11:25:39.362172 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:39.362158    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cf8f06cf-5a6e-45cd-b44d-2817b037ed24-node-exporter-wtmp\") pod \"node-exporter-c5zsn\" (UID: \"cf8f06cf-5a6e-45cd-b44d-2817b037ed24\") " pod="openshift-monitoring/node-exporter-c5zsn"
May 26 11:25:39.362336 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:25:39.362175    2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
May 26 11:25:39.362336 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:39.362203    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf8f06cf-5a6e-45cd-b44d-2817b037ed24-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c5zsn\" (UID: \"cf8f06cf-5a6e-45cd-b44d-2817b037ed24\") " pod="openshift-monitoring/node-exporter-c5zsn"
May 26 11:25:39.362336 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:25:39.362245    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf8f06cf-5a6e-45cd-b44d-2817b037ed24-node-exporter-tls podName:cf8f06cf-5a6e-45cd-b44d-2817b037ed24 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:39.862221009 +0000 UTC m=+72.053663195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/cf8f06cf-5a6e-45cd-b44d-2817b037ed24-node-exporter-tls") pod "node-exporter-c5zsn" (UID: "cf8f06cf-5a6e-45cd-b44d-2817b037ed24") : secret "node-exporter-tls" not found
May 26 11:25:39.362336 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:39.362274    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nj922\" (UniqueName: \"kubernetes.io/projected/cf8f06cf-5a6e-45cd-b44d-2817b037ed24-kube-api-access-nj922\") pod \"node-exporter-c5zsn\" (UID: \"cf8f06cf-5a6e-45cd-b44d-2817b037ed24\") " pod="openshift-monitoring/node-exporter-c5zsn"
May 26 11:25:39.362336 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:39.362309    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cf8f06cf-5a6e-45cd-b44d-2817b037ed24-root\") pod \"node-exporter-c5zsn\" (UID: \"cf8f06cf-5a6e-45cd-b44d-2817b037ed24\") " pod="openshift-monitoring/node-exporter-c5zsn"
May 26 11:25:39.362581 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:39.362341    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf8f06cf-5a6e-45cd-b44d-2817b037ed24-sys\") pod \"node-exporter-c5zsn\" (UID: \"cf8f06cf-5a6e-45cd-b44d-2817b037ed24\") " pod="openshift-monitoring/node-exporter-c5zsn"
May 26 11:25:39.362581 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:39.362350    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cf8f06cf-5a6e-45cd-b44d-2817b037ed24-node-exporter-wtmp\") pod \"node-exporter-c5zsn\" (UID: \"cf8f06cf-5a6e-45cd-b44d-2817b037ed24\") " pod="openshift-monitoring/node-exporter-c5zsn"
May 26 11:25:39.362581 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:39.362367    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf8f06cf-5a6e-45cd-b44d-2817b037ed24-metrics-client-ca\") pod \"node-exporter-c5zsn\" (UID: \"cf8f06cf-5a6e-45cd-b44d-2817b037ed24\") " pod="openshift-monitoring/node-exporter-c5zsn"
May 26 11:25:39.362581 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:39.362412    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cf8f06cf-5a6e-45cd-b44d-2817b037ed24-root\") pod \"node-exporter-c5zsn\" (UID: \"cf8f06cf-5a6e-45cd-b44d-2817b037ed24\") " pod="openshift-monitoring/node-exporter-c5zsn"
May 26 11:25:39.362581 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:39.362416    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cf8f06cf-5a6e-45cd-b44d-2817b037ed24-node-exporter-textfile\") pod \"node-exporter-c5zsn\" (UID: \"cf8f06cf-5a6e-45cd-b44d-2817b037ed24\") " pod="openshift-monitoring/node-exporter-c5zsn"
May 26 11:25:39.362581 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:39.362471    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf8f06cf-5a6e-45cd-b44d-2817b037ed24-sys\") pod \"node-exporter-c5zsn\" (UID: \"cf8f06cf-5a6e-45cd-b44d-2817b037ed24\") " pod="openshift-monitoring/node-exporter-c5zsn"
May 26 11:25:39.362851 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:39.362635    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cf8f06cf-5a6e-45cd-b44d-2817b037ed24-node-exporter-accelerators-collector-config\") pod \"node-exporter-c5zsn\" (UID: \"cf8f06cf-5a6e-45cd-b44d-2817b037ed24\") " pod="openshift-monitoring/node-exporter-c5zsn"
May 26 11:25:39.362929 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:39.362909    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf8f06cf-5a6e-45cd-b44d-2817b037ed24-metrics-client-ca\") pod \"node-exporter-c5zsn\" (UID: \"cf8f06cf-5a6e-45cd-b44d-2817b037ed24\") " pod="openshift-monitoring/node-exporter-c5zsn"
May 26 11:25:39.365249 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:39.365225    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf8f06cf-5a6e-45cd-b44d-2817b037ed24-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c5zsn\" (UID: \"cf8f06cf-5a6e-45cd-b44d-2817b037ed24\") " pod="openshift-monitoring/node-exporter-c5zsn"
May 26 11:25:39.371714 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:39.371689    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj922\" (UniqueName: \"kubernetes.io/projected/cf8f06cf-5a6e-45cd-b44d-2817b037ed24-kube-api-access-nj922\") pod \"node-exporter-c5zsn\" (UID: \"cf8f06cf-5a6e-45cd-b44d-2817b037ed24\") " pod="openshift-monitoring/node-exporter-c5zsn"
May 26 11:25:39.867185 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:39.867139    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cf8f06cf-5a6e-45cd-b44d-2817b037ed24-node-exporter-tls\") pod \"node-exporter-c5zsn\" (UID: \"cf8f06cf-5a6e-45cd-b44d-2817b037ed24\") " pod="openshift-monitoring/node-exporter-c5zsn"
May 26 11:25:39.869949 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:39.869915    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cf8f06cf-5a6e-45cd-b44d-2817b037ed24-node-exporter-tls\") pod \"node-exporter-c5zsn\" (UID: \"cf8f06cf-5a6e-45cd-b44d-2817b037ed24\") " pod="openshift-monitoring/node-exporter-c5zsn"
May 26 11:25:40.098316 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:40.098279    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c5zsn"
May 26 11:25:40.109270 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:25:40.109234    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf8f06cf_5a6e_45cd_b44d_2817b037ed24.slice/crio-50d0b7b9e4ce8a7c8202d79bc5ce77615ffeee09438c918fa8f689aa29411ec6 WatchSource:0}: Error finding container 50d0b7b9e4ce8a7c8202d79bc5ce77615ffeee09438c918fa8f689aa29411ec6: Status 404 returned error can't find the container with id 50d0b7b9e4ce8a7c8202d79bc5ce77615ffeee09438c918fa8f689aa29411ec6
May 26 11:25:40.708888 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:40.708835    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c5zsn" event={"ID":"cf8f06cf-5a6e-45cd-b44d-2817b037ed24","Type":"ContainerStarted","Data":"50d0b7b9e4ce8a7c8202d79bc5ce77615ffeee09438c918fa8f689aa29411ec6"}
May 26 11:25:43.645177 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:43.645055    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-79c798ccf6-c9h4r"]
May 26 11:25:43.647198 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:43.647180    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-79c798ccf6-c9h4r"
May 26 11:25:43.649963 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:43.649938    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-79re3mkev525k\""
May 26 11:25:43.651196 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:43.651170    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
May 26 11:25:43.651310 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:43.651236    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-62w6v\""
May 26 11:25:43.651310 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:43.651173    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
May 26 11:25:43.651409 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:43.651372    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
May 26 11:25:43.651606 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:43.651588    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
May 26 11:25:43.657033 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:43.656612    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-79c798ccf6-c9h4r"]
May 26 11:25:43.698493 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:43.698446    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5-metrics-server-audit-profiles\") pod \"metrics-server-79c798ccf6-c9h4r\" (UID: \"8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5\") " pod="openshift-monitoring/metrics-server-79c798ccf6-c9h4r"
May 26 11:25:43.698689 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:43.698509    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5-secret-metrics-server-client-certs\") pod \"metrics-server-79c798ccf6-c9h4r\" (UID: \"8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5\") " pod="openshift-monitoring/metrics-server-79c798ccf6-c9h4r"
May 26 11:25:43.698689 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:43.698596    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zscgf\" (UniqueName: \"kubernetes.io/projected/8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5-kube-api-access-zscgf\") pod \"metrics-server-79c798ccf6-c9h4r\" (UID: \"8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5\") " pod="openshift-monitoring/metrics-server-79c798ccf6-c9h4r"
May 26 11:25:43.698689 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:43.698650    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5-audit-log\") pod \"metrics-server-79c798ccf6-c9h4r\" (UID: \"8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5\") " pod="openshift-monitoring/metrics-server-79c798ccf6-c9h4r"
May 26 11:25:43.698837 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:43.698743    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-79c798ccf6-c9h4r\" (UID: \"8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5\") " pod="openshift-monitoring/metrics-server-79c798ccf6-c9h4r"
May 26 11:25:43.698837 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:43.698793    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5-secret-metrics-server-tls\") pod \"metrics-server-79c798ccf6-c9h4r\" (UID: \"8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5\") " pod="openshift-monitoring/metrics-server-79c798ccf6-c9h4r"
May 26 11:25:43.698837 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:43.698828    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5-client-ca-bundle\") pod \"metrics-server-79c798ccf6-c9h4r\" (UID: \"8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5\") " pod="openshift-monitoring/metrics-server-79c798ccf6-c9h4r"
May 26 11:25:43.799575 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:43.799537    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5-metrics-server-audit-profiles\") pod \"metrics-server-79c798ccf6-c9h4r\" (UID: \"8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5\") " pod="openshift-monitoring/metrics-server-79c798ccf6-c9h4r"
May 26 11:25:43.799754 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:43.799591    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5-secret-metrics-server-client-certs\") pod \"metrics-server-79c798ccf6-c9h4r\" (UID: \"8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5\") " pod="openshift-monitoring/metrics-server-79c798ccf6-c9h4r"
May 26 11:25:43.799754 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:43.799628    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zscgf\" (UniqueName: \"kubernetes.io/projected/8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5-kube-api-access-zscgf\") pod \"metrics-server-79c798ccf6-c9h4r\" (UID: \"8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5\") " pod="openshift-monitoring/metrics-server-79c798ccf6-c9h4r"
May 26 11:25:43.799754 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:43.799660    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5-audit-log\") pod \"metrics-server-79c798ccf6-c9h4r\" (UID: \"8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5\") " pod="openshift-monitoring/metrics-server-79c798ccf6-c9h4r"
May 26 11:25:43.799754 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:43.799700    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-79c798ccf6-c9h4r\" (UID: \"8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5\") " pod="openshift-monitoring/metrics-server-79c798ccf6-c9h4r"
May 26 11:25:43.799754 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:43.799728    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5-secret-metrics-server-tls\") pod \"metrics-server-79c798ccf6-c9h4r\" (UID: \"8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5\") " pod="openshift-monitoring/metrics-server-79c798ccf6-c9h4r"
May 26 11:25:43.800030 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:43.799756    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5-client-ca-bundle\") pod \"metrics-server-79c798ccf6-c9h4r\" (UID: \"8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5\") " pod="openshift-monitoring/metrics-server-79c798ccf6-c9h4r"
May 26 11:25:43.800131 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:43.800104    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5-audit-log\") pod \"metrics-server-79c798ccf6-c9h4r\" (UID: \"8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5\") " pod="openshift-monitoring/metrics-server-79c798ccf6-c9h4r"
May 26 11:25:43.800888 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:43.800859    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-79c798ccf6-c9h4r\" (UID: \"8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5\") " pod="openshift-monitoring/metrics-server-79c798ccf6-c9h4r"
May 26 11:25:43.801630 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:43.801603    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5-metrics-server-audit-profiles\") pod \"metrics-server-79c798ccf6-c9h4r\" (UID: \"8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5\") " pod="openshift-monitoring/metrics-server-79c798ccf6-c9h4r"
May 26 11:25:43.802739 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:43.802716    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5-secret-metrics-server-client-certs\") pod \"metrics-server-79c798ccf6-c9h4r\" (UID: \"8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5\") " pod="openshift-monitoring/metrics-server-79c798ccf6-c9h4r"
May 26 11:25:43.802884 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:43.802864    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5-client-ca-bundle\") pod \"metrics-server-79c798ccf6-c9h4r\" (UID: \"8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5\") " pod="openshift-monitoring/metrics-server-79c798ccf6-c9h4r"
May 26 11:25:43.802963 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:43.802943    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5-secret-metrics-server-tls\") pod \"metrics-server-79c798ccf6-c9h4r\" (UID: \"8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5\") " pod="openshift-monitoring/metrics-server-79c798ccf6-c9h4r"
May 26 11:25:43.808584 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:43.808563    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zscgf\" (UniqueName: \"kubernetes.io/projected/8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5-kube-api-access-zscgf\") pod \"metrics-server-79c798ccf6-c9h4r\" (UID: \"8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5\") " pod="openshift-monitoring/metrics-server-79c798ccf6-c9h4r"
May 26 11:25:43.943247 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:43.943143    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-8t9bl"]
May 26 11:25:43.945690 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:43.945665    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-8t9bl"
May 26 11:25:43.948247 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:43.948223    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
May 26 11:25:43.948247 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:43.948241    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"default-dockercfg-qgllh\""
May 26 11:25:43.953331 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:43.953300    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-8t9bl"]
May 26 11:25:43.959421 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:43.959399    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-79c798ccf6-c9h4r"
May 26 11:25:44.000912 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:44.000864    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/741322d5-b3e1-4de0-9233-976c7fb37c58-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-8t9bl\" (UID: \"741322d5-b3e1-4de0-9233-976c7fb37c58\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-8t9bl"
May 26 11:25:44.101518 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:44.101484    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/741322d5-b3e1-4de0-9233-976c7fb37c58-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-8t9bl\" (UID: \"741322d5-b3e1-4de0-9233-976c7fb37c58\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-8t9bl"
May 26 11:25:44.104344 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:44.104312    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/741322d5-b3e1-4de0-9233-976c7fb37c58-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-8t9bl\" (UID: \"741322d5-b3e1-4de0-9233-976c7fb37c58\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-8t9bl"
May 26 11:25:44.257479 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:44.257386    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-8t9bl"
May 26 11:25:45.404551 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.404508    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 26 11:25:45.407371 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.407343    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.410631 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.410605    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
May 26 11:25:45.410878 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.410859    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
May 26 11:25:45.410981 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.410895    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
May 26 11:25:45.410981 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.410939    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4vkzp\""
May 26 11:25:45.411088 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.410991    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
May 26 11:25:45.411221 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.411177    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
May 26 11:25:45.411980 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.411959    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
May 26 11:25:45.412552 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.412183    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
May 26 11:25:45.412552 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.412217    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
May 26 11:25:45.412552 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.412263    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
May 26 11:25:45.412552 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.412351    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-a0ubpmc3n9ig7\""
May 26 11:25:45.412552 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.412405    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
May 26 11:25:45.412887 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.412714    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
May 26 11:25:45.415150 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.414592    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
May 26 11:25:45.425451 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.425373    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 26 11:25:45.511413 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.511365    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0f6b7d4-fb12-42f6-b251-8de26af96192-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.511580 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.511423    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a0f6b7d4-fb12-42f6-b251-8de26af96192-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.511580 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.511461    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-web-config\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.511580 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.511485    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.511580 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.511510    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.511580 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.511538    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a0f6b7d4-fb12-42f6-b251-8de26af96192-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.511580 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.511561    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0f6b7d4-fb12-42f6-b251-8de26af96192-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.511900 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.511585    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.511900 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.511613    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0f6b7d4-fb12-42f6-b251-8de26af96192-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.511900 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.511642    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0f6b7d4-fb12-42f6-b251-8de26af96192-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.511900 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.511668    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-config\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.511900 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.511691    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.511900 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.511726    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.511900 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.511751    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.511900 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.511790    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0f6b7d4-fb12-42f6-b251-8de26af96192-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.512258 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.511926    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.512258 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.512024    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0f6b7d4-fb12-42f6-b251-8de26af96192-config-out\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.512258 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.512062    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnxm5\" (UniqueName: \"kubernetes.io/projected/a0f6b7d4-fb12-42f6-b251-8de26af96192-kube-api-access-gnxm5\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.612902 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.612859    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a0f6b7d4-fb12-42f6-b251-8de26af96192-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.612902 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.612907    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0f6b7d4-fb12-42f6-b251-8de26af96192-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.613159 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.612940    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.613159 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.612971    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0f6b7d4-fb12-42f6-b251-8de26af96192-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.613159 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.613003    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0f6b7d4-fb12-42f6-b251-8de26af96192-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.613159 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.613030    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-config\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.613159 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.613049    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.613159 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.613088    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.613159 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.613116    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.613159 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.613141    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0f6b7d4-fb12-42f6-b251-8de26af96192-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.613556 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.613175    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.613556 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.613212    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0f6b7d4-fb12-42f6-b251-8de26af96192-config-out\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.613556 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.613236    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gnxm5\" (UniqueName: \"kubernetes.io/projected/a0f6b7d4-fb12-42f6-b251-8de26af96192-kube-api-access-gnxm5\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.613556 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.613274    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0f6b7d4-fb12-42f6-b251-8de26af96192-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.613556 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.613308    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a0f6b7d4-fb12-42f6-b251-8de26af96192-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.613556 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.613328    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a0f6b7d4-fb12-42f6-b251-8de26af96192-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.614581 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.613963    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0f6b7d4-fb12-42f6-b251-8de26af96192-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.616916 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.616883    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a0f6b7d4-fb12-42f6-b251-8de26af96192-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.617947 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.617924    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-web-config\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.619440 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.619409    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.619545 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.619454    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.619673 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.619642    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0f6b7d4-fb12-42f6-b251-8de26af96192-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.619809 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.619029    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0f6b7d4-fb12-42f6-b251-8de26af96192-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.619809 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.619336    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.619982 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.619961    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0f6b7d4-fb12-42f6-b251-8de26af96192-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.621743 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.621720    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.621945 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.621923    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.622260 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.622218    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0f6b7d4-fb12-42f6-b251-8de26af96192-config-out\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.622401 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.622382    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-config\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.622800 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.622484    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.622800 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.622482    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.622800 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.622759    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-web-config\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.623560 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.623332    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.623560 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.623523    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.624130 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.624045    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0f6b7d4-fb12-42f6-b251-8de26af96192-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.626865 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.626847    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnxm5\" (UniqueName: \"kubernetes.io/projected/a0f6b7d4-fb12-42f6-b251-8de26af96192-kube-api-access-gnxm5\") pod \"prometheus-k8s-0\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:45.723272 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:45.723183    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:25:48.658711 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:48.658180    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-79c798ccf6-c9h4r"]
May 26 11:25:48.664947 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:25:48.664919    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c811a1c_f0c8_4f89_8c13_e9adf2a76ba5.slice/crio-9bf0dddfb50e4de55d975e1d0e72513eb1f6524e0574251fcea469fbf8351751 WatchSource:0}: Error finding container 9bf0dddfb50e4de55d975e1d0e72513eb1f6524e0574251fcea469fbf8351751: Status 404 returned error can't find the container with id 9bf0dddfb50e4de55d975e1d0e72513eb1f6524e0574251fcea469fbf8351751
May 26 11:25:48.697259 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:48.697215    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 26 11:25:48.700409 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:25:48.700382    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f6b7d4_fb12_42f6_b251_8de26af96192.slice/crio-3b9db51d286567ae600702b23b571803a2f402fb1dfccbc56f3711cca5f7a214 WatchSource:0}: Error finding container 3b9db51d286567ae600702b23b571803a2f402fb1dfccbc56f3711cca5f7a214: Status 404 returned error can't find the container with id 3b9db51d286567ae600702b23b571803a2f402fb1dfccbc56f3711cca5f7a214
May 26 11:25:48.707364 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:48.707339    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-8t9bl"]
May 26 11:25:48.711029 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:25:48.711006    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod741322d5_b3e1_4de0_9233_976c7fb37c58.slice/crio-00c983f19e40d2342d798986900a9fd6614fa72ba1d33788a3347ce46156bc9b WatchSource:0}: Error finding container 00c983f19e40d2342d798986900a9fd6614fa72ba1d33788a3347ce46156bc9b: Status 404 returned error can't find the container with id 00c983f19e40d2342d798986900a9fd6614fa72ba1d33788a3347ce46156bc9b
May 26 11:25:48.735306 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:48.735273    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-fhfpk" event={"ID":"e0c32514-7fd6-461c-bf92-092ad4ed740e","Type":"ContainerStarted","Data":"4457e42e74343c82c0cd708abe5ac81b01db91cd4739fed3d7d01752699905fd"}
May 26 11:25:48.735614 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:48.735591    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-57bd96b888-fhfpk"
May 26 11:25:48.736737 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:48.736710    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0f6b7d4-fb12-42f6-b251-8de26af96192","Type":"ContainerStarted","Data":"3b9db51d286567ae600702b23b571803a2f402fb1dfccbc56f3711cca5f7a214"}
May 26 11:25:48.736896 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:48.736789    2563 patch_prober.go:28] interesting pod/downloads-57bd96b888-fhfpk container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.10:8080/\": dial tcp 10.134.0.10:8080: connect: connection refused" start-of-body=
May 26 11:25:48.736896 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:48.736854    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-57bd96b888-fhfpk" podUID="e0c32514-7fd6-461c-bf92-092ad4ed740e" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.10:8080/\": dial tcp 10.134.0.10:8080: connect: connection refused"
May 26 11:25:48.737816 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:48.737769    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-79c798ccf6-c9h4r" event={"ID":"8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5","Type":"ContainerStarted","Data":"9bf0dddfb50e4de55d975e1d0e72513eb1f6524e0574251fcea469fbf8351751"}
May 26 11:25:48.739086 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:48.739058    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-8t9bl" event={"ID":"741322d5-b3e1-4de0-9233-976c7fb37c58","Type":"ContainerStarted","Data":"00c983f19e40d2342d798986900a9fd6614fa72ba1d33788a3347ce46156bc9b"}
May 26 11:25:48.740593 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:48.740572    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c5zsn" event={"ID":"cf8f06cf-5a6e-45cd-b44d-2817b037ed24","Type":"ContainerStarted","Data":"1954f2dfb55def020e188e0270d602727275dff3615f1fc967e37fad57767fa8"}
May 26 11:25:48.753464 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:48.753413    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/downloads-57bd96b888-fhfpk" podStartSLOduration=0.667941678 podStartE2EDuration="19.75340334s" podCreationTimestamp="2026-05-26 11:25:29 +0000 UTC" firstStartedPulling="2026-05-26 11:25:29.504461729 +0000 UTC m=+61.695903910" lastFinishedPulling="2026-05-26 11:25:48.589923401 +0000 UTC m=+80.781365572" observedRunningTime="2026-05-26 11:25:48.752548053 +0000 UTC m=+80.943990239" watchObservedRunningTime="2026-05-26 11:25:48.75340334 +0000 UTC m=+80.944845527"
May 26 11:25:49.662821 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:49.662153    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7c89b4489d-2rf2s"
May 26 11:25:49.746020 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:49.745938    2563 generic.go:358] "Generic (PLEG): container finished" podID="cf8f06cf-5a6e-45cd-b44d-2817b037ed24" containerID="1954f2dfb55def020e188e0270d602727275dff3615f1fc967e37fad57767fa8" exitCode=0
May 26 11:25:49.746764 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:49.746721    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c5zsn" event={"ID":"cf8f06cf-5a6e-45cd-b44d-2817b037ed24","Type":"ContainerDied","Data":"1954f2dfb55def020e188e0270d602727275dff3615f1fc967e37fad57767fa8"}
May 26 11:25:49.754735 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:49.754691    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-57bd96b888-fhfpk"
May 26 11:25:52.757914 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:52.757830    2563 generic.go:358] "Generic (PLEG): container finished" podID="a0f6b7d4-fb12-42f6-b251-8de26af96192" containerID="81fae885c52e5fab4f791ca43b4ea8088dc3491be78573f92f1e2ba3aa1472c6" exitCode=0
May 26 11:25:52.758352 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:52.757925    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0f6b7d4-fb12-42f6-b251-8de26af96192","Type":"ContainerDied","Data":"81fae885c52e5fab4f791ca43b4ea8088dc3491be78573f92f1e2ba3aa1472c6"}
May 26 11:25:52.759572 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:52.759528    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-79c798ccf6-c9h4r" event={"ID":"8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5","Type":"ContainerStarted","Data":"8be39fbc60e9e28129f39f34180e87091c939afa330af4f361b2a4673d7126bf"}
May 26 11:25:52.761926 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:52.761899    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-8t9bl" event={"ID":"741322d5-b3e1-4de0-9233-976c7fb37c58","Type":"ContainerStarted","Data":"4760f733c8b578c5318ae023b5deb632daa3776c0097e5a8f11578dd7c74bf68"}
May 26 11:25:52.762347 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:52.762329    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-8t9bl"
May 26 11:25:52.764402 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:52.764377    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c5zsn" event={"ID":"cf8f06cf-5a6e-45cd-b44d-2817b037ed24","Type":"ContainerStarted","Data":"1cba886442a171d7f67b2dd32f134cbf0ab5a1bbcfa26c9b92066104fe1948da"}
May 26 11:25:52.764507 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:52.764411    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c5zsn" event={"ID":"cf8f06cf-5a6e-45cd-b44d-2817b037ed24","Type":"ContainerStarted","Data":"1b5a68a6196aa996e50555b7ad7d1a5c09ae252df9a0bbb1b18209dc5e8668aa"}
May 26 11:25:52.768697 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:52.768674    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-8t9bl"
May 26 11:25:52.818860 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:52.818800    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-665c9c8584-8t9bl" podStartSLOduration=6.491055976 podStartE2EDuration="9.81876429s" podCreationTimestamp="2026-05-26 11:25:43 +0000 UTC" firstStartedPulling="2026-05-26 11:25:48.71321544 +0000 UTC m=+80.904657607" lastFinishedPulling="2026-05-26 11:25:52.040923744 +0000 UTC m=+84.232365921" observedRunningTime="2026-05-26 11:25:52.818359601 +0000 UTC m=+85.009801789" watchObservedRunningTime="2026-05-26 11:25:52.81876429 +0000 UTC m=+85.010206479"
May 26 11:25:52.843836 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:52.843761    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-c5zsn" podStartSLOduration=5.467043833 podStartE2EDuration="13.843741712s" podCreationTimestamp="2026-05-26 11:25:39 +0000 UTC" firstStartedPulling="2026-05-26 11:25:40.111474747 +0000 UTC m=+72.302916912" lastFinishedPulling="2026-05-26 11:25:48.488172613 +0000 UTC m=+80.679614791" observedRunningTime="2026-05-26 11:25:52.841995 +0000 UTC m=+85.033437189" watchObservedRunningTime="2026-05-26 11:25:52.843741712 +0000 UTC m=+85.035183899"
May 26 11:25:52.869815 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:52.869732    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-79c798ccf6-c9h4r" podStartSLOduration=6.494071006 podStartE2EDuration="9.869711722s" podCreationTimestamp="2026-05-26 11:25:43 +0000 UTC" firstStartedPulling="2026-05-26 11:25:48.667735562 +0000 UTC m=+80.859177727" lastFinishedPulling="2026-05-26 11:25:52.043376263 +0000 UTC m=+84.234818443" observedRunningTime="2026-05-26 11:25:52.868609364 +0000 UTC m=+85.060051552" watchObservedRunningTime="2026-05-26 11:25:52.869711722 +0000 UTC m=+85.061153909"
May 26 11:25:57.785817 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:57.785757    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0f6b7d4-fb12-42f6-b251-8de26af96192","Type":"ContainerStarted","Data":"a24f56d7e97649f69432dce8cc89ecf5aa0debd7d2de66ec9408c3f257145cae"}
May 26 11:25:57.785817 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:57.785819    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0f6b7d4-fb12-42f6-b251-8de26af96192","Type":"ContainerStarted","Data":"3444e1cce7f360073ac74d554755a1aeddaf80282671b8986164f87bc9bd618f"}
May 26 11:25:59.796627 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:59.796594    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0f6b7d4-fb12-42f6-b251-8de26af96192","Type":"ContainerStarted","Data":"8d4d79edd934ae24823b368487dd0806f46a3afc5a18545ee81fe1fd8d690448"}
May 26 11:25:59.796957 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:25:59.796640    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0f6b7d4-fb12-42f6-b251-8de26af96192","Type":"ContainerStarted","Data":"3d52a238e5c3b01fc16077b1f7144176405e5350838f7d5e747f43e14958b291"}
May 26 11:26:00.802400 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:26:00.802360    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0f6b7d4-fb12-42f6-b251-8de26af96192","Type":"ContainerStarted","Data":"75a2b0186de56b4978957285cc92dc86c8c5d08be2c935fc86647c1325e92bf7"}
May 26 11:26:00.802400 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:26:00.802397    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0f6b7d4-fb12-42f6-b251-8de26af96192","Type":"ContainerStarted","Data":"bc991c6914a66c64b7013bae520c01e3af7694346dbca7b9efef71e46b8f81fb"}
May 26 11:26:00.833814 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:26:00.833747    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=5.072483259 podStartE2EDuration="15.833731203s" podCreationTimestamp="2026-05-26 11:25:45 +0000 UTC" firstStartedPulling="2026-05-26 11:25:48.702812958 +0000 UTC m=+80.894255123" lastFinishedPulling="2026-05-26 11:25:59.464060901 +0000 UTC m=+91.655503067" observedRunningTime="2026-05-26 11:26:00.83046615 +0000 UTC m=+93.021908337" watchObservedRunningTime="2026-05-26 11:26:00.833731203 +0000 UTC m=+93.025173425"
May 26 11:26:03.960413 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:26:03.960372    2563 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-79c798ccf6-c9h4r"
May 26 11:26:03.960413 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:26:03.960421    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-79c798ccf6-c9h4r"
May 26 11:26:05.724216 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:26:05.724178    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:26:09.707656 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:26:09.707623    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cbxbl"
May 26 11:26:13.541940 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:26:13.541913    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-d72ts_13698e7d-67cc-4e22-afdd-70f4b08b2819/serve-healthcheck-canary/0.log"
May 26 11:26:23.965943 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:26:23.965914    2563 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-79c798ccf6-c9h4r"
May 26 11:26:23.970190 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:26:23.970168    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-79c798ccf6-c9h4r"
May 26 11:26:45.724112 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:26:45.724069    2563 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:26:45.744302 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:26:45.744276    2563 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:26:45.939658 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:26:45.939631    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:03.790872 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:03.790836    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 26 11:27:03.792271 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:03.792236    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a0f6b7d4-fb12-42f6-b251-8de26af96192" containerName="kube-rbac-proxy-thanos" containerID="cri-o://75a2b0186de56b4978957285cc92dc86c8c5d08be2c935fc86647c1325e92bf7" gracePeriod=600
May 26 11:27:03.792562 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:03.792535    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a0f6b7d4-fb12-42f6-b251-8de26af96192" containerName="kube-rbac-proxy" containerID="cri-o://bc991c6914a66c64b7013bae520c01e3af7694346dbca7b9efef71e46b8f81fb" gracePeriod=600
May 26 11:27:03.792685 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:03.792449    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a0f6b7d4-fb12-42f6-b251-8de26af96192" containerName="prometheus" containerID="cri-o://3444e1cce7f360073ac74d554755a1aeddaf80282671b8986164f87bc9bd618f" gracePeriod=600
May 26 11:27:03.792685 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:03.792498    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a0f6b7d4-fb12-42f6-b251-8de26af96192" containerName="config-reloader" containerID="cri-o://a24f56d7e97649f69432dce8cc89ecf5aa0debd7d2de66ec9408c3f257145cae" gracePeriod=600
May 26 11:27:03.792685 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:03.792621    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a0f6b7d4-fb12-42f6-b251-8de26af96192" containerName="kube-rbac-proxy-web" containerID="cri-o://8d4d79edd934ae24823b368487dd0806f46a3afc5a18545ee81fe1fd8d690448" gracePeriod=600
May 26 11:27:03.792872 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:03.792725    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a0f6b7d4-fb12-42f6-b251-8de26af96192" containerName="thanos-sidecar" containerID="cri-o://3d52a238e5c3b01fc16077b1f7144176405e5350838f7d5e747f43e14958b291" gracePeriod=600
May 26 11:27:03.978131 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:03.978101    2563 generic.go:358] "Generic (PLEG): container finished" podID="a0f6b7d4-fb12-42f6-b251-8de26af96192" containerID="8d4d79edd934ae24823b368487dd0806f46a3afc5a18545ee81fe1fd8d690448" exitCode=0
May 26 11:27:03.978131 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:03.978127    2563 generic.go:358] "Generic (PLEG): container finished" podID="a0f6b7d4-fb12-42f6-b251-8de26af96192" containerID="3d52a238e5c3b01fc16077b1f7144176405e5350838f7d5e747f43e14958b291" exitCode=0
May 26 11:27:03.978131 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:03.978133    2563 generic.go:358] "Generic (PLEG): container finished" podID="a0f6b7d4-fb12-42f6-b251-8de26af96192" containerID="a24f56d7e97649f69432dce8cc89ecf5aa0debd7d2de66ec9408c3f257145cae" exitCode=0
May 26 11:27:03.978339 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:03.978157    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0f6b7d4-fb12-42f6-b251-8de26af96192","Type":"ContainerDied","Data":"8d4d79edd934ae24823b368487dd0806f46a3afc5a18545ee81fe1fd8d690448"}
May 26 11:27:03.978339 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:03.978183    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0f6b7d4-fb12-42f6-b251-8de26af96192","Type":"ContainerDied","Data":"3d52a238e5c3b01fc16077b1f7144176405e5350838f7d5e747f43e14958b291"}
May 26 11:27:03.978339 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:03.978192    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0f6b7d4-fb12-42f6-b251-8de26af96192","Type":"ContainerDied","Data":"a24f56d7e97649f69432dce8cc89ecf5aa0debd7d2de66ec9408c3f257145cae"}
May 26 11:27:04.983889 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:04.983860    2563 generic.go:358] "Generic (PLEG): container finished" podID="a0f6b7d4-fb12-42f6-b251-8de26af96192" containerID="75a2b0186de56b4978957285cc92dc86c8c5d08be2c935fc86647c1325e92bf7" exitCode=0
May 26 11:27:04.983889 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:04.983884    2563 generic.go:358] "Generic (PLEG): container finished" podID="a0f6b7d4-fb12-42f6-b251-8de26af96192" containerID="bc991c6914a66c64b7013bae520c01e3af7694346dbca7b9efef71e46b8f81fb" exitCode=0
May 26 11:27:04.984252 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:04.983934    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0f6b7d4-fb12-42f6-b251-8de26af96192","Type":"ContainerDied","Data":"75a2b0186de56b4978957285cc92dc86c8c5d08be2c935fc86647c1325e92bf7"}
May 26 11:27:04.984252 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:04.983968    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0f6b7d4-fb12-42f6-b251-8de26af96192","Type":"ContainerDied","Data":"bc991c6914a66c64b7013bae520c01e3af7694346dbca7b9efef71e46b8f81fb"}
May 26 11:27:05.925107 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:27:05.925062    2563 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="3444e1cce7f360073ac74d554755a1aeddaf80282671b8986164f87bc9bd618f" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
May 26 11:27:05.926095 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:27:05.926055    2563 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="3444e1cce7f360073ac74d554755a1aeddaf80282671b8986164f87bc9bd618f" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
May 26 11:27:05.926892 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:27:05.926859    2563 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="3444e1cce7f360073ac74d554755a1aeddaf80282671b8986164f87bc9bd618f" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
May 26 11:27:05.926980 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:27:05.926906    2563 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="a0f6b7d4-fb12-42f6-b251-8de26af96192" containerName="prometheus" probeResult="unknown"
May 26 11:27:10.924209 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:27:10.924168    2563 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 3444e1cce7f360073ac74d554755a1aeddaf80282671b8986164f87bc9bd618f is running failed: container process not found" containerID="3444e1cce7f360073ac74d554755a1aeddaf80282671b8986164f87bc9bd618f" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
May 26 11:27:10.924548 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:27:10.924510    2563 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 3444e1cce7f360073ac74d554755a1aeddaf80282671b8986164f87bc9bd618f is running failed: container process not found" containerID="3444e1cce7f360073ac74d554755a1aeddaf80282671b8986164f87bc9bd618f" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
May 26 11:27:10.924841 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:27:10.924813    2563 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 3444e1cce7f360073ac74d554755a1aeddaf80282671b8986164f87bc9bd618f is running failed: container process not found" containerID="3444e1cce7f360073ac74d554755a1aeddaf80282671b8986164f87bc9bd618f" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
May 26 11:27:10.924912 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:27:10.924853    2563 prober.go:104] "Probe errored" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 3444e1cce7f360073ac74d554755a1aeddaf80282671b8986164f87bc9bd618f is running failed: container process not found" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="a0f6b7d4-fb12-42f6-b251-8de26af96192" containerName="prometheus" probeResult="unknown"
May 26 11:27:11.134573 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.134549    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:11.209630 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.209544    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0f6b7d4-fb12-42f6-b251-8de26af96192-tls-assets\") pod \"a0f6b7d4-fb12-42f6-b251-8de26af96192\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") "
May 26 11:27:11.209630 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.209585    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"a0f6b7d4-fb12-42f6-b251-8de26af96192\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") "
May 26 11:27:11.209630 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.209618    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0f6b7d4-fb12-42f6-b251-8de26af96192-configmap-metrics-client-ca\") pod \"a0f6b7d4-fb12-42f6-b251-8de26af96192\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") "
May 26 11:27:11.209946 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.209655    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0f6b7d4-fb12-42f6-b251-8de26af96192-prometheus-trusted-ca-bundle\") pod \"a0f6b7d4-fb12-42f6-b251-8de26af96192\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") "
May 26 11:27:11.209946 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.209677    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0f6b7d4-fb12-42f6-b251-8de26af96192-configmap-serving-certs-ca-bundle\") pod \"a0f6b7d4-fb12-42f6-b251-8de26af96192\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") "
May 26 11:27:11.209946 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.209710    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-secret-kube-rbac-proxy\") pod \"a0f6b7d4-fb12-42f6-b251-8de26af96192\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") "
May 26 11:27:11.210090 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.210033    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnxm5\" (UniqueName: \"kubernetes.io/projected/a0f6b7d4-fb12-42f6-b251-8de26af96192-kube-api-access-gnxm5\") pod \"a0f6b7d4-fb12-42f6-b251-8de26af96192\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") "
May 26 11:27:11.210157 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.210092    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-web-config\") pod \"a0f6b7d4-fb12-42f6-b251-8de26af96192\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") "
May 26 11:27:11.210157 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.210132    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0f6b7d4-fb12-42f6-b251-8de26af96192-configmap-kubelet-serving-ca-bundle\") pod \"a0f6b7d4-fb12-42f6-b251-8de26af96192\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") "
May 26 11:27:11.210253 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.210157    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-thanos-prometheus-http-client-file\") pod \"a0f6b7d4-fb12-42f6-b251-8de26af96192\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") "
May 26 11:27:11.210253 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.210172    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0f6b7d4-fb12-42f6-b251-8de26af96192-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "a0f6b7d4-fb12-42f6-b251-8de26af96192" (UID: "a0f6b7d4-fb12-42f6-b251-8de26af96192"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:27:11.210253 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.210183    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-config\") pod \"a0f6b7d4-fb12-42f6-b251-8de26af96192\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") "
May 26 11:27:11.210253 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.210213    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a0f6b7d4-fb12-42f6-b251-8de26af96192-prometheus-k8s-rulefiles-0\") pod \"a0f6b7d4-fb12-42f6-b251-8de26af96192\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") "
May 26 11:27:11.210253 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.210222    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0f6b7d4-fb12-42f6-b251-8de26af96192-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "a0f6b7d4-fb12-42f6-b251-8de26af96192" (UID: "a0f6b7d4-fb12-42f6-b251-8de26af96192"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:27:11.210664 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.210252    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0f6b7d4-fb12-42f6-b251-8de26af96192-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "a0f6b7d4-fb12-42f6-b251-8de26af96192" (UID: "a0f6b7d4-fb12-42f6-b251-8de26af96192"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:27:11.210664 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.210578    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0f6b7d4-fb12-42f6-b251-8de26af96192-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "a0f6b7d4-fb12-42f6-b251-8de26af96192" (UID: "a0f6b7d4-fb12-42f6-b251-8de26af96192"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:27:11.210664 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.210625    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-secret-metrics-client-certs\") pod \"a0f6b7d4-fb12-42f6-b251-8de26af96192\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") "
May 26 11:27:11.210664 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.210655    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-secret-prometheus-k8s-tls\") pod \"a0f6b7d4-fb12-42f6-b251-8de26af96192\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") "
May 26 11:27:11.210899 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.210704    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0f6b7d4-fb12-42f6-b251-8de26af96192-config-out\") pod \"a0f6b7d4-fb12-42f6-b251-8de26af96192\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") "
May 26 11:27:11.210957 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.210916    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-secret-grpc-tls\") pod \"a0f6b7d4-fb12-42f6-b251-8de26af96192\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") "
May 26 11:27:11.210957 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.210949    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a0f6b7d4-fb12-42f6-b251-8de26af96192-prometheus-k8s-db\") pod \"a0f6b7d4-fb12-42f6-b251-8de26af96192\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") "
May 26 11:27:11.211060 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.210976    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"a0f6b7d4-fb12-42f6-b251-8de26af96192\" (UID: \"a0f6b7d4-fb12-42f6-b251-8de26af96192\") "
May 26 11:27:11.211382 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.211203    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0f6b7d4-fb12-42f6-b251-8de26af96192-configmap-metrics-client-ca\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:27:11.211382 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.211226    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0f6b7d4-fb12-42f6-b251-8de26af96192-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:27:11.211382 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.211243    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0f6b7d4-fb12-42f6-b251-8de26af96192-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:27:11.211382 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.211260    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0f6b7d4-fb12-42f6-b251-8de26af96192-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:27:11.213147 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.213098    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0f6b7d4-fb12-42f6-b251-8de26af96192-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "a0f6b7d4-fb12-42f6-b251-8de26af96192" (UID: "a0f6b7d4-fb12-42f6-b251-8de26af96192"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:27:11.213588 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.213541    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "a0f6b7d4-fb12-42f6-b251-8de26af96192" (UID: "a0f6b7d4-fb12-42f6-b251-8de26af96192"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:27:11.214092 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.214067    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-config" (OuterVolumeSpecName: "config") pod "a0f6b7d4-fb12-42f6-b251-8de26af96192" (UID: "a0f6b7d4-fb12-42f6-b251-8de26af96192"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:27:11.214177 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.214134    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "a0f6b7d4-fb12-42f6-b251-8de26af96192" (UID: "a0f6b7d4-fb12-42f6-b251-8de26af96192"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:27:11.214296 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.214274    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f6b7d4-fb12-42f6-b251-8de26af96192-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "a0f6b7d4-fb12-42f6-b251-8de26af96192" (UID: "a0f6b7d4-fb12-42f6-b251-8de26af96192"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:27:11.214418 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.214400    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "a0f6b7d4-fb12-42f6-b251-8de26af96192" (UID: "a0f6b7d4-fb12-42f6-b251-8de26af96192"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:27:11.214732 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.214691    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0f6b7d4-fb12-42f6-b251-8de26af96192-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a0f6b7d4-fb12-42f6-b251-8de26af96192" (UID: "a0f6b7d4-fb12-42f6-b251-8de26af96192"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:27:11.215086 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.215057    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "a0f6b7d4-fb12-42f6-b251-8de26af96192" (UID: "a0f6b7d4-fb12-42f6-b251-8de26af96192"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:27:11.215235 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.215215    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "a0f6b7d4-fb12-42f6-b251-8de26af96192" (UID: "a0f6b7d4-fb12-42f6-b251-8de26af96192"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:27:11.215666 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.215641    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "a0f6b7d4-fb12-42f6-b251-8de26af96192" (UID: "a0f6b7d4-fb12-42f6-b251-8de26af96192"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:27:11.216358 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.216334    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f6b7d4-fb12-42f6-b251-8de26af96192-config-out" (OuterVolumeSpecName: "config-out") pod "a0f6b7d4-fb12-42f6-b251-8de26af96192" (UID: "a0f6b7d4-fb12-42f6-b251-8de26af96192"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:27:11.217049 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.217027    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "a0f6b7d4-fb12-42f6-b251-8de26af96192" (UID: "a0f6b7d4-fb12-42f6-b251-8de26af96192"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:27:11.217301 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.217136    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0f6b7d4-fb12-42f6-b251-8de26af96192-kube-api-access-gnxm5" (OuterVolumeSpecName: "kube-api-access-gnxm5") pod "a0f6b7d4-fb12-42f6-b251-8de26af96192" (UID: "a0f6b7d4-fb12-42f6-b251-8de26af96192"). InnerVolumeSpecName "kube-api-access-gnxm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:27:11.225750 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.225707    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-web-config" (OuterVolumeSpecName: "web-config") pod "a0f6b7d4-fb12-42f6-b251-8de26af96192" (UID: "a0f6b7d4-fb12-42f6-b251-8de26af96192"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:27:11.312586 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.312550    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a0f6b7d4-fb12-42f6-b251-8de26af96192-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:27:11.312586 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.312586    2563 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-secret-metrics-client-certs\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:27:11.312811 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.312598    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:27:11.312811 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.312608    2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0f6b7d4-fb12-42f6-b251-8de26af96192-config-out\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:27:11.312811 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.312618    2563 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-secret-grpc-tls\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:27:11.312811 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.312625    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a0f6b7d4-fb12-42f6-b251-8de26af96192-prometheus-k8s-db\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:27:11.312811 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.312635    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:27:11.312811 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.312644    2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0f6b7d4-fb12-42f6-b251-8de26af96192-tls-assets\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:27:11.312811 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.312653    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:27:11.312811 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.312664    2563 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-secret-kube-rbac-proxy\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:27:11.312811 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.312673    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnxm5\" (UniqueName: \"kubernetes.io/projected/a0f6b7d4-fb12-42f6-b251-8de26af96192-kube-api-access-gnxm5\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:27:11.312811 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.312681    2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-web-config\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:27:11.312811 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.312690    2563 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:27:11.312811 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:11.312698    2563 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/a0f6b7d4-fb12-42f6-b251-8de26af96192-config\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:27:12.005200 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.005160    2563 generic.go:358] "Generic (PLEG): container finished" podID="a0f6b7d4-fb12-42f6-b251-8de26af96192" containerID="3444e1cce7f360073ac74d554755a1aeddaf80282671b8986164f87bc9bd618f" exitCode=0
May 26 11:27:12.005562 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.005213    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0f6b7d4-fb12-42f6-b251-8de26af96192","Type":"ContainerDied","Data":"3444e1cce7f360073ac74d554755a1aeddaf80282671b8986164f87bc9bd618f"}
May 26 11:27:12.005562 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.005237    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0f6b7d4-fb12-42f6-b251-8de26af96192","Type":"ContainerDied","Data":"3b9db51d286567ae600702b23b571803a2f402fb1dfccbc56f3711cca5f7a214"}
May 26 11:27:12.005562 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.005255    2563 scope.go:117] "RemoveContainer" containerID="75a2b0186de56b4978957285cc92dc86c8c5d08be2c935fc86647c1325e92bf7"
May 26 11:27:12.005562 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.005290    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.013156 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.013109    2563 scope.go:117] "RemoveContainer" containerID="bc991c6914a66c64b7013bae520c01e3af7694346dbca7b9efef71e46b8f81fb"
May 26 11:27:12.019990 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.019971    2563 scope.go:117] "RemoveContainer" containerID="8d4d79edd934ae24823b368487dd0806f46a3afc5a18545ee81fe1fd8d690448"
May 26 11:27:12.026278 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.026258    2563 scope.go:117] "RemoveContainer" containerID="3d52a238e5c3b01fc16077b1f7144176405e5350838f7d5e747f43e14958b291"
May 26 11:27:12.032303 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.032275    2563 scope.go:117] "RemoveContainer" containerID="a24f56d7e97649f69432dce8cc89ecf5aa0debd7d2de66ec9408c3f257145cae"
May 26 11:27:12.038592 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.038572    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 26 11:27:12.039279 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.039258    2563 scope.go:117] "RemoveContainer" containerID="3444e1cce7f360073ac74d554755a1aeddaf80282671b8986164f87bc9bd618f"
May 26 11:27:12.045704 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.045688    2563 scope.go:117] "RemoveContainer" containerID="81fae885c52e5fab4f791ca43b4ea8088dc3491be78573f92f1e2ba3aa1472c6"
May 26 11:27:12.053421 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.053402    2563 scope.go:117] "RemoveContainer" containerID="75a2b0186de56b4978957285cc92dc86c8c5d08be2c935fc86647c1325e92bf7"
May 26 11:27:12.053680 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:27:12.053658    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75a2b0186de56b4978957285cc92dc86c8c5d08be2c935fc86647c1325e92bf7\": container with ID starting with 75a2b0186de56b4978957285cc92dc86c8c5d08be2c935fc86647c1325e92bf7 not found: ID does not exist" containerID="75a2b0186de56b4978957285cc92dc86c8c5d08be2c935fc86647c1325e92bf7"
May 26 11:27:12.053790 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.053699    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75a2b0186de56b4978957285cc92dc86c8c5d08be2c935fc86647c1325e92bf7"} err="failed to get container status \"75a2b0186de56b4978957285cc92dc86c8c5d08be2c935fc86647c1325e92bf7\": rpc error: code = NotFound desc = could not find container \"75a2b0186de56b4978957285cc92dc86c8c5d08be2c935fc86647c1325e92bf7\": container with ID starting with 75a2b0186de56b4978957285cc92dc86c8c5d08be2c935fc86647c1325e92bf7 not found: ID does not exist"
May 26 11:27:12.053790 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.053736    2563 scope.go:117] "RemoveContainer" containerID="bc991c6914a66c64b7013bae520c01e3af7694346dbca7b9efef71e46b8f81fb"
May 26 11:27:12.053975 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:27:12.053959    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc991c6914a66c64b7013bae520c01e3af7694346dbca7b9efef71e46b8f81fb\": container with ID starting with bc991c6914a66c64b7013bae520c01e3af7694346dbca7b9efef71e46b8f81fb not found: ID does not exist" containerID="bc991c6914a66c64b7013bae520c01e3af7694346dbca7b9efef71e46b8f81fb"
May 26 11:27:12.054033 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.053981    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc991c6914a66c64b7013bae520c01e3af7694346dbca7b9efef71e46b8f81fb"} err="failed to get container status \"bc991c6914a66c64b7013bae520c01e3af7694346dbca7b9efef71e46b8f81fb\": rpc error: code = NotFound desc = could not find container \"bc991c6914a66c64b7013bae520c01e3af7694346dbca7b9efef71e46b8f81fb\": container with ID starting with bc991c6914a66c64b7013bae520c01e3af7694346dbca7b9efef71e46b8f81fb not found: ID does not exist"
May 26 11:27:12.054033 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.053997    2563 scope.go:117] "RemoveContainer" containerID="8d4d79edd934ae24823b368487dd0806f46a3afc5a18545ee81fe1fd8d690448"
May 26 11:27:12.054251 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:27:12.054232    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d4d79edd934ae24823b368487dd0806f46a3afc5a18545ee81fe1fd8d690448\": container with ID starting with 8d4d79edd934ae24823b368487dd0806f46a3afc5a18545ee81fe1fd8d690448 not found: ID does not exist" containerID="8d4d79edd934ae24823b368487dd0806f46a3afc5a18545ee81fe1fd8d690448"
May 26 11:27:12.054296 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.054259    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d4d79edd934ae24823b368487dd0806f46a3afc5a18545ee81fe1fd8d690448"} err="failed to get container status \"8d4d79edd934ae24823b368487dd0806f46a3afc5a18545ee81fe1fd8d690448\": rpc error: code = NotFound desc = could not find container \"8d4d79edd934ae24823b368487dd0806f46a3afc5a18545ee81fe1fd8d690448\": container with ID starting with 8d4d79edd934ae24823b368487dd0806f46a3afc5a18545ee81fe1fd8d690448 not found: ID does not exist"
May 26 11:27:12.054296 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.054277    2563 scope.go:117] "RemoveContainer" containerID="3d52a238e5c3b01fc16077b1f7144176405e5350838f7d5e747f43e14958b291"
May 26 11:27:12.054516 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:27:12.054496    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d52a238e5c3b01fc16077b1f7144176405e5350838f7d5e747f43e14958b291\": container with ID starting with 3d52a238e5c3b01fc16077b1f7144176405e5350838f7d5e747f43e14958b291 not found: ID does not exist" containerID="3d52a238e5c3b01fc16077b1f7144176405e5350838f7d5e747f43e14958b291"
May 26 11:27:12.054605 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.054515    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d52a238e5c3b01fc16077b1f7144176405e5350838f7d5e747f43e14958b291"} err="failed to get container status \"3d52a238e5c3b01fc16077b1f7144176405e5350838f7d5e747f43e14958b291\": rpc error: code = NotFound desc = could not find container \"3d52a238e5c3b01fc16077b1f7144176405e5350838f7d5e747f43e14958b291\": container with ID starting with 3d52a238e5c3b01fc16077b1f7144176405e5350838f7d5e747f43e14958b291 not found: ID does not exist"
May 26 11:27:12.054605 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.054528    2563 scope.go:117] "RemoveContainer" containerID="a24f56d7e97649f69432dce8cc89ecf5aa0debd7d2de66ec9408c3f257145cae"
May 26 11:27:12.054850 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:27:12.054820    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a24f56d7e97649f69432dce8cc89ecf5aa0debd7d2de66ec9408c3f257145cae\": container with ID starting with a24f56d7e97649f69432dce8cc89ecf5aa0debd7d2de66ec9408c3f257145cae not found: ID does not exist" containerID="a24f56d7e97649f69432dce8cc89ecf5aa0debd7d2de66ec9408c3f257145cae"
May 26 11:27:12.054930 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.054852    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a24f56d7e97649f69432dce8cc89ecf5aa0debd7d2de66ec9408c3f257145cae"} err="failed to get container status \"a24f56d7e97649f69432dce8cc89ecf5aa0debd7d2de66ec9408c3f257145cae\": rpc error: code = NotFound desc = could not find container \"a24f56d7e97649f69432dce8cc89ecf5aa0debd7d2de66ec9408c3f257145cae\": container with ID starting with a24f56d7e97649f69432dce8cc89ecf5aa0debd7d2de66ec9408c3f257145cae not found: ID does not exist"
May 26 11:27:12.054930 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.054868    2563 scope.go:117] "RemoveContainer" containerID="3444e1cce7f360073ac74d554755a1aeddaf80282671b8986164f87bc9bd618f"
May 26 11:27:12.055511 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:27:12.055489    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3444e1cce7f360073ac74d554755a1aeddaf80282671b8986164f87bc9bd618f\": container with ID starting with 3444e1cce7f360073ac74d554755a1aeddaf80282671b8986164f87bc9bd618f not found: ID does not exist" containerID="3444e1cce7f360073ac74d554755a1aeddaf80282671b8986164f87bc9bd618f"
May 26 11:27:12.055588 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.055516    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3444e1cce7f360073ac74d554755a1aeddaf80282671b8986164f87bc9bd618f"} err="failed to get container status \"3444e1cce7f360073ac74d554755a1aeddaf80282671b8986164f87bc9bd618f\": rpc error: code = NotFound desc = could not find container \"3444e1cce7f360073ac74d554755a1aeddaf80282671b8986164f87bc9bd618f\": container with ID starting with 3444e1cce7f360073ac74d554755a1aeddaf80282671b8986164f87bc9bd618f not found: ID does not exist"
May 26 11:27:12.055588 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.055538    2563 scope.go:117] "RemoveContainer" containerID="81fae885c52e5fab4f791ca43b4ea8088dc3491be78573f92f1e2ba3aa1472c6"
May 26 11:27:12.055825 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:27:12.055757    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81fae885c52e5fab4f791ca43b4ea8088dc3491be78573f92f1e2ba3aa1472c6\": container with ID starting with 81fae885c52e5fab4f791ca43b4ea8088dc3491be78573f92f1e2ba3aa1472c6 not found: ID does not exist" containerID="81fae885c52e5fab4f791ca43b4ea8088dc3491be78573f92f1e2ba3aa1472c6"
May 26 11:27:12.055875 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.055829    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81fae885c52e5fab4f791ca43b4ea8088dc3491be78573f92f1e2ba3aa1472c6"} err="failed to get container status \"81fae885c52e5fab4f791ca43b4ea8088dc3491be78573f92f1e2ba3aa1472c6\": rpc error: code = NotFound desc = could not find container \"81fae885c52e5fab4f791ca43b4ea8088dc3491be78573f92f1e2ba3aa1472c6\": container with ID starting with 81fae885c52e5fab4f791ca43b4ea8088dc3491be78573f92f1e2ba3aa1472c6 not found: ID does not exist"
May 26 11:27:12.057052 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.057032    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 26 11:27:12.108235 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.108202    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 26 11:27:12.108504 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.108489    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0f6b7d4-fb12-42f6-b251-8de26af96192" containerName="config-reloader"
May 26 11:27:12.108559 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.108507    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0f6b7d4-fb12-42f6-b251-8de26af96192" containerName="config-reloader"
May 26 11:27:12.108559 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.108515    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0f6b7d4-fb12-42f6-b251-8de26af96192" containerName="kube-rbac-proxy-thanos"
May 26 11:27:12.108559 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.108520    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0f6b7d4-fb12-42f6-b251-8de26af96192" containerName="kube-rbac-proxy-thanos"
May 26 11:27:12.108559 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.108529    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0f6b7d4-fb12-42f6-b251-8de26af96192" containerName="init-config-reloader"
May 26 11:27:12.108559 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.108534    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0f6b7d4-fb12-42f6-b251-8de26af96192" containerName="init-config-reloader"
May 26 11:27:12.108559 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.108541    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0f6b7d4-fb12-42f6-b251-8de26af96192" containerName="prometheus"
May 26 11:27:12.108559 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.108547    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0f6b7d4-fb12-42f6-b251-8de26af96192" containerName="prometheus"
May 26 11:27:12.108559 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.108553    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0f6b7d4-fb12-42f6-b251-8de26af96192" containerName="thanos-sidecar"
May 26 11:27:12.108559 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.108559    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0f6b7d4-fb12-42f6-b251-8de26af96192" containerName="thanos-sidecar"
May 26 11:27:12.108881 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.108568    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0f6b7d4-fb12-42f6-b251-8de26af96192" containerName="kube-rbac-proxy-web"
May 26 11:27:12.108881 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.108576    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0f6b7d4-fb12-42f6-b251-8de26af96192" containerName="kube-rbac-proxy-web"
May 26 11:27:12.108881 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.108583    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0f6b7d4-fb12-42f6-b251-8de26af96192" containerName="kube-rbac-proxy"
May 26 11:27:12.108881 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.108588    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0f6b7d4-fb12-42f6-b251-8de26af96192" containerName="kube-rbac-proxy"
May 26 11:27:12.108881 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.108626    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="a0f6b7d4-fb12-42f6-b251-8de26af96192" containerName="prometheus"
May 26 11:27:12.108881 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.108634    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="a0f6b7d4-fb12-42f6-b251-8de26af96192" containerName="thanos-sidecar"
May 26 11:27:12.108881 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.108642    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="a0f6b7d4-fb12-42f6-b251-8de26af96192" containerName="config-reloader"
May 26 11:27:12.108881 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.108650    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="a0f6b7d4-fb12-42f6-b251-8de26af96192" containerName="kube-rbac-proxy-web"
May 26 11:27:12.108881 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.108659    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="a0f6b7d4-fb12-42f6-b251-8de26af96192" containerName="kube-rbac-proxy-thanos"
May 26 11:27:12.108881 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.108666    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="a0f6b7d4-fb12-42f6-b251-8de26af96192" containerName="kube-rbac-proxy"
May 26 11:27:12.111366 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.111349    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.114824 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.114590    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
May 26 11:27:12.114824 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.114650    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
May 26 11:27:12.114824 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.114690    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
May 26 11:27:12.114824 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.114718    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
May 26 11:27:12.114824 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.114725    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-a0ubpmc3n9ig7\""
May 26 11:27:12.114824 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.114731    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
May 26 11:27:12.114824 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.114743    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
May 26 11:27:12.114824 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.114690    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
May 26 11:27:12.115292 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.115197    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
May 26 11:27:12.115292 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.115251    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
May 26 11:27:12.115292 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.115251    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4vkzp\""
May 26 11:27:12.115597 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.115581    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
May 26 11:27:12.118593 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.118262    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
May 26 11:27:12.120029 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.120010    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
May 26 11:27:12.127466 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.127446    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 26 11:27:12.219827 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.219717    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/924bcc1d-1874-49bd-8348-8a360af67584-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.219827 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.219762    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/924bcc1d-1874-49bd-8348-8a360af67584-config\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.219827 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.219796    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/924bcc1d-1874-49bd-8348-8a360af67584-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.219827 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.219814    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/924bcc1d-1874-49bd-8348-8a360af67584-config-out\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.220145 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.219899    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2626\" (UniqueName: \"kubernetes.io/projected/924bcc1d-1874-49bd-8348-8a360af67584-kube-api-access-g2626\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.220145 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.219942    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/924bcc1d-1874-49bd-8348-8a360af67584-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.220145 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.219978    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/924bcc1d-1874-49bd-8348-8a360af67584-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.220145 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.220030    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/924bcc1d-1874-49bd-8348-8a360af67584-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.220145 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.220057    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/924bcc1d-1874-49bd-8348-8a360af67584-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.220145 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.220107    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/924bcc1d-1874-49bd-8348-8a360af67584-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.220145 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.220143    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/924bcc1d-1874-49bd-8348-8a360af67584-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.220418 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.220166    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/924bcc1d-1874-49bd-8348-8a360af67584-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.220418 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.220193    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/924bcc1d-1874-49bd-8348-8a360af67584-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.220418 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.220243    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/924bcc1d-1874-49bd-8348-8a360af67584-web-config\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.220418 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.220276    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/924bcc1d-1874-49bd-8348-8a360af67584-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.220418 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.220312    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/924bcc1d-1874-49bd-8348-8a360af67584-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.220418 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.220342    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/924bcc1d-1874-49bd-8348-8a360af67584-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.220418 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.220372    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/924bcc1d-1874-49bd-8348-8a360af67584-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.321303 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.321259    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/924bcc1d-1874-49bd-8348-8a360af67584-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.321303 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.321315    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/924bcc1d-1874-49bd-8348-8a360af67584-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.321505 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.321336    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/924bcc1d-1874-49bd-8348-8a360af67584-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.321505 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.321360    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/924bcc1d-1874-49bd-8348-8a360af67584-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.321505 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.321380    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/924bcc1d-1874-49bd-8348-8a360af67584-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.321505 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.321397    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/924bcc1d-1874-49bd-8348-8a360af67584-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.321505 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.321414    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/924bcc1d-1874-49bd-8348-8a360af67584-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.321505 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.321441    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/924bcc1d-1874-49bd-8348-8a360af67584-web-config\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.321505 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.321470    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/924bcc1d-1874-49bd-8348-8a360af67584-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.321505 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.321497    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/924bcc1d-1874-49bd-8348-8a360af67584-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.321955 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.321519    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/924bcc1d-1874-49bd-8348-8a360af67584-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.321955 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.321545    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/924bcc1d-1874-49bd-8348-8a360af67584-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.321955 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.321568    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/924bcc1d-1874-49bd-8348-8a360af67584-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.321955 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.321597    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/924bcc1d-1874-49bd-8348-8a360af67584-config\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.321955 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.321612    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/924bcc1d-1874-49bd-8348-8a360af67584-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.321955 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.321631    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/924bcc1d-1874-49bd-8348-8a360af67584-config-out\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.321955 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.321665    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g2626\" (UniqueName: \"kubernetes.io/projected/924bcc1d-1874-49bd-8348-8a360af67584-kube-api-access-g2626\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.321955 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.321693    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/924bcc1d-1874-49bd-8348-8a360af67584-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.322505 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.322002    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/924bcc1d-1874-49bd-8348-8a360af67584-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.322565 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.322499    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/924bcc1d-1874-49bd-8348-8a360af67584-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.323090 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.323060    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/924bcc1d-1874-49bd-8348-8a360af67584-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.323707 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.323570    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/924bcc1d-1874-49bd-8348-8a360af67584-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.324815 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.324479    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/924bcc1d-1874-49bd-8348-8a360af67584-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.324815 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.324501    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/924bcc1d-1874-49bd-8348-8a360af67584-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.324815 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.324612    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/924bcc1d-1874-49bd-8348-8a360af67584-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.324815 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.324634    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/924bcc1d-1874-49bd-8348-8a360af67584-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.326336 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.326293    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/924bcc1d-1874-49bd-8348-8a360af67584-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.327089 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.327065    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/924bcc1d-1874-49bd-8348-8a360af67584-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.327211 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.327187    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/924bcc1d-1874-49bd-8348-8a360af67584-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.327381 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.327358    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/924bcc1d-1874-49bd-8348-8a360af67584-config\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.327442 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.327367    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/924bcc1d-1874-49bd-8348-8a360af67584-config-out\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.327442 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.327420    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/924bcc1d-1874-49bd-8348-8a360af67584-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.327506 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.327480    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/924bcc1d-1874-49bd-8348-8a360af67584-web-config\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.327579 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.327557    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/924bcc1d-1874-49bd-8348-8a360af67584-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.327924 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.327908    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/924bcc1d-1874-49bd-8348-8a360af67584-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.333630 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.333606    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2626\" (UniqueName: \"kubernetes.io/projected/924bcc1d-1874-49bd-8348-8a360af67584-kube-api-access-g2626\") pod \"prometheus-k8s-0\" (UID: \"924bcc1d-1874-49bd-8348-8a360af67584\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.421087 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.421046    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:27:12.428633 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.428606    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0f6b7d4-fb12-42f6-b251-8de26af96192" path="/var/lib/kubelet/pods/a0f6b7d4-fb12-42f6-b251-8de26af96192/volumes"
May 26 11:27:12.556333 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:12.556300    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 26 11:27:12.560148 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:27:12.560117    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod924bcc1d_1874_49bd_8348_8a360af67584.slice/crio-53fd0108b5ca9b397111e45343ce95684bf56dd98552ad0563fe38342a659e3b WatchSource:0}: Error finding container 53fd0108b5ca9b397111e45343ce95684bf56dd98552ad0563fe38342a659e3b: Status 404 returned error can't find the container with id 53fd0108b5ca9b397111e45343ce95684bf56dd98552ad0563fe38342a659e3b
May 26 11:27:13.010322 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:13.010285    2563 generic.go:358] "Generic (PLEG): container finished" podID="924bcc1d-1874-49bd-8348-8a360af67584" containerID="3e0cdadc60eb8c93460a04b32f4d89981422690464890d6735038905723af7fb" exitCode=0
May 26 11:27:13.010681 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:13.010366    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"924bcc1d-1874-49bd-8348-8a360af67584","Type":"ContainerDied","Data":"3e0cdadc60eb8c93460a04b32f4d89981422690464890d6735038905723af7fb"}
May 26 11:27:13.010681 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:13.010395    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"924bcc1d-1874-49bd-8348-8a360af67584","Type":"ContainerStarted","Data":"53fd0108b5ca9b397111e45343ce95684bf56dd98552ad0563fe38342a659e3b"}
May 26 11:27:14.015694 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:14.015633    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"924bcc1d-1874-49bd-8348-8a360af67584","Type":"ContainerStarted","Data":"2a8a501fc505d3d35c84f5b88baf9da31bedd707603273467e7a231ce7f3b698"}
May 26 11:27:14.015694 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:14.015681    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"924bcc1d-1874-49bd-8348-8a360af67584","Type":"ContainerStarted","Data":"455212c896112bd005035593d4f1168796f0181794212552d73e744170daf055"}
May 26 11:27:14.015694 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:14.015695    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"924bcc1d-1874-49bd-8348-8a360af67584","Type":"ContainerStarted","Data":"4e6b8f84090c76ed83a369ddb1dbaf5b9a0b0731482432a2bfeac6fe74b2be18"}
May 26 11:27:14.016184 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:14.015708    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"924bcc1d-1874-49bd-8348-8a360af67584","Type":"ContainerStarted","Data":"d98650a1683cfae25087b2b780a44dd7a39b7c3e5def9ca28bb6ada630b24dc3"}
May 26 11:27:14.016184 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:14.015721    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"924bcc1d-1874-49bd-8348-8a360af67584","Type":"ContainerStarted","Data":"263ee952bd035a1ad6318d223769897294aa4c4c657c87b6677ead3dbda6c670"}
May 26 11:27:14.016184 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:14.015736    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"924bcc1d-1874-49bd-8348-8a360af67584","Type":"ContainerStarted","Data":"94fccd3ad9377ceeca64e14f7238eb547856070fe78aa7f4740a790b91682d27"}
May 26 11:27:14.046395 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:14.046335    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.046315662 podStartE2EDuration="2.046315662s" podCreationTimestamp="2026-05-26 11:27:12 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:27:14.044523184 +0000 UTC m=+166.235965371" watchObservedRunningTime="2026-05-26 11:27:14.046315662 +0000 UTC m=+166.237757850"
May 26 11:27:17.422059 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:27:17.422016    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:12.421440 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:28:12.421406    2563 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:12.438016 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:28:12.437991    2563 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:13.198536 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:28:13.198511    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:05.678703 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:29:05.678673    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-x9ccg"]
May 26 11:29:05.680847 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:29:05.680823    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9ccg"
May 26 11:29:05.683450 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:29:05.683433    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 26 11:29:05.693095 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:29:05.693072    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x9ccg"]
May 26 11:29:05.805145 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:29:05.805092    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04c86725-6167-4dfc-b259-f7a1423c418d-original-pull-secret\") pod \"global-pull-secret-syncer-x9ccg\" (UID: \"04c86725-6167-4dfc-b259-f7a1423c418d\") " pod="kube-system/global-pull-secret-syncer-x9ccg"
May 26 11:29:05.805316 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:29:05.805172    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04c86725-6167-4dfc-b259-f7a1423c418d-kubelet-config\") pod \"global-pull-secret-syncer-x9ccg\" (UID: \"04c86725-6167-4dfc-b259-f7a1423c418d\") " pod="kube-system/global-pull-secret-syncer-x9ccg"
May 26 11:29:05.805316 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:29:05.805205    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04c86725-6167-4dfc-b259-f7a1423c418d-dbus\") pod \"global-pull-secret-syncer-x9ccg\" (UID: \"04c86725-6167-4dfc-b259-f7a1423c418d\") " pod="kube-system/global-pull-secret-syncer-x9ccg"
May 26 11:29:05.906422 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:29:05.906382    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04c86725-6167-4dfc-b259-f7a1423c418d-original-pull-secret\") pod \"global-pull-secret-syncer-x9ccg\" (UID: \"04c86725-6167-4dfc-b259-f7a1423c418d\") " pod="kube-system/global-pull-secret-syncer-x9ccg"
May 26 11:29:05.906577 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:29:05.906434    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04c86725-6167-4dfc-b259-f7a1423c418d-kubelet-config\") pod \"global-pull-secret-syncer-x9ccg\" (UID: \"04c86725-6167-4dfc-b259-f7a1423c418d\") " pod="kube-system/global-pull-secret-syncer-x9ccg"
May 26 11:29:05.906577 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:29:05.906474    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04c86725-6167-4dfc-b259-f7a1423c418d-dbus\") pod \"global-pull-secret-syncer-x9ccg\" (UID: \"04c86725-6167-4dfc-b259-f7a1423c418d\") " pod="kube-system/global-pull-secret-syncer-x9ccg"
May 26 11:29:05.906650 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:29:05.906586    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04c86725-6167-4dfc-b259-f7a1423c418d-kubelet-config\") pod \"global-pull-secret-syncer-x9ccg\" (UID: \"04c86725-6167-4dfc-b259-f7a1423c418d\") " pod="kube-system/global-pull-secret-syncer-x9ccg"
May 26 11:29:05.906650 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:29:05.906629    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04c86725-6167-4dfc-b259-f7a1423c418d-dbus\") pod \"global-pull-secret-syncer-x9ccg\" (UID: \"04c86725-6167-4dfc-b259-f7a1423c418d\") " pod="kube-system/global-pull-secret-syncer-x9ccg"
May 26 11:29:05.908790 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:29:05.908747    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04c86725-6167-4dfc-b259-f7a1423c418d-original-pull-secret\") pod \"global-pull-secret-syncer-x9ccg\" (UID: \"04c86725-6167-4dfc-b259-f7a1423c418d\") " pod="kube-system/global-pull-secret-syncer-x9ccg"
May 26 11:29:05.990037 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:29:05.989944    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9ccg"
May 26 11:29:06.120070 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:29:06.120035    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x9ccg"]
May 26 11:29:06.123407 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:29:06.123369    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04c86725_6167_4dfc_b259_f7a1423c418d.slice/crio-c8ecdb2bb7607f6f9bb155f233e6b040e9dc9223440cac4c172608340a081f63 WatchSource:0}: Error finding container c8ecdb2bb7607f6f9bb155f233e6b040e9dc9223440cac4c172608340a081f63: Status 404 returned error can't find the container with id c8ecdb2bb7607f6f9bb155f233e6b040e9dc9223440cac4c172608340a081f63
May 26 11:29:06.326186 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:29:06.326092    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x9ccg" event={"ID":"04c86725-6167-4dfc-b259-f7a1423c418d","Type":"ContainerStarted","Data":"c8ecdb2bb7607f6f9bb155f233e6b040e9dc9223440cac4c172608340a081f63"}
May 26 11:29:10.339242 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:29:10.339204    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x9ccg" event={"ID":"04c86725-6167-4dfc-b259-f7a1423c418d","Type":"ContainerStarted","Data":"4786f122ff800b15d90cb3921e3732a222a34ee00562c9dc0733e4cf37a0419f"}
May 26 11:29:10.355329 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:29:10.355281    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-x9ccg" podStartSLOduration=1.580194939 podStartE2EDuration="5.35526569s" podCreationTimestamp="2026-05-26 11:29:05 +0000 UTC" firstStartedPulling="2026-05-26 11:29:06.125393645 +0000 UTC m=+278.316835810" lastFinishedPulling="2026-05-26 11:29:09.900464395 +0000 UTC m=+282.091906561" observedRunningTime="2026-05-26 11:29:10.353918356 +0000 UTC m=+282.545360544" watchObservedRunningTime="2026-05-26 11:29:10.35526569 +0000 UTC m=+282.546707952"
May 26 11:29:28.336163 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:29:28.336137    2563 kubelet.go:1628] "Image garbage collection succeeded"
May 26 11:42:42.938194 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:42:42.938160    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-init-54vkw"]
May 26 11:42:42.940993 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:42:42.940968    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-54vkw"
May 26 11:42:42.943845 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:42:42.943827    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 26 11:42:42.944108 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:42:42.944088    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
May 26 11:42:42.945079 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:42:42.945064    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-w8jd4\""
May 26 11:42:42.945164 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:42:42.945081    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 26 11:42:42.950405 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:42:42.950385    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-54vkw"]
May 26 11:42:43.007026 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:42:43.006988    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfnnq\" (UniqueName: \"kubernetes.io/projected/1342967f-2803-490a-9473-81acff932ca9-kube-api-access-wfnnq\") pod \"s3-init-54vkw\" (UID: \"1342967f-2803-490a-9473-81acff932ca9\") " pod="kserve/s3-init-54vkw"
May 26 11:42:43.107554 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:42:43.107506    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wfnnq\" (UniqueName: \"kubernetes.io/projected/1342967f-2803-490a-9473-81acff932ca9-kube-api-access-wfnnq\") pod \"s3-init-54vkw\" (UID: \"1342967f-2803-490a-9473-81acff932ca9\") " pod="kserve/s3-init-54vkw"
May 26 11:42:43.115881 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:42:43.115856    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfnnq\" (UniqueName: \"kubernetes.io/projected/1342967f-2803-490a-9473-81acff932ca9-kube-api-access-wfnnq\") pod \"s3-init-54vkw\" (UID: \"1342967f-2803-490a-9473-81acff932ca9\") " pod="kserve/s3-init-54vkw"
May 26 11:42:43.261685 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:42:43.261591    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-54vkw"
May 26 11:42:43.392958 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:42:43.392928    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-54vkw"]
May 26 11:42:43.396315 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:42:43.396290    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1342967f_2803_490a_9473_81acff932ca9.slice/crio-13cc8371faf57fdd584d71154f7dcb0915524d250f32b8c225368514b05c662d WatchSource:0}: Error finding container 13cc8371faf57fdd584d71154f7dcb0915524d250f32b8c225368514b05c662d: Status 404 returned error can't find the container with id 13cc8371faf57fdd584d71154f7dcb0915524d250f32b8c225368514b05c662d
May 26 11:42:43.398399 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:42:43.398383    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 11:42:43.543196 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:42:43.543104    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-54vkw" event={"ID":"1342967f-2803-490a-9473-81acff932ca9","Type":"ContainerStarted","Data":"13cc8371faf57fdd584d71154f7dcb0915524d250f32b8c225368514b05c662d"}
May 26 11:42:48.563609 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:42:48.563566    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-54vkw" event={"ID":"1342967f-2803-490a-9473-81acff932ca9","Type":"ContainerStarted","Data":"231436510aed4e9fddf8c512afbdaa719606b03cb89543450bf3fcdd13bf8553"}
May 26 11:42:48.579602 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:42:48.579415    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-init-54vkw" podStartSLOduration=2.110319167 podStartE2EDuration="6.579398182s" podCreationTimestamp="2026-05-26 11:42:42 +0000 UTC" firstStartedPulling="2026-05-26 11:42:43.398506777 +0000 UTC m=+1095.589948942" lastFinishedPulling="2026-05-26 11:42:47.867585789 +0000 UTC m=+1100.059027957" observedRunningTime="2026-05-26 11:42:48.578762869 +0000 UTC m=+1100.770205059" watchObservedRunningTime="2026-05-26 11:42:48.579398182 +0000 UTC m=+1100.770840370"
May 26 11:42:51.573121 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:42:51.573084    2563 generic.go:358] "Generic (PLEG): container finished" podID="1342967f-2803-490a-9473-81acff932ca9" containerID="231436510aed4e9fddf8c512afbdaa719606b03cb89543450bf3fcdd13bf8553" exitCode=0
May 26 11:42:51.573481 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:42:51.573147    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-54vkw" event={"ID":"1342967f-2803-490a-9473-81acff932ca9","Type":"ContainerDied","Data":"231436510aed4e9fddf8c512afbdaa719606b03cb89543450bf3fcdd13bf8553"}
May 26 11:42:52.697456 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:42:52.697434    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-54vkw"
May 26 11:42:52.793390 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:42:52.793356    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfnnq\" (UniqueName: \"kubernetes.io/projected/1342967f-2803-490a-9473-81acff932ca9-kube-api-access-wfnnq\") pod \"1342967f-2803-490a-9473-81acff932ca9\" (UID: \"1342967f-2803-490a-9473-81acff932ca9\") "
May 26 11:42:52.795603 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:42:52.795575    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1342967f-2803-490a-9473-81acff932ca9-kube-api-access-wfnnq" (OuterVolumeSpecName: "kube-api-access-wfnnq") pod "1342967f-2803-490a-9473-81acff932ca9" (UID: "1342967f-2803-490a-9473-81acff932ca9"). InnerVolumeSpecName "kube-api-access-wfnnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:42:52.894783 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:42:52.894748    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfnnq\" (UniqueName: \"kubernetes.io/projected/1342967f-2803-490a-9473-81acff932ca9-kube-api-access-wfnnq\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:42:53.579535 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:42:53.579498    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-54vkw" event={"ID":"1342967f-2803-490a-9473-81acff932ca9","Type":"ContainerDied","Data":"13cc8371faf57fdd584d71154f7dcb0915524d250f32b8c225368514b05c662d"}
May 26 11:42:53.579535 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:42:53.579529    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13cc8371faf57fdd584d71154f7dcb0915524d250f32b8c225368514b05c662d"
May 26 11:42:53.579535 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:42:53.579511    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-54vkw"
May 26 11:43:02.971038 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:02.970997    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468"]
May 26 11:43:02.971399 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:02.971257    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1342967f-2803-490a-9473-81acff932ca9" containerName="s3-init"
May 26 11:43:02.971399 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:02.971267    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1342967f-2803-490a-9473-81acff932ca9" containerName="s3-init"
May 26 11:43:02.971399 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:02.971315    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="1342967f-2803-490a-9473-81acff932ca9" containerName="s3-init"
May 26 11:43:02.974325 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:02.974309    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468"
May 26 11:43:02.977205 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:02.977172    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\""
May 26 11:43:02.977205 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:02.977175    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 26 11:43:02.977374 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:02.977175    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7tgml\""
May 26 11:43:02.977374 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:02.977202    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\""
May 26 11:43:02.977480 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:02.977465    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 26 11:43:02.991191 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:02.991162    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468"]
May 26 11:43:03.058453 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:03.058413    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn"]
May 26 11:43:03.061510 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:03.061484    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn"
May 26 11:43:03.064829 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:03.064806    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-378f2-kube-rbac-proxy-sar-config\""
May 26 11:43:03.064970 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:03.064886    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-378f2-predictor-serving-cert\""
May 26 11:43:03.079138 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:03.079107    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0805d6c1-804e-4bac-93cc-ea3e36a86829-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-678784f46b-8m468\" (UID: \"0805d6c1-804e-4bac-93cc-ea3e36a86829\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468"
May 26 11:43:03.079321 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:03.079174    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0805d6c1-804e-4bac-93cc-ea3e36a86829-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-678784f46b-8m468\" (UID: \"0805d6c1-804e-4bac-93cc-ea3e36a86829\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468"
May 26 11:43:03.079321 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:03.079204    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0805d6c1-804e-4bac-93cc-ea3e36a86829-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-678784f46b-8m468\" (UID: \"0805d6c1-804e-4bac-93cc-ea3e36a86829\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468"
May 26 11:43:03.079321 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:03.079259    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srlrn\" (UniqueName: \"kubernetes.io/projected/0805d6c1-804e-4bac-93cc-ea3e36a86829-kube-api-access-srlrn\") pod \"isvc-sklearn-graph-1-predictor-678784f46b-8m468\" (UID: \"0805d6c1-804e-4bac-93cc-ea3e36a86829\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468"
May 26 11:43:03.080713 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:03.080688    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn"]
May 26 11:43:03.179712 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:03.179678    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0805d6c1-804e-4bac-93cc-ea3e36a86829-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-678784f46b-8m468\" (UID: \"0805d6c1-804e-4bac-93cc-ea3e36a86829\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468"
May 26 11:43:03.179712 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:03.179714    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0805d6c1-804e-4bac-93cc-ea3e36a86829-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-678784f46b-8m468\" (UID: \"0805d6c1-804e-4bac-93cc-ea3e36a86829\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468"
May 26 11:43:03.179974 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:03.179751    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-srlrn\" (UniqueName: \"kubernetes.io/projected/0805d6c1-804e-4bac-93cc-ea3e36a86829-kube-api-access-srlrn\") pod \"isvc-sklearn-graph-1-predictor-678784f46b-8m468\" (UID: \"0805d6c1-804e-4bac-93cc-ea3e36a86829\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468"
May 26 11:43:03.179974 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:03.179807    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8277b22b-acea-47e9-932e-37813db5a360-proxy-tls\") pod \"error-404-isvc-378f2-predictor-d6b46746f-8chzn\" (UID: \"8277b22b-acea-47e9-932e-37813db5a360\") " pod="kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn"
May 26 11:43:03.179974 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:03.179840    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-378f2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8277b22b-acea-47e9-932e-37813db5a360-error-404-isvc-378f2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-378f2-predictor-d6b46746f-8chzn\" (UID: \"8277b22b-acea-47e9-932e-37813db5a360\") " pod="kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn"
May 26 11:43:03.179974 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:03.179878    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq5nd\" (UniqueName: \"kubernetes.io/projected/8277b22b-acea-47e9-932e-37813db5a360-kube-api-access-wq5nd\") pod \"error-404-isvc-378f2-predictor-d6b46746f-8chzn\" (UID: \"8277b22b-acea-47e9-932e-37813db5a360\") " pod="kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn"
May 26 11:43:03.180173 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:03.179975    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0805d6c1-804e-4bac-93cc-ea3e36a86829-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-678784f46b-8m468\" (UID: \"0805d6c1-804e-4bac-93cc-ea3e36a86829\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468"
May 26 11:43:03.180173 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:03.180143    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0805d6c1-804e-4bac-93cc-ea3e36a86829-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-678784f46b-8m468\" (UID: \"0805d6c1-804e-4bac-93cc-ea3e36a86829\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468"
May 26 11:43:03.180547 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:03.180528    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0805d6c1-804e-4bac-93cc-ea3e36a86829-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-678784f46b-8m468\" (UID: \"0805d6c1-804e-4bac-93cc-ea3e36a86829\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468"
May 26 11:43:03.182439 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:03.182416    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0805d6c1-804e-4bac-93cc-ea3e36a86829-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-678784f46b-8m468\" (UID: \"0805d6c1-804e-4bac-93cc-ea3e36a86829\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468"
May 26 11:43:03.188970 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:03.188950    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srlrn\" (UniqueName: \"kubernetes.io/projected/0805d6c1-804e-4bac-93cc-ea3e36a86829-kube-api-access-srlrn\") pod \"isvc-sklearn-graph-1-predictor-678784f46b-8m468\" (UID: \"0805d6c1-804e-4bac-93cc-ea3e36a86829\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468"
May 26 11:43:03.281170 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:03.281068    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8277b22b-acea-47e9-932e-37813db5a360-proxy-tls\") pod \"error-404-isvc-378f2-predictor-d6b46746f-8chzn\" (UID: \"8277b22b-acea-47e9-932e-37813db5a360\") " pod="kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn"
May 26 11:43:03.281170 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:03.281118    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-378f2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8277b22b-acea-47e9-932e-37813db5a360-error-404-isvc-378f2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-378f2-predictor-d6b46746f-8chzn\" (UID: \"8277b22b-acea-47e9-932e-37813db5a360\") " pod="kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn"
May 26 11:43:03.281170 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:03.281141    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wq5nd\" (UniqueName: \"kubernetes.io/projected/8277b22b-acea-47e9-932e-37813db5a360-kube-api-access-wq5nd\") pod \"error-404-isvc-378f2-predictor-d6b46746f-8chzn\" (UID: \"8277b22b-acea-47e9-932e-37813db5a360\") " pod="kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn"
May 26 11:43:03.281818 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:03.281744    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-378f2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8277b22b-acea-47e9-932e-37813db5a360-error-404-isvc-378f2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-378f2-predictor-d6b46746f-8chzn\" (UID: \"8277b22b-acea-47e9-932e-37813db5a360\") " pod="kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn"
May 26 11:43:03.283599 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:03.283576    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8277b22b-acea-47e9-932e-37813db5a360-proxy-tls\") pod \"error-404-isvc-378f2-predictor-d6b46746f-8chzn\" (UID: \"8277b22b-acea-47e9-932e-37813db5a360\") " pod="kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn"
May 26 11:43:03.283699 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:03.283689    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468"
May 26 11:43:03.290396 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:03.290372    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq5nd\" (UniqueName: \"kubernetes.io/projected/8277b22b-acea-47e9-932e-37813db5a360-kube-api-access-wq5nd\") pod \"error-404-isvc-378f2-predictor-d6b46746f-8chzn\" (UID: \"8277b22b-acea-47e9-932e-37813db5a360\") " pod="kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn"
May 26 11:43:03.373228 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:03.373187    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn"
May 26 11:43:03.413505 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:03.413317    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468"]
May 26 11:43:03.416119 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:43:03.416094    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0805d6c1_804e_4bac_93cc_ea3e36a86829.slice/crio-d7eaa4e758da79a187ce0f576da557d9f68bef303f10c3fc27333bae00999e6b WatchSource:0}: Error finding container d7eaa4e758da79a187ce0f576da557d9f68bef303f10c3fc27333bae00999e6b: Status 404 returned error can't find the container with id d7eaa4e758da79a187ce0f576da557d9f68bef303f10c3fc27333bae00999e6b
May 26 11:43:03.527217 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:03.527191    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn"]
May 26 11:43:03.529423 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:43:03.529394    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8277b22b_acea_47e9_932e_37813db5a360.slice/crio-791ea6f57bba63cdf2e598da1794c21f148f2d6b7143569455f0df10d469854e WatchSource:0}: Error finding container 791ea6f57bba63cdf2e598da1794c21f148f2d6b7143569455f0df10d469854e: Status 404 returned error can't find the container with id 791ea6f57bba63cdf2e598da1794c21f148f2d6b7143569455f0df10d469854e
May 26 11:43:03.605966 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:03.605922    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn" event={"ID":"8277b22b-acea-47e9-932e-37813db5a360","Type":"ContainerStarted","Data":"791ea6f57bba63cdf2e598da1794c21f148f2d6b7143569455f0df10d469854e"}
May 26 11:43:03.606964 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:03.606940    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468" event={"ID":"0805d6c1-804e-4bac-93cc-ea3e36a86829","Type":"ContainerStarted","Data":"d7eaa4e758da79a187ce0f576da557d9f68bef303f10c3fc27333bae00999e6b"}
May 26 11:43:08.632102 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:08.632055    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468" event={"ID":"0805d6c1-804e-4bac-93cc-ea3e36a86829","Type":"ContainerStarted","Data":"017e2746322c6124db785987256fa987170a21b1904d14b2a8411d9fe37102a8"}
May 26 11:43:12.644692 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:12.644599    2563 generic.go:358] "Generic (PLEG): container finished" podID="0805d6c1-804e-4bac-93cc-ea3e36a86829" containerID="017e2746322c6124db785987256fa987170a21b1904d14b2a8411d9fe37102a8" exitCode=0
May 26 11:43:12.645134 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:12.644680    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468" event={"ID":"0805d6c1-804e-4bac-93cc-ea3e36a86829","Type":"ContainerDied","Data":"017e2746322c6124db785987256fa987170a21b1904d14b2a8411d9fe37102a8"}
May 26 11:43:15.656007 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:15.655969    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn" event={"ID":"8277b22b-acea-47e9-932e-37813db5a360","Type":"ContainerStarted","Data":"61eb286df20ecd0f6e6be55d51bd74e2b8dd3d23c4e83ac9220d60f0deecdd87"}
May 26 11:43:18.667336 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:18.667297    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn" event={"ID":"8277b22b-acea-47e9-932e-37813db5a360","Type":"ContainerStarted","Data":"3531569f2a90ec9fdc2bf9e95e2b59f20c8127fa8cc9df9e4e370868a9900d55"}
May 26 11:43:18.667836 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:18.667474    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn"
May 26 11:43:18.690758 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:18.690703    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn" podStartSLOduration=1.236660638 podStartE2EDuration="15.69068668s" podCreationTimestamp="2026-05-26 11:43:03 +0000 UTC" firstStartedPulling="2026-05-26 11:43:03.531211297 +0000 UTC m=+1115.722653461" lastFinishedPulling="2026-05-26 11:43:17.985237329 +0000 UTC m=+1130.176679503" observedRunningTime="2026-05-26 11:43:18.690122636 +0000 UTC m=+1130.881564838" watchObservedRunningTime="2026-05-26 11:43:18.69068668 +0000 UTC m=+1130.882128866"
May 26 11:43:19.670901 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:19.670874    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn"
May 26 11:43:19.672360 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:19.672329    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn" podUID="8277b22b-acea-47e9-932e-37813db5a360" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
May 26 11:43:20.674167 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:20.674123    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn" podUID="8277b22b-acea-47e9-932e-37813db5a360" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
May 26 11:43:22.681952 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:22.681920    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468" event={"ID":"0805d6c1-804e-4bac-93cc-ea3e36a86829","Type":"ContainerStarted","Data":"3692cd94cb6dc72f4976da364dc31ebbe8eb6fd436fb8678c9a5e7d314c784d8"}
May 26 11:43:22.682335 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:22.681961    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468" event={"ID":"0805d6c1-804e-4bac-93cc-ea3e36a86829","Type":"ContainerStarted","Data":"50b21847803594aafdaa83774a3cf41e4a56617ea8c5c48c0453d9aca28aadd5"}
May 26 11:43:22.682335 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:22.682227    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468"
May 26 11:43:22.682335 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:22.682254    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468"
May 26 11:43:22.683443 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:22.683417    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468" podUID="0805d6c1-804e-4bac-93cc-ea3e36a86829" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
May 26 11:43:22.701582 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:22.701536    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468" podStartSLOduration=2.055445704 podStartE2EDuration="20.701524454s" podCreationTimestamp="2026-05-26 11:43:02 +0000 UTC" firstStartedPulling="2026-05-26 11:43:03.418374319 +0000 UTC m=+1115.609816484" lastFinishedPulling="2026-05-26 11:43:22.064453062 +0000 UTC m=+1134.255895234" observedRunningTime="2026-05-26 11:43:22.70049708 +0000 UTC m=+1134.891939268" watchObservedRunningTime="2026-05-26 11:43:22.701524454 +0000 UTC m=+1134.892966641"
May 26 11:43:23.684826 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:23.684770    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468" podUID="0805d6c1-804e-4bac-93cc-ea3e36a86829" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
May 26 11:43:25.678477 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:25.678446    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn"
May 26 11:43:25.678947 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:25.678922    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn" podUID="8277b22b-acea-47e9-932e-37813db5a360" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
May 26 11:43:28.689017 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:28.688964    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468"
May 26 11:43:28.689593 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:28.689550    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468" podUID="0805d6c1-804e-4bac-93cc-ea3e36a86829" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
May 26 11:43:35.679546 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:35.679499    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn" podUID="8277b22b-acea-47e9-932e-37813db5a360" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
May 26 11:43:38.689482 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:38.689442    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468" podUID="0805d6c1-804e-4bac-93cc-ea3e36a86829" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
May 26 11:43:45.679610 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:45.679571    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn" podUID="8277b22b-acea-47e9-932e-37813db5a360" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
May 26 11:43:48.689910 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:48.689871    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468" podUID="0805d6c1-804e-4bac-93cc-ea3e36a86829" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
May 26 11:43:55.679353 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:55.679310    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn" podUID="8277b22b-acea-47e9-932e-37813db5a360" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
May 26 11:43:58.689642 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:43:58.689602    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468" podUID="0805d6c1-804e-4bac-93cc-ea3e36a86829" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
May 26 11:44:05.679523 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:05.679489    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn"
May 26 11:44:08.690280 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:08.690245    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468" podUID="0805d6c1-804e-4bac-93cc-ea3e36a86829" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
May 26 11:44:18.690490 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:18.690449    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468" podUID="0805d6c1-804e-4bac-93cc-ea3e36a86829" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
May 26 11:44:22.700565 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:22.700524    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-378f2-5985b954cf-x9762"]
May 26 11:44:22.703925 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:22.703903    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-378f2-5985b954cf-x9762"
May 26 11:44:22.706432 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:22.706407    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-378f2-serving-cert\""
May 26 11:44:22.706539 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:22.706407    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-378f2-kube-rbac-proxy-sar-config\""
May 26 11:44:22.711849 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:22.711827    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-378f2-5985b954cf-x9762"]
May 26 11:44:22.745138 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:22.745086    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e88eb924-4af1-4f62-8ba7-1afaf826f46d-openshift-service-ca-bundle\") pod \"switch-graph-378f2-5985b954cf-x9762\" (UID: \"e88eb924-4af1-4f62-8ba7-1afaf826f46d\") " pod="kserve-ci-e2e-test/switch-graph-378f2-5985b954cf-x9762"
May 26 11:44:22.745303 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:22.745158    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e88eb924-4af1-4f62-8ba7-1afaf826f46d-proxy-tls\") pod \"switch-graph-378f2-5985b954cf-x9762\" (UID: \"e88eb924-4af1-4f62-8ba7-1afaf826f46d\") " pod="kserve-ci-e2e-test/switch-graph-378f2-5985b954cf-x9762"
May 26 11:44:22.845801 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:22.845739    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e88eb924-4af1-4f62-8ba7-1afaf826f46d-openshift-service-ca-bundle\") pod \"switch-graph-378f2-5985b954cf-x9762\" (UID: \"e88eb924-4af1-4f62-8ba7-1afaf826f46d\") " pod="kserve-ci-e2e-test/switch-graph-378f2-5985b954cf-x9762"
May 26 11:44:22.845966 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:22.845841    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e88eb924-4af1-4f62-8ba7-1afaf826f46d-proxy-tls\") pod \"switch-graph-378f2-5985b954cf-x9762\" (UID: \"e88eb924-4af1-4f62-8ba7-1afaf826f46d\") " pod="kserve-ci-e2e-test/switch-graph-378f2-5985b954cf-x9762"
May 26 11:44:22.846011 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:44:22.845962    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-378f2-serving-cert: secret "switch-graph-378f2-serving-cert" not found
May 26 11:44:22.846052 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:44:22.846038    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e88eb924-4af1-4f62-8ba7-1afaf826f46d-proxy-tls podName:e88eb924-4af1-4f62-8ba7-1afaf826f46d nodeName:}" failed. No retries permitted until 2026-05-26 11:44:23.34601984 +0000 UTC m=+1195.537462010 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e88eb924-4af1-4f62-8ba7-1afaf826f46d-proxy-tls") pod "switch-graph-378f2-5985b954cf-x9762" (UID: "e88eb924-4af1-4f62-8ba7-1afaf826f46d") : secret "switch-graph-378f2-serving-cert" not found
May 26 11:44:22.846429 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:22.846413    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e88eb924-4af1-4f62-8ba7-1afaf826f46d-openshift-service-ca-bundle\") pod \"switch-graph-378f2-5985b954cf-x9762\" (UID: \"e88eb924-4af1-4f62-8ba7-1afaf826f46d\") " pod="kserve-ci-e2e-test/switch-graph-378f2-5985b954cf-x9762"
May 26 11:44:23.350001 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:23.349958    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e88eb924-4af1-4f62-8ba7-1afaf826f46d-proxy-tls\") pod \"switch-graph-378f2-5985b954cf-x9762\" (UID: \"e88eb924-4af1-4f62-8ba7-1afaf826f46d\") " pod="kserve-ci-e2e-test/switch-graph-378f2-5985b954cf-x9762"
May 26 11:44:23.352545 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:23.352519    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e88eb924-4af1-4f62-8ba7-1afaf826f46d-proxy-tls\") pod \"switch-graph-378f2-5985b954cf-x9762\" (UID: \"e88eb924-4af1-4f62-8ba7-1afaf826f46d\") " pod="kserve-ci-e2e-test/switch-graph-378f2-5985b954cf-x9762"
May 26 11:44:23.615661 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:23.615572    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-378f2-5985b954cf-x9762"
May 26 11:44:23.745571 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:23.745533    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-378f2-5985b954cf-x9762"]
May 26 11:44:23.748555 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:44:23.748529    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode88eb924_4af1_4f62_8ba7_1afaf826f46d.slice/crio-2ffc95a816fe3b340767f926b39fb8d298e34cc88213462f0bc76561f79aa845 WatchSource:0}: Error finding container 2ffc95a816fe3b340767f926b39fb8d298e34cc88213462f0bc76561f79aa845: Status 404 returned error can't find the container with id 2ffc95a816fe3b340767f926b39fb8d298e34cc88213462f0bc76561f79aa845
May 26 11:44:23.861314 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:23.861279    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-378f2-5985b954cf-x9762" event={"ID":"e88eb924-4af1-4f62-8ba7-1afaf826f46d","Type":"ContainerStarted","Data":"2ffc95a816fe3b340767f926b39fb8d298e34cc88213462f0bc76561f79aa845"}
May 26 11:44:25.867639 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:25.867602    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-378f2-5985b954cf-x9762" event={"ID":"e88eb924-4af1-4f62-8ba7-1afaf826f46d","Type":"ContainerStarted","Data":"128dc5b8fd48a01cd1f968ff77f7adf95d84575aeb0378af3fb170ac310a829b"}
May 26 11:44:25.868024 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:25.867754    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-378f2-5985b954cf-x9762"
May 26 11:44:25.886913 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:25.886869    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-378f2-5985b954cf-x9762" podStartSLOduration=1.881527586 podStartE2EDuration="3.886855463s" podCreationTimestamp="2026-05-26 11:44:22 +0000 UTC" firstStartedPulling="2026-05-26 11:44:23.750377323 +0000 UTC m=+1195.941819487" lastFinishedPulling="2026-05-26 11:44:25.755705196 +0000 UTC m=+1197.947147364" observedRunningTime="2026-05-26 11:44:25.884900516 +0000 UTC m=+1198.076342708" watchObservedRunningTime="2026-05-26 11:44:25.886855463 +0000 UTC m=+1198.078297650"
May 26 11:44:28.690892 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:28.690863    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468"
May 26 11:44:31.876577 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:31.876539    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-378f2-5985b954cf-x9762"
May 26 11:44:32.894877 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:32.894834    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-378f2-5985b954cf-x9762"]
May 26 11:44:32.895335 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:32.895109    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-378f2-5985b954cf-x9762" podUID="e88eb924-4af1-4f62-8ba7-1afaf826f46d" containerName="switch-graph-378f2" containerID="cri-o://128dc5b8fd48a01cd1f968ff77f7adf95d84575aeb0378af3fb170ac310a829b" gracePeriod=30
May 26 11:44:33.082193 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:33.082162    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn"]
May 26 11:44:33.082462 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:33.082436    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn" podUID="8277b22b-acea-47e9-932e-37813db5a360" containerName="kserve-container" containerID="cri-o://61eb286df20ecd0f6e6be55d51bd74e2b8dd3d23c4e83ac9220d60f0deecdd87" gracePeriod=30
May 26 11:44:33.082509 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:33.082488    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn" podUID="8277b22b-acea-47e9-932e-37813db5a360" containerName="kube-rbac-proxy" containerID="cri-o://3531569f2a90ec9fdc2bf9e95e2b59f20c8127fa8cc9df9e4e370868a9900d55" gracePeriod=30
May 26 11:44:33.179629 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:33.179554    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579"]
May 26 11:44:33.182899 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:33.182883    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579"
May 26 11:44:33.185683 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:33.185664    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-90450-predictor-serving-cert\""
May 26 11:44:33.185683 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:33.185673    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-90450-kube-rbac-proxy-sar-config\""
May 26 11:44:33.193378 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:33.193348    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579"]
May 26 11:44:33.233750 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:33.233715    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-90450-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb0ee137-b57f-4795-b42f-c3f26a00639e-error-404-isvc-90450-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-90450-predictor-f47cbcc5c-5s579\" (UID: \"bb0ee137-b57f-4795-b42f-c3f26a00639e\") " pod="kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579"
May 26 11:44:33.233909 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:33.233808    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb0ee137-b57f-4795-b42f-c3f26a00639e-proxy-tls\") pod \"error-404-isvc-90450-predictor-f47cbcc5c-5s579\" (UID: \"bb0ee137-b57f-4795-b42f-c3f26a00639e\") " pod="kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579"
May 26 11:44:33.233909 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:33.233838    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttsq8\" (UniqueName: \"kubernetes.io/projected/bb0ee137-b57f-4795-b42f-c3f26a00639e-kube-api-access-ttsq8\") pod \"error-404-isvc-90450-predictor-f47cbcc5c-5s579\" (UID: \"bb0ee137-b57f-4795-b42f-c3f26a00639e\") " pod="kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579"
May 26 11:44:33.334384 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:33.334352    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-90450-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb0ee137-b57f-4795-b42f-c3f26a00639e-error-404-isvc-90450-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-90450-predictor-f47cbcc5c-5s579\" (UID: \"bb0ee137-b57f-4795-b42f-c3f26a00639e\") " pod="kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579"
May 26 11:44:33.334583 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:33.334407    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb0ee137-b57f-4795-b42f-c3f26a00639e-proxy-tls\") pod \"error-404-isvc-90450-predictor-f47cbcc5c-5s579\" (UID: \"bb0ee137-b57f-4795-b42f-c3f26a00639e\") " pod="kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579"
May 26 11:44:33.334583 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:33.334430    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ttsq8\" (UniqueName: \"kubernetes.io/projected/bb0ee137-b57f-4795-b42f-c3f26a00639e-kube-api-access-ttsq8\") pod \"error-404-isvc-90450-predictor-f47cbcc5c-5s579\" (UID: \"bb0ee137-b57f-4795-b42f-c3f26a00639e\") " pod="kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579"
May 26 11:44:33.335094 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:33.335068    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-90450-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb0ee137-b57f-4795-b42f-c3f26a00639e-error-404-isvc-90450-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-90450-predictor-f47cbcc5c-5s579\" (UID: \"bb0ee137-b57f-4795-b42f-c3f26a00639e\") " pod="kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579"
May 26 11:44:33.337009 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:33.336986    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb0ee137-b57f-4795-b42f-c3f26a00639e-proxy-tls\") pod \"error-404-isvc-90450-predictor-f47cbcc5c-5s579\" (UID: \"bb0ee137-b57f-4795-b42f-c3f26a00639e\") " pod="kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579"
May 26 11:44:33.344188 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:33.344162    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttsq8\" (UniqueName: \"kubernetes.io/projected/bb0ee137-b57f-4795-b42f-c3f26a00639e-kube-api-access-ttsq8\") pod \"error-404-isvc-90450-predictor-f47cbcc5c-5s579\" (UID: \"bb0ee137-b57f-4795-b42f-c3f26a00639e\") " pod="kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579"
May 26 11:44:33.494080 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:33.493988    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579"
May 26 11:44:33.620561 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:33.620527    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579"]
May 26 11:44:33.624055 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:44:33.624031    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb0ee137_b57f_4795_b42f_c3f26a00639e.slice/crio-891c6ce0d96f6e9c1837cd1593021c71765f5281e1b0cad7bde968b35bc0ae15 WatchSource:0}: Error finding container 891c6ce0d96f6e9c1837cd1593021c71765f5281e1b0cad7bde968b35bc0ae15: Status 404 returned error can't find the container with id 891c6ce0d96f6e9c1837cd1593021c71765f5281e1b0cad7bde968b35bc0ae15
May 26 11:44:33.894157 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:33.894116    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579" event={"ID":"bb0ee137-b57f-4795-b42f-c3f26a00639e","Type":"ContainerStarted","Data":"042d6e8a3b68f98ea45b67fcb5aad2a365beb66cb8c3a7cf9a5e49055d2d2535"}
May 26 11:44:33.894345 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:33.894167    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579" event={"ID":"bb0ee137-b57f-4795-b42f-c3f26a00639e","Type":"ContainerStarted","Data":"5074614b2abceade204359602201a1dd06b2f8c4df33cbe127dcf1f186127804"}
May 26 11:44:33.894345 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:33.894185    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579" event={"ID":"bb0ee137-b57f-4795-b42f-c3f26a00639e","Type":"ContainerStarted","Data":"891c6ce0d96f6e9c1837cd1593021c71765f5281e1b0cad7bde968b35bc0ae15"}
May 26 11:44:33.894449 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:33.894430    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579"
May 26 11:44:33.894576 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:33.894546    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579"
May 26 11:44:33.896056 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:33.896027    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579" podUID="bb0ee137-b57f-4795-b42f-c3f26a00639e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
May 26 11:44:33.896428 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:33.896177    2563 generic.go:358] "Generic (PLEG): container finished" podID="8277b22b-acea-47e9-932e-37813db5a360" containerID="3531569f2a90ec9fdc2bf9e95e2b59f20c8127fa8cc9df9e4e370868a9900d55" exitCode=2
May 26 11:44:33.896428 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:33.896224    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn" event={"ID":"8277b22b-acea-47e9-932e-37813db5a360","Type":"ContainerDied","Data":"3531569f2a90ec9fdc2bf9e95e2b59f20c8127fa8cc9df9e4e370868a9900d55"}
May 26 11:44:33.913442 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:33.913390    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579" podStartSLOduration=0.913378595 podStartE2EDuration="913.378595ms" podCreationTimestamp="2026-05-26 11:44:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:44:33.912662549 +0000 UTC m=+1206.104104737" watchObservedRunningTime="2026-05-26 11:44:33.913378595 +0000 UTC m=+1206.104820782"
May 26 11:44:34.899616 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:34.899578    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579" podUID="bb0ee137-b57f-4795-b42f-c3f26a00639e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
May 26 11:44:35.675260 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:35.675213    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn" podUID="8277b22b-acea-47e9-932e-37813db5a360" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
May 26 11:44:35.679589 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:35.679564    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn" podUID="8277b22b-acea-47e9-932e-37813db5a360" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
May 26 11:44:36.427538 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:36.427504    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn"
May 26 11:44:36.460093 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:36.460014    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wq5nd\" (UniqueName: \"kubernetes.io/projected/8277b22b-acea-47e9-932e-37813db5a360-kube-api-access-wq5nd\") pod \"8277b22b-acea-47e9-932e-37813db5a360\" (UID: \"8277b22b-acea-47e9-932e-37813db5a360\") "
May 26 11:44:36.460237 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:36.460114    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8277b22b-acea-47e9-932e-37813db5a360-proxy-tls\") pod \"8277b22b-acea-47e9-932e-37813db5a360\" (UID: \"8277b22b-acea-47e9-932e-37813db5a360\") "
May 26 11:44:36.460237 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:36.460159    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-378f2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8277b22b-acea-47e9-932e-37813db5a360-error-404-isvc-378f2-kube-rbac-proxy-sar-config\") pod \"8277b22b-acea-47e9-932e-37813db5a360\" (UID: \"8277b22b-acea-47e9-932e-37813db5a360\") "
May 26 11:44:36.460560 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:36.460532    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8277b22b-acea-47e9-932e-37813db5a360-error-404-isvc-378f2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-378f2-kube-rbac-proxy-sar-config") pod "8277b22b-acea-47e9-932e-37813db5a360" (UID: "8277b22b-acea-47e9-932e-37813db5a360"). InnerVolumeSpecName "error-404-isvc-378f2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:44:36.462492 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:36.462469    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8277b22b-acea-47e9-932e-37813db5a360-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8277b22b-acea-47e9-932e-37813db5a360" (UID: "8277b22b-acea-47e9-932e-37813db5a360"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:44:36.462573 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:36.462504    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8277b22b-acea-47e9-932e-37813db5a360-kube-api-access-wq5nd" (OuterVolumeSpecName: "kube-api-access-wq5nd") pod "8277b22b-acea-47e9-932e-37813db5a360" (UID: "8277b22b-acea-47e9-932e-37813db5a360"). InnerVolumeSpecName "kube-api-access-wq5nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:44:36.561043 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:36.561012    2563 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-378f2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8277b22b-acea-47e9-932e-37813db5a360-error-404-isvc-378f2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:44:36.561043 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:36.561039    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq5nd\" (UniqueName: \"kubernetes.io/projected/8277b22b-acea-47e9-932e-37813db5a360-kube-api-access-wq5nd\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:44:36.561221 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:36.561050    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8277b22b-acea-47e9-932e-37813db5a360-proxy-tls\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:44:36.874885 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:36.874848    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-378f2-5985b954cf-x9762" podUID="e88eb924-4af1-4f62-8ba7-1afaf826f46d" containerName="switch-graph-378f2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:44:36.906792 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:36.906747    2563 generic.go:358] "Generic (PLEG): container finished" podID="8277b22b-acea-47e9-932e-37813db5a360" containerID="61eb286df20ecd0f6e6be55d51bd74e2b8dd3d23c4e83ac9220d60f0deecdd87" exitCode=0
May 26 11:44:36.906952 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:36.906809    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn" event={"ID":"8277b22b-acea-47e9-932e-37813db5a360","Type":"ContainerDied","Data":"61eb286df20ecd0f6e6be55d51bd74e2b8dd3d23c4e83ac9220d60f0deecdd87"}
May 26 11:44:36.906952 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:36.906846    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn" event={"ID":"8277b22b-acea-47e9-932e-37813db5a360","Type":"ContainerDied","Data":"791ea6f57bba63cdf2e598da1794c21f148f2d6b7143569455f0df10d469854e"}
May 26 11:44:36.906952 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:36.906862    2563 scope.go:117] "RemoveContainer" containerID="3531569f2a90ec9fdc2bf9e95e2b59f20c8127fa8cc9df9e4e370868a9900d55"
May 26 11:44:36.906952 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:36.906859    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn"
May 26 11:44:36.915115 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:36.915084    2563 scope.go:117] "RemoveContainer" containerID="61eb286df20ecd0f6e6be55d51bd74e2b8dd3d23c4e83ac9220d60f0deecdd87"
May 26 11:44:36.922297 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:36.922280    2563 scope.go:117] "RemoveContainer" containerID="3531569f2a90ec9fdc2bf9e95e2b59f20c8127fa8cc9df9e4e370868a9900d55"
May 26 11:44:36.922570 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:44:36.922549    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3531569f2a90ec9fdc2bf9e95e2b59f20c8127fa8cc9df9e4e370868a9900d55\": container with ID starting with 3531569f2a90ec9fdc2bf9e95e2b59f20c8127fa8cc9df9e4e370868a9900d55 not found: ID does not exist" containerID="3531569f2a90ec9fdc2bf9e95e2b59f20c8127fa8cc9df9e4e370868a9900d55"
May 26 11:44:36.922612 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:36.922581    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3531569f2a90ec9fdc2bf9e95e2b59f20c8127fa8cc9df9e4e370868a9900d55"} err="failed to get container status \"3531569f2a90ec9fdc2bf9e95e2b59f20c8127fa8cc9df9e4e370868a9900d55\": rpc error: code = NotFound desc = could not find container \"3531569f2a90ec9fdc2bf9e95e2b59f20c8127fa8cc9df9e4e370868a9900d55\": container with ID starting with 3531569f2a90ec9fdc2bf9e95e2b59f20c8127fa8cc9df9e4e370868a9900d55 not found: ID does not exist"
May 26 11:44:36.922612 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:36.922599    2563 scope.go:117] "RemoveContainer" containerID="61eb286df20ecd0f6e6be55d51bd74e2b8dd3d23c4e83ac9220d60f0deecdd87"
May 26 11:44:36.922842 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:44:36.922827    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61eb286df20ecd0f6e6be55d51bd74e2b8dd3d23c4e83ac9220d60f0deecdd87\": container with ID starting with 61eb286df20ecd0f6e6be55d51bd74e2b8dd3d23c4e83ac9220d60f0deecdd87 not found: ID does not exist" containerID="61eb286df20ecd0f6e6be55d51bd74e2b8dd3d23c4e83ac9220d60f0deecdd87"
May 26 11:44:36.922894 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:36.922847    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61eb286df20ecd0f6e6be55d51bd74e2b8dd3d23c4e83ac9220d60f0deecdd87"} err="failed to get container status \"61eb286df20ecd0f6e6be55d51bd74e2b8dd3d23c4e83ac9220d60f0deecdd87\": rpc error: code = NotFound desc = could not find container \"61eb286df20ecd0f6e6be55d51bd74e2b8dd3d23c4e83ac9220d60f0deecdd87\": container with ID starting with 61eb286df20ecd0f6e6be55d51bd74e2b8dd3d23c4e83ac9220d60f0deecdd87 not found: ID does not exist"
May 26 11:44:36.927851 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:36.927829    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn"]
May 26 11:44:36.932185 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:36.932164    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-378f2-predictor-d6b46746f-8chzn"]
May 26 11:44:38.428413 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:38.428381    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8277b22b-acea-47e9-932e-37813db5a360" path="/var/lib/kubelet/pods/8277b22b-acea-47e9-932e-37813db5a360/volumes"
May 26 11:44:39.904065 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:39.904032    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579"
May 26 11:44:39.904536 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:39.904507    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579" podUID="bb0ee137-b57f-4795-b42f-c3f26a00639e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
May 26 11:44:41.874297 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:41.874258    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-378f2-5985b954cf-x9762" podUID="e88eb924-4af1-4f62-8ba7-1afaf826f46d" containerName="switch-graph-378f2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:44:46.875050 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:46.875008    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-378f2-5985b954cf-x9762" podUID="e88eb924-4af1-4f62-8ba7-1afaf826f46d" containerName="switch-graph-378f2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:44:46.875418 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:46.875115    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-378f2-5985b954cf-x9762"
May 26 11:44:49.905129 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:49.905086    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579" podUID="bb0ee137-b57f-4795-b42f-c3f26a00639e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
May 26 11:44:51.874986 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:51.874945    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-378f2-5985b954cf-x9762" podUID="e88eb924-4af1-4f62-8ba7-1afaf826f46d" containerName="switch-graph-378f2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:44:52.827063 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:52.827028    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-866798f95b-zp2x7"]
May 26 11:44:52.827359 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:52.827346    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8277b22b-acea-47e9-932e-37813db5a360" containerName="kube-rbac-proxy"
May 26 11:44:52.827403 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:52.827361    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8277b22b-acea-47e9-932e-37813db5a360" containerName="kube-rbac-proxy"
May 26 11:44:52.827403 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:52.827370    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8277b22b-acea-47e9-932e-37813db5a360" containerName="kserve-container"
May 26 11:44:52.827403 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:52.827375    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8277b22b-acea-47e9-932e-37813db5a360" containerName="kserve-container"
May 26 11:44:52.827487 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:52.827418    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="8277b22b-acea-47e9-932e-37813db5a360" containerName="kube-rbac-proxy"
May 26 11:44:52.827487 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:52.827429    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="8277b22b-acea-47e9-932e-37813db5a360" containerName="kserve-container"
May 26 11:44:52.830236 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:52.830221    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-866798f95b-zp2x7"
May 26 11:44:52.833472 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:52.833447    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
May 26 11:44:52.833919 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:52.833895    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
May 26 11:44:52.839253 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:52.839232    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-866798f95b-zp2x7"]
May 26 11:44:52.896927 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:52.896892    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27d83a5b-252b-4c74-9487-6656ec81ec5a-openshift-service-ca-bundle\") pod \"model-chainer-866798f95b-zp2x7\" (UID: \"27d83a5b-252b-4c74-9487-6656ec81ec5a\") " pod="kserve-ci-e2e-test/model-chainer-866798f95b-zp2x7"
May 26 11:44:52.896927 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:52.896932    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27d83a5b-252b-4c74-9487-6656ec81ec5a-proxy-tls\") pod \"model-chainer-866798f95b-zp2x7\" (UID: \"27d83a5b-252b-4c74-9487-6656ec81ec5a\") " pod="kserve-ci-e2e-test/model-chainer-866798f95b-zp2x7"
May 26 11:44:52.997721 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:52.997686    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27d83a5b-252b-4c74-9487-6656ec81ec5a-openshift-service-ca-bundle\") pod \"model-chainer-866798f95b-zp2x7\" (UID: \"27d83a5b-252b-4c74-9487-6656ec81ec5a\") " pod="kserve-ci-e2e-test/model-chainer-866798f95b-zp2x7"
May 26 11:44:52.997721 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:52.997723    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27d83a5b-252b-4c74-9487-6656ec81ec5a-proxy-tls\") pod \"model-chainer-866798f95b-zp2x7\" (UID: \"27d83a5b-252b-4c74-9487-6656ec81ec5a\") " pod="kserve-ci-e2e-test/model-chainer-866798f95b-zp2x7"
May 26 11:44:52.998372 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:52.998344    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27d83a5b-252b-4c74-9487-6656ec81ec5a-openshift-service-ca-bundle\") pod \"model-chainer-866798f95b-zp2x7\" (UID: \"27d83a5b-252b-4c74-9487-6656ec81ec5a\") " pod="kserve-ci-e2e-test/model-chainer-866798f95b-zp2x7"
May 26 11:44:53.000414 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:53.000394    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27d83a5b-252b-4c74-9487-6656ec81ec5a-proxy-tls\") pod \"model-chainer-866798f95b-zp2x7\" (UID: \"27d83a5b-252b-4c74-9487-6656ec81ec5a\") " pod="kserve-ci-e2e-test/model-chainer-866798f95b-zp2x7"
May 26 11:44:53.140837 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:53.140795    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-866798f95b-zp2x7"
May 26 11:44:53.271925 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:53.271886    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-866798f95b-zp2x7"]
May 26 11:44:53.274766 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:44:53.274737    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d83a5b_252b_4c74_9487_6656ec81ec5a.slice/crio-e046bed264b92465eeef0f18b522cb4dc2ded03bf25c2eb8ef38ead887da1864 WatchSource:0}: Error finding container e046bed264b92465eeef0f18b522cb4dc2ded03bf25c2eb8ef38ead887da1864: Status 404 returned error can't find the container with id e046bed264b92465eeef0f18b522cb4dc2ded03bf25c2eb8ef38ead887da1864
May 26 11:44:53.958279 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:53.958241    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-866798f95b-zp2x7" event={"ID":"27d83a5b-252b-4c74-9487-6656ec81ec5a","Type":"ContainerStarted","Data":"e33b8e50e407316013a6cfc54276b01c9a2f03569173a9b6021163bb00b7aa90"}
May 26 11:44:53.958279 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:53.958279    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-866798f95b-zp2x7" event={"ID":"27d83a5b-252b-4c74-9487-6656ec81ec5a","Type":"ContainerStarted","Data":"e046bed264b92465eeef0f18b522cb4dc2ded03bf25c2eb8ef38ead887da1864"}
May 26 11:44:53.958706 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:53.958309    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-866798f95b-zp2x7"
May 26 11:44:53.976286 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:53.976232    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-866798f95b-zp2x7" podStartSLOduration=1.976217852 podStartE2EDuration="1.976217852s" podCreationTimestamp="2026-05-26 11:44:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:44:53.974859629 +0000 UTC m=+1226.166301817" watchObservedRunningTime="2026-05-26 11:44:53.976217852 +0000 UTC m=+1226.167660039"
May 26 11:44:56.874959 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:56.874913    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-378f2-5985b954cf-x9762" podUID="e88eb924-4af1-4f62-8ba7-1afaf826f46d" containerName="switch-graph-378f2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:44:59.904824 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:59.904788    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579" podUID="bb0ee137-b57f-4795-b42f-c3f26a00639e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
May 26 11:44:59.966471 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:44:59.966443    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-866798f95b-zp2x7"
May 26 11:45:01.874975 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:01.874932    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-378f2-5985b954cf-x9762" podUID="e88eb924-4af1-4f62-8ba7-1afaf826f46d" containerName="switch-graph-378f2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:45:02.816528 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:02.816497    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-866798f95b-zp2x7"]
May 26 11:45:02.816785 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:02.816720    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-866798f95b-zp2x7" podUID="27d83a5b-252b-4c74-9487-6656ec81ec5a" containerName="model-chainer" containerID="cri-o://e33b8e50e407316013a6cfc54276b01c9a2f03569173a9b6021163bb00b7aa90" gracePeriod=30
May 26 11:45:02.983526 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:02.983482    2563 generic.go:358] "Generic (PLEG): container finished" podID="e88eb924-4af1-4f62-8ba7-1afaf826f46d" containerID="128dc5b8fd48a01cd1f968ff77f7adf95d84575aeb0378af3fb170ac310a829b" exitCode=0
May 26 11:45:02.983925 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:02.983579    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-378f2-5985b954cf-x9762" event={"ID":"e88eb924-4af1-4f62-8ba7-1afaf826f46d","Type":"ContainerDied","Data":"128dc5b8fd48a01cd1f968ff77f7adf95d84575aeb0378af3fb170ac310a829b"}
May 26 11:45:03.016925 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.016893    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468"]
May 26 11:45:03.017239 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.017200    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468" podUID="0805d6c1-804e-4bac-93cc-ea3e36a86829" containerName="kserve-container" containerID="cri-o://50b21847803594aafdaa83774a3cf41e4a56617ea8c5c48c0453d9aca28aadd5" gracePeriod=30
May 26 11:45:03.017312 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.017265    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468" podUID="0805d6c1-804e-4bac-93cc-ea3e36a86829" containerName="kube-rbac-proxy" containerID="cri-o://3692cd94cb6dc72f4976da364dc31ebbe8eb6fd436fb8678c9a5e7d314c784d8" gracePeriod=30
May 26 11:45:03.100188 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.100159    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g"]
May 26 11:45:03.104551 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.104534    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g"
May 26 11:45:03.107454 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.107429    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-5b0de-predictor-serving-cert\""
May 26 11:45:03.107576 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.107459    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-5b0de-kube-rbac-proxy-sar-config\""
May 26 11:45:03.114351 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.114326    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g"]
May 26 11:45:03.184016 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.183981    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxt5c\" (UniqueName: \"kubernetes.io/projected/d85374dd-6464-461d-9e24-50bd9911b878-kube-api-access-pxt5c\") pod \"error-404-isvc-5b0de-predictor-6766f587d9-qlg2g\" (UID: \"d85374dd-6464-461d-9e24-50bd9911b878\") " pod="kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g"
May 26 11:45:03.184173 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.184100    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d85374dd-6464-461d-9e24-50bd9911b878-proxy-tls\") pod \"error-404-isvc-5b0de-predictor-6766f587d9-qlg2g\" (UID: \"d85374dd-6464-461d-9e24-50bd9911b878\") " pod="kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g"
May 26 11:45:03.184217 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.184174    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-5b0de-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d85374dd-6464-461d-9e24-50bd9911b878-error-404-isvc-5b0de-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5b0de-predictor-6766f587d9-qlg2g\" (UID: \"d85374dd-6464-461d-9e24-50bd9911b878\") " pod="kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g"
May 26 11:45:03.284801 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.284747    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pxt5c\" (UniqueName: \"kubernetes.io/projected/d85374dd-6464-461d-9e24-50bd9911b878-kube-api-access-pxt5c\") pod \"error-404-isvc-5b0de-predictor-6766f587d9-qlg2g\" (UID: \"d85374dd-6464-461d-9e24-50bd9911b878\") " pod="kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g"
May 26 11:45:03.285013 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.284826    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d85374dd-6464-461d-9e24-50bd9911b878-proxy-tls\") pod \"error-404-isvc-5b0de-predictor-6766f587d9-qlg2g\" (UID: \"d85374dd-6464-461d-9e24-50bd9911b878\") " pod="kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g"
May 26 11:45:03.285013 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.284868    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-5b0de-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d85374dd-6464-461d-9e24-50bd9911b878-error-404-isvc-5b0de-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5b0de-predictor-6766f587d9-qlg2g\" (UID: \"d85374dd-6464-461d-9e24-50bd9911b878\") " pod="kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g"
May 26 11:45:03.285500 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.285480    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-5b0de-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d85374dd-6464-461d-9e24-50bd9911b878-error-404-isvc-5b0de-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5b0de-predictor-6766f587d9-qlg2g\" (UID: \"d85374dd-6464-461d-9e24-50bd9911b878\") " pod="kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g"
May 26 11:45:03.287418 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.287390    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d85374dd-6464-461d-9e24-50bd9911b878-proxy-tls\") pod \"error-404-isvc-5b0de-predictor-6766f587d9-qlg2g\" (UID: \"d85374dd-6464-461d-9e24-50bd9911b878\") " pod="kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g"
May 26 11:45:03.294366 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.294338    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxt5c\" (UniqueName: \"kubernetes.io/projected/d85374dd-6464-461d-9e24-50bd9911b878-kube-api-access-pxt5c\") pod \"error-404-isvc-5b0de-predictor-6766f587d9-qlg2g\" (UID: \"d85374dd-6464-461d-9e24-50bd9911b878\") " pod="kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g"
May 26 11:45:03.415574 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.415537    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g"
May 26 11:45:03.543872 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.543849    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-378f2-5985b954cf-x9762"
May 26 11:45:03.553154 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.553121    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g"]
May 26 11:45:03.555994 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:45:03.555968    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd85374dd_6464_461d_9e24_50bd9911b878.slice/crio-2ad5bf0d2e021a10d8f552ff5afcfb80c459ea109bf86f2ac316413b8f6a8ff9 WatchSource:0}: Error finding container 2ad5bf0d2e021a10d8f552ff5afcfb80c459ea109bf86f2ac316413b8f6a8ff9: Status 404 returned error can't find the container with id 2ad5bf0d2e021a10d8f552ff5afcfb80c459ea109bf86f2ac316413b8f6a8ff9
May 26 11:45:03.685439 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.685397    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468" podUID="0805d6c1-804e-4bac-93cc-ea3e36a86829" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.18:8643/healthz\": dial tcp 10.134.0.18:8643: connect: connection refused"
May 26 11:45:03.687675 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.687648    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e88eb924-4af1-4f62-8ba7-1afaf826f46d-openshift-service-ca-bundle\") pod \"e88eb924-4af1-4f62-8ba7-1afaf826f46d\" (UID: \"e88eb924-4af1-4f62-8ba7-1afaf826f46d\") "
May 26 11:45:03.687821 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.687703    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e88eb924-4af1-4f62-8ba7-1afaf826f46d-proxy-tls\") pod \"e88eb924-4af1-4f62-8ba7-1afaf826f46d\" (UID: \"e88eb924-4af1-4f62-8ba7-1afaf826f46d\") "
May 26 11:45:03.688053 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.688031    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e88eb924-4af1-4f62-8ba7-1afaf826f46d-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "e88eb924-4af1-4f62-8ba7-1afaf826f46d" (UID: "e88eb924-4af1-4f62-8ba7-1afaf826f46d"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:45:03.690078 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.690058    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e88eb924-4af1-4f62-8ba7-1afaf826f46d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e88eb924-4af1-4f62-8ba7-1afaf826f46d" (UID: "e88eb924-4af1-4f62-8ba7-1afaf826f46d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:45:03.789013 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.788976    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e88eb924-4af1-4f62-8ba7-1afaf826f46d-openshift-service-ca-bundle\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:45:03.789013 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.789010    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e88eb924-4af1-4f62-8ba7-1afaf826f46d-proxy-tls\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:45:03.988801 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.988691    2563 generic.go:358] "Generic (PLEG): container finished" podID="0805d6c1-804e-4bac-93cc-ea3e36a86829" containerID="3692cd94cb6dc72f4976da364dc31ebbe8eb6fd436fb8678c9a5e7d314c784d8" exitCode=2
May 26 11:45:03.988801 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.988769    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468" event={"ID":"0805d6c1-804e-4bac-93cc-ea3e36a86829","Type":"ContainerDied","Data":"3692cd94cb6dc72f4976da364dc31ebbe8eb6fd436fb8678c9a5e7d314c784d8"}
May 26 11:45:03.990383 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.990358    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g" event={"ID":"d85374dd-6464-461d-9e24-50bd9911b878","Type":"ContainerStarted","Data":"7acbf801fb21c14a09c2b1b70d27e070a84e61831c8689bba2866c7c666c7a82"}
May 26 11:45:03.990510 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.990392    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g" event={"ID":"d85374dd-6464-461d-9e24-50bd9911b878","Type":"ContainerStarted","Data":"ac734cfffc0214426cad2770a55e3b995756751363db21591436b42764037a47"}
May 26 11:45:03.990510 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.990412    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g" event={"ID":"d85374dd-6464-461d-9e24-50bd9911b878","Type":"ContainerStarted","Data":"2ad5bf0d2e021a10d8f552ff5afcfb80c459ea109bf86f2ac316413b8f6a8ff9"}
May 26 11:45:03.990629 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.990598    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g"
May 26 11:45:03.990728 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.990712    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g"
May 26 11:45:03.991633 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.991608    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-378f2-5985b954cf-x9762" event={"ID":"e88eb924-4af1-4f62-8ba7-1afaf826f46d","Type":"ContainerDied","Data":"2ffc95a816fe3b340767f926b39fb8d298e34cc88213462f0bc76561f79aa845"}
May 26 11:45:03.991800 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.991648    2563 scope.go:117] "RemoveContainer" containerID="128dc5b8fd48a01cd1f968ff77f7adf95d84575aeb0378af3fb170ac310a829b"
May 26 11:45:03.991800 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.991655    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-378f2-5985b954cf-x9762"
May 26 11:45:03.992517 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:03.992430    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g" podUID="d85374dd-6464-461d-9e24-50bd9911b878" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
May 26 11:45:04.010083 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:04.010030    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g" podStartSLOduration=1.010016267 podStartE2EDuration="1.010016267s" podCreationTimestamp="2026-05-26 11:45:03 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:45:04.008971725 +0000 UTC m=+1236.200413903" watchObservedRunningTime="2026-05-26 11:45:04.010016267 +0000 UTC m=+1236.201458448"
May 26 11:45:04.023526 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:04.023497    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-378f2-5985b954cf-x9762"]
May 26 11:45:04.026522 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:04.026499    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-378f2-5985b954cf-x9762"]
May 26 11:45:04.428696 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:04.428664    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e88eb924-4af1-4f62-8ba7-1afaf826f46d" path="/var/lib/kubelet/pods/e88eb924-4af1-4f62-8ba7-1afaf826f46d/volumes"
May 26 11:45:04.965556 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:04.965520    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-866798f95b-zp2x7" podUID="27d83a5b-252b-4c74-9487-6656ec81ec5a" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:45:04.997375 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:04.997328    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g" podUID="d85374dd-6464-461d-9e24-50bd9911b878" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
May 26 11:45:07.566531 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:07.566509    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468"
May 26 11:45:07.718938 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:07.718844    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0805d6c1-804e-4bac-93cc-ea3e36a86829-kserve-provision-location\") pod \"0805d6c1-804e-4bac-93cc-ea3e36a86829\" (UID: \"0805d6c1-804e-4bac-93cc-ea3e36a86829\") "
May 26 11:45:07.718938 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:07.718902    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0805d6c1-804e-4bac-93cc-ea3e36a86829-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\") pod \"0805d6c1-804e-4bac-93cc-ea3e36a86829\" (UID: \"0805d6c1-804e-4bac-93cc-ea3e36a86829\") "
May 26 11:45:07.718938 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:07.718933    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0805d6c1-804e-4bac-93cc-ea3e36a86829-proxy-tls\") pod \"0805d6c1-804e-4bac-93cc-ea3e36a86829\" (UID: \"0805d6c1-804e-4bac-93cc-ea3e36a86829\") "
May 26 11:45:07.719217 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:07.718975    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srlrn\" (UniqueName: \"kubernetes.io/projected/0805d6c1-804e-4bac-93cc-ea3e36a86829-kube-api-access-srlrn\") pod \"0805d6c1-804e-4bac-93cc-ea3e36a86829\" (UID: \"0805d6c1-804e-4bac-93cc-ea3e36a86829\") "
May 26 11:45:07.719297 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:07.719265    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0805d6c1-804e-4bac-93cc-ea3e36a86829-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0805d6c1-804e-4bac-93cc-ea3e36a86829" (UID: "0805d6c1-804e-4bac-93cc-ea3e36a86829"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:45:07.719344 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:07.719322    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0805d6c1-804e-4bac-93cc-ea3e36a86829-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-graph-1-kube-rbac-proxy-sar-config") pod "0805d6c1-804e-4bac-93cc-ea3e36a86829" (UID: "0805d6c1-804e-4bac-93cc-ea3e36a86829"). InnerVolumeSpecName "isvc-sklearn-graph-1-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:45:07.721294 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:07.721264    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0805d6c1-804e-4bac-93cc-ea3e36a86829-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0805d6c1-804e-4bac-93cc-ea3e36a86829" (UID: "0805d6c1-804e-4bac-93cc-ea3e36a86829"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:45:07.721395 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:07.721309    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0805d6c1-804e-4bac-93cc-ea3e36a86829-kube-api-access-srlrn" (OuterVolumeSpecName: "kube-api-access-srlrn") pod "0805d6c1-804e-4bac-93cc-ea3e36a86829" (UID: "0805d6c1-804e-4bac-93cc-ea3e36a86829"). InnerVolumeSpecName "kube-api-access-srlrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:45:07.819689 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:07.819654    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0805d6c1-804e-4bac-93cc-ea3e36a86829-kserve-provision-location\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:45:07.819689 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:07.819683    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0805d6c1-804e-4bac-93cc-ea3e36a86829-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:45:07.819689 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:07.819694    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0805d6c1-804e-4bac-93cc-ea3e36a86829-proxy-tls\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:45:07.819946 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:07.819704    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srlrn\" (UniqueName: \"kubernetes.io/projected/0805d6c1-804e-4bac-93cc-ea3e36a86829-kube-api-access-srlrn\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:45:08.012480 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:08.012391    2563 generic.go:358] "Generic (PLEG): container finished" podID="0805d6c1-804e-4bac-93cc-ea3e36a86829" containerID="50b21847803594aafdaa83774a3cf41e4a56617ea8c5c48c0453d9aca28aadd5" exitCode=0
May 26 11:45:08.012480 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:08.012453    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468" event={"ID":"0805d6c1-804e-4bac-93cc-ea3e36a86829","Type":"ContainerDied","Data":"50b21847803594aafdaa83774a3cf41e4a56617ea8c5c48c0453d9aca28aadd5"}
May 26 11:45:08.012697 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:08.012485    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468" event={"ID":"0805d6c1-804e-4bac-93cc-ea3e36a86829","Type":"ContainerDied","Data":"d7eaa4e758da79a187ce0f576da557d9f68bef303f10c3fc27333bae00999e6b"}
May 26 11:45:08.012697 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:08.012507    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468"
May 26 11:45:08.012697 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:08.012504    2563 scope.go:117] "RemoveContainer" containerID="3692cd94cb6dc72f4976da364dc31ebbe8eb6fd436fb8678c9a5e7d314c784d8"
May 26 11:45:08.020713 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:08.020680    2563 scope.go:117] "RemoveContainer" containerID="50b21847803594aafdaa83774a3cf41e4a56617ea8c5c48c0453d9aca28aadd5"
May 26 11:45:08.027953 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:08.027935    2563 scope.go:117] "RemoveContainer" containerID="017e2746322c6124db785987256fa987170a21b1904d14b2a8411d9fe37102a8"
May 26 11:45:08.034981 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:08.034965    2563 scope.go:117] "RemoveContainer" containerID="3692cd94cb6dc72f4976da364dc31ebbe8eb6fd436fb8678c9a5e7d314c784d8"
May 26 11:45:08.035155 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:08.035135    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468"]
May 26 11:45:08.035245 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:45:08.035227    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3692cd94cb6dc72f4976da364dc31ebbe8eb6fd436fb8678c9a5e7d314c784d8\": container with ID starting with 3692cd94cb6dc72f4976da364dc31ebbe8eb6fd436fb8678c9a5e7d314c784d8 not found: ID does not exist" containerID="3692cd94cb6dc72f4976da364dc31ebbe8eb6fd436fb8678c9a5e7d314c784d8"
May 26 11:45:08.035311 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:08.035257    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3692cd94cb6dc72f4976da364dc31ebbe8eb6fd436fb8678c9a5e7d314c784d8"} err="failed to get container status \"3692cd94cb6dc72f4976da364dc31ebbe8eb6fd436fb8678c9a5e7d314c784d8\": rpc error: code = NotFound desc = could not find container \"3692cd94cb6dc72f4976da364dc31ebbe8eb6fd436fb8678c9a5e7d314c784d8\": container with ID starting with 3692cd94cb6dc72f4976da364dc31ebbe8eb6fd436fb8678c9a5e7d314c784d8 not found: ID does not exist"
May 26 11:45:08.035311 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:08.035283    2563 scope.go:117] "RemoveContainer" containerID="50b21847803594aafdaa83774a3cf41e4a56617ea8c5c48c0453d9aca28aadd5"
May 26 11:45:08.035569 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:45:08.035547    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50b21847803594aafdaa83774a3cf41e4a56617ea8c5c48c0453d9aca28aadd5\": container with ID starting with 50b21847803594aafdaa83774a3cf41e4a56617ea8c5c48c0453d9aca28aadd5 not found: ID does not exist" containerID="50b21847803594aafdaa83774a3cf41e4a56617ea8c5c48c0453d9aca28aadd5"
May 26 11:45:08.035663 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:08.035573    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50b21847803594aafdaa83774a3cf41e4a56617ea8c5c48c0453d9aca28aadd5"} err="failed to get container status \"50b21847803594aafdaa83774a3cf41e4a56617ea8c5c48c0453d9aca28aadd5\": rpc error: code = NotFound desc = could not find container \"50b21847803594aafdaa83774a3cf41e4a56617ea8c5c48c0453d9aca28aadd5\": container with ID starting with 50b21847803594aafdaa83774a3cf41e4a56617ea8c5c48c0453d9aca28aadd5 not found: ID does not exist"
May 26 11:45:08.035663 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:08.035595    2563 scope.go:117] "RemoveContainer" containerID="017e2746322c6124db785987256fa987170a21b1904d14b2a8411d9fe37102a8"
May 26 11:45:08.035913 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:45:08.035891    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"017e2746322c6124db785987256fa987170a21b1904d14b2a8411d9fe37102a8\": container with ID starting with 017e2746322c6124db785987256fa987170a21b1904d14b2a8411d9fe37102a8 not found: ID does not exist" containerID="017e2746322c6124db785987256fa987170a21b1904d14b2a8411d9fe37102a8"
May 26 11:45:08.035978 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:08.035921    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"017e2746322c6124db785987256fa987170a21b1904d14b2a8411d9fe37102a8"} err="failed to get container status \"017e2746322c6124db785987256fa987170a21b1904d14b2a8411d9fe37102a8\": rpc error: code = NotFound desc = could not find container \"017e2746322c6124db785987256fa987170a21b1904d14b2a8411d9fe37102a8\": container with ID starting with 017e2746322c6124db785987256fa987170a21b1904d14b2a8411d9fe37102a8 not found: ID does not exist"
May 26 11:45:08.038976 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:08.038956    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-8m468"]
May 26 11:45:08.428405 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:08.428366    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0805d6c1-804e-4bac-93cc-ea3e36a86829" path="/var/lib/kubelet/pods/0805d6c1-804e-4bac-93cc-ea3e36a86829/volumes"
May 26 11:45:09.905086 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:09.905050    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579" podUID="bb0ee137-b57f-4795-b42f-c3f26a00639e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
May 26 11:45:09.965896 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:09.965852    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-866798f95b-zp2x7" podUID="27d83a5b-252b-4c74-9487-6656ec81ec5a" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:45:10.007669 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:10.007640    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g"
May 26 11:45:10.008216 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:10.008190    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g" podUID="d85374dd-6464-461d-9e24-50bd9911b878" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
May 26 11:45:14.965622 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:14.965534    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-866798f95b-zp2x7" podUID="27d83a5b-252b-4c74-9487-6656ec81ec5a" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:45:14.965998 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:14.965647    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-866798f95b-zp2x7"
May 26 11:45:19.904947 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:19.904920    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579"
May 26 11:45:19.965716 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:19.965671    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-866798f95b-zp2x7" podUID="27d83a5b-252b-4c74-9487-6656ec81ec5a" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:45:20.009007 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:20.008970    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g" podUID="d85374dd-6464-461d-9e24-50bd9911b878" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
May 26 11:45:24.965734 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:24.965687    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-866798f95b-zp2x7" podUID="27d83a5b-252b-4c74-9487-6656ec81ec5a" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:45:29.965976 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:29.965934    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-866798f95b-zp2x7" podUID="27d83a5b-252b-4c74-9487-6656ec81ec5a" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:45:30.008364 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:30.008326    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g" podUID="d85374dd-6464-461d-9e24-50bd9911b878" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
May 26 11:45:32.856908 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:45:32.856865    2563 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d83a5b_252b_4c74_9487_6656ec81ec5a.slice/crio-conmon-e33b8e50e407316013a6cfc54276b01c9a2f03569173a9b6021163bb00b7aa90.scope\": RecentStats: unable to find data in memory cache]"
May 26 11:45:32.969728 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:32.969705    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-866798f95b-zp2x7"
May 26 11:45:33.022422 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.022388    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27d83a5b-252b-4c74-9487-6656ec81ec5a-openshift-service-ca-bundle\") pod \"27d83a5b-252b-4c74-9487-6656ec81ec5a\" (UID: \"27d83a5b-252b-4c74-9487-6656ec81ec5a\") "
May 26 11:45:33.022574 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.022431    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27d83a5b-252b-4c74-9487-6656ec81ec5a-proxy-tls\") pod \"27d83a5b-252b-4c74-9487-6656ec81ec5a\" (UID: \"27d83a5b-252b-4c74-9487-6656ec81ec5a\") "
May 26 11:45:33.022811 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.022760    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27d83a5b-252b-4c74-9487-6656ec81ec5a-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "27d83a5b-252b-4c74-9487-6656ec81ec5a" (UID: "27d83a5b-252b-4c74-9487-6656ec81ec5a"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:45:33.024681 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.024653    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27d83a5b-252b-4c74-9487-6656ec81ec5a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "27d83a5b-252b-4c74-9487-6656ec81ec5a" (UID: "27d83a5b-252b-4c74-9487-6656ec81ec5a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:45:33.088061 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.088028    2563 generic.go:358] "Generic (PLEG): container finished" podID="27d83a5b-252b-4c74-9487-6656ec81ec5a" containerID="e33b8e50e407316013a6cfc54276b01c9a2f03569173a9b6021163bb00b7aa90" exitCode=0
May 26 11:45:33.088252 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.088093    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-866798f95b-zp2x7"
May 26 11:45:33.088252 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.088122    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-866798f95b-zp2x7" event={"ID":"27d83a5b-252b-4c74-9487-6656ec81ec5a","Type":"ContainerDied","Data":"e33b8e50e407316013a6cfc54276b01c9a2f03569173a9b6021163bb00b7aa90"}
May 26 11:45:33.088252 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.088163    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-866798f95b-zp2x7" event={"ID":"27d83a5b-252b-4c74-9487-6656ec81ec5a","Type":"ContainerDied","Data":"e046bed264b92465eeef0f18b522cb4dc2ded03bf25c2eb8ef38ead887da1864"}
May 26 11:45:33.088252 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.088180    2563 scope.go:117] "RemoveContainer" containerID="e33b8e50e407316013a6cfc54276b01c9a2f03569173a9b6021163bb00b7aa90"
May 26 11:45:33.096698 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.096674    2563 scope.go:117] "RemoveContainer" containerID="e33b8e50e407316013a6cfc54276b01c9a2f03569173a9b6021163bb00b7aa90"
May 26 11:45:33.097051 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:45:33.097029    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e33b8e50e407316013a6cfc54276b01c9a2f03569173a9b6021163bb00b7aa90\": container with ID starting with e33b8e50e407316013a6cfc54276b01c9a2f03569173a9b6021163bb00b7aa90 not found: ID does not exist" containerID="e33b8e50e407316013a6cfc54276b01c9a2f03569173a9b6021163bb00b7aa90"
May 26 11:45:33.097118 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.097061    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e33b8e50e407316013a6cfc54276b01c9a2f03569173a9b6021163bb00b7aa90"} err="failed to get container status \"e33b8e50e407316013a6cfc54276b01c9a2f03569173a9b6021163bb00b7aa90\": rpc error: code = NotFound desc = could not find container \"e33b8e50e407316013a6cfc54276b01c9a2f03569173a9b6021163bb00b7aa90\": container with ID starting with e33b8e50e407316013a6cfc54276b01c9a2f03569173a9b6021163bb00b7aa90 not found: ID does not exist"
May 26 11:45:33.112585 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.112554    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-866798f95b-zp2x7"]
May 26 11:45:33.121927 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.121899    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-866798f95b-zp2x7"]
May 26 11:45:33.122954 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.122930    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27d83a5b-252b-4c74-9487-6656ec81ec5a-openshift-service-ca-bundle\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:45:33.123059 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.122958    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27d83a5b-252b-4c74-9487-6656ec81ec5a-proxy-tls\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:45:33.143649 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.143608    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-90450-79bdccd478-dd87q"]
May 26 11:45:33.143976 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.143961    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e88eb924-4af1-4f62-8ba7-1afaf826f46d" containerName="switch-graph-378f2"
May 26 11:45:33.144029 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.143979    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e88eb924-4af1-4f62-8ba7-1afaf826f46d" containerName="switch-graph-378f2"
May 26 11:45:33.144029 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.143989    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0805d6c1-804e-4bac-93cc-ea3e36a86829" containerName="storage-initializer"
May 26 11:45:33.144029 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.143994    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0805d6c1-804e-4bac-93cc-ea3e36a86829" containerName="storage-initializer"
May 26 11:45:33.144029 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.144003    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0805d6c1-804e-4bac-93cc-ea3e36a86829" containerName="kserve-container"
May 26 11:45:33.144029 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.144012    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0805d6c1-804e-4bac-93cc-ea3e36a86829" containerName="kserve-container"
May 26 11:45:33.144204 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.144032    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0805d6c1-804e-4bac-93cc-ea3e36a86829" containerName="kube-rbac-proxy"
May 26 11:45:33.144204 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.144037    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0805d6c1-804e-4bac-93cc-ea3e36a86829" containerName="kube-rbac-proxy"
May 26 11:45:33.144204 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.144043    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27d83a5b-252b-4c74-9487-6656ec81ec5a" containerName="model-chainer"
May 26 11:45:33.144204 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.144048    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="27d83a5b-252b-4c74-9487-6656ec81ec5a" containerName="model-chainer"
May 26 11:45:33.144204 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.144105    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="0805d6c1-804e-4bac-93cc-ea3e36a86829" containerName="kube-rbac-proxy"
May 26 11:45:33.144204 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.144114    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="0805d6c1-804e-4bac-93cc-ea3e36a86829" containerName="kserve-container"
May 26 11:45:33.144204 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.144120    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="27d83a5b-252b-4c74-9487-6656ec81ec5a" containerName="model-chainer"
May 26 11:45:33.144204 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.144126    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="e88eb924-4af1-4f62-8ba7-1afaf826f46d" containerName="switch-graph-378f2"
May 26 11:45:33.148460 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.148440    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-90450-79bdccd478-dd87q"
May 26 11:45:33.151142 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.151119    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-90450-serving-cert\""
May 26 11:45:33.151218 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.151133    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-90450-kube-rbac-proxy-sar-config\""
May 26 11:45:33.157884 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.157858    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-90450-79bdccd478-dd87q"]
May 26 11:45:33.223470 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.223434    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a31ec7d-e904-45d7-a4e6-ae2c0120a78e-openshift-service-ca-bundle\") pod \"switch-graph-90450-79bdccd478-dd87q\" (UID: \"1a31ec7d-e904-45d7-a4e6-ae2c0120a78e\") " pod="kserve-ci-e2e-test/switch-graph-90450-79bdccd478-dd87q"
May 26 11:45:33.223651 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.223482    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a31ec7d-e904-45d7-a4e6-ae2c0120a78e-proxy-tls\") pod \"switch-graph-90450-79bdccd478-dd87q\" (UID: \"1a31ec7d-e904-45d7-a4e6-ae2c0120a78e\") " pod="kserve-ci-e2e-test/switch-graph-90450-79bdccd478-dd87q"
May 26 11:45:33.324636 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.324602    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a31ec7d-e904-45d7-a4e6-ae2c0120a78e-openshift-service-ca-bundle\") pod \"switch-graph-90450-79bdccd478-dd87q\" (UID: \"1a31ec7d-e904-45d7-a4e6-ae2c0120a78e\") " pod="kserve-ci-e2e-test/switch-graph-90450-79bdccd478-dd87q"
May 26 11:45:33.324825 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.324648    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a31ec7d-e904-45d7-a4e6-ae2c0120a78e-proxy-tls\") pod \"switch-graph-90450-79bdccd478-dd87q\" (UID: \"1a31ec7d-e904-45d7-a4e6-ae2c0120a78e\") " pod="kserve-ci-e2e-test/switch-graph-90450-79bdccd478-dd87q"
May 26 11:45:33.325242 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.325222    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a31ec7d-e904-45d7-a4e6-ae2c0120a78e-openshift-service-ca-bundle\") pod \"switch-graph-90450-79bdccd478-dd87q\" (UID: \"1a31ec7d-e904-45d7-a4e6-ae2c0120a78e\") " pod="kserve-ci-e2e-test/switch-graph-90450-79bdccd478-dd87q"
May 26 11:45:33.327161 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.327140    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a31ec7d-e904-45d7-a4e6-ae2c0120a78e-proxy-tls\") pod \"switch-graph-90450-79bdccd478-dd87q\" (UID: \"1a31ec7d-e904-45d7-a4e6-ae2c0120a78e\") " pod="kserve-ci-e2e-test/switch-graph-90450-79bdccd478-dd87q"
May 26 11:45:33.458892 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.458762    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-90450-79bdccd478-dd87q"
May 26 11:45:33.588620 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:33.588587    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-90450-79bdccd478-dd87q"]
May 26 11:45:33.591648 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:45:33.591618    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a31ec7d_e904_45d7_a4e6_ae2c0120a78e.slice/crio-d697fa9655afce577eca14a7336b5c3ecefdae2e2b9d5f129dcb144dc6b0750d WatchSource:0}: Error finding container d697fa9655afce577eca14a7336b5c3ecefdae2e2b9d5f129dcb144dc6b0750d: Status 404 returned error can't find the container with id d697fa9655afce577eca14a7336b5c3ecefdae2e2b9d5f129dcb144dc6b0750d
May 26 11:45:34.092888 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:34.092849    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-90450-79bdccd478-dd87q" event={"ID":"1a31ec7d-e904-45d7-a4e6-ae2c0120a78e","Type":"ContainerStarted","Data":"019d9210a39bc0927eb7230a5fe35e18032c91386a4029fb375995c5a1757659"}
May 26 11:45:34.092888 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:34.092888    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-90450-79bdccd478-dd87q" event={"ID":"1a31ec7d-e904-45d7-a4e6-ae2c0120a78e","Type":"ContainerStarted","Data":"d697fa9655afce577eca14a7336b5c3ecefdae2e2b9d5f129dcb144dc6b0750d"}
May 26 11:45:34.093282 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:34.092988    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-90450-79bdccd478-dd87q"
May 26 11:45:34.110274 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:34.110210    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-90450-79bdccd478-dd87q" podStartSLOduration=1.110193919 podStartE2EDuration="1.110193919s" podCreationTimestamp="2026-05-26 11:45:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:45:34.109149695 +0000 UTC m=+1266.300591883" watchObservedRunningTime="2026-05-26 11:45:34.110193919 +0000 UTC m=+1266.301636108"
May 26 11:45:34.429442 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:34.429407    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27d83a5b-252b-4c74-9487-6656ec81ec5a" path="/var/lib/kubelet/pods/27d83a5b-252b-4c74-9487-6656ec81ec5a/volumes"
May 26 11:45:40.008439 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:40.008397    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g" podUID="d85374dd-6464-461d-9e24-50bd9911b878" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
May 26 11:45:40.103191 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:40.103163    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-90450-79bdccd478-dd87q"
May 26 11:45:50.009726 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:45:50.009687    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g"
May 26 11:46:02.997038 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:46:02.997000    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-5b0de-94f97df94-8xmqh"]
May 26 11:46:03.000385 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:46:03.000360    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-5b0de-94f97df94-8xmqh"
May 26 11:46:03.003087 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:46:03.003056    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-5b0de-kube-rbac-proxy-sar-config\""
May 26 11:46:03.003087 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:46:03.003077    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-5b0de-serving-cert\""
May 26 11:46:03.007719 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:46:03.007699    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-5b0de-94f97df94-8xmqh"]
May 26 11:46:03.075737 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:46:03.075696    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f724af6-d324-41d0-9ccb-12cdda5bc341-proxy-tls\") pod \"sequence-graph-5b0de-94f97df94-8xmqh\" (UID: \"1f724af6-d324-41d0-9ccb-12cdda5bc341\") " pod="kserve-ci-e2e-test/sequence-graph-5b0de-94f97df94-8xmqh"
May 26 11:46:03.075737 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:46:03.075736    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f724af6-d324-41d0-9ccb-12cdda5bc341-openshift-service-ca-bundle\") pod \"sequence-graph-5b0de-94f97df94-8xmqh\" (UID: \"1f724af6-d324-41d0-9ccb-12cdda5bc341\") " pod="kserve-ci-e2e-test/sequence-graph-5b0de-94f97df94-8xmqh"
May 26 11:46:03.176385 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:46:03.176351    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f724af6-d324-41d0-9ccb-12cdda5bc341-proxy-tls\") pod \"sequence-graph-5b0de-94f97df94-8xmqh\" (UID: \"1f724af6-d324-41d0-9ccb-12cdda5bc341\") " pod="kserve-ci-e2e-test/sequence-graph-5b0de-94f97df94-8xmqh"
May 26 11:46:03.176385 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:46:03.176386    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f724af6-d324-41d0-9ccb-12cdda5bc341-openshift-service-ca-bundle\") pod \"sequence-graph-5b0de-94f97df94-8xmqh\" (UID: \"1f724af6-d324-41d0-9ccb-12cdda5bc341\") " pod="kserve-ci-e2e-test/sequence-graph-5b0de-94f97df94-8xmqh"
May 26 11:46:03.177084 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:46:03.177058    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f724af6-d324-41d0-9ccb-12cdda5bc341-openshift-service-ca-bundle\") pod \"sequence-graph-5b0de-94f97df94-8xmqh\" (UID: \"1f724af6-d324-41d0-9ccb-12cdda5bc341\") " pod="kserve-ci-e2e-test/sequence-graph-5b0de-94f97df94-8xmqh"
May 26 11:46:03.178921 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:46:03.178901    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f724af6-d324-41d0-9ccb-12cdda5bc341-proxy-tls\") pod \"sequence-graph-5b0de-94f97df94-8xmqh\" (UID: \"1f724af6-d324-41d0-9ccb-12cdda5bc341\") " pod="kserve-ci-e2e-test/sequence-graph-5b0de-94f97df94-8xmqh"
May 26 11:46:03.310942 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:46:03.310842    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-5b0de-94f97df94-8xmqh"
May 26 11:46:03.436968 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:46:03.436910    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-5b0de-94f97df94-8xmqh"]
May 26 11:46:03.439656 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:46:03.439627    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f724af6_d324_41d0_9ccb_12cdda5bc341.slice/crio-005dbaf7bf4c89ad547cd22643c14d34ed609186cb981f519fa3ce8fd6928d74 WatchSource:0}: Error finding container 005dbaf7bf4c89ad547cd22643c14d34ed609186cb981f519fa3ce8fd6928d74: Status 404 returned error can't find the container with id 005dbaf7bf4c89ad547cd22643c14d34ed609186cb981f519fa3ce8fd6928d74
May 26 11:46:04.177492 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:46:04.177450    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-5b0de-94f97df94-8xmqh" event={"ID":"1f724af6-d324-41d0-9ccb-12cdda5bc341","Type":"ContainerStarted","Data":"25ecf6185350a856902e7c0fa6a82ed8626ad23d10bb214027b8d4a9bbe48cff"}
May 26 11:46:04.177492 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:46:04.177495    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-5b0de-94f97df94-8xmqh" event={"ID":"1f724af6-d324-41d0-9ccb-12cdda5bc341","Type":"ContainerStarted","Data":"005dbaf7bf4c89ad547cd22643c14d34ed609186cb981f519fa3ce8fd6928d74"}
May 26 11:46:04.178014 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:46:04.177556    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-5b0de-94f97df94-8xmqh"
May 26 11:46:04.199858 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:46:04.199799    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-5b0de-94f97df94-8xmqh" podStartSLOduration=2.199765303 podStartE2EDuration="2.199765303s" podCreationTimestamp="2026-05-26 11:46:02 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:46:04.198571824 +0000 UTC m=+1296.390014012" watchObservedRunningTime="2026-05-26 11:46:04.199765303 +0000 UTC m=+1296.391207490"
May 26 11:46:10.188098 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:46:10.188054    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-5b0de-94f97df94-8xmqh"
May 26 11:53:47.882520 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:47.882486    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-90450-79bdccd478-dd87q"]
May 26 11:53:47.885130 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:47.882833    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-90450-79bdccd478-dd87q" podUID="1a31ec7d-e904-45d7-a4e6-ae2c0120a78e" containerName="switch-graph-90450" containerID="cri-o://019d9210a39bc0927eb7230a5fe35e18032c91386a4029fb375995c5a1757659" gracePeriod=30
May 26 11:53:48.062312 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:48.062274    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579"]
May 26 11:53:48.062594 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:48.062568    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579" podUID="bb0ee137-b57f-4795-b42f-c3f26a00639e" containerName="kserve-container" containerID="cri-o://5074614b2abceade204359602201a1dd06b2f8c4df33cbe127dcf1f186127804" gracePeriod=30
May 26 11:53:48.062670 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:48.062605    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579" podUID="bb0ee137-b57f-4795-b42f-c3f26a00639e" containerName="kube-rbac-proxy" containerID="cri-o://042d6e8a3b68f98ea45b67fcb5aad2a365beb66cb8c3a7cf9a5e49055d2d2535" gracePeriod=30
May 26 11:53:48.142836 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:48.142743    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv"]
May 26 11:53:48.146124 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:48.146103    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv"
May 26 11:53:48.149058 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:48.149038    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-5380a-kube-rbac-proxy-sar-config\""
May 26 11:53:48.149283 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:48.149261    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-5380a-predictor-serving-cert\""
May 26 11:53:48.156248 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:48.156219    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv"]
May 26 11:53:48.224577 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:48.224537    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwp6j\" (UniqueName: \"kubernetes.io/projected/c9325a47-b280-4387-89a6-5e0cba0b0aee-kube-api-access-wwp6j\") pod \"error-404-isvc-5380a-predictor-78c9576cbc-hsjpv\" (UID: \"c9325a47-b280-4387-89a6-5e0cba0b0aee\") " pod="kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv"
May 26 11:53:48.224577 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:48.224579    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9325a47-b280-4387-89a6-5e0cba0b0aee-proxy-tls\") pod \"error-404-isvc-5380a-predictor-78c9576cbc-hsjpv\" (UID: \"c9325a47-b280-4387-89a6-5e0cba0b0aee\") " pod="kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv"
May 26 11:53:48.224831 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:48.224619    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-5380a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9325a47-b280-4387-89a6-5e0cba0b0aee-error-404-isvc-5380a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5380a-predictor-78c9576cbc-hsjpv\" (UID: \"c9325a47-b280-4387-89a6-5e0cba0b0aee\") " pod="kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv"
May 26 11:53:48.325064 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:48.325021    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wwp6j\" (UniqueName: \"kubernetes.io/projected/c9325a47-b280-4387-89a6-5e0cba0b0aee-kube-api-access-wwp6j\") pod \"error-404-isvc-5380a-predictor-78c9576cbc-hsjpv\" (UID: \"c9325a47-b280-4387-89a6-5e0cba0b0aee\") " pod="kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv"
May 26 11:53:48.325272 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:48.325075    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9325a47-b280-4387-89a6-5e0cba0b0aee-proxy-tls\") pod \"error-404-isvc-5380a-predictor-78c9576cbc-hsjpv\" (UID: \"c9325a47-b280-4387-89a6-5e0cba0b0aee\") " pod="kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv"
May 26 11:53:48.325272 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:48.325133    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-5380a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9325a47-b280-4387-89a6-5e0cba0b0aee-error-404-isvc-5380a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5380a-predictor-78c9576cbc-hsjpv\" (UID: \"c9325a47-b280-4387-89a6-5e0cba0b0aee\") " pod="kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv"
May 26 11:53:48.325885 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:48.325858    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-5380a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9325a47-b280-4387-89a6-5e0cba0b0aee-error-404-isvc-5380a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5380a-predictor-78c9576cbc-hsjpv\" (UID: \"c9325a47-b280-4387-89a6-5e0cba0b0aee\") " pod="kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv"
May 26 11:53:48.327692 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:48.327671    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9325a47-b280-4387-89a6-5e0cba0b0aee-proxy-tls\") pod \"error-404-isvc-5380a-predictor-78c9576cbc-hsjpv\" (UID: \"c9325a47-b280-4387-89a6-5e0cba0b0aee\") " pod="kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv"
May 26 11:53:48.334700 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:48.334667    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwp6j\" (UniqueName: \"kubernetes.io/projected/c9325a47-b280-4387-89a6-5e0cba0b0aee-kube-api-access-wwp6j\") pod \"error-404-isvc-5380a-predictor-78c9576cbc-hsjpv\" (UID: \"c9325a47-b280-4387-89a6-5e0cba0b0aee\") " pod="kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv"
May 26 11:53:48.456891 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:48.456758    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv"
May 26 11:53:48.501707 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:48.501665    2563 generic.go:358] "Generic (PLEG): container finished" podID="bb0ee137-b57f-4795-b42f-c3f26a00639e" containerID="042d6e8a3b68f98ea45b67fcb5aad2a365beb66cb8c3a7cf9a5e49055d2d2535" exitCode=2
May 26 11:53:48.501893 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:48.501725    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579" event={"ID":"bb0ee137-b57f-4795-b42f-c3f26a00639e","Type":"ContainerDied","Data":"042d6e8a3b68f98ea45b67fcb5aad2a365beb66cb8c3a7cf9a5e49055d2d2535"}
May 26 11:53:48.594248 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:48.594212    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv"]
May 26 11:53:48.597571 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:53:48.597537    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9325a47_b280_4387_89a6_5e0cba0b0aee.slice/crio-1c21dcccef0dea904e91ea111997c94541b48a31866f6deeed66f7c21637f023 WatchSource:0}: Error finding container 1c21dcccef0dea904e91ea111997c94541b48a31866f6deeed66f7c21637f023: Status 404 returned error can't find the container with id 1c21dcccef0dea904e91ea111997c94541b48a31866f6deeed66f7c21637f023
May 26 11:53:48.599292 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:48.599271    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 11:53:49.505964 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:49.505924    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv" event={"ID":"c9325a47-b280-4387-89a6-5e0cba0b0aee","Type":"ContainerStarted","Data":"04a6605eb1bb30798bbc795324c91a49f485ae5eedee2f409bb5ffc7570dff5e"}
May 26 11:53:49.505964 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:49.505965    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv" event={"ID":"c9325a47-b280-4387-89a6-5e0cba0b0aee","Type":"ContainerStarted","Data":"4bd0381c01abf0d9472792bf86c7696826b7b51cabac12b1153754ca245df425"}
May 26 11:53:49.506374 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:49.505975    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv" event={"ID":"c9325a47-b280-4387-89a6-5e0cba0b0aee","Type":"ContainerStarted","Data":"1c21dcccef0dea904e91ea111997c94541b48a31866f6deeed66f7c21637f023"}
May 26 11:53:49.506374 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:49.506205    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv"
May 26 11:53:49.525338 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:49.525289    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv" podStartSLOduration=1.525276971 podStartE2EDuration="1.525276971s" podCreationTimestamp="2026-05-26 11:53:48 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:53:49.524630922 +0000 UTC m=+1761.716073110" watchObservedRunningTime="2026-05-26 11:53:49.525276971 +0000 UTC m=+1761.716719157"
May 26 11:53:49.900238 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:49.900191    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579" podUID="bb0ee137-b57f-4795-b42f-c3f26a00639e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
May 26 11:53:49.904537 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:49.904506    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579" podUID="bb0ee137-b57f-4795-b42f-c3f26a00639e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
May 26 11:53:50.101836 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:50.101791    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-90450-79bdccd478-dd87q" podUID="1a31ec7d-e904-45d7-a4e6-ae2c0120a78e" containerName="switch-graph-90450" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:53:50.509351 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:50.509319    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv"
May 26 11:53:50.510629 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:50.510601    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv" podUID="c9325a47-b280-4387-89a6-5e0cba0b0aee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 26 11:53:51.309169 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:51.309146    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579"
May 26 11:53:51.352595 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:51.352502    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttsq8\" (UniqueName: \"kubernetes.io/projected/bb0ee137-b57f-4795-b42f-c3f26a00639e-kube-api-access-ttsq8\") pod \"bb0ee137-b57f-4795-b42f-c3f26a00639e\" (UID: \"bb0ee137-b57f-4795-b42f-c3f26a00639e\") "
May 26 11:53:51.352595 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:51.352565    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-90450-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb0ee137-b57f-4795-b42f-c3f26a00639e-error-404-isvc-90450-kube-rbac-proxy-sar-config\") pod \"bb0ee137-b57f-4795-b42f-c3f26a00639e\" (UID: \"bb0ee137-b57f-4795-b42f-c3f26a00639e\") "
May 26 11:53:51.352842 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:51.352643    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb0ee137-b57f-4795-b42f-c3f26a00639e-proxy-tls\") pod \"bb0ee137-b57f-4795-b42f-c3f26a00639e\" (UID: \"bb0ee137-b57f-4795-b42f-c3f26a00639e\") "
May 26 11:53:51.352998 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:51.352967    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb0ee137-b57f-4795-b42f-c3f26a00639e-error-404-isvc-90450-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-90450-kube-rbac-proxy-sar-config") pod "bb0ee137-b57f-4795-b42f-c3f26a00639e" (UID: "bb0ee137-b57f-4795-b42f-c3f26a00639e"). InnerVolumeSpecName "error-404-isvc-90450-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:53:51.354872 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:51.354842    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb0ee137-b57f-4795-b42f-c3f26a00639e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bb0ee137-b57f-4795-b42f-c3f26a00639e" (UID: "bb0ee137-b57f-4795-b42f-c3f26a00639e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:53:51.354965 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:51.354881    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb0ee137-b57f-4795-b42f-c3f26a00639e-kube-api-access-ttsq8" (OuterVolumeSpecName: "kube-api-access-ttsq8") pod "bb0ee137-b57f-4795-b42f-c3f26a00639e" (UID: "bb0ee137-b57f-4795-b42f-c3f26a00639e"). InnerVolumeSpecName "kube-api-access-ttsq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:53:51.453891 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:51.453842    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb0ee137-b57f-4795-b42f-c3f26a00639e-proxy-tls\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:53:51.453891 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:51.453879    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttsq8\" (UniqueName: \"kubernetes.io/projected/bb0ee137-b57f-4795-b42f-c3f26a00639e-kube-api-access-ttsq8\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:53:51.453891 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:51.453891    2563 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-90450-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb0ee137-b57f-4795-b42f-c3f26a00639e-error-404-isvc-90450-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:53:51.515979 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:51.515947    2563 generic.go:358] "Generic (PLEG): container finished" podID="bb0ee137-b57f-4795-b42f-c3f26a00639e" containerID="5074614b2abceade204359602201a1dd06b2f8c4df33cbe127dcf1f186127804" exitCode=0
May 26 11:53:51.516412 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:51.516024    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579"
May 26 11:53:51.516412 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:51.516023    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579" event={"ID":"bb0ee137-b57f-4795-b42f-c3f26a00639e","Type":"ContainerDied","Data":"5074614b2abceade204359602201a1dd06b2f8c4df33cbe127dcf1f186127804"}
May 26 11:53:51.516412 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:51.516120    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579" event={"ID":"bb0ee137-b57f-4795-b42f-c3f26a00639e","Type":"ContainerDied","Data":"891c6ce0d96f6e9c1837cd1593021c71765f5281e1b0cad7bde968b35bc0ae15"}
May 26 11:53:51.516412 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:51.516135    2563 scope.go:117] "RemoveContainer" containerID="042d6e8a3b68f98ea45b67fcb5aad2a365beb66cb8c3a7cf9a5e49055d2d2535"
May 26 11:53:51.516705 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:51.516682    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv" podUID="c9325a47-b280-4387-89a6-5e0cba0b0aee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 26 11:53:51.525019 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:51.525002    2563 scope.go:117] "RemoveContainer" containerID="5074614b2abceade204359602201a1dd06b2f8c4df33cbe127dcf1f186127804"
May 26 11:53:51.532260 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:51.532237    2563 scope.go:117] "RemoveContainer" containerID="042d6e8a3b68f98ea45b67fcb5aad2a365beb66cb8c3a7cf9a5e49055d2d2535"
May 26 11:53:51.532530 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:53:51.532510    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"042d6e8a3b68f98ea45b67fcb5aad2a365beb66cb8c3a7cf9a5e49055d2d2535\": container with ID starting with 042d6e8a3b68f98ea45b67fcb5aad2a365beb66cb8c3a7cf9a5e49055d2d2535 not found: ID does not exist" containerID="042d6e8a3b68f98ea45b67fcb5aad2a365beb66cb8c3a7cf9a5e49055d2d2535"
May 26 11:53:51.532573 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:51.532547    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"042d6e8a3b68f98ea45b67fcb5aad2a365beb66cb8c3a7cf9a5e49055d2d2535"} err="failed to get container status \"042d6e8a3b68f98ea45b67fcb5aad2a365beb66cb8c3a7cf9a5e49055d2d2535\": rpc error: code = NotFound desc = could not find container \"042d6e8a3b68f98ea45b67fcb5aad2a365beb66cb8c3a7cf9a5e49055d2d2535\": container with ID starting with 042d6e8a3b68f98ea45b67fcb5aad2a365beb66cb8c3a7cf9a5e49055d2d2535 not found: ID does not exist"
May 26 11:53:51.532573 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:51.532571    2563 scope.go:117] "RemoveContainer" containerID="5074614b2abceade204359602201a1dd06b2f8c4df33cbe127dcf1f186127804"
May 26 11:53:51.532862 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:53:51.532840    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5074614b2abceade204359602201a1dd06b2f8c4df33cbe127dcf1f186127804\": container with ID starting with 5074614b2abceade204359602201a1dd06b2f8c4df33cbe127dcf1f186127804 not found: ID does not exist" containerID="5074614b2abceade204359602201a1dd06b2f8c4df33cbe127dcf1f186127804"
May 26 11:53:51.532927 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:51.532869    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5074614b2abceade204359602201a1dd06b2f8c4df33cbe127dcf1f186127804"} err="failed to get container status \"5074614b2abceade204359602201a1dd06b2f8c4df33cbe127dcf1f186127804\": rpc error: code = NotFound desc = could not find container \"5074614b2abceade204359602201a1dd06b2f8c4df33cbe127dcf1f186127804\": container with ID starting with 5074614b2abceade204359602201a1dd06b2f8c4df33cbe127dcf1f186127804 not found: ID does not exist"
May 26 11:53:51.538223 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:51.538196    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579"]
May 26 11:53:51.542504 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:51.542483    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-90450-predictor-f47cbcc5c-5s579"]
May 26 11:53:52.429080 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:52.429046    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb0ee137-b57f-4795-b42f-c3f26a00639e" path="/var/lib/kubelet/pods/bb0ee137-b57f-4795-b42f-c3f26a00639e/volumes"
May 26 11:53:55.101057 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:55.101010    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-90450-79bdccd478-dd87q" podUID="1a31ec7d-e904-45d7-a4e6-ae2c0120a78e" containerName="switch-graph-90450" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:53:56.520709 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:56.520681    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv"
May 26 11:53:56.521220 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:53:56.521193    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv" podUID="c9325a47-b280-4387-89a6-5e0cba0b0aee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 26 11:54:00.100140 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:00.100096    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-90450-79bdccd478-dd87q" podUID="1a31ec7d-e904-45d7-a4e6-ae2c0120a78e" containerName="switch-graph-90450" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:54:00.100521 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:00.100200    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-90450-79bdccd478-dd87q"
May 26 11:54:05.100349 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:05.100298    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-90450-79bdccd478-dd87q" podUID="1a31ec7d-e904-45d7-a4e6-ae2c0120a78e" containerName="switch-graph-90450" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:54:06.521250 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:06.521209    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv" podUID="c9325a47-b280-4387-89a6-5e0cba0b0aee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 26 11:54:10.100553 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:10.100506    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-90450-79bdccd478-dd87q" podUID="1a31ec7d-e904-45d7-a4e6-ae2c0120a78e" containerName="switch-graph-90450" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:54:15.100513 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:15.100426    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-90450-79bdccd478-dd87q" podUID="1a31ec7d-e904-45d7-a4e6-ae2c0120a78e" containerName="switch-graph-90450" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:54:16.522104 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:16.522066    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv" podUID="c9325a47-b280-4387-89a6-5e0cba0b0aee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 26 11:54:17.787714 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:17.787679    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-5b0de-94f97df94-8xmqh"]
May 26 11:54:17.788211 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:17.787929    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-5b0de-94f97df94-8xmqh" podUID="1f724af6-d324-41d0-9ccb-12cdda5bc341" containerName="sequence-graph-5b0de" containerID="cri-o://25ecf6185350a856902e7c0fa6a82ed8626ad23d10bb214027b8d4a9bbe48cff" gracePeriod=30
May 26 11:54:17.971586 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:17.971553    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g"]
May 26 11:54:17.971901 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:17.971873    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g" podUID="d85374dd-6464-461d-9e24-50bd9911b878" containerName="kserve-container" containerID="cri-o://ac734cfffc0214426cad2770a55e3b995756751363db21591436b42764037a47" gracePeriod=30
May 26 11:54:17.971999 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:17.971913    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g" podUID="d85374dd-6464-461d-9e24-50bd9911b878" containerName="kube-rbac-proxy" containerID="cri-o://7acbf801fb21c14a09c2b1b70d27e070a84e61831c8689bba2866c7c666c7a82" gracePeriod=30
May 26 11:54:18.037686 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.037646    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb"]
May 26 11:54:18.038026 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.037989    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb0ee137-b57f-4795-b42f-c3f26a00639e" containerName="kube-rbac-proxy"
May 26 11:54:18.038026 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.038003    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb0ee137-b57f-4795-b42f-c3f26a00639e" containerName="kube-rbac-proxy"
May 26 11:54:18.038026 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.038020    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb0ee137-b57f-4795-b42f-c3f26a00639e" containerName="kserve-container"
May 26 11:54:18.038026 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.038026    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb0ee137-b57f-4795-b42f-c3f26a00639e" containerName="kserve-container"
May 26 11:54:18.038167 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.038078    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="bb0ee137-b57f-4795-b42f-c3f26a00639e" containerName="kserve-container"
May 26 11:54:18.038167 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.038087    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="bb0ee137-b57f-4795-b42f-c3f26a00639e" containerName="kube-rbac-proxy"
May 26 11:54:18.041135 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.041118    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb"
May 26 11:54:18.043688 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.043667    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-fbdd2-kube-rbac-proxy-sar-config\""
May 26 11:54:18.043840 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.043820    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-fbdd2-predictor-serving-cert\""
May 26 11:54:18.049885 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.049859    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb"]
May 26 11:54:18.093047 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.093024    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-90450-79bdccd478-dd87q"
May 26 11:54:18.183354 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.183319    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a31ec7d-e904-45d7-a4e6-ae2c0120a78e-openshift-service-ca-bundle\") pod \"1a31ec7d-e904-45d7-a4e6-ae2c0120a78e\" (UID: \"1a31ec7d-e904-45d7-a4e6-ae2c0120a78e\") "
May 26 11:54:18.183513 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.183374    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a31ec7d-e904-45d7-a4e6-ae2c0120a78e-proxy-tls\") pod \"1a31ec7d-e904-45d7-a4e6-ae2c0120a78e\" (UID: \"1a31ec7d-e904-45d7-a4e6-ae2c0120a78e\") "
May 26 11:54:18.183562 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.183537    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b280eea3-83f6-44fe-9ec8-3566464fb8ee-proxy-tls\") pod \"error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb\" (UID: \"b280eea3-83f6-44fe-9ec8-3566464fb8ee\") " pod="kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb"
May 26 11:54:18.183688 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.183666    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a31ec7d-e904-45d7-a4e6-ae2c0120a78e-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "1a31ec7d-e904-45d7-a4e6-ae2c0120a78e" (UID: "1a31ec7d-e904-45d7-a4e6-ae2c0120a78e"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:54:18.183738 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.183672    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-fbdd2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b280eea3-83f6-44fe-9ec8-3566464fb8ee-error-404-isvc-fbdd2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb\" (UID: \"b280eea3-83f6-44fe-9ec8-3566464fb8ee\") " pod="kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb"
May 26 11:54:18.183807 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.183745    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q4gv\" (UniqueName: \"kubernetes.io/projected/b280eea3-83f6-44fe-9ec8-3566464fb8ee-kube-api-access-2q4gv\") pod \"error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb\" (UID: \"b280eea3-83f6-44fe-9ec8-3566464fb8ee\") " pod="kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb"
May 26 11:54:18.183852 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.183820    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a31ec7d-e904-45d7-a4e6-ae2c0120a78e-openshift-service-ca-bundle\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:54:18.185673 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.185652    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a31ec7d-e904-45d7-a4e6-ae2c0120a78e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1a31ec7d-e904-45d7-a4e6-ae2c0120a78e" (UID: "1a31ec7d-e904-45d7-a4e6-ae2c0120a78e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:54:18.284754 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.284708    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b280eea3-83f6-44fe-9ec8-3566464fb8ee-proxy-tls\") pod \"error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb\" (UID: \"b280eea3-83f6-44fe-9ec8-3566464fb8ee\") " pod="kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb"
May 26 11:54:18.284955 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.284838    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-fbdd2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b280eea3-83f6-44fe-9ec8-3566464fb8ee-error-404-isvc-fbdd2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb\" (UID: \"b280eea3-83f6-44fe-9ec8-3566464fb8ee\") " pod="kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb"
May 26 11:54:18.284955 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.284884    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2q4gv\" (UniqueName: \"kubernetes.io/projected/b280eea3-83f6-44fe-9ec8-3566464fb8ee-kube-api-access-2q4gv\") pod \"error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb\" (UID: \"b280eea3-83f6-44fe-9ec8-3566464fb8ee\") " pod="kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb"
May 26 11:54:18.284955 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.284926    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a31ec7d-e904-45d7-a4e6-ae2c0120a78e-proxy-tls\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:54:18.285526 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.285506    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-fbdd2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b280eea3-83f6-44fe-9ec8-3566464fb8ee-error-404-isvc-fbdd2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb\" (UID: \"b280eea3-83f6-44fe-9ec8-3566464fb8ee\") " pod="kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb"
May 26 11:54:18.287283 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.287266    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b280eea3-83f6-44fe-9ec8-3566464fb8ee-proxy-tls\") pod \"error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb\" (UID: \"b280eea3-83f6-44fe-9ec8-3566464fb8ee\") " pod="kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb"
May 26 11:54:18.294332 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.294265    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q4gv\" (UniqueName: \"kubernetes.io/projected/b280eea3-83f6-44fe-9ec8-3566464fb8ee-kube-api-access-2q4gv\") pod \"error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb\" (UID: \"b280eea3-83f6-44fe-9ec8-3566464fb8ee\") " pod="kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb"
May 26 11:54:18.352430 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.352380    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb"
May 26 11:54:18.487136 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.487063    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb"]
May 26 11:54:18.489791 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:54:18.489738    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb280eea3_83f6_44fe_9ec8_3566464fb8ee.slice/crio-5a8f9359bd380e7058d33012d720cb2cf8d6554ed4f33ac515f001c6f3964cbe WatchSource:0}: Error finding container 5a8f9359bd380e7058d33012d720cb2cf8d6554ed4f33ac515f001c6f3964cbe: Status 404 returned error can't find the container with id 5a8f9359bd380e7058d33012d720cb2cf8d6554ed4f33ac515f001c6f3964cbe
May 26 11:54:18.598880 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.598842    2563 generic.go:358] "Generic (PLEG): container finished" podID="d85374dd-6464-461d-9e24-50bd9911b878" containerID="7acbf801fb21c14a09c2b1b70d27e070a84e61831c8689bba2866c7c666c7a82" exitCode=2
May 26 11:54:18.599017 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.598883    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g" event={"ID":"d85374dd-6464-461d-9e24-50bd9911b878","Type":"ContainerDied","Data":"7acbf801fb21c14a09c2b1b70d27e070a84e61831c8689bba2866c7c666c7a82"}
May 26 11:54:18.600551 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.600529    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb" event={"ID":"b280eea3-83f6-44fe-9ec8-3566464fb8ee","Type":"ContainerStarted","Data":"8b13864361fdf23447bd0fc62c5e7aa3bf3b2e194882c73f99edacd2291687cb"}
May 26 11:54:18.600551 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.600560    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb" event={"ID":"b280eea3-83f6-44fe-9ec8-3566464fb8ee","Type":"ContainerStarted","Data":"5a8f9359bd380e7058d33012d720cb2cf8d6554ed4f33ac515f001c6f3964cbe"}
May 26 11:54:18.601606 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.601583    2563 generic.go:358] "Generic (PLEG): container finished" podID="1a31ec7d-e904-45d7-a4e6-ae2c0120a78e" containerID="019d9210a39bc0927eb7230a5fe35e18032c91386a4029fb375995c5a1757659" exitCode=137
May 26 11:54:18.601699 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.601640    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-90450-79bdccd478-dd87q" event={"ID":"1a31ec7d-e904-45d7-a4e6-ae2c0120a78e","Type":"ContainerDied","Data":"019d9210a39bc0927eb7230a5fe35e18032c91386a4029fb375995c5a1757659"}
May 26 11:54:18.601699 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.601665    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-90450-79bdccd478-dd87q" event={"ID":"1a31ec7d-e904-45d7-a4e6-ae2c0120a78e","Type":"ContainerDied","Data":"d697fa9655afce577eca14a7336b5c3ecefdae2e2b9d5f129dcb144dc6b0750d"}
May 26 11:54:18.601699 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.601682    2563 scope.go:117] "RemoveContainer" containerID="019d9210a39bc0927eb7230a5fe35e18032c91386a4029fb375995c5a1757659"
May 26 11:54:18.601699 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.601685    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-90450-79bdccd478-dd87q"
May 26 11:54:18.610704 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.610682    2563 scope.go:117] "RemoveContainer" containerID="019d9210a39bc0927eb7230a5fe35e18032c91386a4029fb375995c5a1757659"
May 26 11:54:18.611100 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:54:18.611071    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"019d9210a39bc0927eb7230a5fe35e18032c91386a4029fb375995c5a1757659\": container with ID starting with 019d9210a39bc0927eb7230a5fe35e18032c91386a4029fb375995c5a1757659 not found: ID does not exist" containerID="019d9210a39bc0927eb7230a5fe35e18032c91386a4029fb375995c5a1757659"
May 26 11:54:18.611188 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.611108    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"019d9210a39bc0927eb7230a5fe35e18032c91386a4029fb375995c5a1757659"} err="failed to get container status \"019d9210a39bc0927eb7230a5fe35e18032c91386a4029fb375995c5a1757659\": rpc error: code = NotFound desc = could not find container \"019d9210a39bc0927eb7230a5fe35e18032c91386a4029fb375995c5a1757659\": container with ID starting with 019d9210a39bc0927eb7230a5fe35e18032c91386a4029fb375995c5a1757659 not found: ID does not exist"
May 26 11:54:18.620379 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.620354    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-90450-79bdccd478-dd87q"]
May 26 11:54:18.623606 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:18.623585    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-90450-79bdccd478-dd87q"]
May 26 11:54:19.607244 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:19.607208    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb" event={"ID":"b280eea3-83f6-44fe-9ec8-3566464fb8ee","Type":"ContainerStarted","Data":"b39fd0810274a489b9305b3d01312d44f02d81b504f2afd453ec9d656a7a8a51"}
May 26 11:54:19.607648 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:19.607456    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb"
May 26 11:54:19.607648 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:19.607587    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb"
May 26 11:54:19.608728 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:19.608706    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb" podUID="b280eea3-83f6-44fe-9ec8-3566464fb8ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
May 26 11:54:19.626065 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:19.626020    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb" podStartSLOduration=1.626009577 podStartE2EDuration="1.626009577s" podCreationTimestamp="2026-05-26 11:54:18 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:54:19.625048028 +0000 UTC m=+1791.816490216" watchObservedRunningTime="2026-05-26 11:54:19.626009577 +0000 UTC m=+1791.817451761"
May 26 11:54:19.998190 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:19.998074    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g" podUID="d85374dd-6464-461d-9e24-50bd9911b878" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
May 26 11:54:20.009175 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:20.009135    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g" podUID="d85374dd-6464-461d-9e24-50bd9911b878" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
May 26 11:54:20.185301 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:20.185254    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-5b0de-94f97df94-8xmqh" podUID="1f724af6-d324-41d0-9ccb-12cdda5bc341" containerName="sequence-graph-5b0de" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:54:20.428488 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:20.428453    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a31ec7d-e904-45d7-a4e6-ae2c0120a78e" path="/var/lib/kubelet/pods/1a31ec7d-e904-45d7-a4e6-ae2c0120a78e/volumes"
May 26 11:54:20.610522 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:20.610486    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb" podUID="b280eea3-83f6-44fe-9ec8-3566464fb8ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
May 26 11:54:21.426177 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:21.426156    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g"
May 26 11:54:21.512070 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:21.511999    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d85374dd-6464-461d-9e24-50bd9911b878-proxy-tls\") pod \"d85374dd-6464-461d-9e24-50bd9911b878\" (UID: \"d85374dd-6464-461d-9e24-50bd9911b878\") "
May 26 11:54:21.512070 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:21.512060    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxt5c\" (UniqueName: \"kubernetes.io/projected/d85374dd-6464-461d-9e24-50bd9911b878-kube-api-access-pxt5c\") pod \"d85374dd-6464-461d-9e24-50bd9911b878\" (UID: \"d85374dd-6464-461d-9e24-50bd9911b878\") "
May 26 11:54:21.512265 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:21.512126    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-5b0de-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d85374dd-6464-461d-9e24-50bd9911b878-error-404-isvc-5b0de-kube-rbac-proxy-sar-config\") pod \"d85374dd-6464-461d-9e24-50bd9911b878\" (UID: \"d85374dd-6464-461d-9e24-50bd9911b878\") "
May 26 11:54:21.513196 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:21.512548    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d85374dd-6464-461d-9e24-50bd9911b878-error-404-isvc-5b0de-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-5b0de-kube-rbac-proxy-sar-config") pod "d85374dd-6464-461d-9e24-50bd9911b878" (UID: "d85374dd-6464-461d-9e24-50bd9911b878"). InnerVolumeSpecName "error-404-isvc-5b0de-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:54:21.513196 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:21.512971    2563 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-5b0de-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d85374dd-6464-461d-9e24-50bd9911b878-error-404-isvc-5b0de-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:54:21.514401 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:21.514377    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d85374dd-6464-461d-9e24-50bd9911b878-kube-api-access-pxt5c" (OuterVolumeSpecName: "kube-api-access-pxt5c") pod "d85374dd-6464-461d-9e24-50bd9911b878" (UID: "d85374dd-6464-461d-9e24-50bd9911b878"). InnerVolumeSpecName "kube-api-access-pxt5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:54:21.514548 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:21.514527    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d85374dd-6464-461d-9e24-50bd9911b878-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d85374dd-6464-461d-9e24-50bd9911b878" (UID: "d85374dd-6464-461d-9e24-50bd9911b878"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:54:21.613516 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:21.613483    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d85374dd-6464-461d-9e24-50bd9911b878-proxy-tls\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:54:21.613516 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:21.613509    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxt5c\" (UniqueName: \"kubernetes.io/projected/d85374dd-6464-461d-9e24-50bd9911b878-kube-api-access-pxt5c\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:54:21.614693 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:21.614668    2563 generic.go:358] "Generic (PLEG): container finished" podID="d85374dd-6464-461d-9e24-50bd9911b878" containerID="ac734cfffc0214426cad2770a55e3b995756751363db21591436b42764037a47" exitCode=0
May 26 11:54:21.614768 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:21.614721    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g" event={"ID":"d85374dd-6464-461d-9e24-50bd9911b878","Type":"ContainerDied","Data":"ac734cfffc0214426cad2770a55e3b995756751363db21591436b42764037a47"}
May 26 11:54:21.614768 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:21.614743    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g"
May 26 11:54:21.614768 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:21.614753    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g" event={"ID":"d85374dd-6464-461d-9e24-50bd9911b878","Type":"ContainerDied","Data":"2ad5bf0d2e021a10d8f552ff5afcfb80c459ea109bf86f2ac316413b8f6a8ff9"}
May 26 11:54:21.614768 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:21.614768    2563 scope.go:117] "RemoveContainer" containerID="7acbf801fb21c14a09c2b1b70d27e070a84e61831c8689bba2866c7c666c7a82"
May 26 11:54:21.622909 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:21.622892    2563 scope.go:117] "RemoveContainer" containerID="ac734cfffc0214426cad2770a55e3b995756751363db21591436b42764037a47"
May 26 11:54:21.631916 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:21.631895    2563 scope.go:117] "RemoveContainer" containerID="7acbf801fb21c14a09c2b1b70d27e070a84e61831c8689bba2866c7c666c7a82"
May 26 11:54:21.632260 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:54:21.632235    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7acbf801fb21c14a09c2b1b70d27e070a84e61831c8689bba2866c7c666c7a82\": container with ID starting with 7acbf801fb21c14a09c2b1b70d27e070a84e61831c8689bba2866c7c666c7a82 not found: ID does not exist" containerID="7acbf801fb21c14a09c2b1b70d27e070a84e61831c8689bba2866c7c666c7a82"
May 26 11:54:21.632311 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:21.632271    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7acbf801fb21c14a09c2b1b70d27e070a84e61831c8689bba2866c7c666c7a82"} err="failed to get container status \"7acbf801fb21c14a09c2b1b70d27e070a84e61831c8689bba2866c7c666c7a82\": rpc error: code = NotFound desc = could not find container \"7acbf801fb21c14a09c2b1b70d27e070a84e61831c8689bba2866c7c666c7a82\": container with ID starting with 7acbf801fb21c14a09c2b1b70d27e070a84e61831c8689bba2866c7c666c7a82 not found: ID does not exist"
May 26 11:54:21.632311 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:21.632290    2563 scope.go:117] "RemoveContainer" containerID="ac734cfffc0214426cad2770a55e3b995756751363db21591436b42764037a47"
May 26 11:54:21.632554 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:54:21.632535    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac734cfffc0214426cad2770a55e3b995756751363db21591436b42764037a47\": container with ID starting with ac734cfffc0214426cad2770a55e3b995756751363db21591436b42764037a47 not found: ID does not exist" containerID="ac734cfffc0214426cad2770a55e3b995756751363db21591436b42764037a47"
May 26 11:54:21.632603 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:21.632563    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac734cfffc0214426cad2770a55e3b995756751363db21591436b42764037a47"} err="failed to get container status \"ac734cfffc0214426cad2770a55e3b995756751363db21591436b42764037a47\": rpc error: code = NotFound desc = could not find container \"ac734cfffc0214426cad2770a55e3b995756751363db21591436b42764037a47\": container with ID starting with ac734cfffc0214426cad2770a55e3b995756751363db21591436b42764037a47 not found: ID does not exist"
May 26 11:54:21.636446 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:21.636420    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g"]
May 26 11:54:21.640407 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:21.640384    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5b0de-predictor-6766f587d9-qlg2g"]
May 26 11:54:22.429518 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:22.429487    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d85374dd-6464-461d-9e24-50bd9911b878" path="/var/lib/kubelet/pods/d85374dd-6464-461d-9e24-50bd9911b878/volumes"
May 26 11:54:25.184927 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:25.184877    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-5b0de-94f97df94-8xmqh" podUID="1f724af6-d324-41d0-9ccb-12cdda5bc341" containerName="sequence-graph-5b0de" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:54:25.619339 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:25.619310    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb"
May 26 11:54:25.619895 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:25.619865    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb" podUID="b280eea3-83f6-44fe-9ec8-3566464fb8ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
May 26 11:54:26.521949 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:26.521905    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv" podUID="c9325a47-b280-4387-89a6-5e0cba0b0aee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 26 11:54:30.184859 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:30.184810    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-5b0de-94f97df94-8xmqh" podUID="1f724af6-d324-41d0-9ccb-12cdda5bc341" containerName="sequence-graph-5b0de" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:54:30.185304 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:30.184971    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-5b0de-94f97df94-8xmqh"
May 26 11:54:35.185335 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:35.185299    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-5b0de-94f97df94-8xmqh" podUID="1f724af6-d324-41d0-9ccb-12cdda5bc341" containerName="sequence-graph-5b0de" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:54:35.620612 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:35.620569    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb" podUID="b280eea3-83f6-44fe-9ec8-3566464fb8ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
May 26 11:54:36.522402 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:36.522376    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv"
May 26 11:54:40.185361 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:40.185310    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-5b0de-94f97df94-8xmqh" podUID="1f724af6-d324-41d0-9ccb-12cdda5bc341" containerName="sequence-graph-5b0de" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:54:45.184523 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:45.184486    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-5b0de-94f97df94-8xmqh" podUID="1f724af6-d324-41d0-9ccb-12cdda5bc341" containerName="sequence-graph-5b0de" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:54:45.620466 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:45.620429    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb" podUID="b280eea3-83f6-44fe-9ec8-3566464fb8ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
May 26 11:54:47.935985 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:47.935959    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-5b0de-94f97df94-8xmqh"
May 26 11:54:48.022119 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.022085    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f724af6-d324-41d0-9ccb-12cdda5bc341-openshift-service-ca-bundle\") pod \"1f724af6-d324-41d0-9ccb-12cdda5bc341\" (UID: \"1f724af6-d324-41d0-9ccb-12cdda5bc341\") "
May 26 11:54:48.022291 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.022188    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f724af6-d324-41d0-9ccb-12cdda5bc341-proxy-tls\") pod \"1f724af6-d324-41d0-9ccb-12cdda5bc341\" (UID: \"1f724af6-d324-41d0-9ccb-12cdda5bc341\") "
May 26 11:54:48.022470 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.022437    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f724af6-d324-41d0-9ccb-12cdda5bc341-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "1f724af6-d324-41d0-9ccb-12cdda5bc341" (UID: "1f724af6-d324-41d0-9ccb-12cdda5bc341"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:54:48.024349 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.024329    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f724af6-d324-41d0-9ccb-12cdda5bc341-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1f724af6-d324-41d0-9ccb-12cdda5bc341" (UID: "1f724af6-d324-41d0-9ccb-12cdda5bc341"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:54:48.123514 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.123467    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f724af6-d324-41d0-9ccb-12cdda5bc341-proxy-tls\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:54:48.123514 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.123510    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f724af6-d324-41d0-9ccb-12cdda5bc341-openshift-service-ca-bundle\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:54:48.133051 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.133019    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-5380a-676cb95c4b-phg74"]
May 26 11:54:48.133322 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.133310    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d85374dd-6464-461d-9e24-50bd9911b878" containerName="kube-rbac-proxy"
May 26 11:54:48.133364 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.133324    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d85374dd-6464-461d-9e24-50bd9911b878" containerName="kube-rbac-proxy"
May 26 11:54:48.133364 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.133341    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d85374dd-6464-461d-9e24-50bd9911b878" containerName="kserve-container"
May 26 11:54:48.133364 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.133346    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d85374dd-6464-461d-9e24-50bd9911b878" containerName="kserve-container"
May 26 11:54:48.133364 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.133353    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f724af6-d324-41d0-9ccb-12cdda5bc341" containerName="sequence-graph-5b0de"
May 26 11:54:48.133364 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.133360    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f724af6-d324-41d0-9ccb-12cdda5bc341" containerName="sequence-graph-5b0de"
May 26 11:54:48.133513 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.133367    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a31ec7d-e904-45d7-a4e6-ae2c0120a78e" containerName="switch-graph-90450"
May 26 11:54:48.133513 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.133372    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a31ec7d-e904-45d7-a4e6-ae2c0120a78e" containerName="switch-graph-90450"
May 26 11:54:48.133513 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.133418    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="1f724af6-d324-41d0-9ccb-12cdda5bc341" containerName="sequence-graph-5b0de"
May 26 11:54:48.133513 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.133429    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="d85374dd-6464-461d-9e24-50bd9911b878" containerName="kube-rbac-proxy"
May 26 11:54:48.133513 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.133435    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="1a31ec7d-e904-45d7-a4e6-ae2c0120a78e" containerName="switch-graph-90450"
May 26 11:54:48.133513 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.133441    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="d85374dd-6464-461d-9e24-50bd9911b878" containerName="kserve-container"
May 26 11:54:48.136419 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.136403    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-5380a-676cb95c4b-phg74"
May 26 11:54:48.138949 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.138926    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-5380a-serving-cert\""
May 26 11:54:48.139188 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.139174    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-5380a-kube-rbac-proxy-sar-config\""
May 26 11:54:48.151149 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.151122    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-5380a-676cb95c4b-phg74"]
May 26 11:54:48.224583 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.224545    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d1acaa3-e946-40d3-9937-ef994cf95b0a-openshift-service-ca-bundle\") pod \"ensemble-graph-5380a-676cb95c4b-phg74\" (UID: \"7d1acaa3-e946-40d3-9937-ef994cf95b0a\") " pod="kserve-ci-e2e-test/ensemble-graph-5380a-676cb95c4b-phg74"
May 26 11:54:48.224583 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.224591    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d1acaa3-e946-40d3-9937-ef994cf95b0a-proxy-tls\") pod \"ensemble-graph-5380a-676cb95c4b-phg74\" (UID: \"7d1acaa3-e946-40d3-9937-ef994cf95b0a\") " pod="kserve-ci-e2e-test/ensemble-graph-5380a-676cb95c4b-phg74"
May 26 11:54:48.325459 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.325420    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d1acaa3-e946-40d3-9937-ef994cf95b0a-openshift-service-ca-bundle\") pod \"ensemble-graph-5380a-676cb95c4b-phg74\" (UID: \"7d1acaa3-e946-40d3-9937-ef994cf95b0a\") " pod="kserve-ci-e2e-test/ensemble-graph-5380a-676cb95c4b-phg74"
May 26 11:54:48.325459 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.325459    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d1acaa3-e946-40d3-9937-ef994cf95b0a-proxy-tls\") pod \"ensemble-graph-5380a-676cb95c4b-phg74\" (UID: \"7d1acaa3-e946-40d3-9937-ef994cf95b0a\") " pod="kserve-ci-e2e-test/ensemble-graph-5380a-676cb95c4b-phg74"
May 26 11:54:48.326117 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.326096    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d1acaa3-e946-40d3-9937-ef994cf95b0a-openshift-service-ca-bundle\") pod \"ensemble-graph-5380a-676cb95c4b-phg74\" (UID: \"7d1acaa3-e946-40d3-9937-ef994cf95b0a\") " pod="kserve-ci-e2e-test/ensemble-graph-5380a-676cb95c4b-phg74"
May 26 11:54:48.328005 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.327982    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d1acaa3-e946-40d3-9937-ef994cf95b0a-proxy-tls\") pod \"ensemble-graph-5380a-676cb95c4b-phg74\" (UID: \"7d1acaa3-e946-40d3-9937-ef994cf95b0a\") " pod="kserve-ci-e2e-test/ensemble-graph-5380a-676cb95c4b-phg74"
May 26 11:54:48.450854 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.450820    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-5380a-676cb95c4b-phg74"
May 26 11:54:48.584653 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.584629    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-5380a-676cb95c4b-phg74"]
May 26 11:54:48.587353 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:54:48.587325    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d1acaa3_e946_40d3_9937_ef994cf95b0a.slice/crio-9930b50a433edbe43c5ebf583ea20d5f628b608c8d733711e7e312ae35f55f98 WatchSource:0}: Error finding container 9930b50a433edbe43c5ebf583ea20d5f628b608c8d733711e7e312ae35f55f98: Status 404 returned error can't find the container with id 9930b50a433edbe43c5ebf583ea20d5f628b608c8d733711e7e312ae35f55f98
May 26 11:54:48.695498 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.695413    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-5380a-676cb95c4b-phg74" event={"ID":"7d1acaa3-e946-40d3-9937-ef994cf95b0a","Type":"ContainerStarted","Data":"89129a975e0e3aa234a5803889c229f86dc8da317f22ebf4a6d9d12067e91467"}
May 26 11:54:48.695498 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.695456    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-5380a-676cb95c4b-phg74" event={"ID":"7d1acaa3-e946-40d3-9937-ef994cf95b0a","Type":"ContainerStarted","Data":"9930b50a433edbe43c5ebf583ea20d5f628b608c8d733711e7e312ae35f55f98"}
May 26 11:54:48.695705 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.695496    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-5380a-676cb95c4b-phg74"
May 26 11:54:48.696518 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.696494    2563 generic.go:358] "Generic (PLEG): container finished" podID="1f724af6-d324-41d0-9ccb-12cdda5bc341" containerID="25ecf6185350a856902e7c0fa6a82ed8626ad23d10bb214027b8d4a9bbe48cff" exitCode=0
May 26 11:54:48.696607 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.696540    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-5b0de-94f97df94-8xmqh" event={"ID":"1f724af6-d324-41d0-9ccb-12cdda5bc341","Type":"ContainerDied","Data":"25ecf6185350a856902e7c0fa6a82ed8626ad23d10bb214027b8d4a9bbe48cff"}
May 26 11:54:48.696607 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.696567    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-5b0de-94f97df94-8xmqh" event={"ID":"1f724af6-d324-41d0-9ccb-12cdda5bc341","Type":"ContainerDied","Data":"005dbaf7bf4c89ad547cd22643c14d34ed609186cb981f519fa3ce8fd6928d74"}
May 26 11:54:48.696607 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.696584    2563 scope.go:117] "RemoveContainer" containerID="25ecf6185350a856902e7c0fa6a82ed8626ad23d10bb214027b8d4a9bbe48cff"
May 26 11:54:48.696607 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.696585    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-5b0de-94f97df94-8xmqh"
May 26 11:54:48.704450 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.704434    2563 scope.go:117] "RemoveContainer" containerID="25ecf6185350a856902e7c0fa6a82ed8626ad23d10bb214027b8d4a9bbe48cff"
May 26 11:54:48.704693 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:54:48.704668    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25ecf6185350a856902e7c0fa6a82ed8626ad23d10bb214027b8d4a9bbe48cff\": container with ID starting with 25ecf6185350a856902e7c0fa6a82ed8626ad23d10bb214027b8d4a9bbe48cff not found: ID does not exist" containerID="25ecf6185350a856902e7c0fa6a82ed8626ad23d10bb214027b8d4a9bbe48cff"
May 26 11:54:48.704819 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.704698    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25ecf6185350a856902e7c0fa6a82ed8626ad23d10bb214027b8d4a9bbe48cff"} err="failed to get container status \"25ecf6185350a856902e7c0fa6a82ed8626ad23d10bb214027b8d4a9bbe48cff\": rpc error: code = NotFound desc = could not find container \"25ecf6185350a856902e7c0fa6a82ed8626ad23d10bb214027b8d4a9bbe48cff\": container with ID starting with 25ecf6185350a856902e7c0fa6a82ed8626ad23d10bb214027b8d4a9bbe48cff not found: ID does not exist"
May 26 11:54:48.715818 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.715757    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-5380a-676cb95c4b-phg74" podStartSLOduration=0.71574593 podStartE2EDuration="715.74593ms" podCreationTimestamp="2026-05-26 11:54:48 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:54:48.714844225 +0000 UTC m=+1820.906286415" watchObservedRunningTime="2026-05-26 11:54:48.71574593 +0000 UTC m=+1820.907188117"
May 26 11:54:48.730081 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.730054    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-5b0de-94f97df94-8xmqh"]
May 26 11:54:48.733642 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:48.733621    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-5b0de-94f97df94-8xmqh"]
May 26 11:54:50.429064 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:50.429018    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f724af6-d324-41d0-9ccb-12cdda5bc341" path="/var/lib/kubelet/pods/1f724af6-d324-41d0-9ccb-12cdda5bc341/volumes"
May 26 11:54:54.706098 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:54.706072    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-5380a-676cb95c4b-phg74"
May 26 11:54:55.620348 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:55.620313    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb" podUID="b280eea3-83f6-44fe-9ec8-3566464fb8ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
May 26 11:54:58.242074 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:58.241993    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-5380a-676cb95c4b-phg74"]
May 26 11:54:58.242443 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:58.242200    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-5380a-676cb95c4b-phg74" podUID="7d1acaa3-e946-40d3-9937-ef994cf95b0a" containerName="ensemble-graph-5380a" containerID="cri-o://89129a975e0e3aa234a5803889c229f86dc8da317f22ebf4a6d9d12067e91467" gracePeriod=30
May 26 11:54:58.308235 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:58.308201    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv"]
May 26 11:54:58.308594 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:58.308568    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv" podUID="c9325a47-b280-4387-89a6-5e0cba0b0aee" containerName="kserve-container" containerID="cri-o://4bd0381c01abf0d9472792bf86c7696826b7b51cabac12b1153754ca245df425" gracePeriod=30
May 26 11:54:58.308731 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:58.308653    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv" podUID="c9325a47-b280-4387-89a6-5e0cba0b0aee" containerName="kube-rbac-proxy" containerID="cri-o://04a6605eb1bb30798bbc795324c91a49f485ae5eedee2f409bb5ffc7570dff5e" gracePeriod=30
May 26 11:54:58.515807 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:58.515699    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph"]
May 26 11:54:58.519122 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:58.519103    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph"
May 26 11:54:58.523753 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:58.523733    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-0a5df-kube-rbac-proxy-sar-config\""
May 26 11:54:58.523875 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:58.523739    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-0a5df-predictor-serving-cert\""
May 26 11:54:58.530167 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:58.530143    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph"]
May 26 11:54:58.605946 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:58.605906    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68b2c4c5-b1c4-4441-b57d-8dac155f8799-proxy-tls\") pod \"error-404-isvc-0a5df-predictor-c47858789-v9hph\" (UID: \"68b2c4c5-b1c4-4441-b57d-8dac155f8799\") " pod="kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph"
May 26 11:54:58.605946 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:58.605942    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvjxr\" (UniqueName: \"kubernetes.io/projected/68b2c4c5-b1c4-4441-b57d-8dac155f8799-kube-api-access-pvjxr\") pod \"error-404-isvc-0a5df-predictor-c47858789-v9hph\" (UID: \"68b2c4c5-b1c4-4441-b57d-8dac155f8799\") " pod="kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph"
May 26 11:54:58.606156 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:58.606038    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-0a5df-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68b2c4c5-b1c4-4441-b57d-8dac155f8799-error-404-isvc-0a5df-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0a5df-predictor-c47858789-v9hph\" (UID: \"68b2c4c5-b1c4-4441-b57d-8dac155f8799\") " pod="kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph"
May 26 11:54:58.707088 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:58.707056    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68b2c4c5-b1c4-4441-b57d-8dac155f8799-proxy-tls\") pod \"error-404-isvc-0a5df-predictor-c47858789-v9hph\" (UID: \"68b2c4c5-b1c4-4441-b57d-8dac155f8799\") " pod="kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph"
May 26 11:54:58.707088 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:58.707092    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pvjxr\" (UniqueName: \"kubernetes.io/projected/68b2c4c5-b1c4-4441-b57d-8dac155f8799-kube-api-access-pvjxr\") pod \"error-404-isvc-0a5df-predictor-c47858789-v9hph\" (UID: \"68b2c4c5-b1c4-4441-b57d-8dac155f8799\") " pod="kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph"
May 26 11:54:58.707299 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:54:58.707203    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-serving-cert: secret "error-404-isvc-0a5df-predictor-serving-cert" not found
May 26 11:54:58.707299 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:54:58.707266    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68b2c4c5-b1c4-4441-b57d-8dac155f8799-proxy-tls podName:68b2c4c5-b1c4-4441-b57d-8dac155f8799 nodeName:}" failed. No retries permitted until 2026-05-26 11:54:59.207249676 +0000 UTC m=+1831.398691841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/68b2c4c5-b1c4-4441-b57d-8dac155f8799-proxy-tls") pod "error-404-isvc-0a5df-predictor-c47858789-v9hph" (UID: "68b2c4c5-b1c4-4441-b57d-8dac155f8799") : secret "error-404-isvc-0a5df-predictor-serving-cert" not found
May 26 11:54:58.707299 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:58.707287    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-0a5df-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68b2c4c5-b1c4-4441-b57d-8dac155f8799-error-404-isvc-0a5df-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0a5df-predictor-c47858789-v9hph\" (UID: \"68b2c4c5-b1c4-4441-b57d-8dac155f8799\") " pod="kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph"
May 26 11:54:58.707858 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:58.707840    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-0a5df-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68b2c4c5-b1c4-4441-b57d-8dac155f8799-error-404-isvc-0a5df-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0a5df-predictor-c47858789-v9hph\" (UID: \"68b2c4c5-b1c4-4441-b57d-8dac155f8799\") " pod="kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph"
May 26 11:54:58.717018 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:58.716996    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvjxr\" (UniqueName: \"kubernetes.io/projected/68b2c4c5-b1c4-4441-b57d-8dac155f8799-kube-api-access-pvjxr\") pod \"error-404-isvc-0a5df-predictor-c47858789-v9hph\" (UID: \"68b2c4c5-b1c4-4441-b57d-8dac155f8799\") " pod="kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph"
May 26 11:54:58.725032 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:58.725005    2563 generic.go:358] "Generic (PLEG): container finished" podID="c9325a47-b280-4387-89a6-5e0cba0b0aee" containerID="04a6605eb1bb30798bbc795324c91a49f485ae5eedee2f409bb5ffc7570dff5e" exitCode=2
May 26 11:54:58.725162 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:58.725105    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv" event={"ID":"c9325a47-b280-4387-89a6-5e0cba0b0aee","Type":"ContainerDied","Data":"04a6605eb1bb30798bbc795324c91a49f485ae5eedee2f409bb5ffc7570dff5e"}
May 26 11:54:59.211115 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:59.211076    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68b2c4c5-b1c4-4441-b57d-8dac155f8799-proxy-tls\") pod \"error-404-isvc-0a5df-predictor-c47858789-v9hph\" (UID: \"68b2c4c5-b1c4-4441-b57d-8dac155f8799\") " pod="kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph"
May 26 11:54:59.213715 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:59.213694    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68b2c4c5-b1c4-4441-b57d-8dac155f8799-proxy-tls\") pod \"error-404-isvc-0a5df-predictor-c47858789-v9hph\" (UID: \"68b2c4c5-b1c4-4441-b57d-8dac155f8799\") " pod="kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph"
May 26 11:54:59.429897 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:59.429851    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph"
May 26 11:54:59.557324 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:59.557284    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph"]
May 26 11:54:59.560168 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:54:59.560139    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68b2c4c5_b1c4_4441_b57d_8dac155f8799.slice/crio-efd57671adf152cb4d8a67eaafb84e03383174b053a65a4160fe54c56de424a3 WatchSource:0}: Error finding container efd57671adf152cb4d8a67eaafb84e03383174b053a65a4160fe54c56de424a3: Status 404 returned error can't find the container with id efd57671adf152cb4d8a67eaafb84e03383174b053a65a4160fe54c56de424a3
May 26 11:54:59.704595 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:59.704558    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-5380a-676cb95c4b-phg74" podUID="7d1acaa3-e946-40d3-9937-ef994cf95b0a" containerName="ensemble-graph-5380a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:54:59.729857 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:59.729759    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph" event={"ID":"68b2c4c5-b1c4-4441-b57d-8dac155f8799","Type":"ContainerStarted","Data":"7fc5ad4ec4650dc3e67731f970b3c8196bd54392fda3d3e4ed86cc4f1b7e1df1"}
May 26 11:54:59.729857 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:59.729812    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph" event={"ID":"68b2c4c5-b1c4-4441-b57d-8dac155f8799","Type":"ContainerStarted","Data":"3a9ea443cad48d055af0bdff5970081b9eea10eda5e2e7d6388f6ebd23e61a58"}
May 26 11:54:59.729857 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:59.729824    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph" event={"ID":"68b2c4c5-b1c4-4441-b57d-8dac155f8799","Type":"ContainerStarted","Data":"efd57671adf152cb4d8a67eaafb84e03383174b053a65a4160fe54c56de424a3"}
May 26 11:54:59.730104 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:59.730005    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph"
May 26 11:54:59.730104 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:59.730030    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph"
May 26 11:54:59.731048 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:59.731023    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph" podUID="68b2c4c5-b1c4-4441-b57d-8dac155f8799" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 26 11:54:59.747788 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:54:59.747636    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph" podStartSLOduration=1.7476201040000001 podStartE2EDuration="1.747620104s" podCreationTimestamp="2026-05-26 11:54:58 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:54:59.746941274 +0000 UTC m=+1831.938383460" watchObservedRunningTime="2026-05-26 11:54:59.747620104 +0000 UTC m=+1831.939062292"
May 26 11:55:00.733305 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:00.733256    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph" podUID="68b2c4c5-b1c4-4441-b57d-8dac155f8799" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 26 11:55:01.553421 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:01.553396    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv"
May 26 11:55:01.632825 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:01.632768    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwp6j\" (UniqueName: \"kubernetes.io/projected/c9325a47-b280-4387-89a6-5e0cba0b0aee-kube-api-access-wwp6j\") pod \"c9325a47-b280-4387-89a6-5e0cba0b0aee\" (UID: \"c9325a47-b280-4387-89a6-5e0cba0b0aee\") "
May 26 11:55:01.633019 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:01.632835    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-5380a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9325a47-b280-4387-89a6-5e0cba0b0aee-error-404-isvc-5380a-kube-rbac-proxy-sar-config\") pod \"c9325a47-b280-4387-89a6-5e0cba0b0aee\" (UID: \"c9325a47-b280-4387-89a6-5e0cba0b0aee\") "
May 26 11:55:01.633019 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:01.632881    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9325a47-b280-4387-89a6-5e0cba0b0aee-proxy-tls\") pod \"c9325a47-b280-4387-89a6-5e0cba0b0aee\" (UID: \"c9325a47-b280-4387-89a6-5e0cba0b0aee\") "
May 26 11:55:01.633235 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:01.633210    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9325a47-b280-4387-89a6-5e0cba0b0aee-error-404-isvc-5380a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-5380a-kube-rbac-proxy-sar-config") pod "c9325a47-b280-4387-89a6-5e0cba0b0aee" (UID: "c9325a47-b280-4387-89a6-5e0cba0b0aee"). InnerVolumeSpecName "error-404-isvc-5380a-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:55:01.634945 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:01.634914    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9325a47-b280-4387-89a6-5e0cba0b0aee-kube-api-access-wwp6j" (OuterVolumeSpecName: "kube-api-access-wwp6j") pod "c9325a47-b280-4387-89a6-5e0cba0b0aee" (UID: "c9325a47-b280-4387-89a6-5e0cba0b0aee"). InnerVolumeSpecName "kube-api-access-wwp6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:55:01.635042 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:01.634971    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9325a47-b280-4387-89a6-5e0cba0b0aee-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c9325a47-b280-4387-89a6-5e0cba0b0aee" (UID: "c9325a47-b280-4387-89a6-5e0cba0b0aee"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:55:01.733630 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:01.733535    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9325a47-b280-4387-89a6-5e0cba0b0aee-proxy-tls\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:55:01.733630 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:01.733570    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwp6j\" (UniqueName: \"kubernetes.io/projected/c9325a47-b280-4387-89a6-5e0cba0b0aee-kube-api-access-wwp6j\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:55:01.733630 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:01.733587    2563 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-5380a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9325a47-b280-4387-89a6-5e0cba0b0aee-error-404-isvc-5380a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:55:01.737026 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:01.737000    2563 generic.go:358] "Generic (PLEG): container finished" podID="c9325a47-b280-4387-89a6-5e0cba0b0aee" containerID="4bd0381c01abf0d9472792bf86c7696826b7b51cabac12b1153754ca245df425" exitCode=0
May 26 11:55:01.737142 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:01.737086    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv"
May 26 11:55:01.737142 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:01.737084    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv" event={"ID":"c9325a47-b280-4387-89a6-5e0cba0b0aee","Type":"ContainerDied","Data":"4bd0381c01abf0d9472792bf86c7696826b7b51cabac12b1153754ca245df425"}
May 26 11:55:01.737142 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:01.737128    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv" event={"ID":"c9325a47-b280-4387-89a6-5e0cba0b0aee","Type":"ContainerDied","Data":"1c21dcccef0dea904e91ea111997c94541b48a31866f6deeed66f7c21637f023"}
May 26 11:55:01.737238 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:01.737148    2563 scope.go:117] "RemoveContainer" containerID="04a6605eb1bb30798bbc795324c91a49f485ae5eedee2f409bb5ffc7570dff5e"
May 26 11:55:01.748217 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:01.748193    2563 scope.go:117] "RemoveContainer" containerID="4bd0381c01abf0d9472792bf86c7696826b7b51cabac12b1153754ca245df425"
May 26 11:55:01.756015 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:01.755990    2563 scope.go:117] "RemoveContainer" containerID="04a6605eb1bb30798bbc795324c91a49f485ae5eedee2f409bb5ffc7570dff5e"
May 26 11:55:01.756257 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:55:01.756242    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04a6605eb1bb30798bbc795324c91a49f485ae5eedee2f409bb5ffc7570dff5e\": container with ID starting with 04a6605eb1bb30798bbc795324c91a49f485ae5eedee2f409bb5ffc7570dff5e not found: ID does not exist" containerID="04a6605eb1bb30798bbc795324c91a49f485ae5eedee2f409bb5ffc7570dff5e"
May 26 11:55:01.756320 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:01.756265    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04a6605eb1bb30798bbc795324c91a49f485ae5eedee2f409bb5ffc7570dff5e"} err="failed to get container status \"04a6605eb1bb30798bbc795324c91a49f485ae5eedee2f409bb5ffc7570dff5e\": rpc error: code = NotFound desc = could not find container \"04a6605eb1bb30798bbc795324c91a49f485ae5eedee2f409bb5ffc7570dff5e\": container with ID starting with 04a6605eb1bb30798bbc795324c91a49f485ae5eedee2f409bb5ffc7570dff5e not found: ID does not exist"
May 26 11:55:01.756320 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:01.756280    2563 scope.go:117] "RemoveContainer" containerID="4bd0381c01abf0d9472792bf86c7696826b7b51cabac12b1153754ca245df425"
May 26 11:55:01.756479 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:55:01.756456    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bd0381c01abf0d9472792bf86c7696826b7b51cabac12b1153754ca245df425\": container with ID starting with 4bd0381c01abf0d9472792bf86c7696826b7b51cabac12b1153754ca245df425 not found: ID does not exist" containerID="4bd0381c01abf0d9472792bf86c7696826b7b51cabac12b1153754ca245df425"
May 26 11:55:01.756547 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:01.756480    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bd0381c01abf0d9472792bf86c7696826b7b51cabac12b1153754ca245df425"} err="failed to get container status \"4bd0381c01abf0d9472792bf86c7696826b7b51cabac12b1153754ca245df425\": rpc error: code = NotFound desc = could not find container \"4bd0381c01abf0d9472792bf86c7696826b7b51cabac12b1153754ca245df425\": container with ID starting with 4bd0381c01abf0d9472792bf86c7696826b7b51cabac12b1153754ca245df425 not found: ID does not exist"
May 26 11:55:01.765808 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:01.765764    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv"]
May 26 11:55:01.774599 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:01.774579    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv"]
May 26 11:55:02.428267 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:02.428235    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9325a47-b280-4387-89a6-5e0cba0b0aee" path="/var/lib/kubelet/pods/c9325a47-b280-4387-89a6-5e0cba0b0aee/volumes"
May 26 11:55:02.517830 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:02.517759    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5380a-predictor-78c9576cbc-hsjpv" podUID="c9325a47-b280-4387-89a6-5e0cba0b0aee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
May 26 11:55:04.704902 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:04.704858    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-5380a-676cb95c4b-phg74" podUID="7d1acaa3-e946-40d3-9937-ef994cf95b0a" containerName="ensemble-graph-5380a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:55:05.620561 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:05.620532    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb"
May 26 11:55:05.737229 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:05.737201    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph"
May 26 11:55:05.737660 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:05.737632    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph" podUID="68b2c4c5-b1c4-4441-b57d-8dac155f8799" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 26 11:55:09.704917 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:09.704871    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-5380a-676cb95c4b-phg74" podUID="7d1acaa3-e946-40d3-9937-ef994cf95b0a" containerName="ensemble-graph-5380a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:55:09.705369 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:09.704983    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-5380a-676cb95c4b-phg74"
May 26 11:55:14.704520 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:14.704475    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-5380a-676cb95c4b-phg74" podUID="7d1acaa3-e946-40d3-9937-ef994cf95b0a" containerName="ensemble-graph-5380a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:55:15.738242 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:15.738193    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph" podUID="68b2c4c5-b1c4-4441-b57d-8dac155f8799" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 26 11:55:17.949047 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:17.949015    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-fbdd2-58d5d54c6b-lps2t"]
May 26 11:55:17.949647 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:17.949350    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9325a47-b280-4387-89a6-5e0cba0b0aee" containerName="kserve-container"
May 26 11:55:17.949647 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:17.949362    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9325a47-b280-4387-89a6-5e0cba0b0aee" containerName="kserve-container"
May 26 11:55:17.949647 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:17.949373    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9325a47-b280-4387-89a6-5e0cba0b0aee" containerName="kube-rbac-proxy"
May 26 11:55:17.949647 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:17.949379    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9325a47-b280-4387-89a6-5e0cba0b0aee" containerName="kube-rbac-proxy"
May 26 11:55:17.949647 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:17.949425    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="c9325a47-b280-4387-89a6-5e0cba0b0aee" containerName="kserve-container"
May 26 11:55:17.949647 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:17.949434    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="c9325a47-b280-4387-89a6-5e0cba0b0aee" containerName="kube-rbac-proxy"
May 26 11:55:17.953446 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:17.953426    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-fbdd2-58d5d54c6b-lps2t"
May 26 11:55:17.955939 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:17.955917    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-fbdd2-serving-cert\""
May 26 11:55:17.956041 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:17.955941    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-fbdd2-kube-rbac-proxy-sar-config\""
May 26 11:55:17.960135 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:17.960112    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-fbdd2-58d5d54c6b-lps2t"]
May 26 11:55:18.069173 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:18.069136    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d6808ed-f916-452d-a0bc-bd782dda39a1-proxy-tls\") pod \"sequence-graph-fbdd2-58d5d54c6b-lps2t\" (UID: \"7d6808ed-f916-452d-a0bc-bd782dda39a1\") " pod="kserve-ci-e2e-test/sequence-graph-fbdd2-58d5d54c6b-lps2t"
May 26 11:55:18.069355 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:18.069193    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d6808ed-f916-452d-a0bc-bd782dda39a1-openshift-service-ca-bundle\") pod \"sequence-graph-fbdd2-58d5d54c6b-lps2t\" (UID: \"7d6808ed-f916-452d-a0bc-bd782dda39a1\") " pod="kserve-ci-e2e-test/sequence-graph-fbdd2-58d5d54c6b-lps2t"
May 26 11:55:18.170572 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:18.170534    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d6808ed-f916-452d-a0bc-bd782dda39a1-openshift-service-ca-bundle\") pod \"sequence-graph-fbdd2-58d5d54c6b-lps2t\" (UID: \"7d6808ed-f916-452d-a0bc-bd782dda39a1\") " pod="kserve-ci-e2e-test/sequence-graph-fbdd2-58d5d54c6b-lps2t"
May 26 11:55:18.170748 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:18.170611    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d6808ed-f916-452d-a0bc-bd782dda39a1-proxy-tls\") pod \"sequence-graph-fbdd2-58d5d54c6b-lps2t\" (UID: \"7d6808ed-f916-452d-a0bc-bd782dda39a1\") " pod="kserve-ci-e2e-test/sequence-graph-fbdd2-58d5d54c6b-lps2t"
May 26 11:55:18.170748 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:55:18.170734    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-fbdd2-serving-cert: secret "sequence-graph-fbdd2-serving-cert" not found
May 26 11:55:18.170848 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:55:18.170822    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d6808ed-f916-452d-a0bc-bd782dda39a1-proxy-tls podName:7d6808ed-f916-452d-a0bc-bd782dda39a1 nodeName:}" failed. No retries permitted until 2026-05-26 11:55:18.670799638 +0000 UTC m=+1850.862241807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7d6808ed-f916-452d-a0bc-bd782dda39a1-proxy-tls") pod "sequence-graph-fbdd2-58d5d54c6b-lps2t" (UID: "7d6808ed-f916-452d-a0bc-bd782dda39a1") : secret "sequence-graph-fbdd2-serving-cert" not found
May 26 11:55:18.171307 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:18.171287    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d6808ed-f916-452d-a0bc-bd782dda39a1-openshift-service-ca-bundle\") pod \"sequence-graph-fbdd2-58d5d54c6b-lps2t\" (UID: \"7d6808ed-f916-452d-a0bc-bd782dda39a1\") " pod="kserve-ci-e2e-test/sequence-graph-fbdd2-58d5d54c6b-lps2t"
May 26 11:55:18.675165 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:18.675108    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d6808ed-f916-452d-a0bc-bd782dda39a1-proxy-tls\") pod \"sequence-graph-fbdd2-58d5d54c6b-lps2t\" (UID: \"7d6808ed-f916-452d-a0bc-bd782dda39a1\") " pod="kserve-ci-e2e-test/sequence-graph-fbdd2-58d5d54c6b-lps2t"
May 26 11:55:18.677753 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:18.677734    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d6808ed-f916-452d-a0bc-bd782dda39a1-proxy-tls\") pod \"sequence-graph-fbdd2-58d5d54c6b-lps2t\" (UID: \"7d6808ed-f916-452d-a0bc-bd782dda39a1\") " pod="kserve-ci-e2e-test/sequence-graph-fbdd2-58d5d54c6b-lps2t"
May 26 11:55:18.865308 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:18.865265    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-fbdd2-58d5d54c6b-lps2t"
May 26 11:55:19.018589 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:19.018554    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-fbdd2-58d5d54c6b-lps2t"]
May 26 11:55:19.021531 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:55:19.021505    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d6808ed_f916_452d_a0bc_bd782dda39a1.slice/crio-849555cf44dd1bfe7e2083aafe0dcfa36f2ec446444783cd94ade4e0e355d14a WatchSource:0}: Error finding container 849555cf44dd1bfe7e2083aafe0dcfa36f2ec446444783cd94ade4e0e355d14a: Status 404 returned error can't find the container with id 849555cf44dd1bfe7e2083aafe0dcfa36f2ec446444783cd94ade4e0e355d14a
May 26 11:55:19.704064 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:19.704025    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-5380a-676cb95c4b-phg74" podUID="7d1acaa3-e946-40d3-9937-ef994cf95b0a" containerName="ensemble-graph-5380a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:55:19.793066 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:19.793029    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-fbdd2-58d5d54c6b-lps2t" event={"ID":"7d6808ed-f916-452d-a0bc-bd782dda39a1","Type":"ContainerStarted","Data":"aa07c366e8ec39284a71a8d96c7097b842ab7dd291d03c43bd96bf747fee6768"}
May 26 11:55:19.793066 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:19.793073    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-fbdd2-58d5d54c6b-lps2t" event={"ID":"7d6808ed-f916-452d-a0bc-bd782dda39a1","Type":"ContainerStarted","Data":"849555cf44dd1bfe7e2083aafe0dcfa36f2ec446444783cd94ade4e0e355d14a"}
May 26 11:55:19.793340 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:19.793160    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-fbdd2-58d5d54c6b-lps2t"
May 26 11:55:19.811167 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:19.811109    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-fbdd2-58d5d54c6b-lps2t" podStartSLOduration=2.811095553 podStartE2EDuration="2.811095553s" podCreationTimestamp="2026-05-26 11:55:17 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:55:19.808881293 +0000 UTC m=+1852.000323481" watchObservedRunningTime="2026-05-26 11:55:19.811095553 +0000 UTC m=+1852.002537740"
May 26 11:55:24.704670 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:24.704631    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-5380a-676cb95c4b-phg74" podUID="7d1acaa3-e946-40d3-9937-ef994cf95b0a" containerName="ensemble-graph-5380a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:55:25.737849 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:25.737810    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph" podUID="68b2c4c5-b1c4-4441-b57d-8dac155f8799" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 26 11:55:25.801895 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:25.801861    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-fbdd2-58d5d54c6b-lps2t"
May 26 11:55:28.073080 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.073047    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-fbdd2-58d5d54c6b-lps2t"]
May 26 11:55:28.073449 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.073300    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-fbdd2-58d5d54c6b-lps2t" podUID="7d6808ed-f916-452d-a0bc-bd782dda39a1" containerName="sequence-graph-fbdd2" containerID="cri-o://aa07c366e8ec39284a71a8d96c7097b842ab7dd291d03c43bd96bf747fee6768" gracePeriod=30
May 26 11:55:28.247466 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.247432    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb"]
May 26 11:55:28.247815 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.247742    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb" podUID="b280eea3-83f6-44fe-9ec8-3566464fb8ee" containerName="kserve-container" containerID="cri-o://8b13864361fdf23447bd0fc62c5e7aa3bf3b2e194882c73f99edacd2291687cb" gracePeriod=30
May 26 11:55:28.247927 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.247828    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb" podUID="b280eea3-83f6-44fe-9ec8-3566464fb8ee" containerName="kube-rbac-proxy" containerID="cri-o://b39fd0810274a489b9305b3d01312d44f02d81b504f2afd453ec9d656a7a8a51" gracePeriod=30
May 26 11:55:28.293347 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.293312    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw"]
May 26 11:55:28.296644 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.296620    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw"
May 26 11:55:28.299473 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.299447    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3c458-kube-rbac-proxy-sar-config\""
May 26 11:55:28.299793 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.299752    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3c458-predictor-serving-cert\""
May 26 11:55:28.310081 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.310038    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw"]
May 26 11:55:28.354505 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.354473    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtlkj\" (UniqueName: \"kubernetes.io/projected/b21b40a6-9db5-4117-8ad1-b1f087ed9ecc-kube-api-access-rtlkj\") pod \"error-404-isvc-3c458-predictor-64658f5dcf-rcnxw\" (UID: \"b21b40a6-9db5-4117-8ad1-b1f087ed9ecc\") " pod="kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw"
May 26 11:55:28.354688 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.354523    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-3c458-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b21b40a6-9db5-4117-8ad1-b1f087ed9ecc-error-404-isvc-3c458-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3c458-predictor-64658f5dcf-rcnxw\" (UID: \"b21b40a6-9db5-4117-8ad1-b1f087ed9ecc\") " pod="kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw"
May 26 11:55:28.354688 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.354577    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b21b40a6-9db5-4117-8ad1-b1f087ed9ecc-proxy-tls\") pod \"error-404-isvc-3c458-predictor-64658f5dcf-rcnxw\" (UID: \"b21b40a6-9db5-4117-8ad1-b1f087ed9ecc\") " pod="kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw"
May 26 11:55:28.397958 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.397922    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-5380a-676cb95c4b-phg74"
May 26 11:55:28.455736 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.455699    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rtlkj\" (UniqueName: \"kubernetes.io/projected/b21b40a6-9db5-4117-8ad1-b1f087ed9ecc-kube-api-access-rtlkj\") pod \"error-404-isvc-3c458-predictor-64658f5dcf-rcnxw\" (UID: \"b21b40a6-9db5-4117-8ad1-b1f087ed9ecc\") " pod="kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw"
May 26 11:55:28.455937 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.455745    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-3c458-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b21b40a6-9db5-4117-8ad1-b1f087ed9ecc-error-404-isvc-3c458-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3c458-predictor-64658f5dcf-rcnxw\" (UID: \"b21b40a6-9db5-4117-8ad1-b1f087ed9ecc\") " pod="kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw"
May 26 11:55:28.455937 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.455811    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b21b40a6-9db5-4117-8ad1-b1f087ed9ecc-proxy-tls\") pod \"error-404-isvc-3c458-predictor-64658f5dcf-rcnxw\" (UID: \"b21b40a6-9db5-4117-8ad1-b1f087ed9ecc\") " pod="kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw"
May 26 11:55:28.460729 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.460709    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3c458-predictor-serving-cert\""
May 26 11:55:28.460840 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.460767    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3c458-kube-rbac-proxy-sar-config\""
May 26 11:55:28.466533 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:55:28.466510    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-3c458-predictor-serving-cert: secret "error-404-isvc-3c458-predictor-serving-cert" not found
May 26 11:55:28.466619 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:55:28.466584    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b21b40a6-9db5-4117-8ad1-b1f087ed9ecc-proxy-tls podName:b21b40a6-9db5-4117-8ad1-b1f087ed9ecc nodeName:}" failed. No retries permitted until 2026-05-26 11:55:28.966563271 +0000 UTC m=+1861.158005450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b21b40a6-9db5-4117-8ad1-b1f087ed9ecc-proxy-tls") pod "error-404-isvc-3c458-predictor-64658f5dcf-rcnxw" (UID: "b21b40a6-9db5-4117-8ad1-b1f087ed9ecc") : secret "error-404-isvc-3c458-predictor-serving-cert" not found
May 26 11:55:28.467119 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.467099    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-3c458-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b21b40a6-9db5-4117-8ad1-b1f087ed9ecc-error-404-isvc-3c458-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3c458-predictor-64658f5dcf-rcnxw\" (UID: \"b21b40a6-9db5-4117-8ad1-b1f087ed9ecc\") " pod="kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw"
May 26 11:55:28.469526 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.469504    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtlkj\" (UniqueName: \"kubernetes.io/projected/b21b40a6-9db5-4117-8ad1-b1f087ed9ecc-kube-api-access-rtlkj\") pod \"error-404-isvc-3c458-predictor-64658f5dcf-rcnxw\" (UID: \"b21b40a6-9db5-4117-8ad1-b1f087ed9ecc\") " pod="kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw"
May 26 11:55:28.556559 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.556531    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d1acaa3-e946-40d3-9937-ef994cf95b0a-proxy-tls\") pod \"7d1acaa3-e946-40d3-9937-ef994cf95b0a\" (UID: \"7d1acaa3-e946-40d3-9937-ef994cf95b0a\") "
May 26 11:55:28.556717 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.556588    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d1acaa3-e946-40d3-9937-ef994cf95b0a-openshift-service-ca-bundle\") pod \"7d1acaa3-e946-40d3-9937-ef994cf95b0a\" (UID: \"7d1acaa3-e946-40d3-9937-ef994cf95b0a\") "
May 26 11:55:28.556977 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.556953    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d1acaa3-e946-40d3-9937-ef994cf95b0a-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "7d1acaa3-e946-40d3-9937-ef994cf95b0a" (UID: "7d1acaa3-e946-40d3-9937-ef994cf95b0a"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:55:28.558729 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.558711    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d1acaa3-e946-40d3-9937-ef994cf95b0a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7d1acaa3-e946-40d3-9937-ef994cf95b0a" (UID: "7d1acaa3-e946-40d3-9937-ef994cf95b0a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:55:28.657896 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.657807    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d1acaa3-e946-40d3-9937-ef994cf95b0a-proxy-tls\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:55:28.657896 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.657848    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d1acaa3-e946-40d3-9937-ef994cf95b0a-openshift-service-ca-bundle\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:55:28.819903 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.819861    2563 generic.go:358] "Generic (PLEG): container finished" podID="b280eea3-83f6-44fe-9ec8-3566464fb8ee" containerID="b39fd0810274a489b9305b3d01312d44f02d81b504f2afd453ec9d656a7a8a51" exitCode=2
May 26 11:55:28.820087 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.819939    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb" event={"ID":"b280eea3-83f6-44fe-9ec8-3566464fb8ee","Type":"ContainerDied","Data":"b39fd0810274a489b9305b3d01312d44f02d81b504f2afd453ec9d656a7a8a51"}
May 26 11:55:28.820994 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.820973    2563 generic.go:358] "Generic (PLEG): container finished" podID="7d1acaa3-e946-40d3-9937-ef994cf95b0a" containerID="89129a975e0e3aa234a5803889c229f86dc8da317f22ebf4a6d9d12067e91467" exitCode=137
May 26 11:55:28.821101 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.821002    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-5380a-676cb95c4b-phg74" event={"ID":"7d1acaa3-e946-40d3-9937-ef994cf95b0a","Type":"ContainerDied","Data":"89129a975e0e3aa234a5803889c229f86dc8da317f22ebf4a6d9d12067e91467"}
May 26 11:55:28.821101 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.821035    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-5380a-676cb95c4b-phg74" event={"ID":"7d1acaa3-e946-40d3-9937-ef994cf95b0a","Type":"ContainerDied","Data":"9930b50a433edbe43c5ebf583ea20d5f628b608c8d733711e7e312ae35f55f98"}
May 26 11:55:28.821101 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.821053    2563 scope.go:117] "RemoveContainer" containerID="89129a975e0e3aa234a5803889c229f86dc8da317f22ebf4a6d9d12067e91467"
May 26 11:55:28.821101 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.821064    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-5380a-676cb95c4b-phg74"
May 26 11:55:28.828993 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.828975    2563 scope.go:117] "RemoveContainer" containerID="89129a975e0e3aa234a5803889c229f86dc8da317f22ebf4a6d9d12067e91467"
May 26 11:55:28.829270 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:55:28.829245    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89129a975e0e3aa234a5803889c229f86dc8da317f22ebf4a6d9d12067e91467\": container with ID starting with 89129a975e0e3aa234a5803889c229f86dc8da317f22ebf4a6d9d12067e91467 not found: ID does not exist" containerID="89129a975e0e3aa234a5803889c229f86dc8da317f22ebf4a6d9d12067e91467"
May 26 11:55:28.829316 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.829281    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89129a975e0e3aa234a5803889c229f86dc8da317f22ebf4a6d9d12067e91467"} err="failed to get container status \"89129a975e0e3aa234a5803889c229f86dc8da317f22ebf4a6d9d12067e91467\": rpc error: code = NotFound desc = could not find container \"89129a975e0e3aa234a5803889c229f86dc8da317f22ebf4a6d9d12067e91467\": container with ID starting with 89129a975e0e3aa234a5803889c229f86dc8da317f22ebf4a6d9d12067e91467 not found: ID does not exist"
May 26 11:55:28.845028 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.844992    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-5380a-676cb95c4b-phg74"]
May 26 11:55:28.849077 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:28.849054    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-5380a-676cb95c4b-phg74"]
May 26 11:55:29.062115 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:29.062008    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b21b40a6-9db5-4117-8ad1-b1f087ed9ecc-proxy-tls\") pod \"error-404-isvc-3c458-predictor-64658f5dcf-rcnxw\" (UID: \"b21b40a6-9db5-4117-8ad1-b1f087ed9ecc\") " pod="kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw"
May 26 11:55:29.064546 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:29.064515    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b21b40a6-9db5-4117-8ad1-b1f087ed9ecc-proxy-tls\") pod \"error-404-isvc-3c458-predictor-64658f5dcf-rcnxw\" (UID: \"b21b40a6-9db5-4117-8ad1-b1f087ed9ecc\") " pod="kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw"
May 26 11:55:29.222669 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:29.222619    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw"
May 26 11:55:29.348547 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:29.348512    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw"]
May 26 11:55:29.826409 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:29.826325    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw" event={"ID":"b21b40a6-9db5-4117-8ad1-b1f087ed9ecc","Type":"ContainerStarted","Data":"cc3afe0ac12c582f628e9a39ef8ac73d2e0e8678838d194d46ccf067e17d781e"}
May 26 11:55:29.826409 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:29.826365    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw" event={"ID":"b21b40a6-9db5-4117-8ad1-b1f087ed9ecc","Type":"ContainerStarted","Data":"a91dcc99d1b5654c131ed7a10c96c2a268d4f535efa6f42102e03288aa680604"}
May 26 11:55:29.826409 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:29.826381    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw" event={"ID":"b21b40a6-9db5-4117-8ad1-b1f087ed9ecc","Type":"ContainerStarted","Data":"3881af3bcade4b942e3a6eb9241da165b4cbe42deea044a490100341f4d41181"}
May 26 11:55:29.826607 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:29.826478    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw"
May 26 11:55:29.848292 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:29.848247    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw" podStartSLOduration=1.8482339479999998 podStartE2EDuration="1.848233948s" podCreationTimestamp="2026-05-26 11:55:28 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:55:29.84722998 +0000 UTC m=+1862.038672178" watchObservedRunningTime="2026-05-26 11:55:29.848233948 +0000 UTC m=+1862.039676135"
May 26 11:55:30.429522 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:30.429492    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d1acaa3-e946-40d3-9937-ef994cf95b0a" path="/var/lib/kubelet/pods/7d1acaa3-e946-40d3-9937-ef994cf95b0a/volumes"
May 26 11:55:30.610936 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:30.610891    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb" podUID="b280eea3-83f6-44fe-9ec8-3566464fb8ee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
May 26 11:55:30.801125 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:30.801044    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-fbdd2-58d5d54c6b-lps2t" podUID="7d6808ed-f916-452d-a0bc-bd782dda39a1" containerName="sequence-graph-fbdd2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:55:30.829203 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:30.829175    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw"
May 26 11:55:30.830359 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:30.830334    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw" podUID="b21b40a6-9db5-4117-8ad1-b1f087ed9ecc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
May 26 11:55:31.504027 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:31.504004    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb"
May 26 11:55:31.582108 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:31.582027    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2q4gv\" (UniqueName: \"kubernetes.io/projected/b280eea3-83f6-44fe-9ec8-3566464fb8ee-kube-api-access-2q4gv\") pod \"b280eea3-83f6-44fe-9ec8-3566464fb8ee\" (UID: \"b280eea3-83f6-44fe-9ec8-3566464fb8ee\") "
May 26 11:55:31.582254 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:31.582125    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-fbdd2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b280eea3-83f6-44fe-9ec8-3566464fb8ee-error-404-isvc-fbdd2-kube-rbac-proxy-sar-config\") pod \"b280eea3-83f6-44fe-9ec8-3566464fb8ee\" (UID: \"b280eea3-83f6-44fe-9ec8-3566464fb8ee\") "
May 26 11:55:31.582254 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:31.582157    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b280eea3-83f6-44fe-9ec8-3566464fb8ee-proxy-tls\") pod \"b280eea3-83f6-44fe-9ec8-3566464fb8ee\" (UID: \"b280eea3-83f6-44fe-9ec8-3566464fb8ee\") "
May 26 11:55:31.582515 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:31.582488    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b280eea3-83f6-44fe-9ec8-3566464fb8ee-error-404-isvc-fbdd2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-fbdd2-kube-rbac-proxy-sar-config") pod "b280eea3-83f6-44fe-9ec8-3566464fb8ee" (UID: "b280eea3-83f6-44fe-9ec8-3566464fb8ee"). InnerVolumeSpecName "error-404-isvc-fbdd2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:55:31.584314 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:31.584293    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b280eea3-83f6-44fe-9ec8-3566464fb8ee-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b280eea3-83f6-44fe-9ec8-3566464fb8ee" (UID: "b280eea3-83f6-44fe-9ec8-3566464fb8ee"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:55:31.584395 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:31.584372    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b280eea3-83f6-44fe-9ec8-3566464fb8ee-kube-api-access-2q4gv" (OuterVolumeSpecName: "kube-api-access-2q4gv") pod "b280eea3-83f6-44fe-9ec8-3566464fb8ee" (UID: "b280eea3-83f6-44fe-9ec8-3566464fb8ee"). InnerVolumeSpecName "kube-api-access-2q4gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:55:31.682788 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:31.682752    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q4gv\" (UniqueName: \"kubernetes.io/projected/b280eea3-83f6-44fe-9ec8-3566464fb8ee-kube-api-access-2q4gv\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:55:31.682950 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:31.682802    2563 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-fbdd2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b280eea3-83f6-44fe-9ec8-3566464fb8ee-error-404-isvc-fbdd2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:55:31.682950 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:31.682813    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b280eea3-83f6-44fe-9ec8-3566464fb8ee-proxy-tls\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:55:31.833692 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:31.833601    2563 generic.go:358] "Generic (PLEG): container finished" podID="b280eea3-83f6-44fe-9ec8-3566464fb8ee" containerID="8b13864361fdf23447bd0fc62c5e7aa3bf3b2e194882c73f99edacd2291687cb" exitCode=0
May 26 11:55:31.833692 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:31.833674    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb"
May 26 11:55:31.833960 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:31.833691    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb" event={"ID":"b280eea3-83f6-44fe-9ec8-3566464fb8ee","Type":"ContainerDied","Data":"8b13864361fdf23447bd0fc62c5e7aa3bf3b2e194882c73f99edacd2291687cb"}
May 26 11:55:31.833960 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:31.833732    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb" event={"ID":"b280eea3-83f6-44fe-9ec8-3566464fb8ee","Type":"ContainerDied","Data":"5a8f9359bd380e7058d33012d720cb2cf8d6554ed4f33ac515f001c6f3964cbe"}
May 26 11:55:31.833960 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:31.833755    2563 scope.go:117] "RemoveContainer" containerID="b39fd0810274a489b9305b3d01312d44f02d81b504f2afd453ec9d656a7a8a51"
May 26 11:55:31.834339 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:31.834303    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw" podUID="b21b40a6-9db5-4117-8ad1-b1f087ed9ecc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
May 26 11:55:31.842398 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:31.842378    2563 scope.go:117] "RemoveContainer" containerID="8b13864361fdf23447bd0fc62c5e7aa3bf3b2e194882c73f99edacd2291687cb"
May 26 11:55:31.849709 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:31.849690    2563 scope.go:117] "RemoveContainer" containerID="b39fd0810274a489b9305b3d01312d44f02d81b504f2afd453ec9d656a7a8a51"
May 26 11:55:31.849971 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:55:31.849952    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b39fd0810274a489b9305b3d01312d44f02d81b504f2afd453ec9d656a7a8a51\": container with ID starting with b39fd0810274a489b9305b3d01312d44f02d81b504f2afd453ec9d656a7a8a51 not found: ID does not exist" containerID="b39fd0810274a489b9305b3d01312d44f02d81b504f2afd453ec9d656a7a8a51"
May 26 11:55:31.850051 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:31.849984    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b39fd0810274a489b9305b3d01312d44f02d81b504f2afd453ec9d656a7a8a51"} err="failed to get container status \"b39fd0810274a489b9305b3d01312d44f02d81b504f2afd453ec9d656a7a8a51\": rpc error: code = NotFound desc = could not find container \"b39fd0810274a489b9305b3d01312d44f02d81b504f2afd453ec9d656a7a8a51\": container with ID starting with b39fd0810274a489b9305b3d01312d44f02d81b504f2afd453ec9d656a7a8a51 not found: ID does not exist"
May 26 11:55:31.850051 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:31.850013    2563 scope.go:117] "RemoveContainer" containerID="8b13864361fdf23447bd0fc62c5e7aa3bf3b2e194882c73f99edacd2291687cb"
May 26 11:55:31.850266 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:55:31.850243    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b13864361fdf23447bd0fc62c5e7aa3bf3b2e194882c73f99edacd2291687cb\": container with ID starting with 8b13864361fdf23447bd0fc62c5e7aa3bf3b2e194882c73f99edacd2291687cb not found: ID does not exist" containerID="8b13864361fdf23447bd0fc62c5e7aa3bf3b2e194882c73f99edacd2291687cb"
May 26 11:55:31.850301 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:31.850273    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b13864361fdf23447bd0fc62c5e7aa3bf3b2e194882c73f99edacd2291687cb"} err="failed to get container status \"8b13864361fdf23447bd0fc62c5e7aa3bf3b2e194882c73f99edacd2291687cb\": rpc error: code = NotFound desc = could not find container \"8b13864361fdf23447bd0fc62c5e7aa3bf3b2e194882c73f99edacd2291687cb\": container with ID starting with 8b13864361fdf23447bd0fc62c5e7aa3bf3b2e194882c73f99edacd2291687cb not found: ID does not exist"
May 26 11:55:31.855541 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:31.855519    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb"]
May 26 11:55:31.858897 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:31.858874    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fbdd2-predictor-5d98fbb9bf-p4zsb"]
May 26 11:55:32.429020 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:32.428988    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b280eea3-83f6-44fe-9ec8-3566464fb8ee" path="/var/lib/kubelet/pods/b280eea3-83f6-44fe-9ec8-3566464fb8ee/volumes"
May 26 11:55:35.737705 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:35.737669    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph" podUID="68b2c4c5-b1c4-4441-b57d-8dac155f8799" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 26 11:55:35.799622 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:35.799584    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-fbdd2-58d5d54c6b-lps2t" podUID="7d6808ed-f916-452d-a0bc-bd782dda39a1" containerName="sequence-graph-fbdd2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:55:36.839154 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:36.839127    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw"
May 26 11:55:36.839692 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:36.839662    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw" podUID="b21b40a6-9db5-4117-8ad1-b1f087ed9ecc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
May 26 11:55:40.800390 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:40.800326    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-fbdd2-58d5d54c6b-lps2t" podUID="7d6808ed-f916-452d-a0bc-bd782dda39a1" containerName="sequence-graph-fbdd2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:55:40.800767 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:40.800490    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-fbdd2-58d5d54c6b-lps2t"
May 26 11:55:45.738868 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:45.738841    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph"
May 26 11:55:45.800968 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:45.800926    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-fbdd2-58d5d54c6b-lps2t" podUID="7d6808ed-f916-452d-a0bc-bd782dda39a1" containerName="sequence-graph-fbdd2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:55:46.840631 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:46.840591    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw" podUID="b21b40a6-9db5-4117-8ad1-b1f087ed9ecc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
May 26 11:55:50.799520 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:50.799480    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-fbdd2-58d5d54c6b-lps2t" podUID="7d6808ed-f916-452d-a0bc-bd782dda39a1" containerName="sequence-graph-fbdd2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:55:55.800252 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:55.800212    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-fbdd2-58d5d54c6b-lps2t" podUID="7d6808ed-f916-452d-a0bc-bd782dda39a1" containerName="sequence-graph-fbdd2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:55:56.840017 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:56.839975    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw" podUID="b21b40a6-9db5-4117-8ad1-b1f087ed9ecc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
May 26 11:55:58.092071 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:55:58.092043    2563 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d1acaa3_e946_40d3_9937_ef994cf95b0a.slice\": RecentStats: unable to find data in memory cache]"
May 26 11:55:58.092656 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:55:58.092092    2563 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d6808ed_f916_452d_a0bc_bd782dda39a1.slice/crio-849555cf44dd1bfe7e2083aafe0dcfa36f2ec446444783cd94ade4e0e355d14a\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d1acaa3_e946_40d3_9937_ef994cf95b0a.slice\": RecentStats: unable to find data in memory cache]"
May 26 11:55:58.092976 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:55:58.092180    2563 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d1acaa3_e946_40d3_9937_ef994cf95b0a.slice\": RecentStats: unable to find data in memory cache]"
May 26 11:55:58.093187 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:55:58.092180    2563 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d6808ed_f916_452d_a0bc_bd782dda39a1.slice/crio-849555cf44dd1bfe7e2083aafe0dcfa36f2ec446444783cd94ade4e0e355d14a\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d1acaa3_e946_40d3_9937_ef994cf95b0a.slice\": RecentStats: unable to find data in memory cache]"
May 26 11:55:58.094950 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:55:58.094900    2563 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d1acaa3_e946_40d3_9937_ef994cf95b0a.slice/crio-9930b50a433edbe43c5ebf583ea20d5f628b608c8d733711e7e312ae35f55f98\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d6808ed_f916_452d_a0bc_bd782dda39a1.slice/crio-conmon-aa07c366e8ec39284a71a8d96c7097b842ab7dd291d03c43bd96bf747fee6768.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb280eea3_83f6_44fe_9ec8_3566464fb8ee.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d6808ed_f916_452d_a0bc_bd782dda39a1.slice/crio-aa07c366e8ec39284a71a8d96c7097b842ab7dd291d03c43bd96bf747fee6768.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb280eea3_83f6_44fe_9ec8_3566464fb8ee.slice/crio-8b13864361fdf23447bd0fc62c5e7aa3bf3b2e194882c73f99edacd2291687cb.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb280eea3_83f6_44fe_9ec8_3566464fb8ee.slice/crio-conmon-8b13864361fdf23447bd0fc62c5e7aa3bf3b2e194882c73f99edacd2291687cb.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb280eea3_83f6_44fe_9ec8_3566464fb8ee.slice/crio-5a8f9359bd380e7058d33012d720cb2cf8d6554ed4f33ac515f001c6f3964cbe\": RecentStats: unable to find data in memory cache]"
May 26 11:55:58.215420 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.215399    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-fbdd2-58d5d54c6b-lps2t"
May 26 11:55:58.298100 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.298064    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d6808ed-f916-452d-a0bc-bd782dda39a1-openshift-service-ca-bundle\") pod \"7d6808ed-f916-452d-a0bc-bd782dda39a1\" (UID: \"7d6808ed-f916-452d-a0bc-bd782dda39a1\") "
May 26 11:55:58.298262 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.298114    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d6808ed-f916-452d-a0bc-bd782dda39a1-proxy-tls\") pod \"7d6808ed-f916-452d-a0bc-bd782dda39a1\" (UID: \"7d6808ed-f916-452d-a0bc-bd782dda39a1\") "
May 26 11:55:58.298430 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.298404    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d6808ed-f916-452d-a0bc-bd782dda39a1-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "7d6808ed-f916-452d-a0bc-bd782dda39a1" (UID: "7d6808ed-f916-452d-a0bc-bd782dda39a1"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:55:58.300392 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.300369    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d6808ed-f916-452d-a0bc-bd782dda39a1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7d6808ed-f916-452d-a0bc-bd782dda39a1" (UID: "7d6808ed-f916-452d-a0bc-bd782dda39a1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:55:58.399158 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.399076    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d6808ed-f916-452d-a0bc-bd782dda39a1-openshift-service-ca-bundle\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:55:58.399158 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.399107    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d6808ed-f916-452d-a0bc-bd782dda39a1-proxy-tls\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 11:55:58.454038 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.454008    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-0a5df-77865c759f-kmm24"]
May 26 11:55:58.454309 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.454297    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b280eea3-83f6-44fe-9ec8-3566464fb8ee" containerName="kube-rbac-proxy"
May 26 11:55:58.454372 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.454311    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b280eea3-83f6-44fe-9ec8-3566464fb8ee" containerName="kube-rbac-proxy"
May 26 11:55:58.454372 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.454322    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d6808ed-f916-452d-a0bc-bd782dda39a1" containerName="sequence-graph-fbdd2"
May 26 11:55:58.454372 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.454327    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d6808ed-f916-452d-a0bc-bd782dda39a1" containerName="sequence-graph-fbdd2"
May 26 11:55:58.454372 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.454340    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b280eea3-83f6-44fe-9ec8-3566464fb8ee" containerName="kserve-container"
May 26 11:55:58.454372 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.454346    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b280eea3-83f6-44fe-9ec8-3566464fb8ee" containerName="kserve-container"
May 26 11:55:58.454372 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.454358    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d1acaa3-e946-40d3-9937-ef994cf95b0a" containerName="ensemble-graph-5380a"
May 26 11:55:58.454372 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.454363    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d1acaa3-e946-40d3-9937-ef994cf95b0a" containerName="ensemble-graph-5380a"
May 26 11:55:58.454600 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.454401    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="b280eea3-83f6-44fe-9ec8-3566464fb8ee" containerName="kserve-container"
May 26 11:55:58.454600 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.454409    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="b280eea3-83f6-44fe-9ec8-3566464fb8ee" containerName="kube-rbac-proxy"
May 26 11:55:58.454600 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.454418    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="7d1acaa3-e946-40d3-9937-ef994cf95b0a" containerName="ensemble-graph-5380a"
May 26 11:55:58.454600 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.454424    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="7d6808ed-f916-452d-a0bc-bd782dda39a1" containerName="sequence-graph-fbdd2"
May 26 11:55:58.457439 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.457424    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-0a5df-77865c759f-kmm24"
May 26 11:55:58.461651 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.461632    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-0a5df-serving-cert\""
May 26 11:55:58.461744 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.461635    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-0a5df-kube-rbac-proxy-sar-config\""
May 26 11:55:58.467832 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.467812    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-0a5df-77865c759f-kmm24"]
May 26 11:55:58.500092 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.500054    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ddb2c1f-bbdf-48ee-a623-3791529de7f6-openshift-service-ca-bundle\") pod \"ensemble-graph-0a5df-77865c759f-kmm24\" (UID: \"9ddb2c1f-bbdf-48ee-a623-3791529de7f6\") " pod="kserve-ci-e2e-test/ensemble-graph-0a5df-77865c759f-kmm24"
May 26 11:55:58.500092 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.500091    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ddb2c1f-bbdf-48ee-a623-3791529de7f6-proxy-tls\") pod \"ensemble-graph-0a5df-77865c759f-kmm24\" (UID: \"9ddb2c1f-bbdf-48ee-a623-3791529de7f6\") " pod="kserve-ci-e2e-test/ensemble-graph-0a5df-77865c759f-kmm24"
May 26 11:55:58.601434 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.601397    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ddb2c1f-bbdf-48ee-a623-3791529de7f6-openshift-service-ca-bundle\") pod \"ensemble-graph-0a5df-77865c759f-kmm24\" (UID: \"9ddb2c1f-bbdf-48ee-a623-3791529de7f6\") " pod="kserve-ci-e2e-test/ensemble-graph-0a5df-77865c759f-kmm24"
May 26 11:55:58.601611 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.601449    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ddb2c1f-bbdf-48ee-a623-3791529de7f6-proxy-tls\") pod \"ensemble-graph-0a5df-77865c759f-kmm24\" (UID: \"9ddb2c1f-bbdf-48ee-a623-3791529de7f6\") " pod="kserve-ci-e2e-test/ensemble-graph-0a5df-77865c759f-kmm24"
May 26 11:55:58.601678 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:55:58.601624    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-0a5df-serving-cert: secret "ensemble-graph-0a5df-serving-cert" not found
May 26 11:55:58.601732 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:55:58.601720    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ddb2c1f-bbdf-48ee-a623-3791529de7f6-proxy-tls podName:9ddb2c1f-bbdf-48ee-a623-3791529de7f6 nodeName:}" failed. No retries permitted until 2026-05-26 11:55:59.101696283 +0000 UTC m=+1891.293138450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9ddb2c1f-bbdf-48ee-a623-3791529de7f6-proxy-tls") pod "ensemble-graph-0a5df-77865c759f-kmm24" (UID: "9ddb2c1f-bbdf-48ee-a623-3791529de7f6") : secret "ensemble-graph-0a5df-serving-cert" not found
May 26 11:55:58.602069 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.602048    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ddb2c1f-bbdf-48ee-a623-3791529de7f6-openshift-service-ca-bundle\") pod \"ensemble-graph-0a5df-77865c759f-kmm24\" (UID: \"9ddb2c1f-bbdf-48ee-a623-3791529de7f6\") " pod="kserve-ci-e2e-test/ensemble-graph-0a5df-77865c759f-kmm24"
May 26 11:55:58.920631 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.920593    2563 generic.go:358] "Generic (PLEG): container finished" podID="7d6808ed-f916-452d-a0bc-bd782dda39a1" containerID="aa07c366e8ec39284a71a8d96c7097b842ab7dd291d03c43bd96bf747fee6768" exitCode=0
May 26 11:55:58.920868 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.920675    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-fbdd2-58d5d54c6b-lps2t"
May 26 11:55:58.920868 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.920681    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-fbdd2-58d5d54c6b-lps2t" event={"ID":"7d6808ed-f916-452d-a0bc-bd782dda39a1","Type":"ContainerDied","Data":"aa07c366e8ec39284a71a8d96c7097b842ab7dd291d03c43bd96bf747fee6768"}
May 26 11:55:58.920868 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.920728    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-fbdd2-58d5d54c6b-lps2t" event={"ID":"7d6808ed-f916-452d-a0bc-bd782dda39a1","Type":"ContainerDied","Data":"849555cf44dd1bfe7e2083aafe0dcfa36f2ec446444783cd94ade4e0e355d14a"}
May 26 11:55:58.920868 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.920744    2563 scope.go:117] "RemoveContainer" containerID="aa07c366e8ec39284a71a8d96c7097b842ab7dd291d03c43bd96bf747fee6768"
May 26 11:55:58.928555 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.928534    2563 scope.go:117] "RemoveContainer" containerID="aa07c366e8ec39284a71a8d96c7097b842ab7dd291d03c43bd96bf747fee6768"
May 26 11:55:58.928862 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:55:58.928843    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa07c366e8ec39284a71a8d96c7097b842ab7dd291d03c43bd96bf747fee6768\": container with ID starting with aa07c366e8ec39284a71a8d96c7097b842ab7dd291d03c43bd96bf747fee6768 not found: ID does not exist" containerID="aa07c366e8ec39284a71a8d96c7097b842ab7dd291d03c43bd96bf747fee6768"
May 26 11:55:58.928919 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.928871    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa07c366e8ec39284a71a8d96c7097b842ab7dd291d03c43bd96bf747fee6768"} err="failed to get container status \"aa07c366e8ec39284a71a8d96c7097b842ab7dd291d03c43bd96bf747fee6768\": rpc error: code = NotFound desc = could not find container \"aa07c366e8ec39284a71a8d96c7097b842ab7dd291d03c43bd96bf747fee6768\": container with ID starting with aa07c366e8ec39284a71a8d96c7097b842ab7dd291d03c43bd96bf747fee6768 not found: ID does not exist"
May 26 11:55:58.939564 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.939540    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-fbdd2-58d5d54c6b-lps2t"]
May 26 11:55:58.944084 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:58.944062    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-fbdd2-58d5d54c6b-lps2t"]
May 26 11:55:59.105603 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:59.105572    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ddb2c1f-bbdf-48ee-a623-3791529de7f6-proxy-tls\") pod \"ensemble-graph-0a5df-77865c759f-kmm24\" (UID: \"9ddb2c1f-bbdf-48ee-a623-3791529de7f6\") " pod="kserve-ci-e2e-test/ensemble-graph-0a5df-77865c759f-kmm24"
May 26 11:55:59.108193 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:59.108171    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ddb2c1f-bbdf-48ee-a623-3791529de7f6-proxy-tls\") pod \"ensemble-graph-0a5df-77865c759f-kmm24\" (UID: \"9ddb2c1f-bbdf-48ee-a623-3791529de7f6\") " pod="kserve-ci-e2e-test/ensemble-graph-0a5df-77865c759f-kmm24"
May 26 11:55:59.367487 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:59.367454    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-0a5df-77865c759f-kmm24"
May 26 11:55:59.490755 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:59.490730    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-0a5df-77865c759f-kmm24"]
May 26 11:55:59.492994 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:55:59.492966    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ddb2c1f_bbdf_48ee_a623_3791529de7f6.slice/crio-958ee7ffbda05da71d3b6bc35c5b9a22cbfd23961011b146df193f3a32dda485 WatchSource:0}: Error finding container 958ee7ffbda05da71d3b6bc35c5b9a22cbfd23961011b146df193f3a32dda485: Status 404 returned error can't find the container with id 958ee7ffbda05da71d3b6bc35c5b9a22cbfd23961011b146df193f3a32dda485
May 26 11:55:59.924352 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:59.924313    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-0a5df-77865c759f-kmm24" event={"ID":"9ddb2c1f-bbdf-48ee-a623-3791529de7f6","Type":"ContainerStarted","Data":"0ac67c9e4d752947934d85a233a54e66b5b838f3c331deba6c26df11d07dc58c"}
May 26 11:55:59.924352 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:59.924353    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-0a5df-77865c759f-kmm24" event={"ID":"9ddb2c1f-bbdf-48ee-a623-3791529de7f6","Type":"ContainerStarted","Data":"958ee7ffbda05da71d3b6bc35c5b9a22cbfd23961011b146df193f3a32dda485"}
May 26 11:55:59.924637 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:59.924394    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-0a5df-77865c759f-kmm24"
May 26 11:55:59.941409 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:55:59.941363    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-0a5df-77865c759f-kmm24" podStartSLOduration=1.941349109 podStartE2EDuration="1.941349109s" podCreationTimestamp="2026-05-26 11:55:58 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:55:59.940123467 +0000 UTC m=+1892.131565655" watchObservedRunningTime="2026-05-26 11:55:59.941349109 +0000 UTC m=+1892.132791298"
May 26 11:56:00.429064 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:56:00.429027    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d6808ed-f916-452d-a0bc-bd782dda39a1" path="/var/lib/kubelet/pods/7d6808ed-f916-452d-a0bc-bd782dda39a1/volumes"
May 26 11:56:05.933438 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:56:05.933412    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-0a5df-77865c759f-kmm24"
May 26 11:56:06.840033 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:56:06.839987    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw" podUID="b21b40a6-9db5-4117-8ad1-b1f087ed9ecc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
May 26 11:56:08.051716 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:56:08.051686    2563 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d1acaa3_e946_40d3_9937_ef994cf95b0a.slice/crio-9930b50a433edbe43c5ebf583ea20d5f628b608c8d733711e7e312ae35f55f98\": RecentStats: unable to find data in memory cache]"
May 26 11:56:08.103460 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:56:08.103428    2563 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d1acaa3_e946_40d3_9937_ef994cf95b0a.slice/crio-9930b50a433edbe43c5ebf583ea20d5f628b608c8d733711e7e312ae35f55f98\": RecentStats: unable to find data in memory cache]"
May 26 11:56:16.840817 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:56:16.840758    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw"
May 26 11:56:18.138833 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:56:18.138795    2563 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d1acaa3_e946_40d3_9937_ef994cf95b0a.slice/crio-9930b50a433edbe43c5ebf583ea20d5f628b608c8d733711e7e312ae35f55f98\": RecentStats: unable to find data in memory cache]"
May 26 11:56:23.036488 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:56:23.036447    2563 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d1acaa3_e946_40d3_9937_ef994cf95b0a.slice/crio-9930b50a433edbe43c5ebf583ea20d5f628b608c8d733711e7e312ae35f55f98\": RecentStats: unable to find data in memory cache]"
May 26 11:56:28.146731 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:56:28.146697    2563 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d1acaa3_e946_40d3_9937_ef994cf95b0a.slice/crio-9930b50a433edbe43c5ebf583ea20d5f628b608c8d733711e7e312ae35f55f98\": RecentStats: unable to find data in memory cache]"
May 26 11:56:28.269344 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:56:28.269305    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-3c458-6bcbbc655c-q2t5j"]
May 26 11:56:28.272336 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:56:28.272320    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-3c458-6bcbbc655c-q2t5j"
May 26 11:56:28.276354 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:56:28.276332    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-3c458-serving-cert\""
May 26 11:56:28.276471 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:56:28.276336    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-3c458-kube-rbac-proxy-sar-config\""
May 26 11:56:28.283283 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:56:28.283249    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-3c458-6bcbbc655c-q2t5j"]
May 26 11:56:28.353628 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:56:28.353595    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bab8610-ad1e-4408-8df0-13d0cfbd7440-openshift-service-ca-bundle\") pod \"sequence-graph-3c458-6bcbbc655c-q2t5j\" (UID: \"3bab8610-ad1e-4408-8df0-13d0cfbd7440\") " pod="kserve-ci-e2e-test/sequence-graph-3c458-6bcbbc655c-q2t5j"
May 26 11:56:28.353803 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:56:28.353634    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bab8610-ad1e-4408-8df0-13d0cfbd7440-proxy-tls\") pod \"sequence-graph-3c458-6bcbbc655c-q2t5j\" (UID: \"3bab8610-ad1e-4408-8df0-13d0cfbd7440\") " pod="kserve-ci-e2e-test/sequence-graph-3c458-6bcbbc655c-q2t5j"
May 26 11:56:28.454258 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:56:28.454184    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bab8610-ad1e-4408-8df0-13d0cfbd7440-openshift-service-ca-bundle\") pod \"sequence-graph-3c458-6bcbbc655c-q2t5j\" (UID: \"3bab8610-ad1e-4408-8df0-13d0cfbd7440\") " pod="kserve-ci-e2e-test/sequence-graph-3c458-6bcbbc655c-q2t5j"
May 26 11:56:28.454258 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:56:28.454229    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bab8610-ad1e-4408-8df0-13d0cfbd7440-proxy-tls\") pod \"sequence-graph-3c458-6bcbbc655c-q2t5j\" (UID: \"3bab8610-ad1e-4408-8df0-13d0cfbd7440\") " pod="kserve-ci-e2e-test/sequence-graph-3c458-6bcbbc655c-q2t5j"
May 26 11:56:28.454787 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:56:28.454753    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bab8610-ad1e-4408-8df0-13d0cfbd7440-openshift-service-ca-bundle\") pod \"sequence-graph-3c458-6bcbbc655c-q2t5j\" (UID: \"3bab8610-ad1e-4408-8df0-13d0cfbd7440\") " pod="kserve-ci-e2e-test/sequence-graph-3c458-6bcbbc655c-q2t5j"
May 26 11:56:28.456883 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:56:28.456866    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-3c458-serving-cert\""
May 26 11:56:28.465436 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:56:28.465418    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-3c458-serving-cert: secret "sequence-graph-3c458-serving-cert" not found
May 26 11:56:28.465493 ip-10-0-129-97 kubenswrapper[2563]: E0526 11:56:28.465483    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bab8610-ad1e-4408-8df0-13d0cfbd7440-proxy-tls podName:3bab8610-ad1e-4408-8df0-13d0cfbd7440 nodeName:}" failed. No retries permitted until 2026-05-26 11:56:28.965467693 +0000 UTC m=+1921.156909858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3bab8610-ad1e-4408-8df0-13d0cfbd7440-proxy-tls") pod "sequence-graph-3c458-6bcbbc655c-q2t5j" (UID: "3bab8610-ad1e-4408-8df0-13d0cfbd7440") : secret "sequence-graph-3c458-serving-cert" not found
May 26 11:56:29.059659 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:56:29.059613    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bab8610-ad1e-4408-8df0-13d0cfbd7440-proxy-tls\") pod \"sequence-graph-3c458-6bcbbc655c-q2t5j\" (UID: \"3bab8610-ad1e-4408-8df0-13d0cfbd7440\") " pod="kserve-ci-e2e-test/sequence-graph-3c458-6bcbbc655c-q2t5j"
May 26 11:56:29.062273 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:56:29.062251    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bab8610-ad1e-4408-8df0-13d0cfbd7440-proxy-tls\") pod \"sequence-graph-3c458-6bcbbc655c-q2t5j\" (UID: \"3bab8610-ad1e-4408-8df0-13d0cfbd7440\") " pod="kserve-ci-e2e-test/sequence-graph-3c458-6bcbbc655c-q2t5j"
May 26 11:56:29.182663 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:56:29.182610    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-3c458-6bcbbc655c-q2t5j"
May 26 11:56:29.311657 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:56:29.311586    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-3c458-6bcbbc655c-q2t5j"]
May 26 11:56:29.314667 ip-10-0-129-97 kubenswrapper[2563]: W0526 11:56:29.314641    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bab8610_ad1e_4408_8df0_13d0cfbd7440.slice/crio-42c426a21790fcf0b33ecfbe1c239286455143c701b0e4aa01f0732962b7d476 WatchSource:0}: Error finding container 42c426a21790fcf0b33ecfbe1c239286455143c701b0e4aa01f0732962b7d476: Status 404 returned error can't find the container with id 42c426a21790fcf0b33ecfbe1c239286455143c701b0e4aa01f0732962b7d476
May 26 11:56:30.008046 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:56:30.008010    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-3c458-6bcbbc655c-q2t5j" event={"ID":"3bab8610-ad1e-4408-8df0-13d0cfbd7440","Type":"ContainerStarted","Data":"179e5a4dd0f28b92455afb01f19942fb961f3fd0567d847b4b0e67a6df1554ad"}
May 26 11:56:30.008046 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:56:30.008048    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-3c458-6bcbbc655c-q2t5j" event={"ID":"3bab8610-ad1e-4408-8df0-13d0cfbd7440","Type":"ContainerStarted","Data":"42c426a21790fcf0b33ecfbe1c239286455143c701b0e4aa01f0732962b7d476"}
May 26 11:56:30.008268 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:56:30.008073    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-3c458-6bcbbc655c-q2t5j"
May 26 11:56:30.026028 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:56:30.025983    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-3c458-6bcbbc655c-q2t5j" podStartSLOduration=2.025969785 podStartE2EDuration="2.025969785s" podCreationTimestamp="2026-05-26 11:56:28 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:56:30.024273805 +0000 UTC m=+1922.215715992" watchObservedRunningTime="2026-05-26 11:56:30.025969785 +0000 UTC m=+1922.217411971"
May 26 11:56:36.017300 ip-10-0-129-97 kubenswrapper[2563]: I0526 11:56:36.017267    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-3c458-6bcbbc655c-q2t5j"
May 26 12:04:13.160935 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:13.160902    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-0a5df-77865c759f-kmm24"]
May 26 12:04:13.163488 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:13.161129    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-0a5df-77865c759f-kmm24" podUID="9ddb2c1f-bbdf-48ee-a623-3791529de7f6" containerName="ensemble-graph-0a5df" containerID="cri-o://0ac67c9e4d752947934d85a233a54e66b5b838f3c331deba6c26df11d07dc58c" gracePeriod=30
May 26 12:04:13.401966 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:13.401936    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph"]
May 26 12:04:13.402233 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:13.402211    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph" podUID="68b2c4c5-b1c4-4441-b57d-8dac155f8799" containerName="kserve-container" containerID="cri-o://3a9ea443cad48d055af0bdff5970081b9eea10eda5e2e7d6388f6ebd23e61a58" gracePeriod=30
May 26 12:04:13.402362 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:13.402308    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph" podUID="68b2c4c5-b1c4-4441-b57d-8dac155f8799" containerName="kube-rbac-proxy" containerID="cri-o://7fc5ad4ec4650dc3e67731f970b3c8196bd54392fda3d3e4ed86cc4f1b7e1df1" gracePeriod=30
May 26 12:04:13.491945 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:13.491909    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m"]
May 26 12:04:13.495244 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:13.495227    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m"
May 26 12:04:13.497872 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:13.497844    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-81833-predictor-serving-cert\""
May 26 12:04:13.497995 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:13.497941    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-81833-kube-rbac-proxy-sar-config\""
May 26 12:04:13.509651 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:13.509625    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m"]
May 26 12:04:13.511309 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:13.511282    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-81833-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0462e858-a74c-468c-a60d-71276b56c110-error-404-isvc-81833-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-81833-predictor-6db55d4ff5-xfl9m\" (UID: \"0462e858-a74c-468c-a60d-71276b56c110\") " pod="kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m"
May 26 12:04:13.511424 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:13.511325    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m56vq\" (UniqueName: \"kubernetes.io/projected/0462e858-a74c-468c-a60d-71276b56c110-kube-api-access-m56vq\") pod \"error-404-isvc-81833-predictor-6db55d4ff5-xfl9m\" (UID: \"0462e858-a74c-468c-a60d-71276b56c110\") " pod="kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m"
May 26 12:04:13.511424 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:13.511391    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0462e858-a74c-468c-a60d-71276b56c110-proxy-tls\") pod \"error-404-isvc-81833-predictor-6db55d4ff5-xfl9m\" (UID: \"0462e858-a74c-468c-a60d-71276b56c110\") " pod="kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m"
May 26 12:04:13.611879 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:13.611838    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-81833-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0462e858-a74c-468c-a60d-71276b56c110-error-404-isvc-81833-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-81833-predictor-6db55d4ff5-xfl9m\" (UID: \"0462e858-a74c-468c-a60d-71276b56c110\") " pod="kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m"
May 26 12:04:13.611879 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:13.611875    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-m56vq\" (UniqueName: \"kubernetes.io/projected/0462e858-a74c-468c-a60d-71276b56c110-kube-api-access-m56vq\") pod \"error-404-isvc-81833-predictor-6db55d4ff5-xfl9m\" (UID: \"0462e858-a74c-468c-a60d-71276b56c110\") " pod="kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m"
May 26 12:04:13.612112 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:13.612008    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0462e858-a74c-468c-a60d-71276b56c110-proxy-tls\") pod \"error-404-isvc-81833-predictor-6db55d4ff5-xfl9m\" (UID: \"0462e858-a74c-468c-a60d-71276b56c110\") " pod="kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m"
May 26 12:04:13.612175 ip-10-0-129-97 kubenswrapper[2563]: E0526 12:04:13.612153    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-81833-predictor-serving-cert: secret "error-404-isvc-81833-predictor-serving-cert" not found
May 26 12:04:13.612249 ip-10-0-129-97 kubenswrapper[2563]: E0526 12:04:13.612239    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0462e858-a74c-468c-a60d-71276b56c110-proxy-tls podName:0462e858-a74c-468c-a60d-71276b56c110 nodeName:}" failed. No retries permitted until 2026-05-26 12:04:14.112218258 +0000 UTC m=+2386.303660424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0462e858-a74c-468c-a60d-71276b56c110-proxy-tls") pod "error-404-isvc-81833-predictor-6db55d4ff5-xfl9m" (UID: "0462e858-a74c-468c-a60d-71276b56c110") : secret "error-404-isvc-81833-predictor-serving-cert" not found
May 26 12:04:13.612520 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:13.612502    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-81833-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0462e858-a74c-468c-a60d-71276b56c110-error-404-isvc-81833-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-81833-predictor-6db55d4ff5-xfl9m\" (UID: \"0462e858-a74c-468c-a60d-71276b56c110\") " pod="kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m"
May 26 12:04:13.621139 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:13.621109    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m56vq\" (UniqueName: \"kubernetes.io/projected/0462e858-a74c-468c-a60d-71276b56c110-kube-api-access-m56vq\") pod \"error-404-isvc-81833-predictor-6db55d4ff5-xfl9m\" (UID: \"0462e858-a74c-468c-a60d-71276b56c110\") " pod="kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m"
May 26 12:04:14.115196 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:14.115157    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0462e858-a74c-468c-a60d-71276b56c110-proxy-tls\") pod \"error-404-isvc-81833-predictor-6db55d4ff5-xfl9m\" (UID: \"0462e858-a74c-468c-a60d-71276b56c110\") " pod="kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m"
May 26 12:04:14.117804 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:14.117747    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0462e858-a74c-468c-a60d-71276b56c110-proxy-tls\") pod \"error-404-isvc-81833-predictor-6db55d4ff5-xfl9m\" (UID: \"0462e858-a74c-468c-a60d-71276b56c110\") " pod="kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m"
May 26 12:04:14.333464 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:14.333430    2563 generic.go:358] "Generic (PLEG): container finished" podID="68b2c4c5-b1c4-4441-b57d-8dac155f8799" containerID="7fc5ad4ec4650dc3e67731f970b3c8196bd54392fda3d3e4ed86cc4f1b7e1df1" exitCode=2
May 26 12:04:14.333838 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:14.333496    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph" event={"ID":"68b2c4c5-b1c4-4441-b57d-8dac155f8799","Type":"ContainerDied","Data":"7fc5ad4ec4650dc3e67731f970b3c8196bd54392fda3d3e4ed86cc4f1b7e1df1"}
May 26 12:04:14.405852 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:14.405755    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m"
May 26 12:04:14.543571 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:14.543498    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m"]
May 26 12:04:14.547450 ip-10-0-129-97 kubenswrapper[2563]: W0526 12:04:14.547418    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0462e858_a74c_468c_a60d_71276b56c110.slice/crio-5d20c009e6a6bd016125a6f60f1c1b0d71e5febc306e9c72c7a5ca9c96161726 WatchSource:0}: Error finding container 5d20c009e6a6bd016125a6f60f1c1b0d71e5febc306e9c72c7a5ca9c96161726: Status 404 returned error can't find the container with id 5d20c009e6a6bd016125a6f60f1c1b0d71e5febc306e9c72c7a5ca9c96161726
May 26 12:04:14.549203 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:14.549173    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 12:04:15.337752 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:15.337712    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m" event={"ID":"0462e858-a74c-468c-a60d-71276b56c110","Type":"ContainerStarted","Data":"039e9cadb5687d1ffac7859b73fd24d3162217d1c7d0504aeca3ce434848544c"}
May 26 12:04:15.337752 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:15.337752    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m" event={"ID":"0462e858-a74c-468c-a60d-71276b56c110","Type":"ContainerStarted","Data":"35a0aad95e8941c6d1557837868edb4ba7dfca04761a582c2f089f7e160d7a7d"}
May 26 12:04:15.338225 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:15.337765    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m" event={"ID":"0462e858-a74c-468c-a60d-71276b56c110","Type":"ContainerStarted","Data":"5d20c009e6a6bd016125a6f60f1c1b0d71e5febc306e9c72c7a5ca9c96161726"}
May 26 12:04:15.338225 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:15.337915    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m"
May 26 12:04:15.338225 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:15.338056    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m"
May 26 12:04:15.339145 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:15.339120    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m" podUID="0462e858-a74c-468c-a60d-71276b56c110" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 26 12:04:15.734031 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:15.733942    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph" podUID="68b2c4c5-b1c4-4441-b57d-8dac155f8799" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
May 26 12:04:15.738398 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:15.738366    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph" podUID="68b2c4c5-b1c4-4441-b57d-8dac155f8799" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 26 12:04:15.932679 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:15.932641    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-0a5df-77865c759f-kmm24" podUID="9ddb2c1f-bbdf-48ee-a623-3791529de7f6" containerName="ensemble-graph-0a5df" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 12:04:16.341477 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:16.341438    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m" podUID="0462e858-a74c-468c-a60d-71276b56c110" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 26 12:04:16.550527 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:16.550502    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph"
May 26 12:04:16.572471 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:16.572424    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m" podStartSLOduration=3.572403874 podStartE2EDuration="3.572403874s" podCreationTimestamp="2026-05-26 12:04:13 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:04:15.357873758 +0000 UTC m=+2387.549315946" watchObservedRunningTime="2026-05-26 12:04:16.572403874 +0000 UTC m=+2388.763846083"
May 26 12:04:16.635150 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:16.635049    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvjxr\" (UniqueName: \"kubernetes.io/projected/68b2c4c5-b1c4-4441-b57d-8dac155f8799-kube-api-access-pvjxr\") pod \"68b2c4c5-b1c4-4441-b57d-8dac155f8799\" (UID: \"68b2c4c5-b1c4-4441-b57d-8dac155f8799\") "
May 26 12:04:16.635150 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:16.635109    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-0a5df-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68b2c4c5-b1c4-4441-b57d-8dac155f8799-error-404-isvc-0a5df-kube-rbac-proxy-sar-config\") pod \"68b2c4c5-b1c4-4441-b57d-8dac155f8799\" (UID: \"68b2c4c5-b1c4-4441-b57d-8dac155f8799\") "
May 26 12:04:16.635150 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:16.635129    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68b2c4c5-b1c4-4441-b57d-8dac155f8799-proxy-tls\") pod \"68b2c4c5-b1c4-4441-b57d-8dac155f8799\" (UID: \"68b2c4c5-b1c4-4441-b57d-8dac155f8799\") "
May 26 12:04:16.635527 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:16.635498    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68b2c4c5-b1c4-4441-b57d-8dac155f8799-error-404-isvc-0a5df-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-0a5df-kube-rbac-proxy-sar-config") pod "68b2c4c5-b1c4-4441-b57d-8dac155f8799" (UID: "68b2c4c5-b1c4-4441-b57d-8dac155f8799"). InnerVolumeSpecName "error-404-isvc-0a5df-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:04:16.637322 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:16.637287    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b2c4c5-b1c4-4441-b57d-8dac155f8799-kube-api-access-pvjxr" (OuterVolumeSpecName: "kube-api-access-pvjxr") pod "68b2c4c5-b1c4-4441-b57d-8dac155f8799" (UID: "68b2c4c5-b1c4-4441-b57d-8dac155f8799"). InnerVolumeSpecName "kube-api-access-pvjxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:04:16.637322 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:16.637304    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68b2c4c5-b1c4-4441-b57d-8dac155f8799-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "68b2c4c5-b1c4-4441-b57d-8dac155f8799" (UID: "68b2c4c5-b1c4-4441-b57d-8dac155f8799"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:04:16.735946 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:16.735906    2563 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-0a5df-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68b2c4c5-b1c4-4441-b57d-8dac155f8799-error-404-isvc-0a5df-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 12:04:16.735946 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:16.735938    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68b2c4c5-b1c4-4441-b57d-8dac155f8799-proxy-tls\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 12:04:16.735946 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:16.735951    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvjxr\" (UniqueName: \"kubernetes.io/projected/68b2c4c5-b1c4-4441-b57d-8dac155f8799-kube-api-access-pvjxr\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 12:04:17.345406 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:17.345369    2563 generic.go:358] "Generic (PLEG): container finished" podID="68b2c4c5-b1c4-4441-b57d-8dac155f8799" containerID="3a9ea443cad48d055af0bdff5970081b9eea10eda5e2e7d6388f6ebd23e61a58" exitCode=0
May 26 12:04:17.345848 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:17.345431    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph" event={"ID":"68b2c4c5-b1c4-4441-b57d-8dac155f8799","Type":"ContainerDied","Data":"3a9ea443cad48d055af0bdff5970081b9eea10eda5e2e7d6388f6ebd23e61a58"}
May 26 12:04:17.345848 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:17.345449    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph"
May 26 12:04:17.345848 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:17.345465    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph" event={"ID":"68b2c4c5-b1c4-4441-b57d-8dac155f8799","Type":"ContainerDied","Data":"efd57671adf152cb4d8a67eaafb84e03383174b053a65a4160fe54c56de424a3"}
May 26 12:04:17.345848 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:17.345486    2563 scope.go:117] "RemoveContainer" containerID="7fc5ad4ec4650dc3e67731f970b3c8196bd54392fda3d3e4ed86cc4f1b7e1df1"
May 26 12:04:17.355201 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:17.355178    2563 scope.go:117] "RemoveContainer" containerID="3a9ea443cad48d055af0bdff5970081b9eea10eda5e2e7d6388f6ebd23e61a58"
May 26 12:04:17.363291 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:17.363264    2563 scope.go:117] "RemoveContainer" containerID="7fc5ad4ec4650dc3e67731f970b3c8196bd54392fda3d3e4ed86cc4f1b7e1df1"
May 26 12:04:17.363596 ip-10-0-129-97 kubenswrapper[2563]: E0526 12:04:17.363577    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fc5ad4ec4650dc3e67731f970b3c8196bd54392fda3d3e4ed86cc4f1b7e1df1\": container with ID starting with 7fc5ad4ec4650dc3e67731f970b3c8196bd54392fda3d3e4ed86cc4f1b7e1df1 not found: ID does not exist" containerID="7fc5ad4ec4650dc3e67731f970b3c8196bd54392fda3d3e4ed86cc4f1b7e1df1"
May 26 12:04:17.363659 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:17.363605    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fc5ad4ec4650dc3e67731f970b3c8196bd54392fda3d3e4ed86cc4f1b7e1df1"} err="failed to get container status \"7fc5ad4ec4650dc3e67731f970b3c8196bd54392fda3d3e4ed86cc4f1b7e1df1\": rpc error: code = NotFound desc = could not find container \"7fc5ad4ec4650dc3e67731f970b3c8196bd54392fda3d3e4ed86cc4f1b7e1df1\": container with ID starting with 7fc5ad4ec4650dc3e67731f970b3c8196bd54392fda3d3e4ed86cc4f1b7e1df1 not found: ID does not exist"
May 26 12:04:17.363659 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:17.363626    2563 scope.go:117] "RemoveContainer" containerID="3a9ea443cad48d055af0bdff5970081b9eea10eda5e2e7d6388f6ebd23e61a58"
May 26 12:04:17.363910 ip-10-0-129-97 kubenswrapper[2563]: E0526 12:04:17.363895    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a9ea443cad48d055af0bdff5970081b9eea10eda5e2e7d6388f6ebd23e61a58\": container with ID starting with 3a9ea443cad48d055af0bdff5970081b9eea10eda5e2e7d6388f6ebd23e61a58 not found: ID does not exist" containerID="3a9ea443cad48d055af0bdff5970081b9eea10eda5e2e7d6388f6ebd23e61a58"
May 26 12:04:17.363957 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:17.363916    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a9ea443cad48d055af0bdff5970081b9eea10eda5e2e7d6388f6ebd23e61a58"} err="failed to get container status \"3a9ea443cad48d055af0bdff5970081b9eea10eda5e2e7d6388f6ebd23e61a58\": rpc error: code = NotFound desc = could not find container \"3a9ea443cad48d055af0bdff5970081b9eea10eda5e2e7d6388f6ebd23e61a58\": container with ID starting with 3a9ea443cad48d055af0bdff5970081b9eea10eda5e2e7d6388f6ebd23e61a58 not found: ID does not exist"
May 26 12:04:17.368879 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:17.368839    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph"]
May 26 12:04:17.376360 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:17.376331    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0a5df-predictor-c47858789-v9hph"]
May 26 12:04:18.428450 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:18.428413    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68b2c4c5-b1c4-4441-b57d-8dac155f8799" path="/var/lib/kubelet/pods/68b2c4c5-b1c4-4441-b57d-8dac155f8799/volumes"
May 26 12:04:20.932287 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:20.932247    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-0a5df-77865c759f-kmm24" podUID="9ddb2c1f-bbdf-48ee-a623-3791529de7f6" containerName="ensemble-graph-0a5df" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 12:04:21.345837 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:21.345811    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m"
May 26 12:04:21.346441 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:21.346399    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m" podUID="0462e858-a74c-468c-a60d-71276b56c110" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 26 12:04:25.932197 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:25.932151    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-0a5df-77865c759f-kmm24" podUID="9ddb2c1f-bbdf-48ee-a623-3791529de7f6" containerName="ensemble-graph-0a5df" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 12:04:25.932654 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:25.932278    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-0a5df-77865c759f-kmm24"
May 26 12:04:30.932328 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:30.932283    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-0a5df-77865c759f-kmm24" podUID="9ddb2c1f-bbdf-48ee-a623-3791529de7f6" containerName="ensemble-graph-0a5df" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 12:04:31.347236 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:31.347198    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m" podUID="0462e858-a74c-468c-a60d-71276b56c110" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 26 12:04:35.932578 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:35.932535    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-0a5df-77865c759f-kmm24" podUID="9ddb2c1f-bbdf-48ee-a623-3791529de7f6" containerName="ensemble-graph-0a5df" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 12:04:40.932451 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:40.932404    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-0a5df-77865c759f-kmm24" podUID="9ddb2c1f-bbdf-48ee-a623-3791529de7f6" containerName="ensemble-graph-0a5df" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 12:04:41.346707 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:41.346671    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m" podUID="0462e858-a74c-468c-a60d-71276b56c110" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 26 12:04:42.787278 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:42.787246    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-3c458-6bcbbc655c-q2t5j"]
May 26 12:04:42.787663 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:42.787472    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-3c458-6bcbbc655c-q2t5j" podUID="3bab8610-ad1e-4408-8df0-13d0cfbd7440" containerName="sequence-graph-3c458" containerID="cri-o://179e5a4dd0f28b92455afb01f19942fb961f3fd0567d847b4b0e67a6df1554ad" gracePeriod=30
May 26 12:04:42.859119 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:42.859088    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw"]
May 26 12:04:42.859404 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:42.859381    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw" podUID="b21b40a6-9db5-4117-8ad1-b1f087ed9ecc" containerName="kserve-container" containerID="cri-o://a91dcc99d1b5654c131ed7a10c96c2a268d4f535efa6f42102e03288aa680604" gracePeriod=30
May 26 12:04:42.859480 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:42.859401    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw" podUID="b21b40a6-9db5-4117-8ad1-b1f087ed9ecc" containerName="kube-rbac-proxy" containerID="cri-o://cc3afe0ac12c582f628e9a39ef8ac73d2e0e8678838d194d46ccf067e17d781e" gracePeriod=30
May 26 12:04:43.051201 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.051119    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9"]
May 26 12:04:43.051443 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.051430    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68b2c4c5-b1c4-4441-b57d-8dac155f8799" containerName="kube-rbac-proxy"
May 26 12:04:43.051485 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.051444    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="68b2c4c5-b1c4-4441-b57d-8dac155f8799" containerName="kube-rbac-proxy"
May 26 12:04:43.051485 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.051463    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68b2c4c5-b1c4-4441-b57d-8dac155f8799" containerName="kserve-container"
May 26 12:04:43.051485 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.051470    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="68b2c4c5-b1c4-4441-b57d-8dac155f8799" containerName="kserve-container"
May 26 12:04:43.051579 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.051518    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="68b2c4c5-b1c4-4441-b57d-8dac155f8799" containerName="kserve-container"
May 26 12:04:43.051579 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.051530    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="68b2c4c5-b1c4-4441-b57d-8dac155f8799" containerName="kube-rbac-proxy"
May 26 12:04:43.054320 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.054304    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9"
May 26 12:04:43.056886 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.056867    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-152bc-predictor-serving-cert\""
May 26 12:04:43.056987 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.056913    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-152bc-kube-rbac-proxy-sar-config\""
May 26 12:04:43.065359 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.065334    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9"]
May 26 12:04:43.158710 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.158675    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f7d4ea2-96ee-47b8-87f0-2cdd2e684186-proxy-tls\") pod \"error-404-isvc-152bc-predictor-6b9cb766b7-4krt9\" (UID: \"7f7d4ea2-96ee-47b8-87f0-2cdd2e684186\") " pod="kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9"
May 26 12:04:43.158891 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.158716    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-152bc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f7d4ea2-96ee-47b8-87f0-2cdd2e684186-error-404-isvc-152bc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-152bc-predictor-6b9cb766b7-4krt9\" (UID: \"7f7d4ea2-96ee-47b8-87f0-2cdd2e684186\") " pod="kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9"
May 26 12:04:43.158891 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.158738    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bczbr\" (UniqueName: \"kubernetes.io/projected/7f7d4ea2-96ee-47b8-87f0-2cdd2e684186-kube-api-access-bczbr\") pod \"error-404-isvc-152bc-predictor-6b9cb766b7-4krt9\" (UID: \"7f7d4ea2-96ee-47b8-87f0-2cdd2e684186\") " pod="kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9"
May 26 12:04:43.259390 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.259357    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f7d4ea2-96ee-47b8-87f0-2cdd2e684186-proxy-tls\") pod \"error-404-isvc-152bc-predictor-6b9cb766b7-4krt9\" (UID: \"7f7d4ea2-96ee-47b8-87f0-2cdd2e684186\") " pod="kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9"
May 26 12:04:43.259535 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.259397    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-152bc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f7d4ea2-96ee-47b8-87f0-2cdd2e684186-error-404-isvc-152bc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-152bc-predictor-6b9cb766b7-4krt9\" (UID: \"7f7d4ea2-96ee-47b8-87f0-2cdd2e684186\") " pod="kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9"
May 26 12:04:43.259535 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.259416    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bczbr\" (UniqueName: \"kubernetes.io/projected/7f7d4ea2-96ee-47b8-87f0-2cdd2e684186-kube-api-access-bczbr\") pod \"error-404-isvc-152bc-predictor-6b9cb766b7-4krt9\" (UID: \"7f7d4ea2-96ee-47b8-87f0-2cdd2e684186\") " pod="kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9"
May 26 12:04:43.260083 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.260062    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-152bc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f7d4ea2-96ee-47b8-87f0-2cdd2e684186-error-404-isvc-152bc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-152bc-predictor-6b9cb766b7-4krt9\" (UID: \"7f7d4ea2-96ee-47b8-87f0-2cdd2e684186\") " pod="kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9"
May 26 12:04:43.262040 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.262018    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f7d4ea2-96ee-47b8-87f0-2cdd2e684186-proxy-tls\") pod \"error-404-isvc-152bc-predictor-6b9cb766b7-4krt9\" (UID: \"7f7d4ea2-96ee-47b8-87f0-2cdd2e684186\") " pod="kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9"
May 26 12:04:43.268164 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.268136    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bczbr\" (UniqueName: \"kubernetes.io/projected/7f7d4ea2-96ee-47b8-87f0-2cdd2e684186-kube-api-access-bczbr\") pod \"error-404-isvc-152bc-predictor-6b9cb766b7-4krt9\" (UID: \"7f7d4ea2-96ee-47b8-87f0-2cdd2e684186\") " pod="kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9"
May 26 12:04:43.314732 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.314710    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-0a5df-77865c759f-kmm24"
May 26 12:04:43.360380 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.360349    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ddb2c1f-bbdf-48ee-a623-3791529de7f6-proxy-tls\") pod \"9ddb2c1f-bbdf-48ee-a623-3791529de7f6\" (UID: \"9ddb2c1f-bbdf-48ee-a623-3791529de7f6\") "
May 26 12:04:43.360517 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.360397    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ddb2c1f-bbdf-48ee-a623-3791529de7f6-openshift-service-ca-bundle\") pod \"9ddb2c1f-bbdf-48ee-a623-3791529de7f6\" (UID: \"9ddb2c1f-bbdf-48ee-a623-3791529de7f6\") "
May 26 12:04:43.360822 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.360769    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ddb2c1f-bbdf-48ee-a623-3791529de7f6-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "9ddb2c1f-bbdf-48ee-a623-3791529de7f6" (UID: "9ddb2c1f-bbdf-48ee-a623-3791529de7f6"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:04:43.362550 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.362522    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ddb2c1f-bbdf-48ee-a623-3791529de7f6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9ddb2c1f-bbdf-48ee-a623-3791529de7f6" (UID: "9ddb2c1f-bbdf-48ee-a623-3791529de7f6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:04:43.364736 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.364715    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9"
May 26 12:04:43.426197 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.426159    2563 generic.go:358] "Generic (PLEG): container finished" podID="9ddb2c1f-bbdf-48ee-a623-3791529de7f6" containerID="0ac67c9e4d752947934d85a233a54e66b5b838f3c331deba6c26df11d07dc58c" exitCode=0
May 26 12:04:43.426330 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.426198    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-0a5df-77865c759f-kmm24" event={"ID":"9ddb2c1f-bbdf-48ee-a623-3791529de7f6","Type":"ContainerDied","Data":"0ac67c9e4d752947934d85a233a54e66b5b838f3c331deba6c26df11d07dc58c"}
May 26 12:04:43.426330 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.426252    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-0a5df-77865c759f-kmm24" event={"ID":"9ddb2c1f-bbdf-48ee-a623-3791529de7f6","Type":"ContainerDied","Data":"958ee7ffbda05da71d3b6bc35c5b9a22cbfd23961011b146df193f3a32dda485"}
May 26 12:04:43.426330 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.426258    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-0a5df-77865c759f-kmm24"
May 26 12:04:43.426330 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.426273    2563 scope.go:117] "RemoveContainer" containerID="0ac67c9e4d752947934d85a233a54e66b5b838f3c331deba6c26df11d07dc58c"
May 26 12:04:43.428500 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.428473    2563 generic.go:358] "Generic (PLEG): container finished" podID="b21b40a6-9db5-4117-8ad1-b1f087ed9ecc" containerID="cc3afe0ac12c582f628e9a39ef8ac73d2e0e8678838d194d46ccf067e17d781e" exitCode=2
May 26 12:04:43.428628 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.428549    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw" event={"ID":"b21b40a6-9db5-4117-8ad1-b1f087ed9ecc","Type":"ContainerDied","Data":"cc3afe0ac12c582f628e9a39ef8ac73d2e0e8678838d194d46ccf067e17d781e"}
May 26 12:04:43.434740 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.434722    2563 scope.go:117] "RemoveContainer" containerID="0ac67c9e4d752947934d85a233a54e66b5b838f3c331deba6c26df11d07dc58c"
May 26 12:04:43.435011 ip-10-0-129-97 kubenswrapper[2563]: E0526 12:04:43.434992    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ac67c9e4d752947934d85a233a54e66b5b838f3c331deba6c26df11d07dc58c\": container with ID starting with 0ac67c9e4d752947934d85a233a54e66b5b838f3c331deba6c26df11d07dc58c not found: ID does not exist" containerID="0ac67c9e4d752947934d85a233a54e66b5b838f3c331deba6c26df11d07dc58c"
May 26 12:04:43.435048 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.435020    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ac67c9e4d752947934d85a233a54e66b5b838f3c331deba6c26df11d07dc58c"} err="failed to get container status \"0ac67c9e4d752947934d85a233a54e66b5b838f3c331deba6c26df11d07dc58c\": rpc error: code = NotFound desc = could not find container \"0ac67c9e4d752947934d85a233a54e66b5b838f3c331deba6c26df11d07dc58c\": container with ID starting with 0ac67c9e4d752947934d85a233a54e66b5b838f3c331deba6c26df11d07dc58c not found: ID does not exist"
May 26 12:04:43.451329 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.451298    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-0a5df-77865c759f-kmm24"]
May 26 12:04:43.454948 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.454921    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-0a5df-77865c759f-kmm24"]
May 26 12:04:43.461323 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.461300    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ddb2c1f-bbdf-48ee-a623-3791529de7f6-proxy-tls\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 12:04:43.461323 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.461324    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ddb2c1f-bbdf-48ee-a623-3791529de7f6-openshift-service-ca-bundle\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 12:04:43.513406 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:43.513367    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9"]
May 26 12:04:43.518362 ip-10-0-129-97 kubenswrapper[2563]: W0526 12:04:43.518330    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f7d4ea2_96ee_47b8_87f0_2cdd2e684186.slice/crio-64cc1bd465e693b966791b22e72fb22c407f93de4d6ffc7e3b7daa7e61afa92e WatchSource:0}: Error finding container 64cc1bd465e693b966791b22e72fb22c407f93de4d6ffc7e3b7daa7e61afa92e: Status 404 returned error can't find the container with id 64cc1bd465e693b966791b22e72fb22c407f93de4d6ffc7e3b7daa7e61afa92e
May 26 12:04:44.428212 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:44.428169    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ddb2c1f-bbdf-48ee-a623-3791529de7f6" path="/var/lib/kubelet/pods/9ddb2c1f-bbdf-48ee-a623-3791529de7f6/volumes"
May 26 12:04:44.433402 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:44.433370    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9" event={"ID":"7f7d4ea2-96ee-47b8-87f0-2cdd2e684186","Type":"ContainerStarted","Data":"9f3fb060c32a3e005b80e439bf2d32c0a7230d21788e9af389abb52e0522a783"}
May 26 12:04:44.433542 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:44.433410    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9" event={"ID":"7f7d4ea2-96ee-47b8-87f0-2cdd2e684186","Type":"ContainerStarted","Data":"e6a8ef2a98fde879086fae48143fd419e9678fc7c677973b8f22b4190b930611"}
May 26 12:04:44.433542 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:44.433425    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9" event={"ID":"7f7d4ea2-96ee-47b8-87f0-2cdd2e684186","Type":"ContainerStarted","Data":"64cc1bd465e693b966791b22e72fb22c407f93de4d6ffc7e3b7daa7e61afa92e"}
May 26 12:04:44.433542 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:44.433520    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9"
May 26 12:04:44.451567 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:44.451524    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9" podStartSLOduration=1.4515114009999999 podStartE2EDuration="1.451511401s" podCreationTimestamp="2026-05-26 12:04:43 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:04:44.450741501 +0000 UTC m=+2416.642183688" watchObservedRunningTime="2026-05-26 12:04:44.451511401 +0000 UTC m=+2416.642953638"
May 26 12:04:45.438027 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:45.437996    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9"
May 26 12:04:45.439276 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:45.439247    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9" podUID="7f7d4ea2-96ee-47b8-87f0-2cdd2e684186" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 26 12:04:46.015228 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:46.015190    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-3c458-6bcbbc655c-q2t5j" podUID="3bab8610-ad1e-4408-8df0-13d0cfbd7440" containerName="sequence-graph-3c458" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 12:04:46.317751 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:46.317725    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw"
May 26 12:04:46.388514 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:46.388475    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-3c458-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b21b40a6-9db5-4117-8ad1-b1f087ed9ecc-error-404-isvc-3c458-kube-rbac-proxy-sar-config\") pod \"b21b40a6-9db5-4117-8ad1-b1f087ed9ecc\" (UID: \"b21b40a6-9db5-4117-8ad1-b1f087ed9ecc\") "
May 26 12:04:46.388514 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:46.388520    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b21b40a6-9db5-4117-8ad1-b1f087ed9ecc-proxy-tls\") pod \"b21b40a6-9db5-4117-8ad1-b1f087ed9ecc\" (UID: \"b21b40a6-9db5-4117-8ad1-b1f087ed9ecc\") "
May 26 12:04:46.388804 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:46.388564    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtlkj\" (UniqueName: \"kubernetes.io/projected/b21b40a6-9db5-4117-8ad1-b1f087ed9ecc-kube-api-access-rtlkj\") pod \"b21b40a6-9db5-4117-8ad1-b1f087ed9ecc\" (UID: \"b21b40a6-9db5-4117-8ad1-b1f087ed9ecc\") "
May 26 12:04:46.388985 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:46.388956    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b21b40a6-9db5-4117-8ad1-b1f087ed9ecc-error-404-isvc-3c458-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-3c458-kube-rbac-proxy-sar-config") pod "b21b40a6-9db5-4117-8ad1-b1f087ed9ecc" (UID: "b21b40a6-9db5-4117-8ad1-b1f087ed9ecc"). InnerVolumeSpecName "error-404-isvc-3c458-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:04:46.390955 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:46.390926    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b21b40a6-9db5-4117-8ad1-b1f087ed9ecc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b21b40a6-9db5-4117-8ad1-b1f087ed9ecc" (UID: "b21b40a6-9db5-4117-8ad1-b1f087ed9ecc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:04:46.390955 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:46.390937    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b21b40a6-9db5-4117-8ad1-b1f087ed9ecc-kube-api-access-rtlkj" (OuterVolumeSpecName: "kube-api-access-rtlkj") pod "b21b40a6-9db5-4117-8ad1-b1f087ed9ecc" (UID: "b21b40a6-9db5-4117-8ad1-b1f087ed9ecc"). InnerVolumeSpecName "kube-api-access-rtlkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:04:46.442079 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:46.442051    2563 generic.go:358] "Generic (PLEG): container finished" podID="b21b40a6-9db5-4117-8ad1-b1f087ed9ecc" containerID="a91dcc99d1b5654c131ed7a10c96c2a268d4f535efa6f42102e03288aa680604" exitCode=0
May 26 12:04:46.442492 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:46.442128    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw"
May 26 12:04:46.442492 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:46.442137    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw" event={"ID":"b21b40a6-9db5-4117-8ad1-b1f087ed9ecc","Type":"ContainerDied","Data":"a91dcc99d1b5654c131ed7a10c96c2a268d4f535efa6f42102e03288aa680604"}
May 26 12:04:46.442492 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:46.442178    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw" event={"ID":"b21b40a6-9db5-4117-8ad1-b1f087ed9ecc","Type":"ContainerDied","Data":"3881af3bcade4b942e3a6eb9241da165b4cbe42deea044a490100341f4d41181"}
May 26 12:04:46.442492 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:46.442202    2563 scope.go:117] "RemoveContainer" containerID="cc3afe0ac12c582f628e9a39ef8ac73d2e0e8678838d194d46ccf067e17d781e"
May 26 12:04:46.442737 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:46.442706    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9" podUID="7f7d4ea2-96ee-47b8-87f0-2cdd2e684186" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 26 12:04:46.453063 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:46.453043    2563 scope.go:117] "RemoveContainer" containerID="a91dcc99d1b5654c131ed7a10c96c2a268d4f535efa6f42102e03288aa680604"
May 26 12:04:46.460905 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:46.460864    2563 scope.go:117] "RemoveContainer" containerID="cc3afe0ac12c582f628e9a39ef8ac73d2e0e8678838d194d46ccf067e17d781e"
May 26 12:04:46.461520 ip-10-0-129-97 kubenswrapper[2563]: E0526 12:04:46.461490    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc3afe0ac12c582f628e9a39ef8ac73d2e0e8678838d194d46ccf067e17d781e\": container with ID starting with cc3afe0ac12c582f628e9a39ef8ac73d2e0e8678838d194d46ccf067e17d781e not found: ID does not exist" containerID="cc3afe0ac12c582f628e9a39ef8ac73d2e0e8678838d194d46ccf067e17d781e"
May 26 12:04:46.461604 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:46.461532    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc3afe0ac12c582f628e9a39ef8ac73d2e0e8678838d194d46ccf067e17d781e"} err="failed to get container status \"cc3afe0ac12c582f628e9a39ef8ac73d2e0e8678838d194d46ccf067e17d781e\": rpc error: code = NotFound desc = could not find container \"cc3afe0ac12c582f628e9a39ef8ac73d2e0e8678838d194d46ccf067e17d781e\": container with ID starting with cc3afe0ac12c582f628e9a39ef8ac73d2e0e8678838d194d46ccf067e17d781e not found: ID does not exist"
May 26 12:04:46.461604 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:46.461558    2563 scope.go:117] "RemoveContainer" containerID="a91dcc99d1b5654c131ed7a10c96c2a268d4f535efa6f42102e03288aa680604"
May 26 12:04:46.461913 ip-10-0-129-97 kubenswrapper[2563]: E0526 12:04:46.461886    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a91dcc99d1b5654c131ed7a10c96c2a268d4f535efa6f42102e03288aa680604\": container with ID starting with a91dcc99d1b5654c131ed7a10c96c2a268d4f535efa6f42102e03288aa680604 not found: ID does not exist" containerID="a91dcc99d1b5654c131ed7a10c96c2a268d4f535efa6f42102e03288aa680604"
May 26 12:04:46.462007 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:46.461912    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a91dcc99d1b5654c131ed7a10c96c2a268d4f535efa6f42102e03288aa680604"} err="failed to get container status \"a91dcc99d1b5654c131ed7a10c96c2a268d4f535efa6f42102e03288aa680604\": rpc error: code = NotFound desc = could not find container \"a91dcc99d1b5654c131ed7a10c96c2a268d4f535efa6f42102e03288aa680604\": container with ID starting with a91dcc99d1b5654c131ed7a10c96c2a268d4f535efa6f42102e03288aa680604 not found: ID does not exist"
May 26 12:04:46.462758 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:46.462739    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw"]
May 26 12:04:46.467374 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:46.467350    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3c458-predictor-64658f5dcf-rcnxw"]
May 26 12:04:46.489251 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:46.489221    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b21b40a6-9db5-4117-8ad1-b1f087ed9ecc-proxy-tls\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 12:04:46.489251 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:46.489251    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtlkj\" (UniqueName: \"kubernetes.io/projected/b21b40a6-9db5-4117-8ad1-b1f087ed9ecc-kube-api-access-rtlkj\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 12:04:46.489408 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:46.489263    2563 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-3c458-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b21b40a6-9db5-4117-8ad1-b1f087ed9ecc-error-404-isvc-3c458-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 12:04:48.430066 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:48.430038    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b21b40a6-9db5-4117-8ad1-b1f087ed9ecc" path="/var/lib/kubelet/pods/b21b40a6-9db5-4117-8ad1-b1f087ed9ecc/volumes"
May 26 12:04:51.015117 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:51.015076    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-3c458-6bcbbc655c-q2t5j" podUID="3bab8610-ad1e-4408-8df0-13d0cfbd7440" containerName="sequence-graph-3c458" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 12:04:51.351055 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:51.351014    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m" podUID="0462e858-a74c-468c-a60d-71276b56c110" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 26 12:04:51.447446 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:51.447417    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9"
May 26 12:04:51.448051 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:51.448012    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9" podUID="7f7d4ea2-96ee-47b8-87f0-2cdd2e684186" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 26 12:04:56.015042 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:56.014998    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-3c458-6bcbbc655c-q2t5j" podUID="3bab8610-ad1e-4408-8df0-13d0cfbd7440" containerName="sequence-graph-3c458" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 12:04:56.015414 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:04:56.015108    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-3c458-6bcbbc655c-q2t5j"
May 26 12:05:01.014920 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:01.014880    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-3c458-6bcbbc655c-q2t5j" podUID="3bab8610-ad1e-4408-8df0-13d0cfbd7440" containerName="sequence-graph-3c458" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 12:05:01.347260 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:01.347230    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m"
May 26 12:05:01.448314 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:01.448272    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9" podUID="7f7d4ea2-96ee-47b8-87f0-2cdd2e684186" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 26 12:05:06.014906 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:06.014856    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-3c458-6bcbbc655c-q2t5j" podUID="3bab8610-ad1e-4408-8df0-13d0cfbd7440" containerName="sequence-graph-3c458" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 12:05:11.014705 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:11.014667    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-3c458-6bcbbc655c-q2t5j" podUID="3bab8610-ad1e-4408-8df0-13d0cfbd7440" containerName="sequence-graph-3c458" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 12:05:11.448201 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:11.448163    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9" podUID="7f7d4ea2-96ee-47b8-87f0-2cdd2e684186" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 26 12:05:12.924222 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:12.924195    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-3c458-6bcbbc655c-q2t5j"
May 26 12:05:13.117433 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.117408    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bab8610-ad1e-4408-8df0-13d0cfbd7440-openshift-service-ca-bundle\") pod \"3bab8610-ad1e-4408-8df0-13d0cfbd7440\" (UID: \"3bab8610-ad1e-4408-8df0-13d0cfbd7440\") "
May 26 12:05:13.117596 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.117510    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bab8610-ad1e-4408-8df0-13d0cfbd7440-proxy-tls\") pod \"3bab8610-ad1e-4408-8df0-13d0cfbd7440\" (UID: \"3bab8610-ad1e-4408-8df0-13d0cfbd7440\") "
May 26 12:05:13.117875 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.117847    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bab8610-ad1e-4408-8df0-13d0cfbd7440-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "3bab8610-ad1e-4408-8df0-13d0cfbd7440" (UID: "3bab8610-ad1e-4408-8df0-13d0cfbd7440"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:05:13.119698 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.119679    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bab8610-ad1e-4408-8df0-13d0cfbd7440-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3bab8610-ad1e-4408-8df0-13d0cfbd7440" (UID: "3bab8610-ad1e-4408-8df0-13d0cfbd7440"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:05:13.218389 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.218357    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bab8610-ad1e-4408-8df0-13d0cfbd7440-proxy-tls\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 12:05:13.218389 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.218387    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bab8610-ad1e-4408-8df0-13d0cfbd7440-openshift-service-ca-bundle\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 12:05:13.462287 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.462202    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-81833-78975b54d7-2xl89"]
May 26 12:05:13.462512 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.462500    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ddb2c1f-bbdf-48ee-a623-3791529de7f6" containerName="ensemble-graph-0a5df"
May 26 12:05:13.462554 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.462514    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ddb2c1f-bbdf-48ee-a623-3791529de7f6" containerName="ensemble-graph-0a5df"
May 26 12:05:13.462554 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.462525    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b21b40a6-9db5-4117-8ad1-b1f087ed9ecc" containerName="kserve-container"
May 26 12:05:13.462554 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.462531    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b21b40a6-9db5-4117-8ad1-b1f087ed9ecc" containerName="kserve-container"
May 26 12:05:13.462554 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.462541    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b21b40a6-9db5-4117-8ad1-b1f087ed9ecc" containerName="kube-rbac-proxy"
May 26 12:05:13.462554 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.462547    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b21b40a6-9db5-4117-8ad1-b1f087ed9ecc" containerName="kube-rbac-proxy"
May 26 12:05:13.462720 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.462556    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bab8610-ad1e-4408-8df0-13d0cfbd7440" containerName="sequence-graph-3c458"
May 26 12:05:13.462720 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.462561    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bab8610-ad1e-4408-8df0-13d0cfbd7440" containerName="sequence-graph-3c458"
May 26 12:05:13.462720 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.462602    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="b21b40a6-9db5-4117-8ad1-b1f087ed9ecc" containerName="kserve-container"
May 26 12:05:13.462720 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.462610    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="3bab8610-ad1e-4408-8df0-13d0cfbd7440" containerName="sequence-graph-3c458"
May 26 12:05:13.462720 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.462618    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="b21b40a6-9db5-4117-8ad1-b1f087ed9ecc" containerName="kube-rbac-proxy"
May 26 12:05:13.462720 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.462626    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="9ddb2c1f-bbdf-48ee-a623-3791529de7f6" containerName="ensemble-graph-0a5df"
May 26 12:05:13.465882 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.465866    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-81833-78975b54d7-2xl89"
May 26 12:05:13.468490 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.468465    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-81833-kube-rbac-proxy-sar-config\""
May 26 12:05:13.468626 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.468464    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-81833-serving-cert\""
May 26 12:05:13.474222 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.474195    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-81833-78975b54d7-2xl89"]
May 26 12:05:13.528076 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.528043    2563 generic.go:358] "Generic (PLEG): container finished" podID="3bab8610-ad1e-4408-8df0-13d0cfbd7440" containerID="179e5a4dd0f28b92455afb01f19942fb961f3fd0567d847b4b0e67a6df1554ad" exitCode=0
May 26 12:05:13.528265 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.528104    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-3c458-6bcbbc655c-q2t5j"
May 26 12:05:13.528265 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.528129    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-3c458-6bcbbc655c-q2t5j" event={"ID":"3bab8610-ad1e-4408-8df0-13d0cfbd7440","Type":"ContainerDied","Data":"179e5a4dd0f28b92455afb01f19942fb961f3fd0567d847b4b0e67a6df1554ad"}
May 26 12:05:13.528265 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.528166    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-3c458-6bcbbc655c-q2t5j" event={"ID":"3bab8610-ad1e-4408-8df0-13d0cfbd7440","Type":"ContainerDied","Data":"42c426a21790fcf0b33ecfbe1c239286455143c701b0e4aa01f0732962b7d476"}
May 26 12:05:13.528265 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.528185    2563 scope.go:117] "RemoveContainer" containerID="179e5a4dd0f28b92455afb01f19942fb961f3fd0567d847b4b0e67a6df1554ad"
May 26 12:05:13.536689 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.536647    2563 scope.go:117] "RemoveContainer" containerID="179e5a4dd0f28b92455afb01f19942fb961f3fd0567d847b4b0e67a6df1554ad"
May 26 12:05:13.537017 ip-10-0-129-97 kubenswrapper[2563]: E0526 12:05:13.536997    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"179e5a4dd0f28b92455afb01f19942fb961f3fd0567d847b4b0e67a6df1554ad\": container with ID starting with 179e5a4dd0f28b92455afb01f19942fb961f3fd0567d847b4b0e67a6df1554ad not found: ID does not exist" containerID="179e5a4dd0f28b92455afb01f19942fb961f3fd0567d847b4b0e67a6df1554ad"
May 26 12:05:13.537096 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.537026    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"179e5a4dd0f28b92455afb01f19942fb961f3fd0567d847b4b0e67a6df1554ad"} err="failed to get container status \"179e5a4dd0f28b92455afb01f19942fb961f3fd0567d847b4b0e67a6df1554ad\": rpc error: code = NotFound desc = could not find container \"179e5a4dd0f28b92455afb01f19942fb961f3fd0567d847b4b0e67a6df1554ad\": container with ID starting with 179e5a4dd0f28b92455afb01f19942fb961f3fd0567d847b4b0e67a6df1554ad not found: ID does not exist"
May 26 12:05:13.550409 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.550385    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-3c458-6bcbbc655c-q2t5j"]
May 26 12:05:13.554640 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.554619    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-3c458-6bcbbc655c-q2t5j"]
May 26 12:05:13.621365 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.621308    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/214626d8-620f-4fb6-ae8b-d24a040dac22-proxy-tls\") pod \"splitter-graph-81833-78975b54d7-2xl89\" (UID: \"214626d8-620f-4fb6-ae8b-d24a040dac22\") " pod="kserve-ci-e2e-test/splitter-graph-81833-78975b54d7-2xl89"
May 26 12:05:13.621513 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.621380    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/214626d8-620f-4fb6-ae8b-d24a040dac22-openshift-service-ca-bundle\") pod \"splitter-graph-81833-78975b54d7-2xl89\" (UID: \"214626d8-620f-4fb6-ae8b-d24a040dac22\") " pod="kserve-ci-e2e-test/splitter-graph-81833-78975b54d7-2xl89"
May 26 12:05:13.722803 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.722689    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/214626d8-620f-4fb6-ae8b-d24a040dac22-proxy-tls\") pod \"splitter-graph-81833-78975b54d7-2xl89\" (UID: \"214626d8-620f-4fb6-ae8b-d24a040dac22\") " pod="kserve-ci-e2e-test/splitter-graph-81833-78975b54d7-2xl89"
May 26 12:05:13.722803 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.722762    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/214626d8-620f-4fb6-ae8b-d24a040dac22-openshift-service-ca-bundle\") pod \"splitter-graph-81833-78975b54d7-2xl89\" (UID: \"214626d8-620f-4fb6-ae8b-d24a040dac22\") " pod="kserve-ci-e2e-test/splitter-graph-81833-78975b54d7-2xl89"
May 26 12:05:13.723412 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.723392    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/214626d8-620f-4fb6-ae8b-d24a040dac22-openshift-service-ca-bundle\") pod \"splitter-graph-81833-78975b54d7-2xl89\" (UID: \"214626d8-620f-4fb6-ae8b-d24a040dac22\") " pod="kserve-ci-e2e-test/splitter-graph-81833-78975b54d7-2xl89"
May 26 12:05:13.725302 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.725283    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/214626d8-620f-4fb6-ae8b-d24a040dac22-proxy-tls\") pod \"splitter-graph-81833-78975b54d7-2xl89\" (UID: \"214626d8-620f-4fb6-ae8b-d24a040dac22\") " pod="kserve-ci-e2e-test/splitter-graph-81833-78975b54d7-2xl89"
May 26 12:05:13.776452 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.776405    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-81833-78975b54d7-2xl89"
May 26 12:05:13.913957 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:13.913626    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-81833-78975b54d7-2xl89"]
May 26 12:05:13.916506 ip-10-0-129-97 kubenswrapper[2563]: W0526 12:05:13.916459    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod214626d8_620f_4fb6_ae8b_d24a040dac22.slice/crio-b676b6a6eda5a7a0981fe06889a939f33924db621d5e9205c0a293669071f3ca WatchSource:0}: Error finding container b676b6a6eda5a7a0981fe06889a939f33924db621d5e9205c0a293669071f3ca: Status 404 returned error can't find the container with id b676b6a6eda5a7a0981fe06889a939f33924db621d5e9205c0a293669071f3ca
May 26 12:05:14.428524 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:14.428491    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bab8610-ad1e-4408-8df0-13d0cfbd7440" path="/var/lib/kubelet/pods/3bab8610-ad1e-4408-8df0-13d0cfbd7440/volumes"
May 26 12:05:14.532357 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:14.532324    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-81833-78975b54d7-2xl89" event={"ID":"214626d8-620f-4fb6-ae8b-d24a040dac22","Type":"ContainerStarted","Data":"2469fd60a6d1554602165a6c9e56f010471f4b142350c8e2a413b1c329133b33"}
May 26 12:05:14.532357 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:14.532362    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-81833-78975b54d7-2xl89" event={"ID":"214626d8-620f-4fb6-ae8b-d24a040dac22","Type":"ContainerStarted","Data":"b676b6a6eda5a7a0981fe06889a939f33924db621d5e9205c0a293669071f3ca"}
May 26 12:05:14.532610 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:14.532419    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-81833-78975b54d7-2xl89"
May 26 12:05:14.549395 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:14.549341    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-81833-78975b54d7-2xl89" podStartSLOduration=1.549325043 podStartE2EDuration="1.549325043s" podCreationTimestamp="2026-05-26 12:05:13 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:05:14.548351265 +0000 UTC m=+2446.739793455" watchObservedRunningTime="2026-05-26 12:05:14.549325043 +0000 UTC m=+2446.740767229"
May 26 12:05:20.541705 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:20.541678    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-81833-78975b54d7-2xl89"
May 26 12:05:21.448025 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:21.447981    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9" podUID="7f7d4ea2-96ee-47b8-87f0-2cdd2e684186" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 26 12:05:23.533244 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:23.533153    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-81833-78975b54d7-2xl89"]
May 26 12:05:23.533673 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:23.533352    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-81833-78975b54d7-2xl89" podUID="214626d8-620f-4fb6-ae8b-d24a040dac22" containerName="splitter-graph-81833" containerID="cri-o://2469fd60a6d1554602165a6c9e56f010471f4b142350c8e2a413b1c329133b33" gracePeriod=30
May 26 12:05:23.713966 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:23.713934    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m"]
May 26 12:05:23.714230 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:23.714203    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m" podUID="0462e858-a74c-468c-a60d-71276b56c110" containerName="kserve-container" containerID="cri-o://35a0aad95e8941c6d1557837868edb4ba7dfca04761a582c2f089f7e160d7a7d" gracePeriod=30
May 26 12:05:23.714381 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:23.714270    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m" podUID="0462e858-a74c-468c-a60d-71276b56c110" containerName="kube-rbac-proxy" containerID="cri-o://039e9cadb5687d1ffac7859b73fd24d3162217d1c7d0504aeca3ce434848544c" gracePeriod=30
May 26 12:05:23.818609 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:23.818513    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5"]
May 26 12:05:23.822064 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:23.822037    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5"
May 26 12:05:23.825456 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:23.825432    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-52624-predictor-serving-cert\""
May 26 12:05:23.826374 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:23.826358    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-52624-kube-rbac-proxy-sar-config\""
May 26 12:05:23.834524 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:23.834496    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5"]
May 26 12:05:23.902959 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:23.902915    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c560caa-7292-4467-9656-8c68722a1166-proxy-tls\") pod \"error-404-isvc-52624-predictor-5c7c48695f-2g7f5\" (UID: \"7c560caa-7292-4467-9656-8c68722a1166\") " pod="kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5"
May 26 12:05:23.903155 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:23.903010    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-52624-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c560caa-7292-4467-9656-8c68722a1166-error-404-isvc-52624-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-52624-predictor-5c7c48695f-2g7f5\" (UID: \"7c560caa-7292-4467-9656-8c68722a1166\") " pod="kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5"
May 26 12:05:23.903155 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:23.903070    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th94t\" (UniqueName: \"kubernetes.io/projected/7c560caa-7292-4467-9656-8c68722a1166-kube-api-access-th94t\") pod \"error-404-isvc-52624-predictor-5c7c48695f-2g7f5\" (UID: \"7c560caa-7292-4467-9656-8c68722a1166\") " pod="kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5"
May 26 12:05:24.003996 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:24.003962    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-th94t\" (UniqueName: \"kubernetes.io/projected/7c560caa-7292-4467-9656-8c68722a1166-kube-api-access-th94t\") pod \"error-404-isvc-52624-predictor-5c7c48695f-2g7f5\" (UID: \"7c560caa-7292-4467-9656-8c68722a1166\") " pod="kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5"
May 26 12:05:24.004167 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:24.004016    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c560caa-7292-4467-9656-8c68722a1166-proxy-tls\") pod \"error-404-isvc-52624-predictor-5c7c48695f-2g7f5\" (UID: \"7c560caa-7292-4467-9656-8c68722a1166\") " pod="kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5"
May 26 12:05:24.004167 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:24.004055    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-52624-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c560caa-7292-4467-9656-8c68722a1166-error-404-isvc-52624-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-52624-predictor-5c7c48695f-2g7f5\" (UID: \"7c560caa-7292-4467-9656-8c68722a1166\") " pod="kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5"
May 26 12:05:24.004254 ip-10-0-129-97 kubenswrapper[2563]: E0526 12:05:24.004189    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-52624-predictor-serving-cert: secret "error-404-isvc-52624-predictor-serving-cert" not found
May 26 12:05:24.004254 ip-10-0-129-97 kubenswrapper[2563]: E0526 12:05:24.004247    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c560caa-7292-4467-9656-8c68722a1166-proxy-tls podName:7c560caa-7292-4467-9656-8c68722a1166 nodeName:}" failed. No retries permitted until 2026-05-26 12:05:24.504229397 +0000 UTC m=+2456.695671561 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7c560caa-7292-4467-9656-8c68722a1166-proxy-tls") pod "error-404-isvc-52624-predictor-5c7c48695f-2g7f5" (UID: "7c560caa-7292-4467-9656-8c68722a1166") : secret "error-404-isvc-52624-predictor-serving-cert" not found
May 26 12:05:24.004620 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:24.004602    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-52624-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c560caa-7292-4467-9656-8c68722a1166-error-404-isvc-52624-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-52624-predictor-5c7c48695f-2g7f5\" (UID: \"7c560caa-7292-4467-9656-8c68722a1166\") " pod="kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5"
May 26 12:05:24.016569 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:24.016533    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th94t\" (UniqueName: \"kubernetes.io/projected/7c560caa-7292-4467-9656-8c68722a1166-kube-api-access-th94t\") pod \"error-404-isvc-52624-predictor-5c7c48695f-2g7f5\" (UID: \"7c560caa-7292-4467-9656-8c68722a1166\") " pod="kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5"
May 26 12:05:24.508321 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:24.508285    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c560caa-7292-4467-9656-8c68722a1166-proxy-tls\") pod \"error-404-isvc-52624-predictor-5c7c48695f-2g7f5\" (UID: \"7c560caa-7292-4467-9656-8c68722a1166\") " pod="kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5"
May 26 12:05:24.510973 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:24.510939    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c560caa-7292-4467-9656-8c68722a1166-proxy-tls\") pod \"error-404-isvc-52624-predictor-5c7c48695f-2g7f5\" (UID: \"7c560caa-7292-4467-9656-8c68722a1166\") " pod="kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5"
May 26 12:05:24.563045 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:24.563007    2563 generic.go:358] "Generic (PLEG): container finished" podID="0462e858-a74c-468c-a60d-71276b56c110" containerID="039e9cadb5687d1ffac7859b73fd24d3162217d1c7d0504aeca3ce434848544c" exitCode=2
May 26 12:05:24.563398 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:24.563078    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m" event={"ID":"0462e858-a74c-468c-a60d-71276b56c110","Type":"ContainerDied","Data":"039e9cadb5687d1ffac7859b73fd24d3162217d1c7d0504aeca3ce434848544c"}
May 26 12:05:24.733490 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:24.733449    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5"
May 26 12:05:24.870253 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:24.870219    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5"]
May 26 12:05:24.873256 ip-10-0-129-97 kubenswrapper[2563]: W0526 12:05:24.873224    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c560caa_7292_4467_9656_8c68722a1166.slice/crio-e5ba0a35d66e8d6357fe9f971854ffdb890148b48aa626ece6498a6de0f877ad WatchSource:0}: Error finding container e5ba0a35d66e8d6357fe9f971854ffdb890148b48aa626ece6498a6de0f877ad: Status 404 returned error can't find the container with id e5ba0a35d66e8d6357fe9f971854ffdb890148b48aa626ece6498a6de0f877ad
May 26 12:05:25.540218 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:25.540173    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-81833-78975b54d7-2xl89" podUID="214626d8-620f-4fb6-ae8b-d24a040dac22" containerName="splitter-graph-81833" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 12:05:25.567976 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:25.567942    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5" event={"ID":"7c560caa-7292-4467-9656-8c68722a1166","Type":"ContainerStarted","Data":"7c5d6de6fea870f7a1a90320788083155d1110555df35c1677c991c614b247c3"}
May 26 12:05:25.567976 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:25.567980    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5" event={"ID":"7c560caa-7292-4467-9656-8c68722a1166","Type":"ContainerStarted","Data":"9dae4f77d9164d0566af3503e711e5a2628c1d0657961cbb1c9841e7bb0ed822"}
May 26 12:05:25.568470 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:25.567989    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5" event={"ID":"7c560caa-7292-4467-9656-8c68722a1166","Type":"ContainerStarted","Data":"e5ba0a35d66e8d6357fe9f971854ffdb890148b48aa626ece6498a6de0f877ad"}
May 26 12:05:25.568470 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:25.568108    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5"
May 26 12:05:25.568470 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:25.568139    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5"
May 26 12:05:25.569520 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:25.569488    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5" podUID="7c560caa-7292-4467-9656-8c68722a1166" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 26 12:05:25.593043 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:25.592971    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5" podStartSLOduration=2.5929525079999998 podStartE2EDuration="2.592952508s" podCreationTimestamp="2026-05-26 12:05:23 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:05:25.590458215 +0000 UTC m=+2457.781900402" watchObservedRunningTime="2026-05-26 12:05:25.592952508 +0000 UTC m=+2457.784394696"
May 26 12:05:26.342224 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:26.342178    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m" podUID="0462e858-a74c-468c-a60d-71276b56c110" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
May 26 12:05:26.571363 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:26.571323    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5" podUID="7c560caa-7292-4467-9656-8c68722a1166" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 26 12:05:27.270676 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:27.270648    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m"
May 26 12:05:27.331281 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:27.331237    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0462e858-a74c-468c-a60d-71276b56c110-proxy-tls\") pod \"0462e858-a74c-468c-a60d-71276b56c110\" (UID: \"0462e858-a74c-468c-a60d-71276b56c110\") "
May 26 12:05:27.331449 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:27.331300    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-81833-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0462e858-a74c-468c-a60d-71276b56c110-error-404-isvc-81833-kube-rbac-proxy-sar-config\") pod \"0462e858-a74c-468c-a60d-71276b56c110\" (UID: \"0462e858-a74c-468c-a60d-71276b56c110\") "
May 26 12:05:27.331449 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:27.331335    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m56vq\" (UniqueName: \"kubernetes.io/projected/0462e858-a74c-468c-a60d-71276b56c110-kube-api-access-m56vq\") pod \"0462e858-a74c-468c-a60d-71276b56c110\" (UID: \"0462e858-a74c-468c-a60d-71276b56c110\") "
May 26 12:05:27.331682 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:27.331652    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0462e858-a74c-468c-a60d-71276b56c110-error-404-isvc-81833-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-81833-kube-rbac-proxy-sar-config") pod "0462e858-a74c-468c-a60d-71276b56c110" (UID: "0462e858-a74c-468c-a60d-71276b56c110"). InnerVolumeSpecName "error-404-isvc-81833-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:05:27.333562 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:27.333538    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0462e858-a74c-468c-a60d-71276b56c110-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0462e858-a74c-468c-a60d-71276b56c110" (UID: "0462e858-a74c-468c-a60d-71276b56c110"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:05:27.333663 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:27.333633    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0462e858-a74c-468c-a60d-71276b56c110-kube-api-access-m56vq" (OuterVolumeSpecName: "kube-api-access-m56vq") pod "0462e858-a74c-468c-a60d-71276b56c110" (UID: "0462e858-a74c-468c-a60d-71276b56c110"). InnerVolumeSpecName "kube-api-access-m56vq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:05:27.432813 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:27.432688    2563 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-81833-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0462e858-a74c-468c-a60d-71276b56c110-error-404-isvc-81833-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 12:05:27.432813 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:27.432715    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m56vq\" (UniqueName: \"kubernetes.io/projected/0462e858-a74c-468c-a60d-71276b56c110-kube-api-access-m56vq\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 12:05:27.432813 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:27.432726    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0462e858-a74c-468c-a60d-71276b56c110-proxy-tls\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 12:05:27.576205 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:27.576165    2563 generic.go:358] "Generic (PLEG): container finished" podID="0462e858-a74c-468c-a60d-71276b56c110" containerID="35a0aad95e8941c6d1557837868edb4ba7dfca04761a582c2f089f7e160d7a7d" exitCode=0
May 26 12:05:27.576596 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:27.576206    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m" event={"ID":"0462e858-a74c-468c-a60d-71276b56c110","Type":"ContainerDied","Data":"35a0aad95e8941c6d1557837868edb4ba7dfca04761a582c2f089f7e160d7a7d"}
May 26 12:05:27.576596 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:27.576251    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m"
May 26 12:05:27.576596 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:27.576266    2563 scope.go:117] "RemoveContainer" containerID="039e9cadb5687d1ffac7859b73fd24d3162217d1c7d0504aeca3ce434848544c"
May 26 12:05:27.576596 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:27.576255    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m" event={"ID":"0462e858-a74c-468c-a60d-71276b56c110","Type":"ContainerDied","Data":"5d20c009e6a6bd016125a6f60f1c1b0d71e5febc306e9c72c7a5ca9c96161726"}
May 26 12:05:27.584954 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:27.584934    2563 scope.go:117] "RemoveContainer" containerID="35a0aad95e8941c6d1557837868edb4ba7dfca04761a582c2f089f7e160d7a7d"
May 26 12:05:27.593382 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:27.593355    2563 scope.go:117] "RemoveContainer" containerID="039e9cadb5687d1ffac7859b73fd24d3162217d1c7d0504aeca3ce434848544c"
May 26 12:05:27.593703 ip-10-0-129-97 kubenswrapper[2563]: E0526 12:05:27.593682    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"039e9cadb5687d1ffac7859b73fd24d3162217d1c7d0504aeca3ce434848544c\": container with ID starting with 039e9cadb5687d1ffac7859b73fd24d3162217d1c7d0504aeca3ce434848544c not found: ID does not exist" containerID="039e9cadb5687d1ffac7859b73fd24d3162217d1c7d0504aeca3ce434848544c"
May 26 12:05:27.593805 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:27.593712    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"039e9cadb5687d1ffac7859b73fd24d3162217d1c7d0504aeca3ce434848544c"} err="failed to get container status \"039e9cadb5687d1ffac7859b73fd24d3162217d1c7d0504aeca3ce434848544c\": rpc error: code = NotFound desc = could not find container \"039e9cadb5687d1ffac7859b73fd24d3162217d1c7d0504aeca3ce434848544c\": container with ID starting with 039e9cadb5687d1ffac7859b73fd24d3162217d1c7d0504aeca3ce434848544c not found: ID does not exist"
May 26 12:05:27.593805 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:27.593731    2563 scope.go:117] "RemoveContainer" containerID="35a0aad95e8941c6d1557837868edb4ba7dfca04761a582c2f089f7e160d7a7d"
May 26 12:05:27.594000 ip-10-0-129-97 kubenswrapper[2563]: E0526 12:05:27.593982    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35a0aad95e8941c6d1557837868edb4ba7dfca04761a582c2f089f7e160d7a7d\": container with ID starting with 35a0aad95e8941c6d1557837868edb4ba7dfca04761a582c2f089f7e160d7a7d not found: ID does not exist" containerID="35a0aad95e8941c6d1557837868edb4ba7dfca04761a582c2f089f7e160d7a7d"
May 26 12:05:27.594067 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:27.594004    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35a0aad95e8941c6d1557837868edb4ba7dfca04761a582c2f089f7e160d7a7d"} err="failed to get container status \"35a0aad95e8941c6d1557837868edb4ba7dfca04761a582c2f089f7e160d7a7d\": rpc error: code = NotFound desc = could not find container \"35a0aad95e8941c6d1557837868edb4ba7dfca04761a582c2f089f7e160d7a7d\": container with ID starting with 35a0aad95e8941c6d1557837868edb4ba7dfca04761a582c2f089f7e160d7a7d not found: ID does not exist"
May 26 12:05:27.600035 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:27.600007    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m"]
May 26 12:05:27.604180 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:27.604156    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-81833-predictor-6db55d4ff5-xfl9m"]
May 26 12:05:28.429299 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:28.429264    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0462e858-a74c-468c-a60d-71276b56c110" path="/var/lib/kubelet/pods/0462e858-a74c-468c-a60d-71276b56c110/volumes"
May 26 12:05:30.540466 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:30.540427    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-81833-78975b54d7-2xl89" podUID="214626d8-620f-4fb6-ae8b-d24a040dac22" containerName="splitter-graph-81833" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 12:05:31.449510 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:31.449481    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9"
May 26 12:05:31.576404 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:31.576380    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5"
May 26 12:05:31.576996 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:31.576971    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5" podUID="7c560caa-7292-4467-9656-8c68722a1166" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 26 12:05:35.540194 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:35.540153    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-81833-78975b54d7-2xl89" podUID="214626d8-620f-4fb6-ae8b-d24a040dac22" containerName="splitter-graph-81833" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 12:05:35.540614 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:35.540297    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-81833-78975b54d7-2xl89"
May 26 12:05:40.539549 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:40.539509    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-81833-78975b54d7-2xl89" podUID="214626d8-620f-4fb6-ae8b-d24a040dac22" containerName="splitter-graph-81833" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 12:05:41.577411 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:41.577367    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5" podUID="7c560caa-7292-4467-9656-8c68722a1166" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 26 12:05:42.991761 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:42.991720    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-152bc-5b8ffd9c8b-kvgnq"]
May 26 12:05:42.992167 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:42.992130    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0462e858-a74c-468c-a60d-71276b56c110" containerName="kserve-container"
May 26 12:05:42.992167 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:42.992150    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0462e858-a74c-468c-a60d-71276b56c110" containerName="kserve-container"
May 26 12:05:42.992253 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:42.992171    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0462e858-a74c-468c-a60d-71276b56c110" containerName="kube-rbac-proxy"
May 26 12:05:42.992253 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:42.992177    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0462e858-a74c-468c-a60d-71276b56c110" containerName="kube-rbac-proxy"
May 26 12:05:42.992253 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:42.992232    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="0462e858-a74c-468c-a60d-71276b56c110" containerName="kserve-container"
May 26 12:05:42.992253 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:42.992242    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="0462e858-a74c-468c-a60d-71276b56c110" containerName="kube-rbac-proxy"
May 26 12:05:42.995213 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:42.995193    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-152bc-5b8ffd9c8b-kvgnq"
May 26 12:05:42.998057 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:42.998039    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-152bc-kube-rbac-proxy-sar-config\""
May 26 12:05:42.998257 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:42.998244    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-152bc-serving-cert\""
May 26 12:05:43.003498 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:43.003479    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-152bc-5b8ffd9c8b-kvgnq"]
May 26 12:05:43.058792 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:43.058735    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6745d2d9-a284-42e7-8e35-a3e2d249de82-proxy-tls\") pod \"switch-graph-152bc-5b8ffd9c8b-kvgnq\" (UID: \"6745d2d9-a284-42e7-8e35-a3e2d249de82\") " pod="kserve-ci-e2e-test/switch-graph-152bc-5b8ffd9c8b-kvgnq"
May 26 12:05:43.058980 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:43.058817    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6745d2d9-a284-42e7-8e35-a3e2d249de82-openshift-service-ca-bundle\") pod \"switch-graph-152bc-5b8ffd9c8b-kvgnq\" (UID: \"6745d2d9-a284-42e7-8e35-a3e2d249de82\") " pod="kserve-ci-e2e-test/switch-graph-152bc-5b8ffd9c8b-kvgnq"
May 26 12:05:43.160060 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:43.160024    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6745d2d9-a284-42e7-8e35-a3e2d249de82-openshift-service-ca-bundle\") pod \"switch-graph-152bc-5b8ffd9c8b-kvgnq\" (UID: \"6745d2d9-a284-42e7-8e35-a3e2d249de82\") " pod="kserve-ci-e2e-test/switch-graph-152bc-5b8ffd9c8b-kvgnq"
May 26 12:05:43.160272 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:43.160128    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6745d2d9-a284-42e7-8e35-a3e2d249de82-proxy-tls\") pod \"switch-graph-152bc-5b8ffd9c8b-kvgnq\" (UID: \"6745d2d9-a284-42e7-8e35-a3e2d249de82\") " pod="kserve-ci-e2e-test/switch-graph-152bc-5b8ffd9c8b-kvgnq"
May 26 12:05:43.160272 ip-10-0-129-97 kubenswrapper[2563]: E0526 12:05:43.160224    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-152bc-serving-cert: secret "switch-graph-152bc-serving-cert" not found
May 26 12:05:43.160396 ip-10-0-129-97 kubenswrapper[2563]: E0526 12:05:43.160287    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6745d2d9-a284-42e7-8e35-a3e2d249de82-proxy-tls podName:6745d2d9-a284-42e7-8e35-a3e2d249de82 nodeName:}" failed. No retries permitted until 2026-05-26 12:05:43.660271275 +0000 UTC m=+2475.851713440 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6745d2d9-a284-42e7-8e35-a3e2d249de82-proxy-tls") pod "switch-graph-152bc-5b8ffd9c8b-kvgnq" (UID: "6745d2d9-a284-42e7-8e35-a3e2d249de82") : secret "switch-graph-152bc-serving-cert" not found
May 26 12:05:43.160806 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:43.160749    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6745d2d9-a284-42e7-8e35-a3e2d249de82-openshift-service-ca-bundle\") pod \"switch-graph-152bc-5b8ffd9c8b-kvgnq\" (UID: \"6745d2d9-a284-42e7-8e35-a3e2d249de82\") " pod="kserve-ci-e2e-test/switch-graph-152bc-5b8ffd9c8b-kvgnq"
May 26 12:05:43.665357 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:43.665311    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6745d2d9-a284-42e7-8e35-a3e2d249de82-proxy-tls\") pod \"switch-graph-152bc-5b8ffd9c8b-kvgnq\" (UID: \"6745d2d9-a284-42e7-8e35-a3e2d249de82\") " pod="kserve-ci-e2e-test/switch-graph-152bc-5b8ffd9c8b-kvgnq"
May 26 12:05:43.667894 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:43.667860    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6745d2d9-a284-42e7-8e35-a3e2d249de82-proxy-tls\") pod \"switch-graph-152bc-5b8ffd9c8b-kvgnq\" (UID: \"6745d2d9-a284-42e7-8e35-a3e2d249de82\") " pod="kserve-ci-e2e-test/switch-graph-152bc-5b8ffd9c8b-kvgnq"
May 26 12:05:43.906490 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:43.906455    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-152bc-5b8ffd9c8b-kvgnq"
May 26 12:05:44.038406 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:44.038377    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-152bc-5b8ffd9c8b-kvgnq"]
May 26 12:05:44.624319 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:44.624279    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-152bc-5b8ffd9c8b-kvgnq" event={"ID":"6745d2d9-a284-42e7-8e35-a3e2d249de82","Type":"ContainerStarted","Data":"ac89b7fb98cf8a67c3a6c9cae4986dbeee49935b0a36909bb64b7750bf8ed568"}
May 26 12:05:44.624319 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:44.624315    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-152bc-5b8ffd9c8b-kvgnq" event={"ID":"6745d2d9-a284-42e7-8e35-a3e2d249de82","Type":"ContainerStarted","Data":"9b96eb0530e2999311197cfa4fb3e9ad1ef69c7a1b6b0fe6e89b5dc06b937d59"}
May 26 12:05:44.624492 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:44.624436    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-152bc-5b8ffd9c8b-kvgnq"
May 26 12:05:44.641596 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:44.641549    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-152bc-5b8ffd9c8b-kvgnq" podStartSLOduration=2.641535972 podStartE2EDuration="2.641535972s" podCreationTimestamp="2026-05-26 12:05:42 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:05:44.640995342 +0000 UTC m=+2476.832437530" watchObservedRunningTime="2026-05-26 12:05:44.641535972 +0000 UTC m=+2476.832978158"
May 26 12:05:45.540141 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:45.540102    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-81833-78975b54d7-2xl89" podUID="214626d8-620f-4fb6-ae8b-d24a040dac22" containerName="splitter-graph-81833" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 12:05:50.539945 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:50.539910    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-81833-78975b54d7-2xl89" podUID="214626d8-620f-4fb6-ae8b-d24a040dac22" containerName="splitter-graph-81833" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 12:05:50.634242 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:50.634215    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-152bc-5b8ffd9c8b-kvgnq"
May 26 12:05:51.577646 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:51.577599    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5" podUID="7c560caa-7292-4467-9656-8c68722a1166" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 26 12:05:53.552973 ip-10-0-129-97 kubenswrapper[2563]: E0526 12:05:53.552930    2563 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod214626d8_620f_4fb6_ae8b_d24a040dac22.slice/crio-b676b6a6eda5a7a0981fe06889a939f33924db621d5e9205c0a293669071f3ca\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod214626d8_620f_4fb6_ae8b_d24a040dac22.slice/crio-2469fd60a6d1554602165a6c9e56f010471f4b142350c8e2a413b1c329133b33.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod214626d8_620f_4fb6_ae8b_d24a040dac22.slice/crio-conmon-2469fd60a6d1554602165a6c9e56f010471f4b142350c8e2a413b1c329133b33.scope\": RecentStats: unable to find data in memory cache]"
May 26 12:05:53.553348 ip-10-0-129-97 kubenswrapper[2563]: E0526 12:05:53.553054    2563 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod214626d8_620f_4fb6_ae8b_d24a040dac22.slice/crio-2469fd60a6d1554602165a6c9e56f010471f4b142350c8e2a413b1c329133b33.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod214626d8_620f_4fb6_ae8b_d24a040dac22.slice/crio-conmon-2469fd60a6d1554602165a6c9e56f010471f4b142350c8e2a413b1c329133b33.scope\": RecentStats: unable to find data in memory cache]"
May 26 12:05:53.553348 ip-10-0-129-97 kubenswrapper[2563]: E0526 12:05:53.553077    2563 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod214626d8_620f_4fb6_ae8b_d24a040dac22.slice/crio-conmon-2469fd60a6d1554602165a6c9e56f010471f4b142350c8e2a413b1c329133b33.scope\": RecentStats: unable to find data in memory cache]"
May 26 12:05:53.553348 ip-10-0-129-97 kubenswrapper[2563]: E0526 12:05:53.553148    2563 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod214626d8_620f_4fb6_ae8b_d24a040dac22.slice/crio-b676b6a6eda5a7a0981fe06889a939f33924db621d5e9205c0a293669071f3ca\": RecentStats: unable to find data in memory cache]"
May 26 12:05:53.651942 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:53.651908    2563 generic.go:358] "Generic (PLEG): container finished" podID="214626d8-620f-4fb6-ae8b-d24a040dac22" containerID="2469fd60a6d1554602165a6c9e56f010471f4b142350c8e2a413b1c329133b33" exitCode=0
May 26 12:05:53.652119 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:53.651982    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-81833-78975b54d7-2xl89" event={"ID":"214626d8-620f-4fb6-ae8b-d24a040dac22","Type":"ContainerDied","Data":"2469fd60a6d1554602165a6c9e56f010471f4b142350c8e2a413b1c329133b33"}
May 26 12:05:53.676995 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:53.676969    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-81833-78975b54d7-2xl89"
May 26 12:05:53.751720 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:53.751681    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/214626d8-620f-4fb6-ae8b-d24a040dac22-openshift-service-ca-bundle\") pod \"214626d8-620f-4fb6-ae8b-d24a040dac22\" (UID: \"214626d8-620f-4fb6-ae8b-d24a040dac22\") "
May 26 12:05:53.751937 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:53.751738    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/214626d8-620f-4fb6-ae8b-d24a040dac22-proxy-tls\") pod \"214626d8-620f-4fb6-ae8b-d24a040dac22\" (UID: \"214626d8-620f-4fb6-ae8b-d24a040dac22\") "
May 26 12:05:53.752168 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:53.752141    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/214626d8-620f-4fb6-ae8b-d24a040dac22-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "214626d8-620f-4fb6-ae8b-d24a040dac22" (UID: "214626d8-620f-4fb6-ae8b-d24a040dac22"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:05:53.754026 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:53.754006    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/214626d8-620f-4fb6-ae8b-d24a040dac22-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "214626d8-620f-4fb6-ae8b-d24a040dac22" (UID: "214626d8-620f-4fb6-ae8b-d24a040dac22"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:05:53.852649 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:53.852612    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/214626d8-620f-4fb6-ae8b-d24a040dac22-openshift-service-ca-bundle\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 12:05:53.852649 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:53.852646    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/214626d8-620f-4fb6-ae8b-d24a040dac22-proxy-tls\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 12:05:54.655565 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:54.655541    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-81833-78975b54d7-2xl89"
May 26 12:05:54.656036 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:54.655540    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-81833-78975b54d7-2xl89" event={"ID":"214626d8-620f-4fb6-ae8b-d24a040dac22","Type":"ContainerDied","Data":"b676b6a6eda5a7a0981fe06889a939f33924db621d5e9205c0a293669071f3ca"}
May 26 12:05:54.656036 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:54.655657    2563 scope.go:117] "RemoveContainer" containerID="2469fd60a6d1554602165a6c9e56f010471f4b142350c8e2a413b1c329133b33"
May 26 12:05:54.672188 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:54.672160    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-81833-78975b54d7-2xl89"]
May 26 12:05:54.676698 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:54.676676    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-81833-78975b54d7-2xl89"]
May 26 12:05:56.428320 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:05:56.428288    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="214626d8-620f-4fb6-ae8b-d24a040dac22" path="/var/lib/kubelet/pods/214626d8-620f-4fb6-ae8b-d24a040dac22/volumes"
May 26 12:06:01.577114 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:06:01.577073    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5" podUID="7c560caa-7292-4467-9656-8c68722a1166" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 26 12:06:11.577974 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:06:11.577943    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5"
May 26 12:06:23.742083 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:06:23.742049    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-52624-856bdd7d4b-swwrx"]
May 26 12:06:23.742444 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:06:23.742382    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="214626d8-620f-4fb6-ae8b-d24a040dac22" containerName="splitter-graph-81833"
May 26 12:06:23.742444 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:06:23.742394    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="214626d8-620f-4fb6-ae8b-d24a040dac22" containerName="splitter-graph-81833"
May 26 12:06:23.742530 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:06:23.742457    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="214626d8-620f-4fb6-ae8b-d24a040dac22" containerName="splitter-graph-81833"
May 26 12:06:23.745503 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:06:23.745485    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-52624-856bdd7d4b-swwrx"
May 26 12:06:23.748262 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:06:23.748241    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-52624-kube-rbac-proxy-sar-config\""
May 26 12:06:23.748377 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:06:23.748239    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-52624-serving-cert\""
May 26 12:06:23.754958 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:06:23.754933    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-52624-856bdd7d4b-swwrx"]
May 26 12:06:23.803117 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:06:23.803081    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb9ca510-dcf7-4b6a-9065-b2abb57680ff-openshift-service-ca-bundle\") pod \"splitter-graph-52624-856bdd7d4b-swwrx\" (UID: \"bb9ca510-dcf7-4b6a-9065-b2abb57680ff\") " pod="kserve-ci-e2e-test/splitter-graph-52624-856bdd7d4b-swwrx"
May 26 12:06:23.803117 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:06:23.803129    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb9ca510-dcf7-4b6a-9065-b2abb57680ff-proxy-tls\") pod \"splitter-graph-52624-856bdd7d4b-swwrx\" (UID: \"bb9ca510-dcf7-4b6a-9065-b2abb57680ff\") " pod="kserve-ci-e2e-test/splitter-graph-52624-856bdd7d4b-swwrx"
May 26 12:06:23.904165 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:06:23.904132    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb9ca510-dcf7-4b6a-9065-b2abb57680ff-openshift-service-ca-bundle\") pod \"splitter-graph-52624-856bdd7d4b-swwrx\" (UID: \"bb9ca510-dcf7-4b6a-9065-b2abb57680ff\") " pod="kserve-ci-e2e-test/splitter-graph-52624-856bdd7d4b-swwrx"
May 26 12:06:23.904319 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:06:23.904181    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb9ca510-dcf7-4b6a-9065-b2abb57680ff-proxy-tls\") pod \"splitter-graph-52624-856bdd7d4b-swwrx\" (UID: \"bb9ca510-dcf7-4b6a-9065-b2abb57680ff\") " pod="kserve-ci-e2e-test/splitter-graph-52624-856bdd7d4b-swwrx"
May 26 12:06:23.904360 ip-10-0-129-97 kubenswrapper[2563]: E0526 12:06:23.904321    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-52624-serving-cert: secret "splitter-graph-52624-serving-cert" not found
May 26 12:06:23.904400 ip-10-0-129-97 kubenswrapper[2563]: E0526 12:06:23.904383    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb9ca510-dcf7-4b6a-9065-b2abb57680ff-proxy-tls podName:bb9ca510-dcf7-4b6a-9065-b2abb57680ff nodeName:}" failed. No retries permitted until 2026-05-26 12:06:24.404364853 +0000 UTC m=+2516.595807021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bb9ca510-dcf7-4b6a-9065-b2abb57680ff-proxy-tls") pod "splitter-graph-52624-856bdd7d4b-swwrx" (UID: "bb9ca510-dcf7-4b6a-9065-b2abb57680ff") : secret "splitter-graph-52624-serving-cert" not found
May 26 12:06:23.904881 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:06:23.904860    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb9ca510-dcf7-4b6a-9065-b2abb57680ff-openshift-service-ca-bundle\") pod \"splitter-graph-52624-856bdd7d4b-swwrx\" (UID: \"bb9ca510-dcf7-4b6a-9065-b2abb57680ff\") " pod="kserve-ci-e2e-test/splitter-graph-52624-856bdd7d4b-swwrx"
May 26 12:06:24.408295 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:06:24.408242    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb9ca510-dcf7-4b6a-9065-b2abb57680ff-proxy-tls\") pod \"splitter-graph-52624-856bdd7d4b-swwrx\" (UID: \"bb9ca510-dcf7-4b6a-9065-b2abb57680ff\") " pod="kserve-ci-e2e-test/splitter-graph-52624-856bdd7d4b-swwrx"
May 26 12:06:24.410901 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:06:24.410871    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb9ca510-dcf7-4b6a-9065-b2abb57680ff-proxy-tls\") pod \"splitter-graph-52624-856bdd7d4b-swwrx\" (UID: \"bb9ca510-dcf7-4b6a-9065-b2abb57680ff\") " pod="kserve-ci-e2e-test/splitter-graph-52624-856bdd7d4b-swwrx"
May 26 12:06:24.656888 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:06:24.656854    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-52624-856bdd7d4b-swwrx"
May 26 12:06:24.795584 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:06:24.795517    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-52624-856bdd7d4b-swwrx"]
May 26 12:06:24.798314 ip-10-0-129-97 kubenswrapper[2563]: W0526 12:06:24.798284    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb9ca510_dcf7_4b6a_9065_b2abb57680ff.slice/crio-0be03c526d059fd526d5273a36e9ca3de318919b4aa541bcd4a7e00e7436e7d7 WatchSource:0}: Error finding container 0be03c526d059fd526d5273a36e9ca3de318919b4aa541bcd4a7e00e7436e7d7: Status 404 returned error can't find the container with id 0be03c526d059fd526d5273a36e9ca3de318919b4aa541bcd4a7e00e7436e7d7
May 26 12:06:25.754314 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:06:25.754276    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-52624-856bdd7d4b-swwrx" event={"ID":"bb9ca510-dcf7-4b6a-9065-b2abb57680ff","Type":"ContainerStarted","Data":"517ca7f563a4a651ddb6a565665a3cae288ce6f9a8b78c11d71422146ea45045"}
May 26 12:06:25.754314 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:06:25.754319    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-52624-856bdd7d4b-swwrx" event={"ID":"bb9ca510-dcf7-4b6a-9065-b2abb57680ff","Type":"ContainerStarted","Data":"0be03c526d059fd526d5273a36e9ca3de318919b4aa541bcd4a7e00e7436e7d7"}
May 26 12:06:25.754544 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:06:25.754408    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-52624-856bdd7d4b-swwrx"
May 26 12:06:25.772525 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:06:25.772476    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-52624-856bdd7d4b-swwrx" podStartSLOduration=2.772461079 podStartE2EDuration="2.772461079s" podCreationTimestamp="2026-05-26 12:06:23 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:06:25.771067632 +0000 UTC m=+2517.962509829" watchObservedRunningTime="2026-05-26 12:06:25.772461079 +0000 UTC m=+2517.963903279"
May 26 12:06:31.762286 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:06:31.762256    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-52624-856bdd7d4b-swwrx"
May 26 12:14:38.429483 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:38.429453    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-52624-856bdd7d4b-swwrx"]
May 26 12:14:38.430005 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:38.429650    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-52624-856bdd7d4b-swwrx" podUID="bb9ca510-dcf7-4b6a-9065-b2abb57680ff" containerName="splitter-graph-52624" containerID="cri-o://517ca7f563a4a651ddb6a565665a3cae288ce6f9a8b78c11d71422146ea45045" gracePeriod=30
May 26 12:14:38.573583 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:38.573544    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5"]
May 26 12:14:38.573951 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:38.573925    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5" podUID="7c560caa-7292-4467-9656-8c68722a1166" containerName="kserve-container" containerID="cri-o://9dae4f77d9164d0566af3503e711e5a2628c1d0657961cbb1c9841e7bb0ed822" gracePeriod=30
May 26 12:14:38.574039 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:38.573948    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5" podUID="7c560caa-7292-4467-9656-8c68722a1166" containerName="kube-rbac-proxy" containerID="cri-o://7c5d6de6fea870f7a1a90320788083155d1110555df35c1677c991c614b247c3" gracePeriod=30
May 26 12:14:39.184990 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:39.184956    2563 generic.go:358] "Generic (PLEG): container finished" podID="7c560caa-7292-4467-9656-8c68722a1166" containerID="7c5d6de6fea870f7a1a90320788083155d1110555df35c1677c991c614b247c3" exitCode=2
May 26 12:14:39.185169 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:39.185034    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5" event={"ID":"7c560caa-7292-4467-9656-8c68722a1166","Type":"ContainerDied","Data":"7c5d6de6fea870f7a1a90320788083155d1110555df35c1677c991c614b247c3"}
May 26 12:14:41.572031 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:41.571985    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5" podUID="7c560caa-7292-4467-9656-8c68722a1166" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
May 26 12:14:41.577333 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:41.577306    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5" podUID="7c560caa-7292-4467-9656-8c68722a1166" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 26 12:14:41.761212 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:41.761175    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-52624-856bdd7d4b-swwrx" podUID="bb9ca510-dcf7-4b6a-9065-b2abb57680ff" containerName="splitter-graph-52624" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 12:14:41.811106 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:41.811084    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5"
May 26 12:14:41.927787 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:41.927748    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-52624-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c560caa-7292-4467-9656-8c68722a1166-error-404-isvc-52624-kube-rbac-proxy-sar-config\") pod \"7c560caa-7292-4467-9656-8c68722a1166\" (UID: \"7c560caa-7292-4467-9656-8c68722a1166\") "
May 26 12:14:41.927972 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:41.927830    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c560caa-7292-4467-9656-8c68722a1166-proxy-tls\") pod \"7c560caa-7292-4467-9656-8c68722a1166\" (UID: \"7c560caa-7292-4467-9656-8c68722a1166\") "
May 26 12:14:41.927972 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:41.927865    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-th94t\" (UniqueName: \"kubernetes.io/projected/7c560caa-7292-4467-9656-8c68722a1166-kube-api-access-th94t\") pod \"7c560caa-7292-4467-9656-8c68722a1166\" (UID: \"7c560caa-7292-4467-9656-8c68722a1166\") "
May 26 12:14:41.928167 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:41.928137    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c560caa-7292-4467-9656-8c68722a1166-error-404-isvc-52624-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-52624-kube-rbac-proxy-sar-config") pod "7c560caa-7292-4467-9656-8c68722a1166" (UID: "7c560caa-7292-4467-9656-8c68722a1166"). InnerVolumeSpecName "error-404-isvc-52624-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:14:41.930184 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:41.930138    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c560caa-7292-4467-9656-8c68722a1166-kube-api-access-th94t" (OuterVolumeSpecName: "kube-api-access-th94t") pod "7c560caa-7292-4467-9656-8c68722a1166" (UID: "7c560caa-7292-4467-9656-8c68722a1166"). InnerVolumeSpecName "kube-api-access-th94t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:14:41.930307 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:41.930218    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c560caa-7292-4467-9656-8c68722a1166-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7c560caa-7292-4467-9656-8c68722a1166" (UID: "7c560caa-7292-4467-9656-8c68722a1166"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:14:42.028929 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:42.028889    2563 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-52624-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c560caa-7292-4467-9656-8c68722a1166-error-404-isvc-52624-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 12:14:42.028929 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:42.028921    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c560caa-7292-4467-9656-8c68722a1166-proxy-tls\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 12:14:42.028929 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:42.028936    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th94t\" (UniqueName: \"kubernetes.io/projected/7c560caa-7292-4467-9656-8c68722a1166-kube-api-access-th94t\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 12:14:42.195412 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:42.195316    2563 generic.go:358] "Generic (PLEG): container finished" podID="7c560caa-7292-4467-9656-8c68722a1166" containerID="9dae4f77d9164d0566af3503e711e5a2628c1d0657961cbb1c9841e7bb0ed822" exitCode=0
May 26 12:14:42.195412 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:42.195363    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5" event={"ID":"7c560caa-7292-4467-9656-8c68722a1166","Type":"ContainerDied","Data":"9dae4f77d9164d0566af3503e711e5a2628c1d0657961cbb1c9841e7bb0ed822"}
May 26 12:14:42.195412 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:42.195386    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5"
May 26 12:14:42.195412 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:42.195400    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5" event={"ID":"7c560caa-7292-4467-9656-8c68722a1166","Type":"ContainerDied","Data":"e5ba0a35d66e8d6357fe9f971854ffdb890148b48aa626ece6498a6de0f877ad"}
May 26 12:14:42.195716 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:42.195421    2563 scope.go:117] "RemoveContainer" containerID="7c5d6de6fea870f7a1a90320788083155d1110555df35c1677c991c614b247c3"
May 26 12:14:42.206501 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:42.205934    2563 scope.go:117] "RemoveContainer" containerID="9dae4f77d9164d0566af3503e711e5a2628c1d0657961cbb1c9841e7bb0ed822"
May 26 12:14:42.213490 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:42.213474    2563 scope.go:117] "RemoveContainer" containerID="7c5d6de6fea870f7a1a90320788083155d1110555df35c1677c991c614b247c3"
May 26 12:14:42.213736 ip-10-0-129-97 kubenswrapper[2563]: E0526 12:14:42.213718    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c5d6de6fea870f7a1a90320788083155d1110555df35c1677c991c614b247c3\": container with ID starting with 7c5d6de6fea870f7a1a90320788083155d1110555df35c1677c991c614b247c3 not found: ID does not exist" containerID="7c5d6de6fea870f7a1a90320788083155d1110555df35c1677c991c614b247c3"
May 26 12:14:42.213904 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:42.213742    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c5d6de6fea870f7a1a90320788083155d1110555df35c1677c991c614b247c3"} err="failed to get container status \"7c5d6de6fea870f7a1a90320788083155d1110555df35c1677c991c614b247c3\": rpc error: code = NotFound desc = could not find container \"7c5d6de6fea870f7a1a90320788083155d1110555df35c1677c991c614b247c3\": container with ID starting with 7c5d6de6fea870f7a1a90320788083155d1110555df35c1677c991c614b247c3 not found: ID does not exist"
May 26 12:14:42.213904 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:42.213759    2563 scope.go:117] "RemoveContainer" containerID="9dae4f77d9164d0566af3503e711e5a2628c1d0657961cbb1c9841e7bb0ed822"
May 26 12:14:42.214333 ip-10-0-129-97 kubenswrapper[2563]: E0526 12:14:42.214315    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dae4f77d9164d0566af3503e711e5a2628c1d0657961cbb1c9841e7bb0ed822\": container with ID starting with 9dae4f77d9164d0566af3503e711e5a2628c1d0657961cbb1c9841e7bb0ed822 not found: ID does not exist" containerID="9dae4f77d9164d0566af3503e711e5a2628c1d0657961cbb1c9841e7bb0ed822"
May 26 12:14:42.214377 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:42.214340    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dae4f77d9164d0566af3503e711e5a2628c1d0657961cbb1c9841e7bb0ed822"} err="failed to get container status \"9dae4f77d9164d0566af3503e711e5a2628c1d0657961cbb1c9841e7bb0ed822\": rpc error: code = NotFound desc = could not find container \"9dae4f77d9164d0566af3503e711e5a2628c1d0657961cbb1c9841e7bb0ed822\": container with ID starting with 9dae4f77d9164d0566af3503e711e5a2628c1d0657961cbb1c9841e7bb0ed822 not found: ID does not exist"
May 26 12:14:42.217373 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:42.217350    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5"]
May 26 12:14:42.221241 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:42.221219    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-52624-predictor-5c7c48695f-2g7f5"]
May 26 12:14:42.428672 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:42.428640    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c560caa-7292-4467-9656-8c68722a1166" path="/var/lib/kubelet/pods/7c560caa-7292-4467-9656-8c68722a1166/volumes"
May 26 12:14:46.761215 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:46.761173    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-52624-856bdd7d4b-swwrx" podUID="bb9ca510-dcf7-4b6a-9065-b2abb57680ff" containerName="splitter-graph-52624" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 12:14:51.761563 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:51.761525    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-52624-856bdd7d4b-swwrx" podUID="bb9ca510-dcf7-4b6a-9065-b2abb57680ff" containerName="splitter-graph-52624" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 12:14:51.761999 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:51.761634    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-52624-856bdd7d4b-swwrx"
May 26 12:14:56.762504 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:14:56.762458    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-52624-856bdd7d4b-swwrx" podUID="bb9ca510-dcf7-4b6a-9065-b2abb57680ff" containerName="splitter-graph-52624" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 12:15:01.761531 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:15:01.761490    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-52624-856bdd7d4b-swwrx" podUID="bb9ca510-dcf7-4b6a-9065-b2abb57680ff" containerName="splitter-graph-52624" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 12:15:06.761095 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:15:06.761054    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-52624-856bdd7d4b-swwrx" podUID="bb9ca510-dcf7-4b6a-9065-b2abb57680ff" containerName="splitter-graph-52624" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 12:15:08.574145 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:15:08.574118    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-52624-856bdd7d4b-swwrx"
May 26 12:15:08.626723 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:15:08.626686    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb9ca510-dcf7-4b6a-9065-b2abb57680ff-openshift-service-ca-bundle\") pod \"bb9ca510-dcf7-4b6a-9065-b2abb57680ff\" (UID: \"bb9ca510-dcf7-4b6a-9065-b2abb57680ff\") "
May 26 12:15:08.626916 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:15:08.626825    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb9ca510-dcf7-4b6a-9065-b2abb57680ff-proxy-tls\") pod \"bb9ca510-dcf7-4b6a-9065-b2abb57680ff\" (UID: \"bb9ca510-dcf7-4b6a-9065-b2abb57680ff\") "
May 26 12:15:08.627105 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:15:08.627082    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb9ca510-dcf7-4b6a-9065-b2abb57680ff-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "bb9ca510-dcf7-4b6a-9065-b2abb57680ff" (UID: "bb9ca510-dcf7-4b6a-9065-b2abb57680ff"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:15:08.629009 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:15:08.628985    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb9ca510-dcf7-4b6a-9065-b2abb57680ff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bb9ca510-dcf7-4b6a-9065-b2abb57680ff" (UID: "bb9ca510-dcf7-4b6a-9065-b2abb57680ff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:15:08.728291 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:15:08.728198    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb9ca510-dcf7-4b6a-9065-b2abb57680ff-proxy-tls\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 12:15:08.728291 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:15:08.728228    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb9ca510-dcf7-4b6a-9065-b2abb57680ff-openshift-service-ca-bundle\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 12:15:09.270517 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:15:09.270480    2563 generic.go:358] "Generic (PLEG): container finished" podID="bb9ca510-dcf7-4b6a-9065-b2abb57680ff" containerID="517ca7f563a4a651ddb6a565665a3cae288ce6f9a8b78c11d71422146ea45045" exitCode=0
May 26 12:15:09.270694 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:15:09.270544    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-52624-856bdd7d4b-swwrx"
May 26 12:15:09.270694 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:15:09.270562    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-52624-856bdd7d4b-swwrx" event={"ID":"bb9ca510-dcf7-4b6a-9065-b2abb57680ff","Type":"ContainerDied","Data":"517ca7f563a4a651ddb6a565665a3cae288ce6f9a8b78c11d71422146ea45045"}
May 26 12:15:09.270694 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:15:09.270601    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-52624-856bdd7d4b-swwrx" event={"ID":"bb9ca510-dcf7-4b6a-9065-b2abb57680ff","Type":"ContainerDied","Data":"0be03c526d059fd526d5273a36e9ca3de318919b4aa541bcd4a7e00e7436e7d7"}
May 26 12:15:09.270694 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:15:09.270616    2563 scope.go:117] "RemoveContainer" containerID="517ca7f563a4a651ddb6a565665a3cae288ce6f9a8b78c11d71422146ea45045"
May 26 12:15:09.279350 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:15:09.279334    2563 scope.go:117] "RemoveContainer" containerID="517ca7f563a4a651ddb6a565665a3cae288ce6f9a8b78c11d71422146ea45045"
May 26 12:15:09.279638 ip-10-0-129-97 kubenswrapper[2563]: E0526 12:15:09.279618    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"517ca7f563a4a651ddb6a565665a3cae288ce6f9a8b78c11d71422146ea45045\": container with ID starting with 517ca7f563a4a651ddb6a565665a3cae288ce6f9a8b78c11d71422146ea45045 not found: ID does not exist" containerID="517ca7f563a4a651ddb6a565665a3cae288ce6f9a8b78c11d71422146ea45045"
May 26 12:15:09.279714 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:15:09.279645    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"517ca7f563a4a651ddb6a565665a3cae288ce6f9a8b78c11d71422146ea45045"} err="failed to get container status \"517ca7f563a4a651ddb6a565665a3cae288ce6f9a8b78c11d71422146ea45045\": rpc error: code = NotFound desc = could not find container \"517ca7f563a4a651ddb6a565665a3cae288ce6f9a8b78c11d71422146ea45045\": container with ID starting with 517ca7f563a4a651ddb6a565665a3cae288ce6f9a8b78c11d71422146ea45045 not found: ID does not exist"
May 26 12:15:09.292265 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:15:09.292237    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-52624-856bdd7d4b-swwrx"]
May 26 12:15:09.295720 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:15:09.295696    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-52624-856bdd7d4b-swwrx"]
May 26 12:15:10.428730 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:15:10.428683    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb9ca510-dcf7-4b6a-9065-b2abb57680ff" path="/var/lib/kubelet/pods/bb9ca510-dcf7-4b6a-9065-b2abb57680ff/volumes"
May 26 12:22:02.368689 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:02.368653    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-152bc-5b8ffd9c8b-kvgnq"]
May 26 12:22:02.371091 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:02.368927    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-152bc-5b8ffd9c8b-kvgnq" podUID="6745d2d9-a284-42e7-8e35-a3e2d249de82" containerName="switch-graph-152bc" containerID="cri-o://ac89b7fb98cf8a67c3a6c9cae4986dbeee49935b0a36909bb64b7750bf8ed568" gracePeriod=30
May 26 12:22:02.681107 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:02.681017    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9"]
May 26 12:22:02.681388 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:02.681347    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9" podUID="7f7d4ea2-96ee-47b8-87f0-2cdd2e684186" containerName="kserve-container" containerID="cri-o://e6a8ef2a98fde879086fae48143fd419e9678fc7c677973b8f22b4190b930611" gracePeriod=30
May 26 12:22:02.681467 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:02.681400    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9" podUID="7f7d4ea2-96ee-47b8-87f0-2cdd2e684186" containerName="kube-rbac-proxy" containerID="cri-o://9f3fb060c32a3e005b80e439bf2d32c0a7230d21788e9af389abb52e0522a783" gracePeriod=30
May 26 12:22:03.433388 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:03.433352    2563 generic.go:358] "Generic (PLEG): container finished" podID="7f7d4ea2-96ee-47b8-87f0-2cdd2e684186" containerID="9f3fb060c32a3e005b80e439bf2d32c0a7230d21788e9af389abb52e0522a783" exitCode=2
May 26 12:22:03.433770 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:03.433430    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9" event={"ID":"7f7d4ea2-96ee-47b8-87f0-2cdd2e684186","Type":"ContainerDied","Data":"9f3fb060c32a3e005b80e439bf2d32c0a7230d21788e9af389abb52e0522a783"}
May 26 12:22:03.588196 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:03.588163    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b2kcn/must-gather-2x5jj"]
May 26 12:22:03.588449 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:03.588437    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb9ca510-dcf7-4b6a-9065-b2abb57680ff" containerName="splitter-graph-52624"
May 26 12:22:03.588502 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:03.588451    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb9ca510-dcf7-4b6a-9065-b2abb57680ff" containerName="splitter-graph-52624"
May 26 12:22:03.588502 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:03.588460    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c560caa-7292-4467-9656-8c68722a1166" containerName="kserve-container"
May 26 12:22:03.588502 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:03.588466    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c560caa-7292-4467-9656-8c68722a1166" containerName="kserve-container"
May 26 12:22:03.588502 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:03.588477    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c560caa-7292-4467-9656-8c68722a1166" containerName="kube-rbac-proxy"
May 26 12:22:03.588502 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:03.588483    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c560caa-7292-4467-9656-8c68722a1166" containerName="kube-rbac-proxy"
May 26 12:22:03.588680 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:03.588536    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="7c560caa-7292-4467-9656-8c68722a1166" containerName="kserve-container"
May 26 12:22:03.588680 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:03.588543    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="bb9ca510-dcf7-4b6a-9065-b2abb57680ff" containerName="splitter-graph-52624"
May 26 12:22:03.588680 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:03.588552    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="7c560caa-7292-4467-9656-8c68722a1166" containerName="kube-rbac-proxy"
May 26 12:22:03.591495 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:03.591468    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b2kcn/must-gather-2x5jj"
May 26 12:22:03.594266 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:03.594243    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b2kcn\"/\"kube-root-ca.crt\""
May 26 12:22:03.595348 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:03.595327    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-b2kcn\"/\"default-dockercfg-gsx4r\""
May 26 12:22:03.595446 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:03.595335    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b2kcn\"/\"openshift-service-ca.crt\""
May 26 12:22:03.614873 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:03.614833    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b2kcn/must-gather-2x5jj"]
May 26 12:22:03.708607 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:03.708513    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8-must-gather-output\") pod \"must-gather-2x5jj\" (UID: \"cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8\") " pod="openshift-must-gather-b2kcn/must-gather-2x5jj"
May 26 12:22:03.708607 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:03.708560    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28h4l\" (UniqueName: \"kubernetes.io/projected/cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8-kube-api-access-28h4l\") pod \"must-gather-2x5jj\" (UID: \"cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8\") " pod="openshift-must-gather-b2kcn/must-gather-2x5jj"
May 26 12:22:03.809959 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:03.809902    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8-must-gather-output\") pod \"must-gather-2x5jj\" (UID: \"cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8\") " pod="openshift-must-gather-b2kcn/must-gather-2x5jj"
May 26 12:22:03.809959 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:03.809960    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-28h4l\" (UniqueName: \"kubernetes.io/projected/cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8-kube-api-access-28h4l\") pod \"must-gather-2x5jj\" (UID: \"cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8\") " pod="openshift-must-gather-b2kcn/must-gather-2x5jj"
May 26 12:22:03.810262 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:03.810241    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8-must-gather-output\") pod \"must-gather-2x5jj\" (UID: \"cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8\") " pod="openshift-must-gather-b2kcn/must-gather-2x5jj"
May 26 12:22:03.820096 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:03.820064    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28h4l\" (UniqueName: \"kubernetes.io/projected/cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8-kube-api-access-28h4l\") pod \"must-gather-2x5jj\" (UID: \"cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8\") " pod="openshift-must-gather-b2kcn/must-gather-2x5jj"
May 26 12:22:03.913189 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:03.913135    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b2kcn/must-gather-2x5jj"
May 26 12:22:04.049718 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:04.049681    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b2kcn/must-gather-2x5jj"]
May 26 12:22:04.052758 ip-10-0-129-97 kubenswrapper[2563]: W0526 12:22:04.052714    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcae72dd5_2c2e_4463_9da1_e67e7d5c5dc8.slice/crio-81302d17437cfd69a547047e3097d0a6c0631cd7d56484eb5a3875fecbaa8866 WatchSource:0}: Error finding container 81302d17437cfd69a547047e3097d0a6c0631cd7d56484eb5a3875fecbaa8866: Status 404 returned error can't find the container with id 81302d17437cfd69a547047e3097d0a6c0631cd7d56484eb5a3875fecbaa8866
May 26 12:22:04.054449 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:04.054433    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 12:22:04.436442 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:04.436405    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b2kcn/must-gather-2x5jj" event={"ID":"cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8","Type":"ContainerStarted","Data":"81302d17437cfd69a547047e3097d0a6c0631cd7d56484eb5a3875fecbaa8866"}
May 26 12:22:05.633462 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:05.633410    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-152bc-5b8ffd9c8b-kvgnq" podUID="6745d2d9-a284-42e7-8e35-a3e2d249de82" containerName="switch-graph-152bc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 12:22:06.059283 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:06.059248    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9"
May 26 12:22:06.132228 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:06.132123    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f7d4ea2-96ee-47b8-87f0-2cdd2e684186-proxy-tls\") pod \"7f7d4ea2-96ee-47b8-87f0-2cdd2e684186\" (UID: \"7f7d4ea2-96ee-47b8-87f0-2cdd2e684186\") "
May 26 12:22:06.132228 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:06.132182    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-152bc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f7d4ea2-96ee-47b8-87f0-2cdd2e684186-error-404-isvc-152bc-kube-rbac-proxy-sar-config\") pod \"7f7d4ea2-96ee-47b8-87f0-2cdd2e684186\" (UID: \"7f7d4ea2-96ee-47b8-87f0-2cdd2e684186\") "
May 26 12:22:06.132228 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:06.132218    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bczbr\" (UniqueName: \"kubernetes.io/projected/7f7d4ea2-96ee-47b8-87f0-2cdd2e684186-kube-api-access-bczbr\") pod \"7f7d4ea2-96ee-47b8-87f0-2cdd2e684186\" (UID: \"7f7d4ea2-96ee-47b8-87f0-2cdd2e684186\") "
May 26 12:22:06.133010 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:06.132977    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f7d4ea2-96ee-47b8-87f0-2cdd2e684186-error-404-isvc-152bc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-152bc-kube-rbac-proxy-sar-config") pod "7f7d4ea2-96ee-47b8-87f0-2cdd2e684186" (UID: "7f7d4ea2-96ee-47b8-87f0-2cdd2e684186"). InnerVolumeSpecName "error-404-isvc-152bc-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:22:06.135196 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:06.135172    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f7d4ea2-96ee-47b8-87f0-2cdd2e684186-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7f7d4ea2-96ee-47b8-87f0-2cdd2e684186" (UID: "7f7d4ea2-96ee-47b8-87f0-2cdd2e684186"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:22:06.135692 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:06.135665    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f7d4ea2-96ee-47b8-87f0-2cdd2e684186-kube-api-access-bczbr" (OuterVolumeSpecName: "kube-api-access-bczbr") pod "7f7d4ea2-96ee-47b8-87f0-2cdd2e684186" (UID: "7f7d4ea2-96ee-47b8-87f0-2cdd2e684186"). InnerVolumeSpecName "kube-api-access-bczbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:22:06.233197 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:06.233110    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f7d4ea2-96ee-47b8-87f0-2cdd2e684186-proxy-tls\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 12:22:06.233197 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:06.233150    2563 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-152bc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f7d4ea2-96ee-47b8-87f0-2cdd2e684186-error-404-isvc-152bc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 12:22:06.233197 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:06.233162    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bczbr\" (UniqueName: \"kubernetes.io/projected/7f7d4ea2-96ee-47b8-87f0-2cdd2e684186-kube-api-access-bczbr\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 12:22:06.446175 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:06.446137    2563 generic.go:358] "Generic (PLEG): container finished" podID="7f7d4ea2-96ee-47b8-87f0-2cdd2e684186" containerID="e6a8ef2a98fde879086fae48143fd419e9678fc7c677973b8f22b4190b930611" exitCode=0
May 26 12:22:06.446326 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:06.446214    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9" event={"ID":"7f7d4ea2-96ee-47b8-87f0-2cdd2e684186","Type":"ContainerDied","Data":"e6a8ef2a98fde879086fae48143fd419e9678fc7c677973b8f22b4190b930611"}
May 26 12:22:06.446326 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:06.446241    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9"
May 26 12:22:06.446326 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:06.446265    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9" event={"ID":"7f7d4ea2-96ee-47b8-87f0-2cdd2e684186","Type":"ContainerDied","Data":"64cc1bd465e693b966791b22e72fb22c407f93de4d6ffc7e3b7daa7e61afa92e"}
May 26 12:22:06.446326 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:06.446292    2563 scope.go:117] "RemoveContainer" containerID="9f3fb060c32a3e005b80e439bf2d32c0a7230d21788e9af389abb52e0522a783"
May 26 12:22:06.482954 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:06.482907    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9"]
May 26 12:22:06.488457 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:06.488380    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-152bc-predictor-6b9cb766b7-4krt9"]
May 26 12:22:08.429266 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:08.429227    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f7d4ea2-96ee-47b8-87f0-2cdd2e684186" path="/var/lib/kubelet/pods/7f7d4ea2-96ee-47b8-87f0-2cdd2e684186/volumes"
May 26 12:22:09.245873 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:09.245553    2563 scope.go:117] "RemoveContainer" containerID="e6a8ef2a98fde879086fae48143fd419e9678fc7c677973b8f22b4190b930611"
May 26 12:22:10.155803 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:10.155764    2563 scope.go:117] "RemoveContainer" containerID="9f3fb060c32a3e005b80e439bf2d32c0a7230d21788e9af389abb52e0522a783"
May 26 12:22:10.156171 ip-10-0-129-97 kubenswrapper[2563]: E0526 12:22:10.156147    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f3fb060c32a3e005b80e439bf2d32c0a7230d21788e9af389abb52e0522a783\": container with ID starting with 9f3fb060c32a3e005b80e439bf2d32c0a7230d21788e9af389abb52e0522a783 not found: ID does not exist" containerID="9f3fb060c32a3e005b80e439bf2d32c0a7230d21788e9af389abb52e0522a783"
May 26 12:22:10.156218 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:10.156182    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f3fb060c32a3e005b80e439bf2d32c0a7230d21788e9af389abb52e0522a783"} err="failed to get container status \"9f3fb060c32a3e005b80e439bf2d32c0a7230d21788e9af389abb52e0522a783\": rpc error: code = NotFound desc = could not find container \"9f3fb060c32a3e005b80e439bf2d32c0a7230d21788e9af389abb52e0522a783\": container with ID starting with 9f3fb060c32a3e005b80e439bf2d32c0a7230d21788e9af389abb52e0522a783 not found: ID does not exist"
May 26 12:22:10.156218 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:10.156203    2563 scope.go:117] "RemoveContainer" containerID="e6a8ef2a98fde879086fae48143fd419e9678fc7c677973b8f22b4190b930611"
May 26 12:22:10.156460 ip-10-0-129-97 kubenswrapper[2563]: E0526 12:22:10.156440    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6a8ef2a98fde879086fae48143fd419e9678fc7c677973b8f22b4190b930611\": container with ID starting with e6a8ef2a98fde879086fae48143fd419e9678fc7c677973b8f22b4190b930611 not found: ID does not exist" containerID="e6a8ef2a98fde879086fae48143fd419e9678fc7c677973b8f22b4190b930611"
May 26 12:22:10.156514 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:10.156464    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6a8ef2a98fde879086fae48143fd419e9678fc7c677973b8f22b4190b930611"} err="failed to get container status \"e6a8ef2a98fde879086fae48143fd419e9678fc7c677973b8f22b4190b930611\": rpc error: code = NotFound desc = could not find container \"e6a8ef2a98fde879086fae48143fd419e9678fc7c677973b8f22b4190b930611\": container with ID starting with e6a8ef2a98fde879086fae48143fd419e9678fc7c677973b8f22b4190b930611 not found: ID does not exist"
May 26 12:22:10.471521 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:10.471495    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b2kcn/must-gather-2x5jj" event={"ID":"cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8","Type":"ContainerStarted","Data":"0c4c52fb28af25736da0c4988ae7d6dbbb0546169fc0105c31d75f76b6d80aac"}
May 26 12:22:10.632098 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:10.632063    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-152bc-5b8ffd9c8b-kvgnq" podUID="6745d2d9-a284-42e7-8e35-a3e2d249de82" containerName="switch-graph-152bc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 12:22:11.475671 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:11.475637    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b2kcn/must-gather-2x5jj" event={"ID":"cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8","Type":"ContainerStarted","Data":"e540c340d729601d670dcbb298d051a526c2dafa1314a5d40dd58fe1cd2d2277"}
May 26 12:22:11.493196 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:11.493144    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-b2kcn/must-gather-2x5jj" podStartSLOduration=2.276123314 podStartE2EDuration="8.493128936s" podCreationTimestamp="2026-05-26 12:22:03 +0000 UTC" firstStartedPulling="2026-05-26 12:22:04.054560915 +0000 UTC m=+3456.246003080" lastFinishedPulling="2026-05-26 12:22:10.271566537 +0000 UTC m=+3462.463008702" observedRunningTime="2026-05-26 12:22:11.491760822 +0000 UTC m=+3463.683203015" watchObservedRunningTime="2026-05-26 12:22:11.493128936 +0000 UTC m=+3463.684571123"
May 26 12:22:15.632831 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:15.632766    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-152bc-5b8ffd9c8b-kvgnq" podUID="6745d2d9-a284-42e7-8e35-a3e2d249de82" containerName="switch-graph-152bc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 12:22:15.633274 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:15.632901    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-152bc-5b8ffd9c8b-kvgnq"
May 26 12:22:19.617688 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:19.617649    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-152bc-5b8ffd9c8b-kvgnq_6745d2d9-a284-42e7-8e35-a3e2d249de82/switch-graph-152bc/0.log"
May 26 12:22:20.454669 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:20.454638    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-152bc-5b8ffd9c8b-kvgnq_6745d2d9-a284-42e7-8e35-a3e2d249de82/switch-graph-152bc/0.log"
May 26 12:22:20.631819 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:20.631755    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-152bc-5b8ffd9c8b-kvgnq" podUID="6745d2d9-a284-42e7-8e35-a3e2d249de82" containerName="switch-graph-152bc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 12:22:21.270687 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:21.270652    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-152bc-5b8ffd9c8b-kvgnq_6745d2d9-a284-42e7-8e35-a3e2d249de82/switch-graph-152bc/0.log"
May 26 12:22:22.068408 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:22.068350    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-152bc-5b8ffd9c8b-kvgnq_6745d2d9-a284-42e7-8e35-a3e2d249de82/switch-graph-152bc/0.log"
May 26 12:22:22.904748 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:22.904709    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-152bc-5b8ffd9c8b-kvgnq_6745d2d9-a284-42e7-8e35-a3e2d249de82/switch-graph-152bc/0.log"
May 26 12:22:23.719217 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:23.719188    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-152bc-5b8ffd9c8b-kvgnq_6745d2d9-a284-42e7-8e35-a3e2d249de82/switch-graph-152bc/0.log"
May 26 12:22:24.522436 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:24.522410    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-152bc-5b8ffd9c8b-kvgnq_6745d2d9-a284-42e7-8e35-a3e2d249de82/switch-graph-152bc/0.log"
May 26 12:22:25.311437 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:25.311410    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-152bc-5b8ffd9c8b-kvgnq_6745d2d9-a284-42e7-8e35-a3e2d249de82/switch-graph-152bc/0.log"
May 26 12:22:25.633292 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:25.633238    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-152bc-5b8ffd9c8b-kvgnq" podUID="6745d2d9-a284-42e7-8e35-a3e2d249de82" containerName="switch-graph-152bc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 12:22:26.103349 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:26.103323    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-152bc-5b8ffd9c8b-kvgnq_6745d2d9-a284-42e7-8e35-a3e2d249de82/switch-graph-152bc/0.log"
May 26 12:22:26.893556 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:26.893525    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-152bc-5b8ffd9c8b-kvgnq_6745d2d9-a284-42e7-8e35-a3e2d249de82/switch-graph-152bc/0.log"
May 26 12:22:27.691528 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:27.691496    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-152bc-5b8ffd9c8b-kvgnq_6745d2d9-a284-42e7-8e35-a3e2d249de82/switch-graph-152bc/0.log"
May 26 12:22:28.492068 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:28.492038    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-152bc-5b8ffd9c8b-kvgnq_6745d2d9-a284-42e7-8e35-a3e2d249de82/switch-graph-152bc/0.log"
May 26 12:22:30.632080 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:30.632041    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-152bc-5b8ffd9c8b-kvgnq" podUID="6745d2d9-a284-42e7-8e35-a3e2d249de82" containerName="switch-graph-152bc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 12:22:32.510188 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:32.510163    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-152bc-5b8ffd9c8b-kvgnq"
May 26 12:22:32.543847 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:32.543810    2563 generic.go:358] "Generic (PLEG): container finished" podID="cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8" containerID="0c4c52fb28af25736da0c4988ae7d6dbbb0546169fc0105c31d75f76b6d80aac" exitCode=0
May 26 12:22:32.544042 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:32.543884    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b2kcn/must-gather-2x5jj" event={"ID":"cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8","Type":"ContainerDied","Data":"0c4c52fb28af25736da0c4988ae7d6dbbb0546169fc0105c31d75f76b6d80aac"}
May 26 12:22:32.544276 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:32.544258    2563 scope.go:117] "RemoveContainer" containerID="0c4c52fb28af25736da0c4988ae7d6dbbb0546169fc0105c31d75f76b6d80aac"
May 26 12:22:32.545121 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:32.545100    2563 generic.go:358] "Generic (PLEG): container finished" podID="6745d2d9-a284-42e7-8e35-a3e2d249de82" containerID="ac89b7fb98cf8a67c3a6c9cae4986dbeee49935b0a36909bb64b7750bf8ed568" exitCode=0
May 26 12:22:32.545242 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:32.545130    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-152bc-5b8ffd9c8b-kvgnq" event={"ID":"6745d2d9-a284-42e7-8e35-a3e2d249de82","Type":"ContainerDied","Data":"ac89b7fb98cf8a67c3a6c9cae4986dbeee49935b0a36909bb64b7750bf8ed568"}
May 26 12:22:32.545242 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:32.545152    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-152bc-5b8ffd9c8b-kvgnq" event={"ID":"6745d2d9-a284-42e7-8e35-a3e2d249de82","Type":"ContainerDied","Data":"9b96eb0530e2999311197cfa4fb3e9ad1ef69c7a1b6b0fe6e89b5dc06b937d59"}
May 26 12:22:32.545242 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:32.545165    2563 scope.go:117] "RemoveContainer" containerID="ac89b7fb98cf8a67c3a6c9cae4986dbeee49935b0a36909bb64b7750bf8ed568"
May 26 12:22:32.545242 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:32.545211    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-152bc-5b8ffd9c8b-kvgnq"
May 26 12:22:32.553200 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:32.553179    2563 scope.go:117] "RemoveContainer" containerID="ac89b7fb98cf8a67c3a6c9cae4986dbeee49935b0a36909bb64b7750bf8ed568"
May 26 12:22:32.553456 ip-10-0-129-97 kubenswrapper[2563]: E0526 12:22:32.553437    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac89b7fb98cf8a67c3a6c9cae4986dbeee49935b0a36909bb64b7750bf8ed568\": container with ID starting with ac89b7fb98cf8a67c3a6c9cae4986dbeee49935b0a36909bb64b7750bf8ed568 not found: ID does not exist" containerID="ac89b7fb98cf8a67c3a6c9cae4986dbeee49935b0a36909bb64b7750bf8ed568"
May 26 12:22:32.553498 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:32.553465    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac89b7fb98cf8a67c3a6c9cae4986dbeee49935b0a36909bb64b7750bf8ed568"} err="failed to get container status \"ac89b7fb98cf8a67c3a6c9cae4986dbeee49935b0a36909bb64b7750bf8ed568\": rpc error: code = NotFound desc = could not find container \"ac89b7fb98cf8a67c3a6c9cae4986dbeee49935b0a36909bb64b7750bf8ed568\": container with ID starting with ac89b7fb98cf8a67c3a6c9cae4986dbeee49935b0a36909bb64b7750bf8ed568 not found: ID does not exist"
May 26 12:22:32.655956 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:32.655864    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6745d2d9-a284-42e7-8e35-a3e2d249de82-proxy-tls\") pod \"6745d2d9-a284-42e7-8e35-a3e2d249de82\" (UID: \"6745d2d9-a284-42e7-8e35-a3e2d249de82\") "
May 26 12:22:32.655956 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:32.655910    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6745d2d9-a284-42e7-8e35-a3e2d249de82-openshift-service-ca-bundle\") pod \"6745d2d9-a284-42e7-8e35-a3e2d249de82\" (UID: \"6745d2d9-a284-42e7-8e35-a3e2d249de82\") "
May 26 12:22:32.656287 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:32.656261    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6745d2d9-a284-42e7-8e35-a3e2d249de82-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "6745d2d9-a284-42e7-8e35-a3e2d249de82" (UID: "6745d2d9-a284-42e7-8e35-a3e2d249de82"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:22:32.658199 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:32.658181    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6745d2d9-a284-42e7-8e35-a3e2d249de82-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6745d2d9-a284-42e7-8e35-a3e2d249de82" (UID: "6745d2d9-a284-42e7-8e35-a3e2d249de82"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:22:32.756963 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:32.756911    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6745d2d9-a284-42e7-8e35-a3e2d249de82-proxy-tls\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 12:22:32.756963 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:32.756959    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6745d2d9-a284-42e7-8e35-a3e2d249de82-openshift-service-ca-bundle\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 12:22:32.873971 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:32.873936    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-152bc-5b8ffd9c8b-kvgnq"]
May 26 12:22:32.877338 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:32.877311    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-152bc-5b8ffd9c8b-kvgnq"]
May 26 12:22:33.393453 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:33.393428    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-b2kcn_must-gather-2x5jj_cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8/gather/0.log"
May 26 12:22:33.990395 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:33.990357    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zm7c5/must-gather-jzpwd"]
May 26 12:22:33.990754 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:33.990627    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6745d2d9-a284-42e7-8e35-a3e2d249de82" containerName="switch-graph-152bc"
May 26 12:22:33.990754 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:33.990639    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6745d2d9-a284-42e7-8e35-a3e2d249de82" containerName="switch-graph-152bc"
May 26 12:22:33.990754 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:33.990651    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f7d4ea2-96ee-47b8-87f0-2cdd2e684186" containerName="kserve-container"
May 26 12:22:33.990754 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:33.990656    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f7d4ea2-96ee-47b8-87f0-2cdd2e684186" containerName="kserve-container"
May 26 12:22:33.990754 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:33.990664    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f7d4ea2-96ee-47b8-87f0-2cdd2e684186" containerName="kube-rbac-proxy"
May 26 12:22:33.990754 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:33.990670    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f7d4ea2-96ee-47b8-87f0-2cdd2e684186" containerName="kube-rbac-proxy"
May 26 12:22:33.990754 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:33.990710    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="7f7d4ea2-96ee-47b8-87f0-2cdd2e684186" containerName="kserve-container"
May 26 12:22:33.990754 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:33.990720    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="6745d2d9-a284-42e7-8e35-a3e2d249de82" containerName="switch-graph-152bc"
May 26 12:22:33.990754 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:33.990726    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="7f7d4ea2-96ee-47b8-87f0-2cdd2e684186" containerName="kube-rbac-proxy"
May 26 12:22:33.993747 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:33.993730    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zm7c5/must-gather-jzpwd"
May 26 12:22:33.996358 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:33.996325    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zm7c5\"/\"openshift-service-ca.crt\""
May 26 12:22:33.996494 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:33.996437    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zm7c5\"/\"default-dockercfg-kbnnp\""
May 26 12:22:33.997590 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:33.997572    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zm7c5\"/\"kube-root-ca.crt\""
May 26 12:22:34.001465 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:34.001444    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zm7c5/must-gather-jzpwd"]
May 26 12:22:34.069132 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:34.069079    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/faf639d9-1ab1-448d-9ca7-cbfadc7d42de-must-gather-output\") pod \"must-gather-jzpwd\" (UID: \"faf639d9-1ab1-448d-9ca7-cbfadc7d42de\") " pod="openshift-must-gather-zm7c5/must-gather-jzpwd"
May 26 12:22:34.069323 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:34.069150    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzps9\" (UniqueName: \"kubernetes.io/projected/faf639d9-1ab1-448d-9ca7-cbfadc7d42de-kube-api-access-lzps9\") pod \"must-gather-jzpwd\" (UID: \"faf639d9-1ab1-448d-9ca7-cbfadc7d42de\") " pod="openshift-must-gather-zm7c5/must-gather-jzpwd"
May 26 12:22:34.170295 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:34.170255    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/faf639d9-1ab1-448d-9ca7-cbfadc7d42de-must-gather-output\") pod \"must-gather-jzpwd\" (UID: \"faf639d9-1ab1-448d-9ca7-cbfadc7d42de\") " pod="openshift-must-gather-zm7c5/must-gather-jzpwd"
May 26 12:22:34.170295 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:34.170301    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lzps9\" (UniqueName: \"kubernetes.io/projected/faf639d9-1ab1-448d-9ca7-cbfadc7d42de-kube-api-access-lzps9\") pod \"must-gather-jzpwd\" (UID: \"faf639d9-1ab1-448d-9ca7-cbfadc7d42de\") " pod="openshift-must-gather-zm7c5/must-gather-jzpwd"
May 26 12:22:34.170656 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:34.170633    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/faf639d9-1ab1-448d-9ca7-cbfadc7d42de-must-gather-output\") pod \"must-gather-jzpwd\" (UID: \"faf639d9-1ab1-448d-9ca7-cbfadc7d42de\") " pod="openshift-must-gather-zm7c5/must-gather-jzpwd"
May 26 12:22:34.178811 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:34.178768    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzps9\" (UniqueName: \"kubernetes.io/projected/faf639d9-1ab1-448d-9ca7-cbfadc7d42de-kube-api-access-lzps9\") pod \"must-gather-jzpwd\" (UID: \"faf639d9-1ab1-448d-9ca7-cbfadc7d42de\") " pod="openshift-must-gather-zm7c5/must-gather-jzpwd"
May 26 12:22:34.304013 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:34.303906    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zm7c5/must-gather-jzpwd"
May 26 12:22:34.434194 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:34.434150    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6745d2d9-a284-42e7-8e35-a3e2d249de82" path="/var/lib/kubelet/pods/6745d2d9-a284-42e7-8e35-a3e2d249de82/volumes"
May 26 12:22:34.434437 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:34.434419    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zm7c5/must-gather-jzpwd"]
May 26 12:22:34.436184 ip-10-0-129-97 kubenswrapper[2563]: W0526 12:22:34.436158    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfaf639d9_1ab1_448d_9ca7_cbfadc7d42de.slice/crio-7d4b9336ad9721ae2452ab1046af5a16abc6d1a78f288e06254d00e30f2067bb WatchSource:0}: Error finding container 7d4b9336ad9721ae2452ab1046af5a16abc6d1a78f288e06254d00e30f2067bb: Status 404 returned error can't find the container with id 7d4b9336ad9721ae2452ab1046af5a16abc6d1a78f288e06254d00e30f2067bb
May 26 12:22:34.553752 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:34.553716    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zm7c5/must-gather-jzpwd" event={"ID":"faf639d9-1ab1-448d-9ca7-cbfadc7d42de","Type":"ContainerStarted","Data":"7d4b9336ad9721ae2452ab1046af5a16abc6d1a78f288e06254d00e30f2067bb"}
May 26 12:22:35.558671 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:35.558622    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zm7c5/must-gather-jzpwd" event={"ID":"faf639d9-1ab1-448d-9ca7-cbfadc7d42de","Type":"ContainerStarted","Data":"388cf2574c8435b9231dccddae1df8f1c7cc96c0a520ce820eb1b0f67fe2a3f8"}
May 26 12:22:35.558671 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:35.558675    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zm7c5/must-gather-jzpwd" event={"ID":"faf639d9-1ab1-448d-9ca7-cbfadc7d42de","Type":"ContainerStarted","Data":"8d63dd13e5ab0337aea3c30373888365e14ee5db1a6867cf9a75a3c42ea2869b"}
May 26 12:22:35.575954 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:35.575893    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-zm7c5/must-gather-jzpwd" podStartSLOduration=1.661535931 podStartE2EDuration="2.575873384s" podCreationTimestamp="2026-05-26 12:22:33 +0000 UTC" firstStartedPulling="2026-05-26 12:22:34.437664751 +0000 UTC m=+3486.629106916" lastFinishedPulling="2026-05-26 12:22:35.352002204 +0000 UTC m=+3487.543444369" observedRunningTime="2026-05-26 12:22:35.574738472 +0000 UTC m=+3487.766180659" watchObservedRunningTime="2026-05-26 12:22:35.575873384 +0000 UTC m=+3487.767315572"
May 26 12:22:36.845609 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:36.845574    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-x9ccg_04c86725-6167-4dfc-b259-f7a1423c418d/global-pull-secret-syncer/0.log"
May 26 12:22:36.923140 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:36.923106    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xtkmg_b4769eef-c61f-4722-96a1-8461e3cc4801/konnectivity-agent/0.log"
May 26 12:22:36.983646 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:36.983620    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-97.ec2.internal_7b05a01f2c158d2611520e5fe1f839c3/haproxy/0.log"
May 26 12:22:38.834681 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:38.834637    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-b2kcn/must-gather-2x5jj"]
May 26 12:22:38.835367 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:38.834942    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-b2kcn/must-gather-2x5jj" podUID="cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8" containerName="copy" containerID="cri-o://e540c340d729601d670dcbb298d051a526c2dafa1314a5d40dd58fe1cd2d2277" gracePeriod=2
May 26 12:22:38.837457 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:38.837406    2563 status_manager.go:895] "Failed to get status for pod" podUID="cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8" pod="openshift-must-gather-b2kcn/must-gather-2x5jj" err="pods \"must-gather-2x5jj\" is forbidden: User \"system:node:ip-10-0-129-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-b2kcn\": no relationship found between node 'ip-10-0-129-97.ec2.internal' and this object"
May 26 12:22:38.839390 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:38.839363    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-b2kcn/must-gather-2x5jj"]
May 26 12:22:39.208796 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:39.206059    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-b2kcn_must-gather-2x5jj_cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8/copy/0.log"
May 26 12:22:39.208796 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:39.206517    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b2kcn/must-gather-2x5jj"
May 26 12:22:39.213610 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:39.211080    2563 status_manager.go:895] "Failed to get status for pod" podUID="cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8" pod="openshift-must-gather-b2kcn/must-gather-2x5jj" err="pods \"must-gather-2x5jj\" is forbidden: User \"system:node:ip-10-0-129-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-b2kcn\": no relationship found between node 'ip-10-0-129-97.ec2.internal' and this object"
May 26 12:22:39.326066 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:39.326025    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8-must-gather-output\") pod \"cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8\" (UID: \"cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8\") "
May 26 12:22:39.326350 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:39.326126    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28h4l\" (UniqueName: \"kubernetes.io/projected/cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8-kube-api-access-28h4l\") pod \"cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8\" (UID: \"cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8\") "
May 26 12:22:39.329591 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:39.329552    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8" (UID: "cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:22:39.337260 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:39.337219    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8-kube-api-access-28h4l" (OuterVolumeSpecName: "kube-api-access-28h4l") pod "cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8" (UID: "cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8"). InnerVolumeSpecName "kube-api-access-28h4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:22:39.427600 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:39.427510    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28h4l\" (UniqueName: \"kubernetes.io/projected/cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8-kube-api-access-28h4l\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 12:22:39.427600 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:39.427552    2563 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8-must-gather-output\") on node \"ip-10-0-129-97.ec2.internal\" DevicePath \"\""
May 26 12:22:39.575163 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:39.575135    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-b2kcn_must-gather-2x5jj_cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8/copy/0.log"
May 26 12:22:39.575790 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:39.575746    2563 generic.go:358] "Generic (PLEG): container finished" podID="cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8" containerID="e540c340d729601d670dcbb298d051a526c2dafa1314a5d40dd58fe1cd2d2277" exitCode=143
May 26 12:22:39.576026 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:39.576013    2563 scope.go:117] "RemoveContainer" containerID="e540c340d729601d670dcbb298d051a526c2dafa1314a5d40dd58fe1cd2d2277"
May 26 12:22:39.576248 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:39.576235    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b2kcn/must-gather-2x5jj"
May 26 12:22:39.581959 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:39.581911    2563 status_manager.go:895] "Failed to get status for pod" podUID="cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8" pod="openshift-must-gather-b2kcn/must-gather-2x5jj" err="pods \"must-gather-2x5jj\" is forbidden: User \"system:node:ip-10-0-129-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-b2kcn\": no relationship found between node 'ip-10-0-129-97.ec2.internal' and this object"
May 26 12:22:39.595687 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:39.594648    2563 scope.go:117] "RemoveContainer" containerID="0c4c52fb28af25736da0c4988ae7d6dbbb0546169fc0105c31d75f76b6d80aac"
May 26 12:22:39.609364 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:39.609322    2563 status_manager.go:895] "Failed to get status for pod" podUID="cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8" pod="openshift-must-gather-b2kcn/must-gather-2x5jj" err="pods \"must-gather-2x5jj\" is forbidden: User \"system:node:ip-10-0-129-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-b2kcn\": no relationship found between node 'ip-10-0-129-97.ec2.internal' and this object"
May 26 12:22:39.616573 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:39.616550    2563 scope.go:117] "RemoveContainer" containerID="e540c340d729601d670dcbb298d051a526c2dafa1314a5d40dd58fe1cd2d2277"
May 26 12:22:39.617143 ip-10-0-129-97 kubenswrapper[2563]: E0526 12:22:39.617083    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e540c340d729601d670dcbb298d051a526c2dafa1314a5d40dd58fe1cd2d2277\": container with ID starting with e540c340d729601d670dcbb298d051a526c2dafa1314a5d40dd58fe1cd2d2277 not found: ID does not exist" containerID="e540c340d729601d670dcbb298d051a526c2dafa1314a5d40dd58fe1cd2d2277"
May 26 12:22:39.617359 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:39.617281    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e540c340d729601d670dcbb298d051a526c2dafa1314a5d40dd58fe1cd2d2277"} err="failed to get container status \"e540c340d729601d670dcbb298d051a526c2dafa1314a5d40dd58fe1cd2d2277\": rpc error: code = NotFound desc = could not find container \"e540c340d729601d670dcbb298d051a526c2dafa1314a5d40dd58fe1cd2d2277\": container with ID starting with e540c340d729601d670dcbb298d051a526c2dafa1314a5d40dd58fe1cd2d2277 not found: ID does not exist"
May 26 12:22:39.617359 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:39.617322    2563 scope.go:117] "RemoveContainer" containerID="0c4c52fb28af25736da0c4988ae7d6dbbb0546169fc0105c31d75f76b6d80aac"
May 26 12:22:39.617976 ip-10-0-129-97 kubenswrapper[2563]: E0526 12:22:39.617939    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c4c52fb28af25736da0c4988ae7d6dbbb0546169fc0105c31d75f76b6d80aac\": container with ID starting with 0c4c52fb28af25736da0c4988ae7d6dbbb0546169fc0105c31d75f76b6d80aac not found: ID does not exist" containerID="0c4c52fb28af25736da0c4988ae7d6dbbb0546169fc0105c31d75f76b6d80aac"
May 26 12:22:39.618158 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:39.618131    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c4c52fb28af25736da0c4988ae7d6dbbb0546169fc0105c31d75f76b6d80aac"} err="failed to get container status \"0c4c52fb28af25736da0c4988ae7d6dbbb0546169fc0105c31d75f76b6d80aac\": rpc error: code = NotFound desc = could not find container \"0c4c52fb28af25736da0c4988ae7d6dbbb0546169fc0105c31d75f76b6d80aac\": container with ID starting with 0c4c52fb28af25736da0c4988ae7d6dbbb0546169fc0105c31d75f76b6d80aac not found: ID does not exist"
May 26 12:22:40.430215 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:40.430168    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8" path="/var/lib/kubelet/pods/cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8/volumes"
May 26 12:22:40.650657 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:40.650630    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-79c798ccf6-c9h4r_8c811a1c-f0c8-4f89-8c13-e9adf2a76ba5/metrics-server/0.log"
May 26 12:22:40.680200 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:40.680170    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-665c9c8584-8t9bl_741322d5-b3e1-4de0-9233-976c7fb37c58/monitoring-plugin/0.log"
May 26 12:22:40.894306 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:40.894277    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c5zsn_cf8f06cf-5a6e-45cd-b44d-2817b037ed24/node-exporter/0.log"
May 26 12:22:40.916451 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:40.916371    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c5zsn_cf8f06cf-5a6e-45cd-b44d-2817b037ed24/kube-rbac-proxy/0.log"
May 26 12:22:40.938763 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:40.938734    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c5zsn_cf8f06cf-5a6e-45cd-b44d-2817b037ed24/init-textfile/0.log"
May 26 12:22:41.061102 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:41.061071    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_924bcc1d-1874-49bd-8348-8a360af67584/prometheus/0.log"
May 26 12:22:41.079237 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:41.079209    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_924bcc1d-1874-49bd-8348-8a360af67584/config-reloader/0.log"
May 26 12:22:41.102480 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:41.102438    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_924bcc1d-1874-49bd-8348-8a360af67584/thanos-sidecar/0.log"
May 26 12:22:41.125348 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:41.125323    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_924bcc1d-1874-49bd-8348-8a360af67584/kube-rbac-proxy-web/0.log"
May 26 12:22:41.148736 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:41.148675    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_924bcc1d-1874-49bd-8348-8a360af67584/kube-rbac-proxy/0.log"
May 26 12:22:41.172580 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:41.172504    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_924bcc1d-1874-49bd-8348-8a360af67584/kube-rbac-proxy-thanos/0.log"
May 26 12:22:41.196689 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:41.196657    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_924bcc1d-1874-49bd-8348-8a360af67584/init-config-reloader/0.log"
May 26 12:22:41.226106 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:41.226073    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-pcq8j_126ae8b2-7015-4fa0-b662-6a4b3289b160/prometheus-operator/0.log"
May 26 12:22:41.245146 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:41.245107    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-pcq8j_126ae8b2-7015-4fa0-b662-6a4b3289b160/kube-rbac-proxy/0.log"
May 26 12:22:44.030761 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:44.030684    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zm7c5/perf-node-gather-daemonset-dmbk5"]
May 26 12:22:44.031201 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:44.031012    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8" containerName="copy"
May 26 12:22:44.031201 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:44.031025    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8" containerName="copy"
May 26 12:22:44.031201 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:44.031040    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8" containerName="gather"
May 26 12:22:44.031201 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:44.031046    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8" containerName="gather"
May 26 12:22:44.031201 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:44.031096    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8" containerName="gather"
May 26 12:22:44.031201 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:44.031106    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="cae72dd5-2c2e-4463-9da1-e67e7d5c5dc8" containerName="copy"
May 26 12:22:44.036821 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:44.036797    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-dmbk5"
May 26 12:22:44.043418 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:44.043389    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zm7c5/perf-node-gather-daemonset-dmbk5"]
May 26 12:22:44.067437 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:44.067384    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-57bd96b888-fhfpk_e0c32514-7fd6-461c-bf92-092ad4ed740e/download-server/0.log"
May 26 12:22:44.172102 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:44.172067    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/357f0d82-a3dc-4165-b160-529fe97ae885-proc\") pod \"perf-node-gather-daemonset-dmbk5\" (UID: \"357f0d82-a3dc-4165-b160-529fe97ae885\") " pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-dmbk5"
May 26 12:22:44.172102 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:44.172106    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xx5r\" (UniqueName: \"kubernetes.io/projected/357f0d82-a3dc-4165-b160-529fe97ae885-kube-api-access-2xx5r\") pod \"perf-node-gather-daemonset-dmbk5\" (UID: \"357f0d82-a3dc-4165-b160-529fe97ae885\") " pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-dmbk5"
May 26 12:22:44.172314 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:44.172179    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/357f0d82-a3dc-4165-b160-529fe97ae885-lib-modules\") pod \"perf-node-gather-daemonset-dmbk5\" (UID: \"357f0d82-a3dc-4165-b160-529fe97ae885\") " pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-dmbk5"
May 26 12:22:44.172314 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:44.172225    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/357f0d82-a3dc-4165-b160-529fe97ae885-podres\") pod \"perf-node-gather-daemonset-dmbk5\" (UID: \"357f0d82-a3dc-4165-b160-529fe97ae885\") " pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-dmbk5"
May 26 12:22:44.172314 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:44.172241    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/357f0d82-a3dc-4165-b160-529fe97ae885-sys\") pod \"perf-node-gather-daemonset-dmbk5\" (UID: \"357f0d82-a3dc-4165-b160-529fe97ae885\") " pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-dmbk5"
May 26 12:22:44.273219 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:44.273174    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/357f0d82-a3dc-4165-b160-529fe97ae885-lib-modules\") pod \"perf-node-gather-daemonset-dmbk5\" (UID: \"357f0d82-a3dc-4165-b160-529fe97ae885\") " pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-dmbk5"
May 26 12:22:44.273413 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:44.273230    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/357f0d82-a3dc-4165-b160-529fe97ae885-podres\") pod \"perf-node-gather-daemonset-dmbk5\" (UID: \"357f0d82-a3dc-4165-b160-529fe97ae885\") " pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-dmbk5"
May 26 12:22:44.273413 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:44.273247    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/357f0d82-a3dc-4165-b160-529fe97ae885-sys\") pod \"perf-node-gather-daemonset-dmbk5\" (UID: \"357f0d82-a3dc-4165-b160-529fe97ae885\") " pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-dmbk5"
May 26 12:22:44.273413 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:44.273277    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/357f0d82-a3dc-4165-b160-529fe97ae885-proc\") pod \"perf-node-gather-daemonset-dmbk5\" (UID: \"357f0d82-a3dc-4165-b160-529fe97ae885\") " pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-dmbk5"
May 26 12:22:44.273413 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:44.273293    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2xx5r\" (UniqueName: \"kubernetes.io/projected/357f0d82-a3dc-4165-b160-529fe97ae885-kube-api-access-2xx5r\") pod \"perf-node-gather-daemonset-dmbk5\" (UID: \"357f0d82-a3dc-4165-b160-529fe97ae885\") " pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-dmbk5"
May 26 12:22:44.273413 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:44.273382    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/357f0d82-a3dc-4165-b160-529fe97ae885-podres\") pod \"perf-node-gather-daemonset-dmbk5\" (UID: \"357f0d82-a3dc-4165-b160-529fe97ae885\") " pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-dmbk5"
May 26 12:22:44.273413 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:44.273406    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/357f0d82-a3dc-4165-b160-529fe97ae885-proc\") pod \"perf-node-gather-daemonset-dmbk5\" (UID: \"357f0d82-a3dc-4165-b160-529fe97ae885\") " pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-dmbk5"
May 26 12:22:44.273630 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:44.273382    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/357f0d82-a3dc-4165-b160-529fe97ae885-lib-modules\") pod \"perf-node-gather-daemonset-dmbk5\" (UID: \"357f0d82-a3dc-4165-b160-529fe97ae885\") " pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-dmbk5"
May 26 12:22:44.273630 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:44.273387    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/357f0d82-a3dc-4165-b160-529fe97ae885-sys\") pod \"perf-node-gather-daemonset-dmbk5\" (UID: \"357f0d82-a3dc-4165-b160-529fe97ae885\") " pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-dmbk5"
May 26 12:22:44.282679 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:44.282617    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xx5r\" (UniqueName: \"kubernetes.io/projected/357f0d82-a3dc-4165-b160-529fe97ae885-kube-api-access-2xx5r\") pod \"perf-node-gather-daemonset-dmbk5\" (UID: \"357f0d82-a3dc-4165-b160-529fe97ae885\") " pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-dmbk5"
May 26 12:22:44.347630 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:44.347595    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-dmbk5"
May 26 12:22:44.492943 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:44.491476    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zm7c5/perf-node-gather-daemonset-dmbk5"]
May 26 12:22:44.595320 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:44.595288    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-dmbk5" event={"ID":"357f0d82-a3dc-4165-b160-529fe97ae885","Type":"ContainerStarted","Data":"1d302bff91a5f21c3f25ae44d2f716a16add5bb397ef1858ec35eafa1221bcf3"}
May 26 12:22:45.599537 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:45.599495    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-dmbk5" event={"ID":"357f0d82-a3dc-4165-b160-529fe97ae885","Type":"ContainerStarted","Data":"491531a0a52a7df0acb08f425bac712cb5d6969f08cb98bbd656050495812916"}
May 26 12:22:45.599928 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:45.599566    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-dmbk5"
May 26 12:22:45.616262 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:45.616205    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-dmbk5" podStartSLOduration=1.616192548 podStartE2EDuration="1.616192548s" podCreationTimestamp="2026-05-26 12:22:44 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:22:45.613884618 +0000 UTC m=+3497.805326806" watchObservedRunningTime="2026-05-26 12:22:45.616192548 +0000 UTC m=+3497.807634735"
May 26 12:22:45.677847 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:45.677818    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fn6r5_0704d5eb-7568-4943-ad52-70c0f8814792/dns/0.log"
May 26 12:22:45.697334 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:45.697304    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fn6r5_0704d5eb-7568-4943-ad52-70c0f8814792/kube-rbac-proxy/0.log"
May 26 12:22:45.766467 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:45.766436    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tlx86_463206a2-8ca0-4458-ba0b-6c3e6f59b837/dns-node-resolver/0.log"
May 26 12:22:46.171017 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:46.170982    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7c89b4489d-2rf2s_3571472a-95af-4617-9cb4-ac2782af9f74/registry/0.log"
May 26 12:22:46.196733 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:46.196701    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-58x22_f1ab37a2-3fa3-4941-8cc3-0149f5478b36/node-ca/0.log"
May 26 12:22:47.260010 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:47.259971    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-d72ts_13698e7d-67cc-4e22-afdd-70f4b08b2819/serve-healthcheck-canary/0.log"
May 26 12:22:47.763870 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:47.763838    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7phq6_e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4/kube-rbac-proxy/0.log"
May 26 12:22:47.784170 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:47.784125    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7phq6_e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4/exporter/0.log"
May 26 12:22:47.804357 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:47.804307    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7phq6_e0357dfc-ab2b-4b79-8f21-c1ee4e8a66e4/extractor/0.log"
May 26 12:22:50.084932 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:50.084897    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-54vkw_1342967f-2803-490a-9473-81acff932ca9/s3-init/0.log"
May 26 12:22:51.617039 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:51.617004    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-dmbk5"
May 26 12:22:54.112729 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:54.112700    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-6k5q8_47a7e428-2e75-4d78-b7f0-cf4edd466758/migrator/0.log"
May 26 12:22:54.134278 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:54.134244    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-6k5q8_47a7e428-2e75-4d78-b7f0-cf4edd466758/graceful-termination/0.log"
May 26 12:22:56.291504 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:56.291431    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kz76h_b642b5f8-5576-42de-807d-6da907ae897d/kube-multus-additional-cni-plugins/0.log"
May 26 12:22:56.351271 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:56.351242    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kz76h_b642b5f8-5576-42de-807d-6da907ae897d/egress-router-binary-copy/0.log"
May 26 12:22:56.385656 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:56.385627    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kz76h_b642b5f8-5576-42de-807d-6da907ae897d/cni-plugins/0.log"
May 26 12:22:56.413514 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:56.413490    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kz76h_b642b5f8-5576-42de-807d-6da907ae897d/bond-cni-plugin/0.log"
May 26 12:22:56.437762 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:56.437727    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kz76h_b642b5f8-5576-42de-807d-6da907ae897d/routeoverride-cni/0.log"
May 26 12:22:56.462527 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:56.462490    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kz76h_b642b5f8-5576-42de-807d-6da907ae897d/whereabouts-cni-bincopy/0.log"
May 26 12:22:56.495306 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:56.495278    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kz76h_b642b5f8-5576-42de-807d-6da907ae897d/whereabouts-cni/0.log"
May 26 12:22:56.573413 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:56.573326    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-pfkj5_bfdd44fe-70d3-4e4e-864b-f007de978ffe/kube-multus/0.log"
May 26 12:22:56.623595 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:56.623567    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pp9l8_9d573a77-5520-489f-99cb-86419b23cd19/network-metrics-daemon/0.log"
May 26 12:22:56.651244 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:56.651218    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pp9l8_9d573a77-5520-489f-99cb-86419b23cd19/kube-rbac-proxy/0.log"
May 26 12:22:58.169559 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:58.169531    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wxzlw_8526b1f6-8087-4c6e-8cd0-a0188c5db3a2/ovn-controller/0.log"
May 26 12:22:58.218107 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:58.218075    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wxzlw_8526b1f6-8087-4c6e-8cd0-a0188c5db3a2/ovn-acl-logging/0.log"
May 26 12:22:58.240459 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:58.240426    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wxzlw_8526b1f6-8087-4c6e-8cd0-a0188c5db3a2/kube-rbac-proxy-node/0.log"
May 26 12:22:58.264826 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:58.264768    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wxzlw_8526b1f6-8087-4c6e-8cd0-a0188c5db3a2/kube-rbac-proxy-ovn-metrics/0.log"
May 26 12:22:58.288011 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:58.287984    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wxzlw_8526b1f6-8087-4c6e-8cd0-a0188c5db3a2/northd/0.log"
May 26 12:22:58.309689 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:58.309666    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wxzlw_8526b1f6-8087-4c6e-8cd0-a0188c5db3a2/nbdb/0.log"
May 26 12:22:58.330364 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:58.330333    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wxzlw_8526b1f6-8087-4c6e-8cd0-a0188c5db3a2/sbdb/0.log"
May 26 12:22:58.457278 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:58.457179    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wxzlw_8526b1f6-8087-4c6e-8cd0-a0188c5db3a2/ovnkube-controller/0.log"
May 26 12:22:59.466459 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:22:59.466428    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-cbxbl_8528f6a0-0b8d-47f8-a20c-715dd15d1076/network-check-target-container/0.log"
May 26 12:23:00.489107 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:23:00.489083    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-ldk8g_d316eade-d057-4616-b829-4c09f6ea6d16/iptables-alerter/0.log"
May 26 12:23:01.266921 ip-10-0-129-97 kubenswrapper[2563]: I0526 12:23:01.266885    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-njvjv_be852edc-c490-4de5-99b0-a73297a80fa0/tuned/0.log"