May 26 16:00:50.169218 ip-10-0-131-96 systemd[1]: Starting Kubernetes Kubelet...
May 26 16:00:50.637505 ip-10-0-131-96 kubenswrapper[2552]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 16:00:50.637505 ip-10-0-131-96 kubenswrapper[2552]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 26 16:00:50.637505 ip-10-0-131-96 kubenswrapper[2552]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 16:00:50.637505 ip-10-0-131-96 kubenswrapper[2552]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 26 16:00:50.637505 ip-10-0-131-96 kubenswrapper[2552]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 16:00:50.641799 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.641600    2552 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 26 16:00:50.648026 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.647998    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 16:00:50.648026 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648022    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 16:00:50.648026 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648026    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 16:00:50.648026 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648029    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 16:00:50.648026 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648032    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 16:00:50.648026 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648035    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 16:00:50.648252 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648038    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 16:00:50.648252 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648042    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 16:00:50.648252 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648045    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 16:00:50.648252 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648048    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 16:00:50.648252 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648050    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 16:00:50.648252 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648053    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 16:00:50.648252 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648056    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 16:00:50.648252 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648059    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 16:00:50.648252 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648062    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 16:00:50.648252 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648064    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 16:00:50.648252 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648067    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 16:00:50.648252 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648069    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 16:00:50.648252 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648072    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 16:00:50.648252 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648075    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 16:00:50.648252 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648078    2552 feature_gate.go:328] unrecognized feature gate: Example
May 26 16:00:50.648252 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648081    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 16:00:50.648252 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648084    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 16:00:50.648252 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648094    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 16:00:50.648252 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648098    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 16:00:50.648698 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648102    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 16:00:50.648698 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648105    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 16:00:50.648698 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648108    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 16:00:50.648698 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648111    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 16:00:50.648698 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648114    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 16:00:50.648698 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648117    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 16:00:50.648698 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648120    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 16:00:50.648698 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648122    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 16:00:50.648698 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648125    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 16:00:50.648698 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648127    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 16:00:50.648698 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648130    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 16:00:50.648698 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648132    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 16:00:50.648698 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648135    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 16:00:50.648698 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648137    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 16:00:50.648698 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648141    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 16:00:50.648698 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648143    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 16:00:50.648698 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648146    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 16:00:50.648698 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648148    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 16:00:50.648698 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648151    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 16:00:50.648698 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648154    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 16:00:50.649225 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648156    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 16:00:50.649225 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648159    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 16:00:50.649225 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648161    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 16:00:50.649225 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648164    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 16:00:50.649225 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648167    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 16:00:50.649225 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648169    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 16:00:50.649225 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648172    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 16:00:50.649225 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648174    2552 feature_gate.go:328] unrecognized feature gate: Example2
May 26 16:00:50.649225 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648177    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 16:00:50.649225 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648179    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 16:00:50.649225 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648182    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 16:00:50.649225 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648185    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 16:00:50.649225 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648188    2552 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 16:00:50.649225 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648190    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 16:00:50.649225 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648193    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 16:00:50.649225 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648196    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 16:00:50.649225 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648199    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 16:00:50.649225 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648201    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 16:00:50.649225 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648205    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 16:00:50.649225 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648208    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 16:00:50.649724 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648210    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 16:00:50.649724 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648213    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 16:00:50.649724 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648216    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 16:00:50.649724 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648220    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 16:00:50.649724 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648223    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 16:00:50.649724 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648227    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 16:00:50.649724 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648230    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 16:00:50.649724 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648233    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 16:00:50.649724 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648235    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 16:00:50.649724 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648238    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 16:00:50.649724 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648242    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 16:00:50.649724 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648245    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 16:00:50.649724 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648248    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 16:00:50.649724 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648251    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 16:00:50.649724 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648254    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 16:00:50.649724 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648257    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 16:00:50.649724 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648259    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 16:00:50.649724 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648262    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 16:00:50.649724 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648265    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 16:00:50.649724 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648267    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 16:00:50.650212 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648270    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 16:00:50.650212 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648646    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 16:00:50.650212 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648650    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 16:00:50.650212 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648653    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 16:00:50.650212 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648656    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 16:00:50.650212 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648659    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 16:00:50.650212 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648662    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 16:00:50.650212 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648665    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 16:00:50.650212 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648668    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 16:00:50.650212 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648671    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 16:00:50.650212 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648674    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 16:00:50.650212 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648678    2552 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 16:00:50.650212 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648680    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 16:00:50.650212 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648683    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 16:00:50.650212 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648685    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 16:00:50.650212 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648688    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 16:00:50.650212 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648691    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 16:00:50.650212 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648694    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 16:00:50.650212 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648698    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 16:00:50.650665 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648701    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 16:00:50.650665 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648703    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 16:00:50.650665 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648706    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 16:00:50.650665 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648709    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 16:00:50.650665 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648711    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 16:00:50.650665 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648714    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 16:00:50.650665 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648717    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 16:00:50.650665 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648720    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 16:00:50.650665 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648723    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 16:00:50.650665 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648725    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 16:00:50.650665 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648728    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 16:00:50.650665 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648730    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 16:00:50.650665 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648733    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 16:00:50.650665 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648736    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 16:00:50.650665 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648738    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 16:00:50.650665 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648741    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 16:00:50.650665 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648744    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 16:00:50.650665 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648746    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 16:00:50.650665 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648749    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 16:00:50.650665 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648751    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 16:00:50.651211 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648754    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 16:00:50.651211 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648756    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 16:00:50.651211 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648759    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 16:00:50.651211 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648761    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 16:00:50.651211 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648764    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 16:00:50.651211 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648766    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 16:00:50.651211 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648769    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 16:00:50.651211 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648771    2552 feature_gate.go:328] unrecognized feature gate: Example
May 26 16:00:50.651211 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648774    2552 feature_gate.go:328] unrecognized feature gate: Example2
May 26 16:00:50.651211 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648776    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 16:00:50.651211 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648779    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 16:00:50.651211 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648781    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 16:00:50.651211 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648783    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 16:00:50.651211 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648786    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 16:00:50.651211 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648788    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 16:00:50.651211 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648791    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 16:00:50.651211 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648793    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 16:00:50.651211 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648796    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 16:00:50.651211 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648798    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 16:00:50.651211 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648802    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 16:00:50.651740 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648804    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 16:00:50.651740 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648807    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 16:00:50.651740 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648810    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 16:00:50.651740 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648812    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 16:00:50.651740 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648815    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 16:00:50.651740 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648818    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 16:00:50.651740 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648820    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 16:00:50.651740 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648823    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 16:00:50.651740 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648826    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 16:00:50.651740 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648828    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 16:00:50.651740 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648831    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 16:00:50.651740 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648834    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 16:00:50.651740 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648836    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 16:00:50.651740 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648839    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 16:00:50.651740 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648841    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 16:00:50.651740 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648844    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 16:00:50.651740 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648846    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 16:00:50.651740 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648849    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 16:00:50.651740 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648852    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 16:00:50.651740 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648855    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 16:00:50.652242 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648857    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 16:00:50.652242 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648860    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 16:00:50.652242 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648862    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 16:00:50.652242 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648866    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 16:00:50.652242 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648870    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 16:00:50.652242 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648873    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 16:00:50.652242 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648875    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 16:00:50.652242 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.648878    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 16:00:50.652242 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649695    2552 flags.go:64] FLAG: --address="0.0.0.0"
May 26 16:00:50.652242 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649705    2552 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 26 16:00:50.652242 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649713    2552 flags.go:64] FLAG: --anonymous-auth="true"
May 26 16:00:50.652242 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649717    2552 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 26 16:00:50.652242 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649722    2552 flags.go:64] FLAG: --authentication-token-webhook="false"
May 26 16:00:50.652242 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649726    2552 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 26 16:00:50.652242 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649731    2552 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 26 16:00:50.652242 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649735    2552 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 26 16:00:50.652242 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649739    2552 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 26 16:00:50.652242 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649742    2552 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 26 16:00:50.652242 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649747    2552 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 26 16:00:50.652242 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649750    2552 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 26 16:00:50.652242 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649753    2552 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 26 16:00:50.652746 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649756    2552 flags.go:64] FLAG: --cgroup-root=""
May 26 16:00:50.652746 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649759    2552 flags.go:64] FLAG: --cgroups-per-qos="true"
May 26 16:00:50.652746 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649762    2552 flags.go:64] FLAG: --client-ca-file=""
May 26 16:00:50.652746 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649765    2552 flags.go:64] FLAG: --cloud-config=""
May 26 16:00:50.652746 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649768    2552 flags.go:64] FLAG: --cloud-provider="external"
May 26 16:00:50.652746 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649771    2552 flags.go:64] FLAG: --cluster-dns="[]"
May 26 16:00:50.652746 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649775    2552 flags.go:64] FLAG: --cluster-domain=""
May 26 16:00:50.652746 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649778    2552 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 26 16:00:50.652746 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649781    2552 flags.go:64] FLAG: --config-dir=""
May 26 16:00:50.652746 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649784    2552 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 26 16:00:50.652746 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649788    2552 flags.go:64] FLAG: --container-log-max-files="5"
May 26 16:00:50.652746 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649791    2552 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 26 16:00:50.652746 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649794    2552 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 26 16:00:50.652746 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649798    2552 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 26 16:00:50.652746 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649801    2552 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 26 16:00:50.652746 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649804    2552 flags.go:64] FLAG: --contention-profiling="false"
May 26 16:00:50.652746 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649807    2552 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 26 16:00:50.652746 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649810    2552 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 26 16:00:50.652746 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649814    2552 flags.go:64] FLAG: --cpu-manager-policy="none"
May 26 16:00:50.652746 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649817    2552 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 26 16:00:50.652746 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649821    2552 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 26 16:00:50.652746 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649824    2552 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 26 16:00:50.652746 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649827    2552 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 26 16:00:50.652746 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649830    2552 flags.go:64] FLAG: --enable-load-reader="false"
May 26 16:00:50.652746 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649833    2552 flags.go:64] FLAG: --enable-server="true"
May 26 16:00:50.653405 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649836    2552 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 26 16:00:50.653405 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649841    2552 flags.go:64] FLAG: --event-burst="100"
May 26 16:00:50.653405 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649844    2552 flags.go:64] FLAG: --event-qps="50"
May 26 16:00:50.653405 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649847    2552 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 26 16:00:50.653405 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649852    2552 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 26 16:00:50.653405 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649855    2552 flags.go:64] FLAG: --eviction-hard=""
May 26 16:00:50.653405 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649859    2552 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 26 16:00:50.653405 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649862    2552 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 26 16:00:50.653405 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649865    2552 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 26 16:00:50.653405 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649868    2552 flags.go:64] FLAG: --eviction-soft=""
May 26 16:00:50.653405 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649871    2552 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 26 16:00:50.653405 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649874    2552 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 26 16:00:50.653405 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649877    2552 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 26 16:00:50.653405 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649880    2552 flags.go:64] FLAG: --experimental-mounter-path=""
May 26 16:00:50.653405 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649883    2552 flags.go:64] FLAG: --fail-cgroupv1="false"
May 26 16:00:50.653405 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649886    2552 flags.go:64] FLAG: --fail-swap-on="true"
May 26 16:00:50.653405 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649889    2552 flags.go:64] FLAG: --feature-gates=""
May 26 16:00:50.653405 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649893    2552 flags.go:64] FLAG: --file-check-frequency="20s"
May 26 16:00:50.653405 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649896    2552 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 26 16:00:50.653405 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649899    2552 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 26 16:00:50.653405 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649902    2552 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 26 16:00:50.653405 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649905    2552 flags.go:64] FLAG: --healthz-port="10248"
May 26 16:00:50.653405 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649908    2552 flags.go:64] FLAG: --help="false"
May 26 16:00:50.653405 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649911    2552 flags.go:64] FLAG: --hostname-override="ip-10-0-131-96.ec2.internal"
May 26 16:00:50.653405 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649914    2552 flags.go:64] FLAG: --housekeeping-interval="10s"
May 26 16:00:50.654023 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649917    2552 flags.go:64] FLAG: --http-check-frequency="20s"
May 26 16:00:50.654023 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649920    2552 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 26 16:00:50.654023 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649923    2552 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 26 16:00:50.654023 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649940    2552 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 26 16:00:50.654023 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649943    2552 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 26 16:00:50.654023 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649947    2552 flags.go:64] FLAG: --image-service-endpoint=""
May 26 16:00:50.654023 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649950    2552 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 26 16:00:50.654023 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649953    2552 flags.go:64] FLAG: --kube-api-burst="100"
May 26 16:00:50.654023 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649957    2552 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 26 16:00:50.654023 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649960    2552 flags.go:64] FLAG: --kube-api-qps="50"
May 26 16:00:50.654023 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649963    2552 flags.go:64] FLAG: --kube-reserved=""
May 26 16:00:50.654023 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649966    2552 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 26 16:00:50.654023 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649969    2552 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 26 16:00:50.654023 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649972    2552 flags.go:64] FLAG: --kubelet-cgroups=""
May 26 16:00:50.654023 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649976    2552 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 26 16:00:50.654023 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649978    2552 flags.go:64] FLAG: --lock-file=""
May 26 16:00:50.654023 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649981    2552 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 26 16:00:50.654023 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649984    2552 flags.go:64] FLAG: --log-flush-frequency="5s"
May 26 16:00:50.654023 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649987    2552 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 26 16:00:50.654023 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649992    2552 flags.go:64] FLAG: --log-json-split-stream="false"
May 26 16:00:50.654023 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649995    2552 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 26 16:00:50.654023 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.649998    2552 flags.go:64] FLAG: --log-text-split-stream="false"
May 26 16:00:50.654023 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650001    2552 flags.go:64] FLAG: --logging-format="text"
May 26 16:00:50.654582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650004    2552 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 26 16:00:50.654582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650007    2552 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 26 16:00:50.654582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650010    2552 flags.go:64] FLAG: --manifest-url=""
May 26 16:00:50.654582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650013    2552 flags.go:64] FLAG: --manifest-url-header=""
May 26 16:00:50.654582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650018    2552 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 26 16:00:50.654582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650021    2552 flags.go:64] FLAG: --max-open-files="1000000"
May 26 16:00:50.654582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650025    2552 flags.go:64] FLAG: --max-pods="110"
May 26 16:00:50.654582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650027    2552 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 26 16:00:50.654582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650030    2552 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 26 16:00:50.654582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650033    2552 flags.go:64] FLAG: --memory-manager-policy="None"
May 26 16:00:50.654582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650036    2552 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 26 16:00:50.654582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650039    2552 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 26 16:00:50.654582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650043    2552 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 26 16:00:50.654582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650046    2552 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 26 16:00:50.654582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650054    2552 flags.go:64] FLAG: --node-status-max-images="50"
May 26 16:00:50.654582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650057    2552 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 26 16:00:50.654582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650060    2552 flags.go:64] FLAG: --oom-score-adj="-999"
May 26 16:00:50.654582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650063    2552 flags.go:64] FLAG: --pod-cidr=""
May 26 16:00:50.654582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650066    2552 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 26 16:00:50.654582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650072    2552 flags.go:64] FLAG: --pod-manifest-path=""
May 26 16:00:50.654582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650075    2552 flags.go:64] FLAG: --pod-max-pids="-1"
May 26 16:00:50.654582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650079    2552 flags.go:64] FLAG: --pods-per-core="0"
May 26 16:00:50.654582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650082    2552 flags.go:64] FLAG: --port="10250"
May 26 16:00:50.654582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650085    2552 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 26 16:00:50.655174 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650088    2552 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-018962a1c990da7b7"
May 26 16:00:50.655174 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650091    2552 flags.go:64] FLAG: --qos-reserved=""
May 26 16:00:50.655174 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650094    2552 flags.go:64] FLAG: --read-only-port="10255"
May 26 16:00:50.655174 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650097    2552 flags.go:64] FLAG: --register-node="true"
May 26 16:00:50.655174 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650100    2552 flags.go:64] FLAG: --register-schedulable="true"
May 26 16:00:50.655174 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650103    2552 flags.go:64] FLAG: --register-with-taints=""
May 26 16:00:50.655174 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650106    2552 flags.go:64] FLAG: --registry-burst="10"
May 26 16:00:50.655174 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650109    2552 flags.go:64] FLAG: --registry-qps="5"
May 26 16:00:50.655174 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650112    2552 flags.go:64] FLAG: --reserved-cpus=""
May 26 16:00:50.655174 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650115    2552 flags.go:64] FLAG: --reserved-memory=""
May 26 16:00:50.655174 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650118    2552 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 26 16:00:50.655174 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650121    2552 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 26 16:00:50.655174 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650124    2552 flags.go:64] FLAG: --rotate-certificates="false"
May 26 16:00:50.655174 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650127    2552 flags.go:64] FLAG: --rotate-server-certificates="false"
May 26 16:00:50.655174 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650130    2552 flags.go:64] FLAG: --runonce="false"
May 26 16:00:50.655174 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650133    2552 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 26 16:00:50.655174 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650136    2552 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 26 16:00:50.655174 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650139    2552 flags.go:64] FLAG: --seccomp-default="false"
May 26 16:00:50.655174 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650141    2552 flags.go:64] FLAG: --serialize-image-pulls="true"
May 26 16:00:50.655174 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650144    2552 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 26 16:00:50.655174 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650147    2552 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 26 16:00:50.655174 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650151    2552 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 26 16:00:50.655174 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650154    2552 flags.go:64] FLAG: --storage-driver-password="root"
May 26 16:00:50.655174 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650156    2552 flags.go:64] FLAG: --storage-driver-secure="false"
May 26 16:00:50.655174 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650159    2552 flags.go:64] FLAG: --storage-driver-table="stats"
May 26 16:00:50.655174 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650162    2552 flags.go:64] FLAG: --storage-driver-user="root"
May 26 16:00:50.655799 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650165    2552 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 26 16:00:50.655799 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650173    2552 flags.go:64] FLAG: --sync-frequency="1m0s"
May 26 16:00:50.655799 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650176    2552 flags.go:64] FLAG: --system-cgroups=""
May 26 16:00:50.655799 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650179    2552 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 26 16:00:50.655799 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650184    2552 flags.go:64] FLAG: --system-reserved-cgroup=""
May 26 16:00:50.655799 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650187    2552 flags.go:64] FLAG: --tls-cert-file=""
May 26 16:00:50.655799 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650190    2552 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 26 16:00:50.655799 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650194    2552 flags.go:64] FLAG: --tls-min-version=""
May 26 16:00:50.655799 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650196    2552 flags.go:64] FLAG: --tls-private-key-file=""
May 26 16:00:50.655799 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650199    2552 flags.go:64] FLAG: --topology-manager-policy="none"
May 26 16:00:50.655799 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650202    2552 flags.go:64] FLAG: --topology-manager-policy-options=""
May 26 16:00:50.655799 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650205    2552 flags.go:64] FLAG: --topology-manager-scope="container"
May 26 16:00:50.655799 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650208    2552 flags.go:64] FLAG: --v="2"
May 26 16:00:50.655799 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650212    2552 flags.go:64] FLAG: --version="false"
May 26 16:00:50.655799 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650216    2552 flags.go:64] FLAG: --vmodule=""
May 26 16:00:50.655799 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650221    2552 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 26 16:00:50.655799 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.650224    2552 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 26 16:00:50.655799 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650319    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 16:00:50.655799 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650323    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 16:00:50.655799 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650327    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 16:00:50.655799 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650330    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 16:00:50.655799 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650333    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 16:00:50.655799 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650335    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 16:00:50.656367 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650338    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 16:00:50.656367 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650342    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 16:00:50.656367 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650346    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 16:00:50.656367 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650351    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 16:00:50.656367 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650353    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 16:00:50.656367 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650356    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 16:00:50.656367 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650359    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 16:00:50.656367 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650362    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 16:00:50.656367 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650364    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 16:00:50.656367 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650367    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 16:00:50.656367 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650371    2552 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 16:00:50.656367 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650375    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 16:00:50.656367 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650377    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 16:00:50.656367 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650380    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 16:00:50.656367 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650383    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 16:00:50.656367 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650385    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 16:00:50.656367 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650388    2552 feature_gate.go:328] unrecognized feature gate: Example2
May 26 16:00:50.656367 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650391    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 16:00:50.656367 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650393    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 16:00:50.656878 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650396    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 16:00:50.656878 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650398    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 16:00:50.656878 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650401    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 16:00:50.656878 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650403    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 16:00:50.656878 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650406    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 16:00:50.656878 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650408    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 16:00:50.656878 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650411    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 16:00:50.656878 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650414    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 16:00:50.656878 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650416    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 16:00:50.656878 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650419    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 16:00:50.656878 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650421    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 16:00:50.656878 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650424    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 16:00:50.656878 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650426    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 16:00:50.656878 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650429    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 16:00:50.656878 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650431    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 16:00:50.656878 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650434    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 16:00:50.656878 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650439    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 16:00:50.656878 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650441    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 16:00:50.656878 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650444    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 16:00:50.656878 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650446    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 16:00:50.657395 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650449    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 16:00:50.657395 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650451    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 16:00:50.657395 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650454    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 16:00:50.657395 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650457    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 16:00:50.657395 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650462    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 16:00:50.657395 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650465    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 16:00:50.657395 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650467    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 16:00:50.657395 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650470    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 16:00:50.657395 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650472    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 16:00:50.657395 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650475    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 16:00:50.657395 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650478    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 16:00:50.657395 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650480    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 16:00:50.657395 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650483    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 16:00:50.657395 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650485    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 16:00:50.657395 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650488    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 16:00:50.657395 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650490    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 16:00:50.657395 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650493    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 16:00:50.657395 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650495    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 16:00:50.657395 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650498    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 16:00:50.657395 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650500    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 16:00:50.657888 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650503    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 16:00:50.657888 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650505    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 16:00:50.657888 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650508    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 16:00:50.657888 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650510    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 16:00:50.657888 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650513    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 16:00:50.657888 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650515    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 16:00:50.657888 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650518    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 16:00:50.657888 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650520    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 16:00:50.657888 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650524    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 16:00:50.657888 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650526    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 16:00:50.657888 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650529    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 16:00:50.657888 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650531    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 16:00:50.657888 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650534    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 16:00:50.657888 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650537    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 16:00:50.657888 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650539    2552 feature_gate.go:328] unrecognized feature gate: Example
May 26 16:00:50.657888 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650543    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 16:00:50.657888 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650545    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 16:00:50.657888 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650548    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 16:00:50.657888 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650551    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 16:00:50.657888 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650554    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 16:00:50.658391 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.650556    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 16:00:50.658391 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.651864    2552 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 16:00:50.659003 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.658981    2552 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 26 16:00:50.659038 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.659004    2552 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 26 16:00:50.659069 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659056    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 16:00:50.659069 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659061    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 16:00:50.659069 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659064    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 16:00:50.659069 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659067    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 16:00:50.659069 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659070    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 16:00:50.659192 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659073    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 16:00:50.659192 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659076    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 16:00:50.659192 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659079    2552 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 16:00:50.659192 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659082    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 16:00:50.659192 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659085    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 16:00:50.659192 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659088    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 16:00:50.659192 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659090    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 16:00:50.659192 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659093    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 16:00:50.659192 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659096    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 16:00:50.659192 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659098    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 16:00:50.659192 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659101    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 16:00:50.659192 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659104    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 16:00:50.659192 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659106    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 16:00:50.659192 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659109    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 16:00:50.659192 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659111    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 16:00:50.659192 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659114    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 16:00:50.659192 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659117    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 16:00:50.659192 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659120    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 16:00:50.659192 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659122    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 16:00:50.659192 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659125    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 16:00:50.659682 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659129    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 16:00:50.659682 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659131    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 16:00:50.659682 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659134    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 16:00:50.659682 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659136    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 16:00:50.659682 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659139    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 16:00:50.659682 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659141    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 16:00:50.659682 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659144    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 16:00:50.659682 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659147    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 16:00:50.659682 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659149    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 16:00:50.659682 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659152    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 16:00:50.659682 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659155    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 16:00:50.659682 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659157    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 16:00:50.659682 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659160    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 16:00:50.659682 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659163    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 16:00:50.659682 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659166    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 16:00:50.659682 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659169    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 16:00:50.659682 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659172    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 16:00:50.659682 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659175    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 16:00:50.659682 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659177    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 16:00:50.659682 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659180    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 16:00:50.660195 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659183    2552 feature_gate.go:328] unrecognized feature gate: Example2
May 26 16:00:50.660195 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659185    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 16:00:50.660195 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659188    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 16:00:50.660195 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659191    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 16:00:50.660195 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659193    2552 feature_gate.go:328] unrecognized feature gate: Example
May 26 16:00:50.660195 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659196    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 16:00:50.660195 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659199    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 16:00:50.660195 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659201    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 16:00:50.660195 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659204    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 16:00:50.660195 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659207    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 16:00:50.660195 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659209    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 16:00:50.660195 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659212    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 16:00:50.660195 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659215    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 16:00:50.660195 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659219    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 16:00:50.660195 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659223    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 16:00:50.660195 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659228    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 16:00:50.660195 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659232    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 16:00:50.660195 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659235    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 16:00:50.660195 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659238    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 16:00:50.660667 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659241    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 16:00:50.660667 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659244    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 16:00:50.660667 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659246    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 16:00:50.660667 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659249    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 16:00:50.660667 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659251    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 16:00:50.660667 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659254    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 16:00:50.660667 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659257    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 16:00:50.660667 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659260    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 16:00:50.660667 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659263    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 16:00:50.660667 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659266    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 16:00:50.660667 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659268    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 16:00:50.660667 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659271    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 16:00:50.660667 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659274    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 16:00:50.660667 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659276    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 16:00:50.660667 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659279    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 16:00:50.660667 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659281    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 16:00:50.660667 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659284    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 16:00:50.660667 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659286    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 16:00:50.660667 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659289    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 16:00:50.660667 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659292    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 16:00:50.661238 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659294    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 16:00:50.661238 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659296    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 16:00:50.661238 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.659302    2552 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 16:00:50.661238 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659408    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 16:00:50.661238 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659412    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 16:00:50.661238 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659415    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 16:00:50.661238 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659418    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 16:00:50.661238 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659421    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 16:00:50.661238 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659424    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 16:00:50.661238 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659427    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 16:00:50.661238 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659429    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 16:00:50.661238 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659432    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 16:00:50.661238 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659435    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 16:00:50.661238 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659439    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 16:00:50.661238 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659442    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 16:00:50.661602 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659445    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 16:00:50.661602 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659448    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 16:00:50.661602 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659451    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 16:00:50.661602 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659454    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 16:00:50.661602 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659458    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 16:00:50.661602 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659460    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 16:00:50.661602 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659463    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 16:00:50.661602 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659466    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 16:00:50.661602 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659468    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 16:00:50.661602 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659471    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 16:00:50.661602 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659473    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 16:00:50.661602 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659476    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 16:00:50.661602 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659478    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 16:00:50.661602 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659481    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 16:00:50.661602 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659484    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 16:00:50.661602 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659486    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 16:00:50.661602 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659489    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 16:00:50.661602 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659491    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 16:00:50.661602 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659494    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 16:00:50.662071 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659496    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 16:00:50.662071 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659499    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 16:00:50.662071 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659502    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 16:00:50.662071 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659505    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 16:00:50.662071 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659507    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 16:00:50.662071 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659510    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 16:00:50.662071 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659513    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 16:00:50.662071 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659516    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 16:00:50.662071 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659518    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 16:00:50.662071 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659521    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 16:00:50.662071 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659524    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 16:00:50.662071 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659526    2552 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 16:00:50.662071 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659528    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 16:00:50.662071 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659531    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 16:00:50.662071 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659534    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 16:00:50.662071 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659536    2552 feature_gate.go:328] unrecognized feature gate: Example2
May 26 16:00:50.662071 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659539    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 16:00:50.662071 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659542    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 16:00:50.662071 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659544    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 16:00:50.662071 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659546    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 16:00:50.662544 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659549    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 16:00:50.662544 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659552    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 16:00:50.662544 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659554    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 16:00:50.662544 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659557    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 16:00:50.662544 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659559    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 16:00:50.662544 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659562    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 16:00:50.662544 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659565    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 16:00:50.662544 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659568    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 16:00:50.662544 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659571    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 16:00:50.662544 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659573    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 16:00:50.662544 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659576    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 16:00:50.662544 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659578    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 16:00:50.662544 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659580    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 16:00:50.662544 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659583    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 16:00:50.662544 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659585    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 16:00:50.662544 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659588    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 16:00:50.662544 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659590    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 16:00:50.662544 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659593    2552 feature_gate.go:328] unrecognized feature gate: Example
May 26 16:00:50.662544 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659595    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 16:00:50.662544 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659598    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 16:00:50.663039 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659601    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 16:00:50.663039 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659603    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 16:00:50.663039 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659605    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 16:00:50.663039 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659608    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 16:00:50.663039 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659611    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 16:00:50.663039 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659613    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 16:00:50.663039 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659616    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 16:00:50.663039 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659618    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 16:00:50.663039 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659621    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 16:00:50.663039 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659623    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 16:00:50.663039 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659626    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 16:00:50.663039 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659628    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 16:00:50.663039 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659630    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 16:00:50.663039 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659633    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 16:00:50.663039 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:50.659635    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 16:00:50.663039 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.659640    2552 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 16:00:50.663424 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.660388    2552 server.go:962] "Client rotation is on, will bootstrap in background"
May 26 16:00:50.664032 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.664019    2552 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 26 16:00:50.665096 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.665085    2552 server.go:1019] "Starting client certificate rotation"
May 26 16:00:50.665203 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.665187    2552 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 26 16:00:50.665239 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.665227    2552 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 26 16:00:50.692675 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.692647    2552 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 26 16:00:50.694816 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.694799    2552 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 26 16:00:50.713192 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.713171    2552 log.go:25] "Validated CRI v1 runtime API"
May 26 16:00:50.718859 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.718844    2552 log.go:25] "Validated CRI v1 image API"
May 26 16:00:50.720663 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.720643    2552 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 26 16:00:50.723494 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.723478    2552 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 26 16:00:50.723669 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.723651    2552 fs.go:135] Filesystem UUIDs: map[459e3c2d-8da5-44f2-b0a0-c2905eff3b44:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d653df32-cbad-479c-9149-d52cc9f47509:/dev/nvme0n1p3]
May 26 16:00:50.723715 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.723670    2552 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 26 16:00:50.727864 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.727763    2552 manager.go:217] Machine: {Timestamp:2026-05-26 16:00:50.727287468 +0000 UTC m=+0.435742499 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099362 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d9c2201c0df701cf8d5cc9c4cd1fe SystemUUID:ec2d9c22-01c0-df70-1cf8-d5cc9c4cd1fe BootID:6cdb6610-5ede-4a3b-9b95-bae144463f2a Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a8:c4:f7:aa:c3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a8:c4:f7:aa:c3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:cc:0f:d3:a4:d9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 26 16:00:50.728485 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.728474    2552 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 26 16:00:50.728602 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.728590    2552 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 26 16:00:50.730161 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.730142    2552 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 26 16:00:50.730287 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.730163    2552 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-96.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 26 16:00:50.730336 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.730296    2552 topology_manager.go:138] "Creating topology manager with none policy"
May 26 16:00:50.730336 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.730305    2552 container_manager_linux.go:306] "Creating device plugin manager"
May 26 16:00:50.730336 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.730319    2552 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 26 16:00:50.730958 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.730948    2552 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 26 16:00:50.731836 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.731827    2552 state_mem.go:36] "Initialized new in-memory state store"
May 26 16:00:50.731967 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.731958    2552 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 26 16:00:50.734160 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.734150    2552 kubelet.go:491] "Attempting to sync node with API server"
May 26 16:00:50.734201 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.734167    2552 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 26 16:00:50.734201 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.734178    2552 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 26 16:00:50.734201 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.734188    2552 kubelet.go:397] "Adding apiserver pod source"
May 26 16:00:50.734201 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.734196    2552 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 26 16:00:50.735275 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.735263    2552 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 26 16:00:50.735342 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.735282    2552 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 26 16:00:50.738227 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.738210    2552 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 26 16:00:50.739984 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.739970    2552 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 26 16:00:50.741615 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.741604    2552 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 26 16:00:50.741615 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.741621    2552 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 26 16:00:50.741696 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.741627    2552 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 26 16:00:50.741696 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.741632    2552 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 26 16:00:50.741696 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.741641    2552 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 26 16:00:50.741696 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.741658    2552 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 26 16:00:50.741696 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.741664    2552 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 26 16:00:50.741696 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.741669    2552 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 26 16:00:50.741696 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.741675    2552 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 26 16:00:50.741696 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.741681    2552 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 26 16:00:50.741696 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.741692    2552 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 26 16:00:50.741696 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.741698    2552 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 26 16:00:50.742484 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.742475    2552 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 26 16:00:50.742484 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.742484    2552 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 26 16:00:50.745653 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.745641    2552 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 26 16:00:50.745694 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.745675    2552 server.go:1295] "Started kubelet"
May 26 16:00:50.746242 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.746178    2552 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 26 16:00:50.746335 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.746266    2552 server_v1.go:47] "podresources" method="list" useActivePods=true
May 26 16:00:50.746644 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.746617    2552 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 26 16:00:50.746720 ip-10-0-131-96 systemd[1]: Started Kubernetes Kubelet.
May 26 16:00:50.749390 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.749365    2552 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 26 16:00:50.749646 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.749630    2552 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-96.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 26 16:00:50.749965 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:50.749945    2552 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-96.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 26 16:00:50.749965 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:50.749942    2552 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 26 16:00:50.751924 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.751903    2552 server.go:317] "Adding debug handlers to kubelet server"
May 26 16:00:50.754560 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:50.753367    2552 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-96.ec2.internal.18b328e717f039dd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-96.ec2.internal,UID:ip-10-0-131-96.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-96.ec2.internal,},FirstTimestamp:2026-05-26 16:00:50.745653725 +0000 UTC m=+0.454108755,LastTimestamp:2026-05-26 16:00:50.745653725 +0000 UTC m=+0.454108755,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-96.ec2.internal,}"
May 26 16:00:50.756748 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.756730    2552 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 26 16:00:50.757259 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:50.757122    2552 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 26 16:00:50.757259 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.757216    2552 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 26 16:00:50.757712 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.757606    2552 volume_manager.go:295] "The desired_state_of_world populator starts"
May 26 16:00:50.757712 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.757619    2552 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 26 16:00:50.757822 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.757815    2552 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 26 16:00:50.757893 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.757878    2552 reconstruct.go:97] "Volume reconstruction finished"
May 26 16:00:50.757893 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.757892    2552 reconciler.go:26] "Reconciler: start to sync state"
May 26 16:00:50.758180 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:50.758157    2552 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-96.ec2.internal\" not found"
May 26 16:00:50.758581 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.758562    2552 factory.go:55] Registering systemd factory
May 26 16:00:50.758581 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.758584    2552 factory.go:223] Registration of the systemd container factory successfully
May 26 16:00:50.758803 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.758786    2552 factory.go:153] Registering CRI-O factory
May 26 16:00:50.758803 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.758806    2552 factory.go:223] Registration of the crio container factory successfully
May 26 16:00:50.758943 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.758858    2552 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 26 16:00:50.758943 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.758873    2552 factory.go:103] Registering Raw factory
May 26 16:00:50.758943 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.758886    2552 manager.go:1196] Started watching for new ooms in manager
May 26 16:00:50.759294 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.759275    2552 manager.go:319] Starting recovery of all containers
May 26 16:00:50.760488 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:50.760460    2552 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
May 26 16:00:50.760578 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:50.760534    2552 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-96.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
May 26 16:00:50.769198 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.769176    2552 manager.go:324] Recovery completed
May 26 16:00:50.773953 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.773920    2552 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 16:00:50.774541 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.774520    2552 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tsjdb"
May 26 16:00:50.775922 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.775907    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-96.ec2.internal" event="NodeHasSufficientMemory"
May 26 16:00:50.776021 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.775952    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-96.ec2.internal" event="NodeHasNoDiskPressure"
May 26 16:00:50.776021 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.775967    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-96.ec2.internal" event="NodeHasSufficientPID"
May 26 16:00:50.776423 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.776408    2552 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 26 16:00:50.776423 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.776421    2552 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 26 16:00:50.776521 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.776438    2552 state_mem.go:36] "Initialized new in-memory state store"
May 26 16:00:50.778081 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:50.778023    2552 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-96.ec2.internal.18b328e719be19a5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-96.ec2.internal,UID:ip-10-0-131-96.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-96.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-96.ec2.internal,},FirstTimestamp:2026-05-26 16:00:50.775923109 +0000 UTC m=+0.484378147,LastTimestamp:2026-05-26 16:00:50.775923109 +0000 UTC m=+0.484378147,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-96.ec2.internal,}"
May 26 16:00:50.778885 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.778869    2552 policy_none.go:49] "None policy: Start"
May 26 16:00:50.778885 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.778887    2552 memory_manager.go:186] "Starting memorymanager" policy="None"
May 26 16:00:50.779283 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.779269    2552 state_mem.go:35] "Initializing new in-memory state store"
May 26 16:00:50.782466 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.782446    2552 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tsjdb"
May 26 16:00:50.826475 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.826460    2552 manager.go:341] "Starting Device Plugin manager"
May 26 16:00:50.835076 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:50.826489    2552 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 26 16:00:50.835076 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.826499    2552 server.go:85] "Starting device plugin registration server"
May 26 16:00:50.835076 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.826721    2552 eviction_manager.go:189] "Eviction manager: starting control loop"
May 26 16:00:50.835076 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.826734    2552 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 26 16:00:50.835076 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.826847    2552 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 26 16:00:50.835076 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.826915    2552 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 26 16:00:50.835076 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.826923    2552 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 26 16:00:50.835076 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:50.828195    2552 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 26 16:00:50.835076 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:50.828228    2552 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-96.ec2.internal\" not found"
May 26 16:00:50.883770 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.883744    2552 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 26 16:00:50.884892 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.884877    2552 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 26 16:00:50.884968 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.884899    2552 status_manager.go:230] "Starting to sync pod status with apiserver"
May 26 16:00:50.884968 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.884920    2552 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 26 16:00:50.884968 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.884926    2552 kubelet.go:2451] "Starting kubelet main sync loop"
May 26 16:00:50.885113 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:50.885011    2552 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 26 16:00:50.889054 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.889009    2552 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 16:00:50.927253 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.927228    2552 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 16:00:50.929372 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.929359    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-96.ec2.internal" event="NodeHasSufficientMemory"
May 26 16:00:50.929436 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.929384    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-96.ec2.internal" event="NodeHasNoDiskPressure"
May 26 16:00:50.929436 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.929396    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-96.ec2.internal" event="NodeHasSufficientPID"
May 26 16:00:50.929436 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.929417    2552 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-96.ec2.internal"
May 26 16:00:50.939109 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.939096    2552 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-96.ec2.internal"
May 26 16:00:50.939152 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:50.939115    2552 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-96.ec2.internal\": node \"ip-10-0-131-96.ec2.internal\" not found"
May 26 16:00:50.954868 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:50.954852    2552 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-96.ec2.internal\" not found"
May 26 16:00:50.986102 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.986079    2552 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-96.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-96.ec2.internal"]
May 26 16:00:50.986150 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.986138    2552 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 16:00:50.987101 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.987086    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-96.ec2.internal" event="NodeHasSufficientMemory"
May 26 16:00:50.987164 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.987114    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-96.ec2.internal" event="NodeHasNoDiskPressure"
May 26 16:00:50.987164 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.987125    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-96.ec2.internal" event="NodeHasSufficientPID"
May 26 16:00:50.988362 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.988350    2552 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 16:00:50.988546 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.988533    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-96.ec2.internal"
May 26 16:00:50.988597 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.988589    2552 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 16:00:50.989043 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.989030    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-96.ec2.internal" event="NodeHasSufficientMemory"
May 26 16:00:50.989108 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.989041    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-96.ec2.internal" event="NodeHasSufficientMemory"
May 26 16:00:50.989108 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.989054    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-96.ec2.internal" event="NodeHasNoDiskPressure"
May 26 16:00:50.989108 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.989066    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-96.ec2.internal" event="NodeHasNoDiskPressure"
May 26 16:00:50.989108 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.989081    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-96.ec2.internal" event="NodeHasSufficientPID"
May 26 16:00:50.989237 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.989068    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-96.ec2.internal" event="NodeHasSufficientPID"
May 26 16:00:50.990371 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.990358    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-96.ec2.internal"
May 26 16:00:50.990432 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.990379    2552 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 16:00:50.991051 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.991035    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-96.ec2.internal" event="NodeHasSufficientMemory"
May 26 16:00:50.991117 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.991062    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-96.ec2.internal" event="NodeHasNoDiskPressure"
May 26 16:00:50.991117 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:50.991078    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-96.ec2.internal" event="NodeHasSufficientPID"
May 26 16:00:51.004458 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:51.004441    2552 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-96.ec2.internal\" not found" node="ip-10-0-131-96.ec2.internal"
May 26 16:00:51.008565 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:51.008547    2552 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-96.ec2.internal\" not found" node="ip-10-0-131-96.ec2.internal"
May 26 16:00:51.055080 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:51.055054    2552 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-96.ec2.internal\" not found"
May 26 16:00:51.059067 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:51.059052    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/688439714ab8bc3b4b3d0132d8a1b252-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-96.ec2.internal\" (UID: \"688439714ab8bc3b4b3d0132d8a1b252\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-96.ec2.internal"
May 26 16:00:51.059126 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:51.059075    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/688439714ab8bc3b4b3d0132d8a1b252-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-96.ec2.internal\" (UID: \"688439714ab8bc3b4b3d0132d8a1b252\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-96.ec2.internal"
May 26 16:00:51.059126 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:51.059089    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b789aeffe1887cd67b4335801350d604-config\") pod \"kube-apiserver-proxy-ip-10-0-131-96.ec2.internal\" (UID: \"b789aeffe1887cd67b4335801350d604\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-96.ec2.internal"
May 26 16:00:51.155401 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:51.155361    2552 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-96.ec2.internal\" not found"
May 26 16:00:51.159647 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:51.159632    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/688439714ab8bc3b4b3d0132d8a1b252-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-96.ec2.internal\" (UID: \"688439714ab8bc3b4b3d0132d8a1b252\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-96.ec2.internal"
May 26 16:00:51.159713 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:51.159658    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/688439714ab8bc3b4b3d0132d8a1b252-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-96.ec2.internal\" (UID: \"688439714ab8bc3b4b3d0132d8a1b252\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-96.ec2.internal"
May 26 16:00:51.159713 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:51.159675    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b789aeffe1887cd67b4335801350d604-config\") pod \"kube-apiserver-proxy-ip-10-0-131-96.ec2.internal\" (UID: \"b789aeffe1887cd67b4335801350d604\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-96.ec2.internal"
May 26 16:00:51.159778 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:51.159727    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b789aeffe1887cd67b4335801350d604-config\") pod \"kube-apiserver-proxy-ip-10-0-131-96.ec2.internal\" (UID: \"b789aeffe1887cd67b4335801350d604\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-96.ec2.internal"
May 26 16:00:51.159778 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:51.159732    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/688439714ab8bc3b4b3d0132d8a1b252-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-96.ec2.internal\" (UID: \"688439714ab8bc3b4b3d0132d8a1b252\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-96.ec2.internal"
May 26 16:00:51.159778 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:51.159727    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/688439714ab8bc3b4b3d0132d8a1b252-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-96.ec2.internal\" (UID: \"688439714ab8bc3b4b3d0132d8a1b252\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-96.ec2.internal"
May 26 16:00:51.256049 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:51.256023    2552 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-96.ec2.internal\" not found"
May 26 16:00:51.307495 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:51.307474    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-96.ec2.internal"
May 26 16:00:51.311012 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:51.310994    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-96.ec2.internal"
May 26 16:00:51.356775 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:51.356746    2552 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-96.ec2.internal\" not found"
May 26 16:00:51.457215 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:51.457160    2552 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-96.ec2.internal\" not found"
May 26 16:00:51.557574 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:51.557546    2552 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-96.ec2.internal\" not found"
May 26 16:00:51.657981 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:51.657956    2552 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-96.ec2.internal\" not found"
May 26 16:00:51.665269 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:51.665248    2552 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 26 16:00:51.665383 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:51.665369    2552 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 26 16:00:51.757663 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:51.757615    2552 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 26 16:00:51.758059 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:51.758042    2552 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-96.ec2.internal\" not found"
May 26 16:00:51.784271 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:51.784240    2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-25 15:55:50 +0000 UTC" deadline="2028-02-18 02:14:40.874995044 +0000 UTC"
May 26 16:00:51.784271 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:51.784263    2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15178h13m49.090734801s"
May 26 16:00:51.790868 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:51.790850    2552 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 26 16:00:51.838063 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:51.837897    2552 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mnr8p"
May 26 16:00:51.845650 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:51.845632    2552 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mnr8p"
May 26 16:00:51.847275 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:51.847258    2552 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 16:00:51.858516 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:51.858497    2552 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-96.ec2.internal\" not found"
May 26 16:00:51.867777 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:51.867755    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb789aeffe1887cd67b4335801350d604.slice/crio-21f77db4eed67a527ec3e97ff94f7e4aa0c6770bfb0a6674f77bd3415e7c2f5c WatchSource:0}: Error finding container 21f77db4eed67a527ec3e97ff94f7e4aa0c6770bfb0a6674f77bd3415e7c2f5c: Status 404 returned error can't find the container with id 21f77db4eed67a527ec3e97ff94f7e4aa0c6770bfb0a6674f77bd3415e7c2f5c
May 26 16:00:51.868000 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:51.867976    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod688439714ab8bc3b4b3d0132d8a1b252.slice/crio-36e2b4266386f72ed8eeb02015ba5382890d122668914989d8c141237f27b31a WatchSource:0}: Error finding container 36e2b4266386f72ed8eeb02015ba5382890d122668914989d8c141237f27b31a: Status 404 returned error can't find the container with id 36e2b4266386f72ed8eeb02015ba5382890d122668914989d8c141237f27b31a
May 26 16:00:51.873265 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:51.873252    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:00:51.887323 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:51.887285    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-96.ec2.internal" event={"ID":"688439714ab8bc3b4b3d0132d8a1b252","Type":"ContainerStarted","Data":"36e2b4266386f72ed8eeb02015ba5382890d122668914989d8c141237f27b31a"}
May 26 16:00:51.888257 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:51.888235    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-96.ec2.internal" event={"ID":"b789aeffe1887cd67b4335801350d604","Type":"ContainerStarted","Data":"21f77db4eed67a527ec3e97ff94f7e4aa0c6770bfb0a6674f77bd3415e7c2f5c"}
May 26 16:00:51.892426 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:51.892394    2552 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 16:00:51.958670 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:51.958648    2552 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-96.ec2.internal\" not found"
May 26 16:00:52.059117 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:52.059063    2552 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-96.ec2.internal\" not found"
May 26 16:00:52.159504 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:52.159468    2552 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-96.ec2.internal\" not found"
May 26 16:00:52.260210 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:52.260188    2552 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-96.ec2.internal\" not found"
May 26 16:00:52.339456 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.339385    2552 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 16:00:52.357995 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.357801    2552 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-96.ec2.internal"
May 26 16:00:52.369150 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.369127    2552 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 26 16:00:52.370170 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.370146    2552 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-96.ec2.internal"
May 26 16:00:52.385256 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.385222    2552 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 26 16:00:52.735086 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.734992    2552 apiserver.go:52] "Watching apiserver"
May 26 16:00:52.741081 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.741061    2552 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 26 16:00:52.741497 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.741467    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-96.ec2.internal","openshift-network-diagnostics/network-check-target-5v7fc","openshift-ovn-kubernetes/ovnkube-node-sj6db","kube-system/global-pull-secret-syncer-p5zcf","openshift-dns/node-resolver-j7hlf","openshift-multus/multus-additional-cni-plugins-bq8pz","openshift-multus/multus-wpv87","openshift-multus/network-metrics-daemon-gj94f","openshift-network-operator/iptables-alerter-r87kr","kube-system/konnectivity-agent-v7qxk","kube-system/kube-apiserver-proxy-ip-10-0-131-96.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dpgwt","openshift-cluster-node-tuning-operator/tuned-gqgct","openshift-image-registry/node-ca-5xp7f"]
May 26 16:00:52.743065 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.743049    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bq8pz"
May 26 16:00:52.744234 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.744216    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r87kr"
May 26 16:00:52.745377 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.745350    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 26 16:00:52.745539 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.745447    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 26 16:00:52.745539 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.745516    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 26 16:00:52.745649 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.745568    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 26 16:00:52.745944 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.745914    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 26 16:00:52.746047 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.746028    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jczks\""
May 26 16:00:52.746679 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.746662    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gj94f"
May 26 16:00:52.746765 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:52.746741    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gj94f" podUID="655a0641-f708-444d-91bc-fe7518694ed3"
May 26 16:00:52.746843 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.746762    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wpv87"
May 26 16:00:52.747125 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.747109    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 26 16:00:52.747290 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.747271    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vczzv\""
May 26 16:00:52.747399 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.747373    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 26 16:00:52.747611 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.747595    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 26 16:00:52.747896 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.747870    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v7qxk"
May 26 16:00:52.749035 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.749018    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 26 16:00:52.749174 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.749156    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j7hlf"
May 26 16:00:52.749974 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.749956    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-bnzht\""
May 26 16:00:52.751067 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.750541    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 26 16:00:52.751067 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.750561    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 26 16:00:52.751067 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.750596    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.751067 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.750682    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gd4dh\""
May 26 16:00:52.752487 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.752460    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9c79z\""
May 26 16:00:52.752742 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.752721    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5v7fc"
May 26 16:00:52.752982 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.752962    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 26 16:00:52.753064 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.753019    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 26 16:00:52.753207 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.753182    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 26 16:00:52.753313 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:52.752955    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5v7fc" podUID="c4c50203-4f83-413b-b73f-edc0a16e49ee"
May 26 16:00:52.756064 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.754502    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 26 16:00:52.756064 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.754516    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 26 16:00:52.756064 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.754945    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 26 16:00:52.756064 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.755072    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 26 16:00:52.756064 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.755101    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rgtn7\""
May 26 16:00:52.756064 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.755287    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 26 16:00:52.757348 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.757033    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dpgwt"
May 26 16:00:52.758595 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.758576    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.759269 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.759249    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 26 16:00:52.759357 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.759341    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 26 16:00:52.759655 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.759514    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gmxkn\""
May 26 16:00:52.759655 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.759553    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 26 16:00:52.760288 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.760271    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5xp7f"
May 26 16:00:52.760845 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.760827    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 26 16:00:52.760947 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.760911    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 26 16:00:52.761164 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.761144    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p7mrf\""
May 26 16:00:52.761795 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.761777    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5zcf"
May 26 16:00:52.761883 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:52.761830    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5zcf" podUID="3369da97-ee5f-42cb-a95f-e6066680f349"
May 26 16:00:52.762409 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.762347    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 26 16:00:52.762556 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.762543    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 26 16:00:52.762643 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.762613    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fdt45\""
May 26 16:00:52.762774 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.762747    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 26 16:00:52.767892 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.767781    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-multus-cni-dir\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.767892 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.767813    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-host-run-k8s-cni-cncf-io\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.767892 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.767843    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-log-socket\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.767892 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.767870    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/08b4c93f-e507-4b19-8510-9fc8fffb21f4-etc-sysconfig\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.768089 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.767899    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-multus-conf-dir\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.768089 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.767922    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-etc-kubernetes\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.768089 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.767964    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-systemd-units\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.768089 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.767990    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-host-cni-netd\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.768089 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.768013    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08b4c93f-e507-4b19-8510-9fc8fffb21f4-tmp\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.768089 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.768040    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b90e9cb1-f2de-4919-ab89-8bc29c2476ea-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bq8pz\" (UID: \"b90e9cb1-f2de-4919-ab89-8bc29c2476ea\") " pod="openshift-multus/multus-additional-cni-plugins-bq8pz"
May 26 16:00:52.768089 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.768065    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49rm4\" (UniqueName: \"kubernetes.io/projected/b90e9cb1-f2de-4919-ab89-8bc29c2476ea-kube-api-access-49rm4\") pod \"multus-additional-cni-plugins-bq8pz\" (UID: \"b90e9cb1-f2de-4919-ab89-8bc29c2476ea\") " pod="openshift-multus/multus-additional-cni-plugins-bq8pz"
May 26 16:00:52.768089 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.768089    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-os-release\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.768380 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.768110    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-env-overrides\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.768380 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.768133    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b90e9cb1-f2de-4919-ab89-8bc29c2476ea-os-release\") pod \"multus-additional-cni-plugins-bq8pz\" (UID: \"b90e9cb1-f2de-4919-ab89-8bc29c2476ea\") " pod="openshift-multus/multus-additional-cni-plugins-bq8pz"
May 26 16:00:52.768380 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.768156    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-run-systemd\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.768380 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.768180    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-run-openvswitch\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.768380 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.768221    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-ovnkube-script-lib\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.768380 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.768253    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4b3e3775-013c-43d3-9914-c7cc09bd2926-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dpgwt\" (UID: \"4b3e3775-013c-43d3-9914-c7cc09bd2926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dpgwt"
May 26 16:00:52.768380 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.768285    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nszth\" (UniqueName: \"kubernetes.io/projected/4b3e3775-013c-43d3-9914-c7cc09bd2926-kube-api-access-nszth\") pod \"aws-ebs-csi-driver-node-dpgwt\" (UID: \"4b3e3775-013c-43d3-9914-c7cc09bd2926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dpgwt"
May 26 16:00:52.768380 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.768310    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5gbv\" (UniqueName: \"kubernetes.io/projected/655a0641-f708-444d-91bc-fe7518694ed3-kube-api-access-z5gbv\") pod \"network-metrics-daemon-gj94f\" (UID: \"655a0641-f708-444d-91bc-fe7518694ed3\") " pod="openshift-multus/network-metrics-daemon-gj94f"
May 26 16:00:52.768380 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.768337    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-host-var-lib-kubelet\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.768380 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.768364    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4b3e3775-013c-43d3-9914-c7cc09bd2926-socket-dir\") pod \"aws-ebs-csi-driver-node-dpgwt\" (UID: \"4b3e3775-013c-43d3-9914-c7cc09bd2926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dpgwt"
May 26 16:00:52.768735 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.768389    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-etc-openvswitch\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.768735 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.768433    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08b4c93f-e507-4b19-8510-9fc8fffb21f4-var-lib-kubelet\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.768735 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.768467    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/092e4858-1372-4cb9-b51b-dde9e50e0f9d-serviceca\") pod \"node-ca-5xp7f\" (UID: \"092e4858-1372-4cb9-b51b-dde9e50e0f9d\") " pod="openshift-image-registry/node-ca-5xp7f"
May 26 16:00:52.768735 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.768503    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b90e9cb1-f2de-4919-ab89-8bc29c2476ea-cni-binary-copy\") pod \"multus-additional-cni-plugins-bq8pz\" (UID: \"b90e9cb1-f2de-4919-ab89-8bc29c2476ea\") " pod="openshift-multus/multus-additional-cni-plugins-bq8pz"
May 26 16:00:52.768735 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.768554    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-host-var-lib-cni-bin\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.768735 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.768599    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1f8c521d-e87e-4504-af4b-b30fa7ece51b-multus-daemon-config\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.768735 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.768628    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/08b4c93f-e507-4b19-8510-9fc8fffb21f4-lib-modules\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.768735 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.768689    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b90e9cb1-f2de-4919-ab89-8bc29c2476ea-cnibin\") pod \"multus-additional-cni-plugins-bq8pz\" (UID: \"b90e9cb1-f2de-4919-ab89-8bc29c2476ea\") " pod="openshift-multus/multus-additional-cni-plugins-bq8pz"
May 26 16:00:52.769077 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.768747    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-host-run-multus-certs\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.769077 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.768774    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4b3e3775-013c-43d3-9914-c7cc09bd2926-sys-fs\") pod \"aws-ebs-csi-driver-node-dpgwt\" (UID: \"4b3e3775-013c-43d3-9914-c7cc09bd2926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dpgwt"
May 26 16:00:52.769077 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.768800    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-host-run-ovn-kubernetes\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.769077 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.768822    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08b4c93f-e507-4b19-8510-9fc8fffb21f4-sys\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.769077 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.768850    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/655a0641-f708-444d-91bc-fe7518694ed3-metrics-certs\") pod \"network-metrics-daemon-gj94f\" (UID: \"655a0641-f708-444d-91bc-fe7518694ed3\") " pod="openshift-multus/network-metrics-daemon-gj94f"
May 26 16:00:52.769077 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.768874    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-host-run-netns\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.769077 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.768909    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/08b4c93f-e507-4b19-8510-9fc8fffb21f4-etc-sysctl-d\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.769077 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.768960    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd9pn\" (UniqueName: \"kubernetes.io/projected/08b4c93f-e507-4b19-8510-9fc8fffb21f4-kube-api-access-gd9pn\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.769077 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.768992    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b90e9cb1-f2de-4919-ab89-8bc29c2476ea-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bq8pz\" (UID: \"b90e9cb1-f2de-4919-ab89-8bc29c2476ea\") " pod="openshift-multus/multus-additional-cni-plugins-bq8pz"
May 26 16:00:52.769077 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769016    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b90e9cb1-f2de-4919-ab89-8bc29c2476ea-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bq8pz\" (UID: \"b90e9cb1-f2de-4919-ab89-8bc29c2476ea\") " pod="openshift-multus/multus-additional-cni-plugins-bq8pz"
May 26 16:00:52.769077 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769040    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6qxx\" (UniqueName: \"kubernetes.io/projected/c4c50203-4f83-413b-b73f-edc0a16e49ee-kube-api-access-w6qxx\") pod \"network-check-target-5v7fc\" (UID: \"c4c50203-4f83-413b-b73f-edc0a16e49ee\") " pod="openshift-network-diagnostics/network-check-target-5v7fc"
May 26 16:00:52.769077 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769059    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4475d020-5281-4257-bf03-1a264049169d-tmp-dir\") pod \"node-resolver-j7hlf\" (UID: \"4475d020-5281-4257-bf03-1a264049169d\") " pod="openshift-dns/node-resolver-j7hlf"
May 26 16:00:52.769077 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769077    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-host-var-lib-cni-multus\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.769655 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769099    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jjr7\" (UniqueName: \"kubernetes.io/projected/1f8c521d-e87e-4504-af4b-b30fa7ece51b-kube-api-access-8jjr7\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.769655 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769123    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/50bb2f40-6580-4a07-abf3-e6cd8ca11465-konnectivity-ca\") pod \"konnectivity-agent-v7qxk\" (UID: \"50bb2f40-6580-4a07-abf3-e6cd8ca11465\") " pod="kube-system/konnectivity-agent-v7qxk"
May 26 16:00:52.769655 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769144    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5zhx\" (UniqueName: \"kubernetes.io/projected/4475d020-5281-4257-bf03-1a264049169d-kube-api-access-g5zhx\") pod \"node-resolver-j7hlf\" (UID: \"4475d020-5281-4257-bf03-1a264049169d\") " pod="openshift-dns/node-resolver-j7hlf"
May 26 16:00:52.769655 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769168    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-host-kubelet\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.769655 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769189    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/08b4c93f-e507-4b19-8510-9fc8fffb21f4-etc-systemd\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.769655 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769225    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f8c521d-e87e-4504-af4b-b30fa7ece51b-cni-binary-copy\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.769655 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769253    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-multus-socket-dir-parent\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.769655 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769276    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-host-run-netns\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.769655 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769300    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b90e9cb1-f2de-4919-ab89-8bc29c2476ea-system-cni-dir\") pod \"multus-additional-cni-plugins-bq8pz\" (UID: \"b90e9cb1-f2de-4919-ab89-8bc29c2476ea\") " pod="openshift-multus/multus-additional-cni-plugins-bq8pz"
May 26 16:00:52.769655 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769323    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz9k6\" (UniqueName: \"kubernetes.io/projected/6b19d97f-df30-477c-941f-93bcf7e8c5c3-kube-api-access-fz9k6\") pod \"iptables-alerter-r87kr\" (UID: \"6b19d97f-df30-477c-941f-93bcf7e8c5c3\") " pod="openshift-network-operator/iptables-alerter-r87kr"
May 26 16:00:52.769655 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769345    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-cnibin\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.769655 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769369    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-var-lib-openvswitch\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.769655 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769392    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-host-cni-bin\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.769655 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769415    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/08b4c93f-e507-4b19-8510-9fc8fffb21f4-etc-modprobe-d\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.769655 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769437    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/50bb2f40-6580-4a07-abf3-e6cd8ca11465-agent-certs\") pod \"konnectivity-agent-v7qxk\" (UID: \"50bb2f40-6580-4a07-abf3-e6cd8ca11465\") " pod="kube-system/konnectivity-agent-v7qxk"
May 26 16:00:52.769655 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769462    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4475d020-5281-4257-bf03-1a264049169d-hosts-file\") pod \"node-resolver-j7hlf\" (UID: \"4475d020-5281-4257-bf03-1a264049169d\") " pod="openshift-dns/node-resolver-j7hlf"
May 26 16:00:52.769655 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769485    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-run-ovn\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.770512 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769508    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/08b4c93f-e507-4b19-8510-9fc8fffb21f4-etc-sysctl-conf\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.770512 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769530    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/08b4c93f-e507-4b19-8510-9fc8fffb21f4-etc-tuned\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.770512 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769552    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b19d97f-df30-477c-941f-93bcf7e8c5c3-host-slash\") pod \"iptables-alerter-r87kr\" (UID: \"6b19d97f-df30-477c-941f-93bcf7e8c5c3\") " pod="openshift-network-operator/iptables-alerter-r87kr"
May 26 16:00:52.770512 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769584    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4b3e3775-013c-43d3-9914-c7cc09bd2926-device-dir\") pod \"aws-ebs-csi-driver-node-dpgwt\" (UID: \"4b3e3775-013c-43d3-9914-c7cc09bd2926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dpgwt"
May 26 16:00:52.770512 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769609    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4b3e3775-013c-43d3-9914-c7cc09bd2926-etc-selinux\") pod \"aws-ebs-csi-driver-node-dpgwt\" (UID: \"4b3e3775-013c-43d3-9914-c7cc09bd2926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dpgwt"
May 26 16:00:52.770512 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769632    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.770512 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769658    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-ovn-node-metrics-cert\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.770512 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769684    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfp5q\" (UniqueName: \"kubernetes.io/projected/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-kube-api-access-dfp5q\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.770512 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769705    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/08b4c93f-e507-4b19-8510-9fc8fffb21f4-run\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.770512 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769727    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-system-cni-dir\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.770512 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769748    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-hostroot\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.770512 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769769    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4b3e3775-013c-43d3-9914-c7cc09bd2926-registration-dir\") pod \"aws-ebs-csi-driver-node-dpgwt\" (UID: \"4b3e3775-013c-43d3-9914-c7cc09bd2926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dpgwt"
May 26 16:00:52.770512 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769785    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-host-slash\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.770512 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769805    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-node-log\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.770512 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769832    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08b4c93f-e507-4b19-8510-9fc8fffb21f4-etc-kubernetes\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.770512 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769853    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08b4c93f-e507-4b19-8510-9fc8fffb21f4-host\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.771286 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769874    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/092e4858-1372-4cb9-b51b-dde9e50e0f9d-host\") pod \"node-ca-5xp7f\" (UID: \"092e4858-1372-4cb9-b51b-dde9e50e0f9d\") " pod="openshift-image-registry/node-ca-5xp7f"
May 26 16:00:52.771286 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769896    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-ovnkube-config\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.771286 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769921    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-758fk\" (UniqueName: \"kubernetes.io/projected/092e4858-1372-4cb9-b51b-dde9e50e0f9d-kube-api-access-758fk\") pod \"node-ca-5xp7f\" (UID: \"092e4858-1372-4cb9-b51b-dde9e50e0f9d\") " pod="openshift-image-registry/node-ca-5xp7f"
May 26 16:00:52.771286 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.769973    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6b19d97f-df30-477c-941f-93bcf7e8c5c3-iptables-alerter-script\") pod \"iptables-alerter-r87kr\" (UID: \"6b19d97f-df30-477c-941f-93bcf7e8c5c3\") " pod="openshift-network-operator/iptables-alerter-r87kr"
May 26 16:00:52.792981 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.792966    2552 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 16:00:52.847017 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.846981    2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-25 15:55:51 +0000 UTC" deadline="2028-01-04 04:40:55.868047296 +0000 UTC"
May 26 16:00:52.847017 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.847007    2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14100h40m3.02104294s"
May 26 16:00:52.858733 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.858712    2552 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 26 16:00:52.870558 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.870533    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-run-openvswitch\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.870652 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.870573    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-ovnkube-script-lib\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.870652 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.870593    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4b3e3775-013c-43d3-9914-c7cc09bd2926-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dpgwt\" (UID: \"4b3e3775-013c-43d3-9914-c7cc09bd2926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dpgwt"
May 26 16:00:52.870652 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.870614    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nszth\" (UniqueName: \"kubernetes.io/projected/4b3e3775-013c-43d3-9914-c7cc09bd2926-kube-api-access-nszth\") pod \"aws-ebs-csi-driver-node-dpgwt\" (UID: \"4b3e3775-013c-43d3-9914-c7cc09bd2926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dpgwt"
May 26 16:00:52.870797 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.870655    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-run-openvswitch\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.870797 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.870674    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4b3e3775-013c-43d3-9914-c7cc09bd2926-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dpgwt\" (UID: \"4b3e3775-013c-43d3-9914-c7cc09bd2926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dpgwt"
May 26 16:00:52.870797 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.870709    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-z5gbv\" (UniqueName: \"kubernetes.io/projected/655a0641-f708-444d-91bc-fe7518694ed3-kube-api-access-z5gbv\") pod \"network-metrics-daemon-gj94f\" (UID: \"655a0641-f708-444d-91bc-fe7518694ed3\") " pod="openshift-multus/network-metrics-daemon-gj94f"
May 26 16:00:52.870797 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.870738    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-host-var-lib-kubelet\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.870797 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.870765    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4b3e3775-013c-43d3-9914-c7cc09bd2926-socket-dir\") pod \"aws-ebs-csi-driver-node-dpgwt\" (UID: \"4b3e3775-013c-43d3-9914-c7cc09bd2926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dpgwt"
May 26 16:00:52.870797 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.870795    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-etc-openvswitch\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.871043 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.870820    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08b4c93f-e507-4b19-8510-9fc8fffb21f4-var-lib-kubelet\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.871043 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.870822    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-host-var-lib-kubelet\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.871043 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.870843    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/092e4858-1372-4cb9-b51b-dde9e50e0f9d-serviceca\") pod \"node-ca-5xp7f\" (UID: \"092e4858-1372-4cb9-b51b-dde9e50e0f9d\") " pod="openshift-image-registry/node-ca-5xp7f"
May 26 16:00:52.871043 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.870867    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b90e9cb1-f2de-4919-ab89-8bc29c2476ea-cni-binary-copy\") pod \"multus-additional-cni-plugins-bq8pz\" (UID: \"b90e9cb1-f2de-4919-ab89-8bc29c2476ea\") " pod="openshift-multus/multus-additional-cni-plugins-bq8pz"
May 26 16:00:52.871043 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.870891    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-host-var-lib-cni-bin\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.871043 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.870894    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-etc-openvswitch\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.871043 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.870916    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1f8c521d-e87e-4504-af4b-b30fa7ece51b-multus-daemon-config\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.871043 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.870975    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4b3e3775-013c-43d3-9914-c7cc09bd2926-socket-dir\") pod \"aws-ebs-csi-driver-node-dpgwt\" (UID: \"4b3e3775-013c-43d3-9914-c7cc09bd2926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dpgwt"
May 26 16:00:52.871043 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.870981    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/08b4c93f-e507-4b19-8510-9fc8fffb21f4-lib-modules\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.871496 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871019    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b90e9cb1-f2de-4919-ab89-8bc29c2476ea-cnibin\") pod \"multus-additional-cni-plugins-bq8pz\" (UID: \"b90e9cb1-f2de-4919-ab89-8bc29c2476ea\") " pod="openshift-multus/multus-additional-cni-plugins-bq8pz"
May 26 16:00:52.871496 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871086    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-host-run-multus-certs\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.871496 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871114    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4b3e3775-013c-43d3-9914-c7cc09bd2926-sys-fs\") pod \"aws-ebs-csi-driver-node-dpgwt\" (UID: \"4b3e3775-013c-43d3-9914-c7cc09bd2926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dpgwt"
May 26 16:00:52.871496 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871121    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b90e9cb1-f2de-4919-ab89-8bc29c2476ea-cnibin\") pod \"multus-additional-cni-plugins-bq8pz\" (UID: \"b90e9cb1-f2de-4919-ab89-8bc29c2476ea\") " pod="openshift-multus/multus-additional-cni-plugins-bq8pz"
May 26 16:00:52.871496 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871089    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/08b4c93f-e507-4b19-8510-9fc8fffb21f4-lib-modules\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.871496 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871141    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-host-run-ovn-kubernetes\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.871496 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871165    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08b4c93f-e507-4b19-8510-9fc8fffb21f4-sys\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.871496 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871170    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-host-run-multus-certs\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.871496 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871190    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/655a0641-f708-444d-91bc-fe7518694ed3-metrics-certs\") pod \"network-metrics-daemon-gj94f\" (UID: \"655a0641-f708-444d-91bc-fe7518694ed3\") " pod="openshift-multus/network-metrics-daemon-gj94f"
May 26 16:00:52.871496 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871211    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-host-run-ovn-kubernetes\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.871496 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871216    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-host-run-netns\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.871496 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871225    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-ovnkube-script-lib\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.871496 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871260    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4b3e3775-013c-43d3-9914-c7cc09bd2926-sys-fs\") pod \"aws-ebs-csi-driver-node-dpgwt\" (UID: \"4b3e3775-013c-43d3-9914-c7cc09bd2926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dpgwt"
May 26 16:00:52.871496 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871282    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08b4c93f-e507-4b19-8510-9fc8fffb21f4-sys\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.871496 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871284    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-host-run-netns\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.871496 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871287    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/08b4c93f-e507-4b19-8510-9fc8fffb21f4-etc-sysctl-d\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.871496 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871331    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gd9pn\" (UniqueName: \"kubernetes.io/projected/08b4c93f-e507-4b19-8510-9fc8fffb21f4-kube-api-access-gd9pn\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.871496 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871340    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08b4c93f-e507-4b19-8510-9fc8fffb21f4-var-lib-kubelet\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.872323 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871353    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/092e4858-1372-4cb9-b51b-dde9e50e0f9d-serviceca\") pod \"node-ca-5xp7f\" (UID: \"092e4858-1372-4cb9-b51b-dde9e50e0f9d\") " pod="openshift-image-registry/node-ca-5xp7f"
May 26 16:00:52.872323 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871368    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b90e9cb1-f2de-4919-ab89-8bc29c2476ea-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bq8pz\" (UID: \"b90e9cb1-f2de-4919-ab89-8bc29c2476ea\") " pod="openshift-multus/multus-additional-cni-plugins-bq8pz"
May 26 16:00:52.872323 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:52.871378    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:52.872323 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871393    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b90e9cb1-f2de-4919-ab89-8bc29c2476ea-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bq8pz\" (UID: \"b90e9cb1-f2de-4919-ab89-8bc29c2476ea\") " pod="openshift-multus/multus-additional-cni-plugins-bq8pz"
May 26 16:00:52.872323 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871420    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3369da97-ee5f-42cb-a95f-e6066680f349-kubelet-config\") pod \"global-pull-secret-syncer-p5zcf\" (UID: \"3369da97-ee5f-42cb-a95f-e6066680f349\") " pod="kube-system/global-pull-secret-syncer-p5zcf"
May 26 16:00:52.872323 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871456    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/08b4c93f-e507-4b19-8510-9fc8fffb21f4-etc-sysctl-d\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.872323 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871456    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1f8c521d-e87e-4504-af4b-b30fa7ece51b-multus-daemon-config\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.872323 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:52.871477    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/655a0641-f708-444d-91bc-fe7518694ed3-metrics-certs podName:655a0641-f708-444d-91bc-fe7518694ed3 nodeName:}" failed. No retries permitted until 2026-05-26 16:00:53.371426529 +0000 UTC m=+3.079881550 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/655a0641-f708-444d-91bc-fe7518694ed3-metrics-certs") pod "network-metrics-daemon-gj94f" (UID: "655a0641-f708-444d-91bc-fe7518694ed3") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:52.872323 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871497    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-host-var-lib-cni-bin\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.872323 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871528    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b90e9cb1-f2de-4919-ab89-8bc29c2476ea-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bq8pz\" (UID: \"b90e9cb1-f2de-4919-ab89-8bc29c2476ea\") " pod="openshift-multus/multus-additional-cni-plugins-bq8pz"
May 26 16:00:52.872323 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871573    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w6qxx\" (UniqueName: \"kubernetes.io/projected/c4c50203-4f83-413b-b73f-edc0a16e49ee-kube-api-access-w6qxx\") pod \"network-check-target-5v7fc\" (UID: \"c4c50203-4f83-413b-b73f-edc0a16e49ee\") " pod="openshift-network-diagnostics/network-check-target-5v7fc"
May 26 16:00:52.872323 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871597    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4475d020-5281-4257-bf03-1a264049169d-tmp-dir\") pod \"node-resolver-j7hlf\" (UID: \"4475d020-5281-4257-bf03-1a264049169d\") " pod="openshift-dns/node-resolver-j7hlf"
May 26 16:00:52.872323 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871621    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-host-var-lib-cni-multus\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.872323 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871644    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8jjr7\" (UniqueName: \"kubernetes.io/projected/1f8c521d-e87e-4504-af4b-b30fa7ece51b-kube-api-access-8jjr7\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.872323 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871670    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/50bb2f40-6580-4a07-abf3-e6cd8ca11465-konnectivity-ca\") pod \"konnectivity-agent-v7qxk\" (UID: \"50bb2f40-6580-4a07-abf3-e6cd8ca11465\") " pod="kube-system/konnectivity-agent-v7qxk"
May 26 16:00:52.872323 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871704    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g5zhx\" (UniqueName: \"kubernetes.io/projected/4475d020-5281-4257-bf03-1a264049169d-kube-api-access-g5zhx\") pod \"node-resolver-j7hlf\" (UID: \"4475d020-5281-4257-bf03-1a264049169d\") " pod="openshift-dns/node-resolver-j7hlf"
May 26 16:00:52.872323 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871711    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-host-var-lib-cni-multus\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.873140 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871728    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-host-kubelet\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.873140 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871752    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/08b4c93f-e507-4b19-8510-9fc8fffb21f4-etc-systemd\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.873140 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871777    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f8c521d-e87e-4504-af4b-b30fa7ece51b-cni-binary-copy\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.873140 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871803    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-multus-socket-dir-parent\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.873140 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871830    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-host-run-netns\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.873140 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871854    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b90e9cb1-f2de-4919-ab89-8bc29c2476ea-system-cni-dir\") pod \"multus-additional-cni-plugins-bq8pz\" (UID: \"b90e9cb1-f2de-4919-ab89-8bc29c2476ea\") " pod="openshift-multus/multus-additional-cni-plugins-bq8pz"
May 26 16:00:52.873140 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871880    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fz9k6\" (UniqueName: \"kubernetes.io/projected/6b19d97f-df30-477c-941f-93bcf7e8c5c3-kube-api-access-fz9k6\") pod \"iptables-alerter-r87kr\" (UID: \"6b19d97f-df30-477c-941f-93bcf7e8c5c3\") " pod="openshift-network-operator/iptables-alerter-r87kr"
May 26 16:00:52.873140 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871907    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-cnibin\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.873140 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871894    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b90e9cb1-f2de-4919-ab89-8bc29c2476ea-cni-binary-copy\") pod \"multus-additional-cni-plugins-bq8pz\" (UID: \"b90e9cb1-f2de-4919-ab89-8bc29c2476ea\") " pod="openshift-multus/multus-additional-cni-plugins-bq8pz"
May 26 16:00:52.873140 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871904    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b90e9cb1-f2de-4919-ab89-8bc29c2476ea-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bq8pz\" (UID: \"b90e9cb1-f2de-4919-ab89-8bc29c2476ea\") " pod="openshift-multus/multus-additional-cni-plugins-bq8pz"
May 26 16:00:52.873140 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871978    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-host-kubelet\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.873140 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.871997    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-cnibin\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.873140 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872001    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-var-lib-openvswitch\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.873140 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872025    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/08b4c93f-e507-4b19-8510-9fc8fffb21f4-etc-systemd\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.873140 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872036    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-var-lib-openvswitch\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.873140 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872063    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4475d020-5281-4257-bf03-1a264049169d-tmp-dir\") pod \"node-resolver-j7hlf\" (UID: \"4475d020-5281-4257-bf03-1a264049169d\") " pod="openshift-dns/node-resolver-j7hlf"
May 26 16:00:52.873140 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872078    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-host-cni-bin\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.873140 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872083    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-multus-socket-dir-parent\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.873956 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872083    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-host-run-netns\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.873956 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872036    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-host-cni-bin\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.873956 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872125    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/08b4c93f-e507-4b19-8510-9fc8fffb21f4-etc-modprobe-d\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.873956 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872154    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3369da97-ee5f-42cb-a95f-e6066680f349-original-pull-secret\") pod \"global-pull-secret-syncer-p5zcf\" (UID: \"3369da97-ee5f-42cb-a95f-e6066680f349\") " pod="kube-system/global-pull-secret-syncer-p5zcf"
May 26 16:00:52.873956 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872179    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/50bb2f40-6580-4a07-abf3-e6cd8ca11465-agent-certs\") pod \"konnectivity-agent-v7qxk\" (UID: \"50bb2f40-6580-4a07-abf3-e6cd8ca11465\") " pod="kube-system/konnectivity-agent-v7qxk"
May 26 16:00:52.873956 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872201    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4475d020-5281-4257-bf03-1a264049169d-hosts-file\") pod \"node-resolver-j7hlf\" (UID: \"4475d020-5281-4257-bf03-1a264049169d\") " pod="openshift-dns/node-resolver-j7hlf"
May 26 16:00:52.873956 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872226    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-run-ovn\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.873956 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872251    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/08b4c93f-e507-4b19-8510-9fc8fffb21f4-etc-sysctl-conf\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.873956 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872268    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/08b4c93f-e507-4b19-8510-9fc8fffb21f4-etc-modprobe-d\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.873956 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872275    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/08b4c93f-e507-4b19-8510-9fc8fffb21f4-etc-tuned\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.873956 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872302    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b19d97f-df30-477c-941f-93bcf7e8c5c3-host-slash\") pod \"iptables-alerter-r87kr\" (UID: \"6b19d97f-df30-477c-941f-93bcf7e8c5c3\") " pod="openshift-network-operator/iptables-alerter-r87kr"
May 26 16:00:52.873956 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872330    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4475d020-5281-4257-bf03-1a264049169d-hosts-file\") pod \"node-resolver-j7hlf\" (UID: \"4475d020-5281-4257-bf03-1a264049169d\") " pod="openshift-dns/node-resolver-j7hlf"
May 26 16:00:52.873956 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872340    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4b3e3775-013c-43d3-9914-c7cc09bd2926-device-dir\") pod \"aws-ebs-csi-driver-node-dpgwt\" (UID: \"4b3e3775-013c-43d3-9914-c7cc09bd2926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dpgwt"
May 26 16:00:52.873956 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872365    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4b3e3775-013c-43d3-9914-c7cc09bd2926-etc-selinux\") pod \"aws-ebs-csi-driver-node-dpgwt\" (UID: \"4b3e3775-013c-43d3-9914-c7cc09bd2926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dpgwt"
May 26 16:00:52.873956 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872410    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/08b4c93f-e507-4b19-8510-9fc8fffb21f4-etc-sysctl-conf\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.873956 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872417    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.873956 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872427    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b19d97f-df30-477c-941f-93bcf7e8c5c3-host-slash\") pod \"iptables-alerter-r87kr\" (UID: \"6b19d97f-df30-477c-941f-93bcf7e8c5c3\") " pod="openshift-network-operator/iptables-alerter-r87kr"
May 26 16:00:52.874764 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872456    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-run-ovn\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.874764 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872485    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-ovn-node-metrics-cert\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.874764 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872494    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4b3e3775-013c-43d3-9914-c7cc09bd2926-device-dir\") pod \"aws-ebs-csi-driver-node-dpgwt\" (UID: \"4b3e3775-013c-43d3-9914-c7cc09bd2926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dpgwt"
May 26 16:00:52.874764 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872492    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4b3e3775-013c-43d3-9914-c7cc09bd2926-etc-selinux\") pod \"aws-ebs-csi-driver-node-dpgwt\" (UID: \"4b3e3775-013c-43d3-9914-c7cc09bd2926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dpgwt"
May 26 16:00:52.874764 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872517    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dfp5q\" (UniqueName: \"kubernetes.io/projected/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-kube-api-access-dfp5q\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.874764 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872541    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f8c521d-e87e-4504-af4b-b30fa7ece51b-cni-binary-copy\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.874764 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872549    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/08b4c93f-e507-4b19-8510-9fc8fffb21f4-run\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.874764 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872541    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.874764 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872575    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-system-cni-dir\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.874764 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872599    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-hostroot\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.874764 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872602    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/08b4c93f-e507-4b19-8510-9fc8fffb21f4-run\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.874764 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872625    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4b3e3775-013c-43d3-9914-c7cc09bd2926-registration-dir\") pod \"aws-ebs-csi-driver-node-dpgwt\" (UID: \"4b3e3775-013c-43d3-9914-c7cc09bd2926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dpgwt"
May 26 16:00:52.874764 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872649    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-host-slash\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.874764 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872662    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-system-cni-dir\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.874764 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872673    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-node-log\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.874764 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872701    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4b3e3775-013c-43d3-9914-c7cc09bd2926-registration-dir\") pod \"aws-ebs-csi-driver-node-dpgwt\" (UID: \"4b3e3775-013c-43d3-9914-c7cc09bd2926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dpgwt"
May 26 16:00:52.874764 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872717    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-hostroot\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.874764 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872705    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08b4c93f-e507-4b19-8510-9fc8fffb21f4-etc-kubernetes\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.875406 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872647    2552 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 26 16:00:52.875406 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872740    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-host-slash\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.875406 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872742    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-node-log\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.875406 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872760    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08b4c93f-e507-4b19-8510-9fc8fffb21f4-host\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.875406 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872786    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/092e4858-1372-4cb9-b51b-dde9e50e0f9d-host\") pod \"node-ca-5xp7f\" (UID: \"092e4858-1372-4cb9-b51b-dde9e50e0f9d\") " pod="openshift-image-registry/node-ca-5xp7f"
May 26 16:00:52.875406 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872791    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08b4c93f-e507-4b19-8510-9fc8fffb21f4-etc-kubernetes\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.875406 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872812    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-ovnkube-config\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.875406 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872833    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08b4c93f-e507-4b19-8510-9fc8fffb21f4-host\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.875406 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872839    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-758fk\" (UniqueName: \"kubernetes.io/projected/092e4858-1372-4cb9-b51b-dde9e50e0f9d-kube-api-access-758fk\") pod \"node-ca-5xp7f\" (UID: \"092e4858-1372-4cb9-b51b-dde9e50e0f9d\") " pod="openshift-image-registry/node-ca-5xp7f"
May 26 16:00:52.875406 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872864    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/092e4858-1372-4cb9-b51b-dde9e50e0f9d-host\") pod \"node-ca-5xp7f\" (UID: \"092e4858-1372-4cb9-b51b-dde9e50e0f9d\") " pod="openshift-image-registry/node-ca-5xp7f"
May 26 16:00:52.875406 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872869    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6b19d97f-df30-477c-941f-93bcf7e8c5c3-iptables-alerter-script\") pod \"iptables-alerter-r87kr\" (UID: \"6b19d97f-df30-477c-941f-93bcf7e8c5c3\") " pod="openshift-network-operator/iptables-alerter-r87kr"
May 26 16:00:52.875406 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872899    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-multus-cni-dir\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.875406 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872923    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-host-run-k8s-cni-cncf-io\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.875406 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872964    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-log-socket\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.875406 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.872987    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/08b4c93f-e507-4b19-8510-9fc8fffb21f4-etc-sysconfig\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.875406 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.873014    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-multus-conf-dir\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.875406 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.873041    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-etc-kubernetes\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.875406 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.873073    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-systemd-units\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.876083 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.873095    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-host-cni-netd\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.876083 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.873118    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08b4c93f-e507-4b19-8510-9fc8fffb21f4-tmp\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.876083 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.873142    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b90e9cb1-f2de-4919-ab89-8bc29c2476ea-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bq8pz\" (UID: \"b90e9cb1-f2de-4919-ab89-8bc29c2476ea\") " pod="openshift-multus/multus-additional-cni-plugins-bq8pz"
May 26 16:00:52.876083 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.873184    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-49rm4\" (UniqueName: \"kubernetes.io/projected/b90e9cb1-f2de-4919-ab89-8bc29c2476ea-kube-api-access-49rm4\") pod \"multus-additional-cni-plugins-bq8pz\" (UID: \"b90e9cb1-f2de-4919-ab89-8bc29c2476ea\") " pod="openshift-multus/multus-additional-cni-plugins-bq8pz"
May 26 16:00:52.876083 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.873213    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-os-release\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.876083 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.873234    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-env-overrides\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.876083 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.873255    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b90e9cb1-f2de-4919-ab89-8bc29c2476ea-os-release\") pod \"multus-additional-cni-plugins-bq8pz\" (UID: \"b90e9cb1-f2de-4919-ab89-8bc29c2476ea\") " pod="openshift-multus/multus-additional-cni-plugins-bq8pz"
May 26 16:00:52.876083 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.873292    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3369da97-ee5f-42cb-a95f-e6066680f349-dbus\") pod \"global-pull-secret-syncer-p5zcf\" (UID: \"3369da97-ee5f-42cb-a95f-e6066680f349\") " pod="kube-system/global-pull-secret-syncer-p5zcf"
May 26 16:00:52.876083 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.873313    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-run-systemd\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.876083 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.873334    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-ovnkube-config\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.876083 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.873376    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-run-systemd\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.876083 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.873374    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6b19d97f-df30-477c-941f-93bcf7e8c5c3-iptables-alerter-script\") pod \"iptables-alerter-r87kr\" (UID: \"6b19d97f-df30-477c-941f-93bcf7e8c5c3\") " pod="openshift-network-operator/iptables-alerter-r87kr"
May 26 16:00:52.876083 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.873392    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-host-cni-netd\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.876083 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.873438    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-multus-cni-dir\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.876083 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.873472    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-host-run-k8s-cni-cncf-io\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.876083 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.873509    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-log-socket\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.876083 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.873546    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/08b4c93f-e507-4b19-8510-9fc8fffb21f4-etc-sysconfig\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.876902 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.873578    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-multus-conf-dir\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.876902 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.873609    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-etc-kubernetes\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.876902 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.873642    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-systemd-units\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.876902 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.873655    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1f8c521d-e87e-4504-af4b-b30fa7ece51b-os-release\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.876902 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.873878    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b90e9cb1-f2de-4919-ab89-8bc29c2476ea-os-release\") pod \"multus-additional-cni-plugins-bq8pz\" (UID: \"b90e9cb1-f2de-4919-ab89-8bc29c2476ea\") " pod="openshift-multus/multus-additional-cni-plugins-bq8pz"
May 26 16:00:52.876902 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.873883    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/50bb2f40-6580-4a07-abf3-e6cd8ca11465-konnectivity-ca\") pod \"konnectivity-agent-v7qxk\" (UID: \"50bb2f40-6580-4a07-abf3-e6cd8ca11465\") " pod="kube-system/konnectivity-agent-v7qxk"
May 26 16:00:52.876902 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.873960    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b90e9cb1-f2de-4919-ab89-8bc29c2476ea-system-cni-dir\") pod \"multus-additional-cni-plugins-bq8pz\" (UID: \"b90e9cb1-f2de-4919-ab89-8bc29c2476ea\") " pod="openshift-multus/multus-additional-cni-plugins-bq8pz"
May 26 16:00:52.876902 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.874167    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b90e9cb1-f2de-4919-ab89-8bc29c2476ea-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bq8pz\" (UID: \"b90e9cb1-f2de-4919-ab89-8bc29c2476ea\") " pod="openshift-multus/multus-additional-cni-plugins-bq8pz"
May 26 16:00:52.876902 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.874257    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-env-overrides\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.876902 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.876272    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/08b4c93f-e507-4b19-8510-9fc8fffb21f4-etc-tuned\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.876902 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.876299    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08b4c93f-e507-4b19-8510-9fc8fffb21f4-tmp\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.876902 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.876378    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-ovn-node-metrics-cert\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.876902 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.876482    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/50bb2f40-6580-4a07-abf3-e6cd8ca11465-agent-certs\") pod \"konnectivity-agent-v7qxk\" (UID: \"50bb2f40-6580-4a07-abf3-e6cd8ca11465\") " pod="kube-system/konnectivity-agent-v7qxk"
May 26 16:00:52.886429 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.886405    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5gbv\" (UniqueName: \"kubernetes.io/projected/655a0641-f708-444d-91bc-fe7518694ed3-kube-api-access-z5gbv\") pod \"network-metrics-daemon-gj94f\" (UID: \"655a0641-f708-444d-91bc-fe7518694ed3\") " pod="openshift-multus/network-metrics-daemon-gj94f"
May 26 16:00:52.888062 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.888044    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz9k6\" (UniqueName: \"kubernetes.io/projected/6b19d97f-df30-477c-941f-93bcf7e8c5c3-kube-api-access-fz9k6\") pod \"iptables-alerter-r87kr\" (UID: \"6b19d97f-df30-477c-941f-93bcf7e8c5c3\") " pod="openshift-network-operator/iptables-alerter-r87kr"
May 26 16:00:52.888192 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.888083    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jjr7\" (UniqueName: \"kubernetes.io/projected/1f8c521d-e87e-4504-af4b-b30fa7ece51b-kube-api-access-8jjr7\") pod \"multus-wpv87\" (UID: \"1f8c521d-e87e-4504-af4b-b30fa7ece51b\") " pod="openshift-multus/multus-wpv87"
May 26 16:00:52.890305 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:52.890059    2552 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:00:52.890305 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:52.890082    2552 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:00:52.890305 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:52.890095    2552 projected.go:194] Error preparing data for projected volume kube-api-access-w6qxx for pod openshift-network-diagnostics/network-check-target-5v7fc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:52.890305 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:52.890147    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4c50203-4f83-413b-b73f-edc0a16e49ee-kube-api-access-w6qxx podName:c4c50203-4f83-413b-b73f-edc0a16e49ee nodeName:}" failed. No retries permitted until 2026-05-26 16:00:53.390131486 +0000 UTC m=+3.098586527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-w6qxx" (UniqueName: "kubernetes.io/projected/c4c50203-4f83-413b-b73f-edc0a16e49ee-kube-api-access-w6qxx") pod "network-check-target-5v7fc" (UID: "c4c50203-4f83-413b-b73f-edc0a16e49ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:52.890572 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.890510    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49rm4\" (UniqueName: \"kubernetes.io/projected/b90e9cb1-f2de-4919-ab89-8bc29c2476ea-kube-api-access-49rm4\") pod \"multus-additional-cni-plugins-bq8pz\" (UID: \"b90e9cb1-f2de-4919-ab89-8bc29c2476ea\") " pod="openshift-multus/multus-additional-cni-plugins-bq8pz"
May 26 16:00:52.891322 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.891303    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfp5q\" (UniqueName: \"kubernetes.io/projected/5c0f9fc6-2bd6-4370-96d0-5ae315363c7f-kube-api-access-dfp5q\") pod \"ovnkube-node-sj6db\" (UID: \"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:52.893290 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.893268    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nszth\" (UniqueName: \"kubernetes.io/projected/4b3e3775-013c-43d3-9914-c7cc09bd2926-kube-api-access-nszth\") pod \"aws-ebs-csi-driver-node-dpgwt\" (UID: \"4b3e3775-013c-43d3-9914-c7cc09bd2926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dpgwt"
May 26 16:00:52.894021 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.894002    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-758fk\" (UniqueName: \"kubernetes.io/projected/092e4858-1372-4cb9-b51b-dde9e50e0f9d-kube-api-access-758fk\") pod \"node-ca-5xp7f\" (UID: \"092e4858-1372-4cb9-b51b-dde9e50e0f9d\") " pod="openshift-image-registry/node-ca-5xp7f"
May 26 16:00:52.895670 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.895646    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd9pn\" (UniqueName: \"kubernetes.io/projected/08b4c93f-e507-4b19-8510-9fc8fffb21f4-kube-api-access-gd9pn\") pod \"tuned-gqgct\" (UID: \"08b4c93f-e507-4b19-8510-9fc8fffb21f4\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:52.895773 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.895757    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5zhx\" (UniqueName: \"kubernetes.io/projected/4475d020-5281-4257-bf03-1a264049169d-kube-api-access-g5zhx\") pod \"node-resolver-j7hlf\" (UID: \"4475d020-5281-4257-bf03-1a264049169d\") " pod="openshift-dns/node-resolver-j7hlf"
May 26 16:00:52.973554 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.973533    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3369da97-ee5f-42cb-a95f-e6066680f349-kubelet-config\") pod \"global-pull-secret-syncer-p5zcf\" (UID: \"3369da97-ee5f-42cb-a95f-e6066680f349\") " pod="kube-system/global-pull-secret-syncer-p5zcf"
May 26 16:00:52.973649 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.973576    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3369da97-ee5f-42cb-a95f-e6066680f349-original-pull-secret\") pod \"global-pull-secret-syncer-p5zcf\" (UID: \"3369da97-ee5f-42cb-a95f-e6066680f349\") " pod="kube-system/global-pull-secret-syncer-p5zcf"
May 26 16:00:52.973649 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.973622    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3369da97-ee5f-42cb-a95f-e6066680f349-dbus\") pod \"global-pull-secret-syncer-p5zcf\" (UID: \"3369da97-ee5f-42cb-a95f-e6066680f349\") " pod="kube-system/global-pull-secret-syncer-p5zcf"
May 26 16:00:52.973747 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.973659    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3369da97-ee5f-42cb-a95f-e6066680f349-kubelet-config\") pod \"global-pull-secret-syncer-p5zcf\" (UID: \"3369da97-ee5f-42cb-a95f-e6066680f349\") " pod="kube-system/global-pull-secret-syncer-p5zcf"
May 26 16:00:52.973747 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:52.973696    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3369da97-ee5f-42cb-a95f-e6066680f349-dbus\") pod \"global-pull-secret-syncer-p5zcf\" (UID: \"3369da97-ee5f-42cb-a95f-e6066680f349\") " pod="kube-system/global-pull-secret-syncer-p5zcf"
May 26 16:00:52.973811 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:52.973776    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 16:00:52.973903 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:52.973891    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3369da97-ee5f-42cb-a95f-e6066680f349-original-pull-secret podName:3369da97-ee5f-42cb-a95f-e6066680f349 nodeName:}" failed. No retries permitted until 2026-05-26 16:00:53.473872228 +0000 UTC m=+3.182327250 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3369da97-ee5f-42cb-a95f-e6066680f349-original-pull-secret") pod "global-pull-secret-syncer-p5zcf" (UID: "3369da97-ee5f-42cb-a95f-e6066680f349") : object "kube-system"/"original-pull-secret" not registered
May 26 16:00:53.057639 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:53.057555    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bq8pz"
May 26 16:00:53.064386 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:53.064365    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r87kr"
May 26 16:00:53.072839 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:53.072809    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wpv87"
May 26 16:00:53.079482 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:53.079462    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v7qxk"
May 26 16:00:53.086953 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:53.086913    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j7hlf"
May 26 16:00:53.094624 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:53.094604    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:00:53.102300 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:53.102280    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dpgwt"
May 26 16:00:53.108919 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:53.108897    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gqgct"
May 26 16:00:53.114442 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:53.114423    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5xp7f"
May 26 16:00:53.375995 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:53.375900    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/655a0641-f708-444d-91bc-fe7518694ed3-metrics-certs\") pod \"network-metrics-daemon-gj94f\" (UID: \"655a0641-f708-444d-91bc-fe7518694ed3\") " pod="openshift-multus/network-metrics-daemon-gj94f"
May 26 16:00:53.376150 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:53.376056    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:53.376150 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:53.376134    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/655a0641-f708-444d-91bc-fe7518694ed3-metrics-certs podName:655a0641-f708-444d-91bc-fe7518694ed3 nodeName:}" failed. No retries permitted until 2026-05-26 16:00:54.376112791 +0000 UTC m=+4.084567810 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/655a0641-f708-444d-91bc-fe7518694ed3-metrics-certs") pod "network-metrics-daemon-gj94f" (UID: "655a0641-f708-444d-91bc-fe7518694ed3") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:53.476840 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:53.476815    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w6qxx\" (UniqueName: \"kubernetes.io/projected/c4c50203-4f83-413b-b73f-edc0a16e49ee-kube-api-access-w6qxx\") pod \"network-check-target-5v7fc\" (UID: \"c4c50203-4f83-413b-b73f-edc0a16e49ee\") " pod="openshift-network-diagnostics/network-check-target-5v7fc"
May 26 16:00:53.476960 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:53.476852    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3369da97-ee5f-42cb-a95f-e6066680f349-original-pull-secret\") pod \"global-pull-secret-syncer-p5zcf\" (UID: \"3369da97-ee5f-42cb-a95f-e6066680f349\") " pod="kube-system/global-pull-secret-syncer-p5zcf"
May 26 16:00:53.477028 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:53.476973    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 16:00:53.477028 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:53.476995    2552 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:00:53.477028 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:53.477018    2552 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:00:53.477154 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:53.477031    2552 projected.go:194] Error preparing data for projected volume kube-api-access-w6qxx for pod openshift-network-diagnostics/network-check-target-5v7fc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:53.477154 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:53.477037    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3369da97-ee5f-42cb-a95f-e6066680f349-original-pull-secret podName:3369da97-ee5f-42cb-a95f-e6066680f349 nodeName:}" failed. No retries permitted until 2026-05-26 16:00:54.477022034 +0000 UTC m=+4.185477052 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3369da97-ee5f-42cb-a95f-e6066680f349-original-pull-secret") pod "global-pull-secret-syncer-p5zcf" (UID: "3369da97-ee5f-42cb-a95f-e6066680f349") : object "kube-system"/"original-pull-secret" not registered
May 26 16:00:53.477154 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:53.477083    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4c50203-4f83-413b-b73f-edc0a16e49ee-kube-api-access-w6qxx podName:c4c50203-4f83-413b-b73f-edc0a16e49ee nodeName:}" failed. No retries permitted until 2026-05-26 16:00:54.47706698 +0000 UTC m=+4.185522001 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-w6qxx" (UniqueName: "kubernetes.io/projected/c4c50203-4f83-413b-b73f-edc0a16e49ee-kube-api-access-w6qxx") pod "network-check-target-5v7fc" (UID: "c4c50203-4f83-413b-b73f-edc0a16e49ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:53.486872 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:53.486840    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b19d97f_df30_477c_941f_93bcf7e8c5c3.slice/crio-aa721d877c8b5104c7e287a51996c7f706739388b63cecb050e3b46c1e55acd7 WatchSource:0}: Error finding container aa721d877c8b5104c7e287a51996c7f706739388b63cecb050e3b46c1e55acd7: Status 404 returned error can't find the container with id aa721d877c8b5104c7e287a51996c7f706739388b63cecb050e3b46c1e55acd7
May 26 16:00:53.488827 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:53.488779    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c0f9fc6_2bd6_4370_96d0_5ae315363c7f.slice/crio-7195a69d77a5fc75ce4b13bc3afd0345790ff8eb31b388645e27216e6aaea70e WatchSource:0}: Error finding container 7195a69d77a5fc75ce4b13bc3afd0345790ff8eb31b388645e27216e6aaea70e: Status 404 returned error can't find the container with id 7195a69d77a5fc75ce4b13bc3afd0345790ff8eb31b388645e27216e6aaea70e
May 26 16:00:53.493227 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:53.493209    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod092e4858_1372_4cb9_b51b_dde9e50e0f9d.slice/crio-fdb3328343062e0f1119a1c24c7d4043d9d2f09079100a3afd2f32323d7a486d WatchSource:0}: Error finding container fdb3328343062e0f1119a1c24c7d4043d9d2f09079100a3afd2f32323d7a486d: Status 404 returned error can't find the container with id fdb3328343062e0f1119a1c24c7d4043d9d2f09079100a3afd2f32323d7a486d
May 26 16:00:53.493840 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:53.493821    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb90e9cb1_f2de_4919_ab89_8bc29c2476ea.slice/crio-8065db411ebf0bf3a85c9196443f212eaf2fac45b6f71944ed729e30f1f98347 WatchSource:0}: Error finding container 8065db411ebf0bf3a85c9196443f212eaf2fac45b6f71944ed729e30f1f98347: Status 404 returned error can't find the container with id 8065db411ebf0bf3a85c9196443f212eaf2fac45b6f71944ed729e30f1f98347
May 26 16:00:53.494612 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:53.494590    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b3e3775_013c_43d3_9914_c7cc09bd2926.slice/crio-61b9aed3b587948a6beeeaa4a69d7340b306f07ce5137a8757dbd7126768463d WatchSource:0}: Error finding container 61b9aed3b587948a6beeeaa4a69d7340b306f07ce5137a8757dbd7126768463d: Status 404 returned error can't find the container with id 61b9aed3b587948a6beeeaa4a69d7340b306f07ce5137a8757dbd7126768463d
May 26 16:00:53.495944 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:53.495909    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50bb2f40_6580_4a07_abf3_e6cd8ca11465.slice/crio-ba27bbd651ef7a143a4b723178a5e17978119ae37ec34145ce0986901251f7f2 WatchSource:0}: Error finding container ba27bbd651ef7a143a4b723178a5e17978119ae37ec34145ce0986901251f7f2: Status 404 returned error can't find the container with id ba27bbd651ef7a143a4b723178a5e17978119ae37ec34145ce0986901251f7f2
May 26 16:00:53.496517 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:53.496476    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f8c521d_e87e_4504_af4b_b30fa7ece51b.slice/crio-a6371c91533f96cdf9b31dffd65a92c9a842e28a6f42a3a7a5c89b9878d3d6f6 WatchSource:0}: Error finding container a6371c91533f96cdf9b31dffd65a92c9a842e28a6f42a3a7a5c89b9878d3d6f6: Status 404 returned error can't find the container with id a6371c91533f96cdf9b31dffd65a92c9a842e28a6f42a3a7a5c89b9878d3d6f6
May 26 16:00:53.498804 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:53.498722    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4475d020_5281_4257_bf03_1a264049169d.slice/crio-04999119fcff348ce120d2b3fd6af2b6003267d1530d946f38432800dc2bf204 WatchSource:0}: Error finding container 04999119fcff348ce120d2b3fd6af2b6003267d1530d946f38432800dc2bf204: Status 404 returned error can't find the container with id 04999119fcff348ce120d2b3fd6af2b6003267d1530d946f38432800dc2bf204
May 26 16:00:53.499202 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:00:53.499062    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b4c93f_e507_4b19_8510_9fc8fffb21f4.slice/crio-17f8fe9462ae50806e1e108924d99d1a99a8131fedc964fedcea67c1db429031 WatchSource:0}: Error finding container 17f8fe9462ae50806e1e108924d99d1a99a8131fedc964fedcea67c1db429031: Status 404 returned error can't find the container with id 17f8fe9462ae50806e1e108924d99d1a99a8131fedc964fedcea67c1db429031
May 26 16:00:53.847829 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:53.847754    2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-25 15:55:51 +0000 UTC" deadline="2028-02-10 19:00:51.877533178 +0000 UTC"
May 26 16:00:53.847829 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:53.847794    2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15002h59m58.029742564s"
May 26 16:00:53.888812 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:53.888284    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gj94f"
May 26 16:00:53.888812 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:53.888430    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gj94f" podUID="655a0641-f708-444d-91bc-fe7518694ed3"
May 26 16:00:53.899955 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:53.899883    2552 generic.go:358] "Generic (PLEG): container finished" podID="688439714ab8bc3b4b3d0132d8a1b252" containerID="ec901dfcb2878103bc9b4d4a6aef75dae77bc59502be659f01283bf2d8b83627" exitCode=0
May 26 16:00:53.900067 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:53.899995    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-96.ec2.internal" event={"ID":"688439714ab8bc3b4b3d0132d8a1b252","Type":"ContainerDied","Data":"ec901dfcb2878103bc9b4d4a6aef75dae77bc59502be659f01283bf2d8b83627"}
May 26 16:00:53.910413 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:53.910372    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-96.ec2.internal" event={"ID":"b789aeffe1887cd67b4335801350d604","Type":"ContainerStarted","Data":"fbd34ebf4552e18a98145e0d9a53123ef20f482cd896e8ef48dd0bc179de969f"}
May 26 16:00:53.915334 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:53.915310    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gqgct" event={"ID":"08b4c93f-e507-4b19-8510-9fc8fffb21f4","Type":"ContainerStarted","Data":"17f8fe9462ae50806e1e108924d99d1a99a8131fedc964fedcea67c1db429031"}
May 26 16:00:53.918610 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:53.918586    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v7qxk" event={"ID":"50bb2f40-6580-4a07-abf3-e6cd8ca11465","Type":"ContainerStarted","Data":"ba27bbd651ef7a143a4b723178a5e17978119ae37ec34145ce0986901251f7f2"}
May 26 16:00:53.921270 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:53.921245    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5xp7f" event={"ID":"092e4858-1372-4cb9-b51b-dde9e50e0f9d","Type":"ContainerStarted","Data":"fdb3328343062e0f1119a1c24c7d4043d9d2f09079100a3afd2f32323d7a486d"}
May 26 16:00:53.928363 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:53.928341    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sj6db" event={"ID":"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f","Type":"ContainerStarted","Data":"7195a69d77a5fc75ce4b13bc3afd0345790ff8eb31b388645e27216e6aaea70e"}
May 26 16:00:53.930305 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:53.930250    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r87kr" event={"ID":"6b19d97f-df30-477c-941f-93bcf7e8c5c3","Type":"ContainerStarted","Data":"aa721d877c8b5104c7e287a51996c7f706739388b63cecb050e3b46c1e55acd7"}
May 26 16:00:53.933587 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:53.933523    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dpgwt" event={"ID":"4b3e3775-013c-43d3-9914-c7cc09bd2926","Type":"ContainerStarted","Data":"61b9aed3b587948a6beeeaa4a69d7340b306f07ce5137a8757dbd7126768463d"}
May 26 16:00:53.935261 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:53.935205    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bq8pz" event={"ID":"b90e9cb1-f2de-4919-ab89-8bc29c2476ea","Type":"ContainerStarted","Data":"8065db411ebf0bf3a85c9196443f212eaf2fac45b6f71944ed729e30f1f98347"}
May 26 16:00:53.936779 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:53.936705    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j7hlf" event={"ID":"4475d020-5281-4257-bf03-1a264049169d","Type":"ContainerStarted","Data":"04999119fcff348ce120d2b3fd6af2b6003267d1530d946f38432800dc2bf204"}
May 26 16:00:53.942132 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:53.942109    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wpv87" event={"ID":"1f8c521d-e87e-4504-af4b-b30fa7ece51b","Type":"ContainerStarted","Data":"a6371c91533f96cdf9b31dffd65a92c9a842e28a6f42a3a7a5c89b9878d3d6f6"}
May 26 16:00:54.392184 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:54.392099    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/655a0641-f708-444d-91bc-fe7518694ed3-metrics-certs\") pod \"network-metrics-daemon-gj94f\" (UID: \"655a0641-f708-444d-91bc-fe7518694ed3\") " pod="openshift-multus/network-metrics-daemon-gj94f"
May 26 16:00:54.392342 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:54.392234    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:54.392342 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:54.392295    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/655a0641-f708-444d-91bc-fe7518694ed3-metrics-certs podName:655a0641-f708-444d-91bc-fe7518694ed3 nodeName:}" failed. No retries permitted until 2026-05-26 16:00:56.392276548 +0000 UTC m=+6.100731580 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/655a0641-f708-444d-91bc-fe7518694ed3-metrics-certs") pod "network-metrics-daemon-gj94f" (UID: "655a0641-f708-444d-91bc-fe7518694ed3") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:54.493026 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:54.492992    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w6qxx\" (UniqueName: \"kubernetes.io/projected/c4c50203-4f83-413b-b73f-edc0a16e49ee-kube-api-access-w6qxx\") pod \"network-check-target-5v7fc\" (UID: \"c4c50203-4f83-413b-b73f-edc0a16e49ee\") " pod="openshift-network-diagnostics/network-check-target-5v7fc"
May 26 16:00:54.493196 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:54.493046    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3369da97-ee5f-42cb-a95f-e6066680f349-original-pull-secret\") pod \"global-pull-secret-syncer-p5zcf\" (UID: \"3369da97-ee5f-42cb-a95f-e6066680f349\") " pod="kube-system/global-pull-secret-syncer-p5zcf"
May 26 16:00:54.493196 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:54.493166    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 16:00:54.493320 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:54.493229    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3369da97-ee5f-42cb-a95f-e6066680f349-original-pull-secret podName:3369da97-ee5f-42cb-a95f-e6066680f349 nodeName:}" failed. No retries permitted until 2026-05-26 16:00:56.493209544 +0000 UTC m=+6.201664586 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3369da97-ee5f-42cb-a95f-e6066680f349-original-pull-secret") pod "global-pull-secret-syncer-p5zcf" (UID: "3369da97-ee5f-42cb-a95f-e6066680f349") : object "kube-system"/"original-pull-secret" not registered
May 26 16:00:54.493637 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:54.493620    2552 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:00:54.493719 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:54.493639    2552 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:00:54.493719 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:54.493653    2552 projected.go:194] Error preparing data for projected volume kube-api-access-w6qxx for pod openshift-network-diagnostics/network-check-target-5v7fc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:54.493719 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:54.493697    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4c50203-4f83-413b-b73f-edc0a16e49ee-kube-api-access-w6qxx podName:c4c50203-4f83-413b-b73f-edc0a16e49ee nodeName:}" failed. No retries permitted until 2026-05-26 16:00:56.493682939 +0000 UTC m=+6.202137972 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-w6qxx" (UniqueName: "kubernetes.io/projected/c4c50203-4f83-413b-b73f-edc0a16e49ee-kube-api-access-w6qxx") pod "network-check-target-5v7fc" (UID: "c4c50203-4f83-413b-b73f-edc0a16e49ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:54.885947 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:54.885859    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5v7fc"
May 26 16:00:54.886491 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:54.886023    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5v7fc" podUID="c4c50203-4f83-413b-b73f-edc0a16e49ee"
May 26 16:00:54.886491 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:54.885859    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5zcf"
May 26 16:00:54.886607 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:54.886508    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5zcf" podUID="3369da97-ee5f-42cb-a95f-e6066680f349"
May 26 16:00:54.963901 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:54.963839    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-96.ec2.internal" event={"ID":"688439714ab8bc3b4b3d0132d8a1b252","Type":"ContainerStarted","Data":"bdaaef7610e44a78e6381c8389c515ba1212ea56452418fdb898c02851cdb375"}
May 26 16:00:54.979407 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:54.979357    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-96.ec2.internal" podStartSLOduration=2.979339998 podStartE2EDuration="2.979339998s" podCreationTimestamp="2026-05-26 16:00:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:00:53.944274164 +0000 UTC m=+3.652729205" watchObservedRunningTime="2026-05-26 16:00:54.979339998 +0000 UTC m=+4.687795043"
May 26 16:00:55.890085 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:55.889547    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gj94f"
May 26 16:00:55.890085 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:55.890045    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gj94f" podUID="655a0641-f708-444d-91bc-fe7518694ed3"
May 26 16:00:56.409672 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:56.409638    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/655a0641-f708-444d-91bc-fe7518694ed3-metrics-certs\") pod \"network-metrics-daemon-gj94f\" (UID: \"655a0641-f708-444d-91bc-fe7518694ed3\") " pod="openshift-multus/network-metrics-daemon-gj94f"
May 26 16:00:56.409848 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:56.409805    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:56.409921 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:56.409867    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/655a0641-f708-444d-91bc-fe7518694ed3-metrics-certs podName:655a0641-f708-444d-91bc-fe7518694ed3 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:00.409847576 +0000 UTC m=+10.118302604 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/655a0641-f708-444d-91bc-fe7518694ed3-metrics-certs") pod "network-metrics-daemon-gj94f" (UID: "655a0641-f708-444d-91bc-fe7518694ed3") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:56.510362 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:56.510120    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w6qxx\" (UniqueName: \"kubernetes.io/projected/c4c50203-4f83-413b-b73f-edc0a16e49ee-kube-api-access-w6qxx\") pod \"network-check-target-5v7fc\" (UID: \"c4c50203-4f83-413b-b73f-edc0a16e49ee\") " pod="openshift-network-diagnostics/network-check-target-5v7fc"
May 26 16:00:56.510362 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:56.510174    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3369da97-ee5f-42cb-a95f-e6066680f349-original-pull-secret\") pod \"global-pull-secret-syncer-p5zcf\" (UID: \"3369da97-ee5f-42cb-a95f-e6066680f349\") " pod="kube-system/global-pull-secret-syncer-p5zcf"
May 26 16:00:56.510362 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:56.510266    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 16:00:56.510362 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:56.510280    2552 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:00:56.510362 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:56.510295    2552 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:00:56.510362 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:56.510307    2552 projected.go:194] Error preparing data for projected volume kube-api-access-w6qxx for pod openshift-network-diagnostics/network-check-target-5v7fc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:56.510362 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:56.510324    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3369da97-ee5f-42cb-a95f-e6066680f349-original-pull-secret podName:3369da97-ee5f-42cb-a95f-e6066680f349 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:00.510307743 +0000 UTC m=+10.218762784 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3369da97-ee5f-42cb-a95f-e6066680f349-original-pull-secret") pod "global-pull-secret-syncer-p5zcf" (UID: "3369da97-ee5f-42cb-a95f-e6066680f349") : object "kube-system"/"original-pull-secret" not registered
May 26 16:00:56.510362 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:56.510342    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4c50203-4f83-413b-b73f-edc0a16e49ee-kube-api-access-w6qxx podName:c4c50203-4f83-413b-b73f-edc0a16e49ee nodeName:}" failed. No retries permitted until 2026-05-26 16:01:00.510331665 +0000 UTC m=+10.218786705 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-w6qxx" (UniqueName: "kubernetes.io/projected/c4c50203-4f83-413b-b73f-edc0a16e49ee-kube-api-access-w6qxx") pod "network-check-target-5v7fc" (UID: "c4c50203-4f83-413b-b73f-edc0a16e49ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:56.886605 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:56.886067    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5v7fc"
May 26 16:00:56.886605 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:56.886225    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5v7fc" podUID="c4c50203-4f83-413b-b73f-edc0a16e49ee"
May 26 16:00:56.886605 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:56.886275    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5zcf"
May 26 16:00:56.886605 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:56.886398    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5zcf" podUID="3369da97-ee5f-42cb-a95f-e6066680f349"
May 26 16:00:57.886212 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:57.885993    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gj94f"
May 26 16:00:57.886212 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:57.886150    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gj94f" podUID="655a0641-f708-444d-91bc-fe7518694ed3"
May 26 16:00:58.885780 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:58.885298    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5v7fc"
May 26 16:00:58.885780 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:58.885420    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5v7fc" podUID="c4c50203-4f83-413b-b73f-edc0a16e49ee"
May 26 16:00:58.885780 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:58.885763    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5zcf"
May 26 16:00:58.886082 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:58.885845    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5zcf" podUID="3369da97-ee5f-42cb-a95f-e6066680f349"
May 26 16:00:59.885350 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:00:59.885309    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gj94f"
May 26 16:00:59.885815 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:00:59.885438    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gj94f" podUID="655a0641-f708-444d-91bc-fe7518694ed3"
May 26 16:01:00.446053 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:00.445488    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/655a0641-f708-444d-91bc-fe7518694ed3-metrics-certs\") pod \"network-metrics-daemon-gj94f\" (UID: \"655a0641-f708-444d-91bc-fe7518694ed3\") " pod="openshift-multus/network-metrics-daemon-gj94f"
May 26 16:01:00.446053 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:00.445650    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:00.446053 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:00.445713    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/655a0641-f708-444d-91bc-fe7518694ed3-metrics-certs podName:655a0641-f708-444d-91bc-fe7518694ed3 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:08.445694761 +0000 UTC m=+18.154149780 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/655a0641-f708-444d-91bc-fe7518694ed3-metrics-certs") pod "network-metrics-daemon-gj94f" (UID: "655a0641-f708-444d-91bc-fe7518694ed3") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:00.546192 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:00.546156    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w6qxx\" (UniqueName: \"kubernetes.io/projected/c4c50203-4f83-413b-b73f-edc0a16e49ee-kube-api-access-w6qxx\") pod \"network-check-target-5v7fc\" (UID: \"c4c50203-4f83-413b-b73f-edc0a16e49ee\") " pod="openshift-network-diagnostics/network-check-target-5v7fc"
May 26 16:01:00.546367 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:00.546214    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3369da97-ee5f-42cb-a95f-e6066680f349-original-pull-secret\") pod \"global-pull-secret-syncer-p5zcf\" (UID: \"3369da97-ee5f-42cb-a95f-e6066680f349\") " pod="kube-system/global-pull-secret-syncer-p5zcf"
May 26 16:01:00.546367 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:00.546334    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 16:01:00.546478 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:00.546392    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3369da97-ee5f-42cb-a95f-e6066680f349-original-pull-secret podName:3369da97-ee5f-42cb-a95f-e6066680f349 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:08.546373457 +0000 UTC m=+18.254828491 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3369da97-ee5f-42cb-a95f-e6066680f349-original-pull-secret") pod "global-pull-secret-syncer-p5zcf" (UID: "3369da97-ee5f-42cb-a95f-e6066680f349") : object "kube-system"/"original-pull-secret" not registered
May 26 16:01:00.546638 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:00.546604    2552 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:01:00.546638 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:00.546630    2552 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:01:00.546638 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:00.546643    2552 projected.go:194] Error preparing data for projected volume kube-api-access-w6qxx for pod openshift-network-diagnostics/network-check-target-5v7fc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:00.546870 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:00.546687    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4c50203-4f83-413b-b73f-edc0a16e49ee-kube-api-access-w6qxx podName:c4c50203-4f83-413b-b73f-edc0a16e49ee nodeName:}" failed. No retries permitted until 2026-05-26 16:01:08.546671288 +0000 UTC m=+18.255126312 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-w6qxx" (UniqueName: "kubernetes.io/projected/c4c50203-4f83-413b-b73f-edc0a16e49ee-kube-api-access-w6qxx") pod "network-check-target-5v7fc" (UID: "c4c50203-4f83-413b-b73f-edc0a16e49ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:00.886804 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:00.886537    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5v7fc"
May 26 16:01:00.887207 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:00.886846    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5v7fc" podUID="c4c50203-4f83-413b-b73f-edc0a16e49ee"
May 26 16:01:00.887207 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:00.886557    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5zcf"
May 26 16:01:00.887329 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:00.887263    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5zcf" podUID="3369da97-ee5f-42cb-a95f-e6066680f349"
May 26 16:01:01.885660 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:01.885623    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gj94f"
May 26 16:01:01.885825 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:01.885750    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gj94f" podUID="655a0641-f708-444d-91bc-fe7518694ed3"
May 26 16:01:02.886100 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:02.886071    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5v7fc"
May 26 16:01:02.886446 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:02.886118    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5zcf"
May 26 16:01:02.886446 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:02.886273    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5v7fc" podUID="c4c50203-4f83-413b-b73f-edc0a16e49ee"
May 26 16:01:02.886446 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:02.886280    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5zcf" podUID="3369da97-ee5f-42cb-a95f-e6066680f349"
May 26 16:01:03.886075 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:03.886046    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gj94f"
May 26 16:01:03.886205 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:03.886151    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gj94f" podUID="655a0641-f708-444d-91bc-fe7518694ed3"
May 26 16:01:04.885731 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:04.885698    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5v7fc"
May 26 16:01:04.885913 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:04.885698    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5zcf"
May 26 16:01:04.885913 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:04.885824    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5v7fc" podUID="c4c50203-4f83-413b-b73f-edc0a16e49ee"
May 26 16:01:04.885913 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:04.885899    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5zcf" podUID="3369da97-ee5f-42cb-a95f-e6066680f349"
May 26 16:01:05.885768 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:05.885727    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gj94f"
May 26 16:01:05.886203 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:05.885861    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gj94f" podUID="655a0641-f708-444d-91bc-fe7518694ed3"
May 26 16:01:06.886102 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:06.886065    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5v7fc"
May 26 16:01:06.886526 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:06.886184    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5v7fc" podUID="c4c50203-4f83-413b-b73f-edc0a16e49ee"
May 26 16:01:06.886526 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:06.886251    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5zcf"
May 26 16:01:06.886526 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:06.886374    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5zcf" podUID="3369da97-ee5f-42cb-a95f-e6066680f349"
May 26 16:01:07.885944 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:07.885903    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gj94f"
May 26 16:01:07.886139 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:07.886042    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gj94f" podUID="655a0641-f708-444d-91bc-fe7518694ed3"
May 26 16:01:08.508150 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:08.508114    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/655a0641-f708-444d-91bc-fe7518694ed3-metrics-certs\") pod \"network-metrics-daemon-gj94f\" (UID: \"655a0641-f708-444d-91bc-fe7518694ed3\") " pod="openshift-multus/network-metrics-daemon-gj94f"
May 26 16:01:08.508388 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:08.508267    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:08.508388 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:08.508338    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/655a0641-f708-444d-91bc-fe7518694ed3-metrics-certs podName:655a0641-f708-444d-91bc-fe7518694ed3 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:24.508312635 +0000 UTC m=+34.216767653 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/655a0641-f708-444d-91bc-fe7518694ed3-metrics-certs") pod "network-metrics-daemon-gj94f" (UID: "655a0641-f708-444d-91bc-fe7518694ed3") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:08.609052 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:08.609020    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w6qxx\" (UniqueName: \"kubernetes.io/projected/c4c50203-4f83-413b-b73f-edc0a16e49ee-kube-api-access-w6qxx\") pod \"network-check-target-5v7fc\" (UID: \"c4c50203-4f83-413b-b73f-edc0a16e49ee\") " pod="openshift-network-diagnostics/network-check-target-5v7fc"
May 26 16:01:08.609052 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:08.609065    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3369da97-ee5f-42cb-a95f-e6066680f349-original-pull-secret\") pod \"global-pull-secret-syncer-p5zcf\" (UID: \"3369da97-ee5f-42cb-a95f-e6066680f349\") " pod="kube-system/global-pull-secret-syncer-p5zcf"
May 26 16:01:08.609279 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:08.609185    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 16:01:08.609279 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:08.609239    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3369da97-ee5f-42cb-a95f-e6066680f349-original-pull-secret podName:3369da97-ee5f-42cb-a95f-e6066680f349 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:24.60922136 +0000 UTC m=+34.317676382 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3369da97-ee5f-42cb-a95f-e6066680f349-original-pull-secret") pod "global-pull-secret-syncer-p5zcf" (UID: "3369da97-ee5f-42cb-a95f-e6066680f349") : object "kube-system"/"original-pull-secret" not registered
May 26 16:01:08.609279 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:08.609187    2552 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:01:08.609421 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:08.609300    2552 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:01:08.609421 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:08.609316    2552 projected.go:194] Error preparing data for projected volume kube-api-access-w6qxx for pod openshift-network-diagnostics/network-check-target-5v7fc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:08.609421 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:08.609377    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4c50203-4f83-413b-b73f-edc0a16e49ee-kube-api-access-w6qxx podName:c4c50203-4f83-413b-b73f-edc0a16e49ee nodeName:}" failed. No retries permitted until 2026-05-26 16:01:24.60936026 +0000 UTC m=+34.317815294 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-w6qxx" (UniqueName: "kubernetes.io/projected/c4c50203-4f83-413b-b73f-edc0a16e49ee-kube-api-access-w6qxx") pod "network-check-target-5v7fc" (UID: "c4c50203-4f83-413b-b73f-edc0a16e49ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:08.886018 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:08.885986    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5v7fc"
May 26 16:01:08.886183 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:08.886035    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5zcf"
May 26 16:01:08.886183 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:08.886113    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5v7fc" podUID="c4c50203-4f83-413b-b73f-edc0a16e49ee"
May 26 16:01:08.886600 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:08.886246    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5zcf" podUID="3369da97-ee5f-42cb-a95f-e6066680f349"
May 26 16:01:09.886189 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:09.886153    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gj94f"
May 26 16:01:09.886567 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:09.886267    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gj94f" podUID="655a0641-f708-444d-91bc-fe7518694ed3"
May 26 16:01:10.886232 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:10.886206    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5v7fc"
May 26 16:01:10.886661 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:10.886294    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5v7fc" podUID="c4c50203-4f83-413b-b73f-edc0a16e49ee"
May 26 16:01:10.886661 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:10.886367    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5zcf"
May 26 16:01:10.886661 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:10.886454    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5zcf" podUID="3369da97-ee5f-42cb-a95f-e6066680f349"
May 26 16:01:11.885133 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:11.885101    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gj94f"
May 26 16:01:11.885285 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:11.885216    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gj94f" podUID="655a0641-f708-444d-91bc-fe7518694ed3"
May 26 16:01:12.885361 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:12.885192    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5v7fc"
May 26 16:01:12.886028 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:12.885274    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5zcf"
May 26 16:01:12.886028 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:12.885432    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5v7fc" podUID="c4c50203-4f83-413b-b73f-edc0a16e49ee"
May 26 16:01:12.886028 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:12.885510    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5zcf" podUID="3369da97-ee5f-42cb-a95f-e6066680f349"
May 26 16:01:13.002577 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:13.002548    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gqgct" event={"ID":"08b4c93f-e507-4b19-8510-9fc8fffb21f4","Type":"ContainerStarted","Data":"1689f4f003ba63579e76bd38851ca9a3127a0daebc026fa6eaa4150d75d3e0c6"}
May 26 16:01:13.003951 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:13.003905    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v7qxk" event={"ID":"50bb2f40-6580-4a07-abf3-e6cd8ca11465","Type":"ContainerStarted","Data":"59d19a6f79ae7a2d76f380649668ee3a38c0a346a35718e1b4c9f0b5dd531249"}
May 26 16:01:13.005234 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:13.005206    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5xp7f" event={"ID":"092e4858-1372-4cb9-b51b-dde9e50e0f9d","Type":"ContainerStarted","Data":"75511b98cbf583aab3f15904f1d9f58e3180cb9cf3e4068f91b4b622da60681c"}
May 26 16:01:13.007048 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:13.007033    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sj6db_5c0f9fc6-2bd6-4370-96d0-5ae315363c7f/ovn-acl-logging/0.log"
May 26 16:01:13.007327 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:13.007301    2552 generic.go:358] "Generic (PLEG): container finished" podID="5c0f9fc6-2bd6-4370-96d0-5ae315363c7f" containerID="ddccd5ff5cd366f62c669c09bde548e749be919ee0c64e40f4e07c856acef0cd" exitCode=1
May 26 16:01:13.007404 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:13.007360    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sj6db" event={"ID":"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f","Type":"ContainerStarted","Data":"1ea7dcf6933cac9fb91e2e743882f6b45ebfce6033437db2926c99b3c7fb8652"}
May 26 16:01:13.007404 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:13.007381    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sj6db" event={"ID":"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f","Type":"ContainerStarted","Data":"b2cebb8228f77190837999527762c513c576d21e4198fac8e9083cd40262a45e"}
May 26 16:01:13.007404 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:13.007389    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sj6db" event={"ID":"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f","Type":"ContainerDied","Data":"ddccd5ff5cd366f62c669c09bde548e749be919ee0c64e40f4e07c856acef0cd"}
May 26 16:01:13.007404 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:13.007399    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sj6db" event={"ID":"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f","Type":"ContainerStarted","Data":"2a0ee4c1297ad39ea81fb34788a1beb92fd98378d34e52656e0c465a345b1b68"}
May 26 16:01:13.008522 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:13.008506    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dpgwt" event={"ID":"4b3e3775-013c-43d3-9914-c7cc09bd2926","Type":"ContainerStarted","Data":"25e1e0e7bcbb4db7df06777e085f77ba317e189a584db33d4c9bb42c779369d7"}
May 26 16:01:13.009591 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:13.009571    2552 generic.go:358] "Generic (PLEG): container finished" podID="b90e9cb1-f2de-4919-ab89-8bc29c2476ea" containerID="066a99e5bbf7ee58e0997ee331da6c1a4371b13cbadc714b215b90f610bb37db" exitCode=0
May 26 16:01:13.009664 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:13.009639    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bq8pz" event={"ID":"b90e9cb1-f2de-4919-ab89-8bc29c2476ea","Type":"ContainerDied","Data":"066a99e5bbf7ee58e0997ee331da6c1a4371b13cbadc714b215b90f610bb37db"}
May 26 16:01:13.010661 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:13.010644    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j7hlf" event={"ID":"4475d020-5281-4257-bf03-1a264049169d","Type":"ContainerStarted","Data":"5e213e29cabaebebd4a31e764ae8fbe80656f8cad2d0315edbfea9d2abbd7b15"}
May 26 16:01:13.011923 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:13.011861    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wpv87" event={"ID":"1f8c521d-e87e-4504-af4b-b30fa7ece51b","Type":"ContainerStarted","Data":"26f61633a2ee54e8e0c93e2cbf23615483b0413c0cf5c3528c7f38df2b36ef2c"}
May 26 16:01:13.025293 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:13.025274    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-v7qxk"
May 26 16:01:13.026091 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:13.026078    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-v7qxk"
May 26 16:01:13.036532 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:13.036501    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gqgct" podStartSLOduration=3.6527437860000003 podStartE2EDuration="22.036491666s" podCreationTimestamp="2026-05-26 16:00:51 +0000 UTC" firstStartedPulling="2026-05-26 16:00:53.500835036 +0000 UTC m=+3.209290054" lastFinishedPulling="2026-05-26 16:01:11.884582916 +0000 UTC m=+21.593037934" observedRunningTime="2026-05-26 16:01:13.03608755 +0000 UTC m=+22.744542590" watchObservedRunningTime="2026-05-26 16:01:13.036491666 +0000 UTC m=+22.744946703"
May 26 16:01:13.036625 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:13.036607    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-96.ec2.internal" podStartSLOduration=21.036602865 podStartE2EDuration="21.036602865s" podCreationTimestamp="2026-05-26 16:00:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:00:54.979845225 +0000 UTC m=+4.688300266" watchObservedRunningTime="2026-05-26 16:01:13.036602865 +0000 UTC m=+22.745057905"
May 26 16:01:13.051651 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:13.051622    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-v7qxk" podStartSLOduration=3.666635445 podStartE2EDuration="22.051609822s" podCreationTimestamp="2026-05-26 16:00:51 +0000 UTC" firstStartedPulling="2026-05-26 16:00:53.497824558 +0000 UTC m=+3.206279577" lastFinishedPulling="2026-05-26 16:01:11.882798937 +0000 UTC m=+21.591253954" observedRunningTime="2026-05-26 16:01:13.051224086 +0000 UTC m=+22.759679125" watchObservedRunningTime="2026-05-26 16:01:13.051609822 +0000 UTC m=+22.760064861"
May 26 16:01:13.072846 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:13.072769    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5xp7f" podStartSLOduration=8.327589034 podStartE2EDuration="22.072760924s" podCreationTimestamp="2026-05-26 16:00:51 +0000 UTC" firstStartedPulling="2026-05-26 16:00:53.495508945 +0000 UTC m=+3.203963978" lastFinishedPulling="2026-05-26 16:01:07.240680836 +0000 UTC m=+16.949135868" observedRunningTime="2026-05-26 16:01:13.072352578 +0000 UTC m=+22.780807617" watchObservedRunningTime="2026-05-26 16:01:13.072760924 +0000 UTC m=+22.781215992"
May 26 16:01:13.117555 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:13.117519    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-wpv87" podStartSLOduration=4.716739985 podStartE2EDuration="23.117509495s" podCreationTimestamp="2026-05-26 16:00:50 +0000 UTC" firstStartedPulling="2026-05-26 16:00:53.498798595 +0000 UTC m=+3.207253613" lastFinishedPulling="2026-05-26 16:01:11.899568104 +0000 UTC m=+21.608023123" observedRunningTime="2026-05-26 16:01:13.117179927 +0000 UTC m=+22.825634988" watchObservedRunningTime="2026-05-26 16:01:13.117509495 +0000 UTC m=+22.825964533"
May 26 16:01:13.132399 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:13.132369    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-j7hlf" podStartSLOduration=3.74968622 podStartE2EDuration="22.132359535s" podCreationTimestamp="2026-05-26 16:00:51 +0000 UTC" firstStartedPulling="2026-05-26 16:00:53.500123608 +0000 UTC m=+3.208578629" lastFinishedPulling="2026-05-26 16:01:11.882796917 +0000 UTC m=+21.591251944" observedRunningTime="2026-05-26 16:01:13.132063713 +0000 UTC m=+22.840518754" watchObservedRunningTime="2026-05-26 16:01:13.132359535 +0000 UTC m=+22.840814575"
May 26 16:01:13.296964 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:13.296925    2552 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 26 16:01:13.841821 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:13.841675    2552 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-26T16:01:13.29696027Z","UUID":"d9784531-6e9b-44e6-a46e-534d575d5709","Handler":null,"Name":"","Endpoint":""}
May 26 16:01:13.844683 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:13.844661    2552 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 26 16:01:13.844683 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:13.844688    2552 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 26 16:01:13.885251 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:13.885227    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gj94f"
May 26 16:01:13.885464 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:13.885343    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gj94f" podUID="655a0641-f708-444d-91bc-fe7518694ed3"
May 26 16:01:14.017231 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:14.017190    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sj6db_5c0f9fc6-2bd6-4370-96d0-5ae315363c7f/ovn-acl-logging/0.log"
May 26 16:01:14.017597 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:14.017561    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sj6db" event={"ID":"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f","Type":"ContainerStarted","Data":"ac97056e3b3f2d06d27d37ff6cd0b31cbd78d7f88cb90298947ba9716727ec4d"}
May 26 16:01:14.017722 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:14.017600    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sj6db" event={"ID":"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f","Type":"ContainerStarted","Data":"0f851dad66a61adf33f1d417f6f66b94a625715c753e37bf54474266d344c3e5"}
May 26 16:01:14.019176 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:14.019146    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r87kr" event={"ID":"6b19d97f-df30-477c-941f-93bcf7e8c5c3","Type":"ContainerStarted","Data":"b96c37ac79bd55677b1006614496cff50285c4d9b4bcb8e3ba759c18d455c388"}
May 26 16:01:14.021539 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:14.021453    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dpgwt" event={"ID":"4b3e3775-013c-43d3-9914-c7cc09bd2926","Type":"ContainerStarted","Data":"758dfdc0aab0dcaf1a6e933095403a73e96a5443ab33e0647e68bfb354856b0c"}
May 26 16:01:14.039140 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:14.039096    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-r87kr" podStartSLOduration=5.645584974 podStartE2EDuration="24.039083309s" podCreationTimestamp="2026-05-26 16:00:50 +0000 UTC" firstStartedPulling="2026-05-26 16:00:53.489301568 +0000 UTC m=+3.197756594" lastFinishedPulling="2026-05-26 16:01:11.882799911 +0000 UTC m=+21.591254929" observedRunningTime="2026-05-26 16:01:14.03870977 +0000 UTC m=+23.747164832" watchObservedRunningTime="2026-05-26 16:01:14.039083309 +0000 UTC m=+23.747538566"
May 26 16:01:14.886154 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:14.886116    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5v7fc"
May 26 16:01:14.886719 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:14.886162    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5zcf"
May 26 16:01:14.886719 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:14.886247    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5v7fc" podUID="c4c50203-4f83-413b-b73f-edc0a16e49ee"
May 26 16:01:14.886719 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:14.886385    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5zcf" podUID="3369da97-ee5f-42cb-a95f-e6066680f349"
May 26 16:01:15.025094 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:15.025054    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dpgwt" event={"ID":"4b3e3775-013c-43d3-9914-c7cc09bd2926","Type":"ContainerStarted","Data":"6e6c2f443720bb27926c23ebcd9c8e93ec772301a6c8ced9dc4ddb4791d9d352"}
May 26 16:01:15.025269 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:15.025118    2552 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
May 26 16:01:15.042448 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:15.042395    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dpgwt" podStartSLOduration=3.24753645 podStartE2EDuration="24.04238207s" podCreationTimestamp="2026-05-26 16:00:51 +0000 UTC" firstStartedPulling="2026-05-26 16:00:53.496695072 +0000 UTC m=+3.205150093" lastFinishedPulling="2026-05-26 16:01:14.291540679 +0000 UTC m=+23.999995713" observedRunningTime="2026-05-26 16:01:15.042337003 +0000 UTC m=+24.750792044" watchObservedRunningTime="2026-05-26 16:01:15.04238207 +0000 UTC m=+24.750837092"
May 26 16:01:15.885962 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:15.885925    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gj94f"
May 26 16:01:15.886107 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:15.886062    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gj94f" podUID="655a0641-f708-444d-91bc-fe7518694ed3"
May 26 16:01:16.885810 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:16.885784    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5zcf"
May 26 16:01:16.885810 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:16.885804    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5v7fc"
May 26 16:01:16.886324 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:16.885910    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5zcf" podUID="3369da97-ee5f-42cb-a95f-e6066680f349"
May 26 16:01:16.886324 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:16.886006    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5v7fc" podUID="c4c50203-4f83-413b-b73f-edc0a16e49ee"
May 26 16:01:17.031117 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:17.031096    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sj6db_5c0f9fc6-2bd6-4370-96d0-5ae315363c7f/ovn-acl-logging/0.log"
May 26 16:01:17.031361 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:17.031342    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sj6db" event={"ID":"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f","Type":"ContainerStarted","Data":"e1626e3aa7d3bcb5c05118035de75ecfceb10915d246d5b841c82fbf5042e126"}
May 26 16:01:17.885971 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:17.885773    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gj94f"
May 26 16:01:17.886328 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:17.886050    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gj94f" podUID="655a0641-f708-444d-91bc-fe7518694ed3"
May 26 16:01:18.034299 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:18.034267    2552 generic.go:358] "Generic (PLEG): container finished" podID="b90e9cb1-f2de-4919-ab89-8bc29c2476ea" containerID="e5989d747196b56cad41fd33a97f035ad951fd2b622aeb78244f726c147f1017" exitCode=0
May 26 16:01:18.034450 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:18.034311    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bq8pz" event={"ID":"b90e9cb1-f2de-4919-ab89-8bc29c2476ea","Type":"ContainerDied","Data":"e5989d747196b56cad41fd33a97f035ad951fd2b622aeb78244f726c147f1017"}
May 26 16:01:18.885600 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:18.885568    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5v7fc"
May 26 16:01:18.885771 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:18.885671    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5v7fc" podUID="c4c50203-4f83-413b-b73f-edc0a16e49ee"
May 26 16:01:18.885771 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:18.885760    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5zcf"
May 26 16:01:18.885899 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:18.885863    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5zcf" podUID="3369da97-ee5f-42cb-a95f-e6066680f349"
May 26 16:01:19.039238 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:19.039078    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sj6db_5c0f9fc6-2bd6-4370-96d0-5ae315363c7f/ovn-acl-logging/0.log"
May 26 16:01:19.040095 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:19.039395    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sj6db" event={"ID":"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f","Type":"ContainerStarted","Data":"9da9317934b536e6126d54c090e143961bead332d0a5b0aaaf8f7a13b1436759"}
May 26 16:01:19.885680 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:19.885650    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gj94f"
May 26 16:01:19.885815 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:19.885748    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gj94f" podUID="655a0641-f708-444d-91bc-fe7518694ed3"
May 26 16:01:20.042307 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:20.042273    2552 generic.go:358] "Generic (PLEG): container finished" podID="b90e9cb1-f2de-4919-ab89-8bc29c2476ea" containerID="a1b755fa1d55bb1399f9b9f74dcb21f82f3359bc06f7b2abb709d0fdee187d8c" exitCode=0
May 26 16:01:20.042751 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:20.042367    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bq8pz" event={"ID":"b90e9cb1-f2de-4919-ab89-8bc29c2476ea","Type":"ContainerDied","Data":"a1b755fa1d55bb1399f9b9f74dcb21f82f3359bc06f7b2abb709d0fdee187d8c"}
May 26 16:01:20.042751 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:20.042549    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:01:20.042751 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:20.042587    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:01:20.042751 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:20.042678    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:01:20.042751 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:20.042750    2552 scope.go:117] "RemoveContainer" containerID="ddccd5ff5cd366f62c669c09bde548e749be919ee0c64e40f4e07c856acef0cd"
May 26 16:01:20.058857 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:20.058838    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:01:20.060151 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:20.060136    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:01:20.886550 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:20.886344    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5zcf"
May 26 16:01:20.886728 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:20.886549    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5zcf" podUID="3369da97-ee5f-42cb-a95f-e6066680f349"
May 26 16:01:20.886728 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:20.886417    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5v7fc"
May 26 16:01:20.886728 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:20.886669    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5v7fc" podUID="c4c50203-4f83-413b-b73f-edc0a16e49ee"
May 26 16:01:21.050301 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:21.050279    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sj6db_5c0f9fc6-2bd6-4370-96d0-5ae315363c7f/ovn-acl-logging/0.log"
May 26 16:01:21.050747 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:21.050698    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sj6db" event={"ID":"5c0f9fc6-2bd6-4370-96d0-5ae315363c7f","Type":"ContainerStarted","Data":"197322fb2d6de254e415fa3fa5f40a84724af4dfd89c0a59c2b837d23383f5c1"}
May 26 16:01:21.053394 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:21.053369    2552 generic.go:358] "Generic (PLEG): container finished" podID="b90e9cb1-f2de-4919-ab89-8bc29c2476ea" containerID="025c64f2d5d5c6bff9024207cb2d882db44730c2cb8f4b5195ad15618aadcb14" exitCode=0
May 26 16:01:21.053499 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:21.053414    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bq8pz" event={"ID":"b90e9cb1-f2de-4919-ab89-8bc29c2476ea","Type":"ContainerDied","Data":"025c64f2d5d5c6bff9024207cb2d882db44730c2cb8f4b5195ad15618aadcb14"}
May 26 16:01:21.081494 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:21.081453    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-sj6db" podStartSLOduration=11.223581516 podStartE2EDuration="30.081440733s" podCreationTimestamp="2026-05-26 16:00:51 +0000 UTC" firstStartedPulling="2026-05-26 16:00:53.492494558 +0000 UTC m=+3.200949579" lastFinishedPulling="2026-05-26 16:01:12.350353776 +0000 UTC m=+22.058808796" observedRunningTime="2026-05-26 16:01:21.081317854 +0000 UTC m=+30.789772915" watchObservedRunningTime="2026-05-26 16:01:21.081440733 +0000 UTC m=+30.789895771"
May 26 16:01:21.262620 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:21.262583    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p5zcf"]
May 26 16:01:21.262734 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:21.262675    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5zcf"
May 26 16:01:21.262766 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:21.262755    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5zcf" podUID="3369da97-ee5f-42cb-a95f-e6066680f349"
May 26 16:01:21.265822 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:21.265796    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gj94f"]
May 26 16:01:21.265987 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:21.265918    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gj94f"
May 26 16:01:21.266064 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:21.266043    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gj94f" podUID="655a0641-f708-444d-91bc-fe7518694ed3"
May 26 16:01:21.266598 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:21.266577    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5v7fc"]
May 26 16:01:21.266684 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:21.266644    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5v7fc"
May 26 16:01:21.266736 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:21.266706    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5v7fc" podUID="c4c50203-4f83-413b-b73f-edc0a16e49ee"
May 26 16:01:22.337665 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:22.337633    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-v7qxk"
May 26 16:01:22.338293 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:22.337777    2552 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
May 26 16:01:22.338557 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:22.338532    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v7qxk"
May 26 16:01:22.889015 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:22.888983    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5v7fc"
May 26 16:01:22.889015 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:22.889003    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gj94f"
May 26 16:01:22.889237 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:22.888983    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5zcf"
May 26 16:01:22.889237 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:22.889080    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5v7fc" podUID="c4c50203-4f83-413b-b73f-edc0a16e49ee"
May 26 16:01:22.889237 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:22.889154    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gj94f" podUID="655a0641-f708-444d-91bc-fe7518694ed3"
May 26 16:01:22.889237 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:22.889223    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5zcf" podUID="3369da97-ee5f-42cb-a95f-e6066680f349"
May 26 16:01:24.537033 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.536995    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/655a0641-f708-444d-91bc-fe7518694ed3-metrics-certs\") pod \"network-metrics-daemon-gj94f\" (UID: \"655a0641-f708-444d-91bc-fe7518694ed3\") " pod="openshift-multus/network-metrics-daemon-gj94f"
May 26 16:01:24.537524 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:24.537159    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:24.537524 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:24.537226    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/655a0641-f708-444d-91bc-fe7518694ed3-metrics-certs podName:655a0641-f708-444d-91bc-fe7518694ed3 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:56.537208609 +0000 UTC m=+66.245663636 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/655a0641-f708-444d-91bc-fe7518694ed3-metrics-certs") pod "network-metrics-daemon-gj94f" (UID: "655a0641-f708-444d-91bc-fe7518694ed3") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:24.575166 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.575143    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-96.ec2.internal" event="NodeReady"
May 26 16:01:24.575301 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.575280    2552 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 26 16:01:24.624352 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.624272    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f5759674c-t6282"]
May 26 16:01:24.637942 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.637907    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w6qxx\" (UniqueName: \"kubernetes.io/projected/c4c50203-4f83-413b-b73f-edc0a16e49ee-kube-api-access-w6qxx\") pod \"network-check-target-5v7fc\" (UID: \"c4c50203-4f83-413b-b73f-edc0a16e49ee\") " pod="openshift-network-diagnostics/network-check-target-5v7fc"
May 26 16:01:24.638074 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.637968    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3369da97-ee5f-42cb-a95f-e6066680f349-original-pull-secret\") pod \"global-pull-secret-syncer-p5zcf\" (UID: \"3369da97-ee5f-42cb-a95f-e6066680f349\") " pod="kube-system/global-pull-secret-syncer-p5zcf"
May 26 16:01:24.638132 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:24.638085    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 16:01:24.638132 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:24.638085    2552 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:01:24.638233 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:24.638135    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3369da97-ee5f-42cb-a95f-e6066680f349-original-pull-secret podName:3369da97-ee5f-42cb-a95f-e6066680f349 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:56.638119403 +0000 UTC m=+66.346574422 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3369da97-ee5f-42cb-a95f-e6066680f349-original-pull-secret") pod "global-pull-secret-syncer-p5zcf" (UID: "3369da97-ee5f-42cb-a95f-e6066680f349") : object "kube-system"/"original-pull-secret" not registered
May 26 16:01:24.638233 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:24.638153    2552 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:01:24.638233 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:24.638170    2552 projected.go:194] Error preparing data for projected volume kube-api-access-w6qxx for pod openshift-network-diagnostics/network-check-target-5v7fc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:24.638233 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:24.638220    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4c50203-4f83-413b-b73f-edc0a16e49ee-kube-api-access-w6qxx podName:c4c50203-4f83-413b-b73f-edc0a16e49ee nodeName:}" failed. No retries permitted until 2026-05-26 16:01:56.638204864 +0000 UTC m=+66.346659886 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-w6qxx" (UniqueName: "kubernetes.io/projected/c4c50203-4f83-413b-b73f-edc0a16e49ee-kube-api-access-w6qxx") pod "network-check-target-5v7fc" (UID: "c4c50203-4f83-413b-b73f-edc0a16e49ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:24.640734 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.640712    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-59589468bc-7p484"]
May 26 16:01:24.640884 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.640868    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f5759674c-t6282"
May 26 16:01:24.645433 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.645409    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
May 26 16:01:24.645534 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.645423    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
May 26 16:01:24.648029 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.647401    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
May 26 16:01:24.648549 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.648525    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
May 26 16:01:24.653258 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.652863    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8559875f4d-f4x8v"]
May 26 16:01:24.654248 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.654035    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59589468bc-7p484"
May 26 16:01:24.654915 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.654687    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
May 26 16:01:24.654915 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.654712    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
May 26 16:01:24.654915 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.654819    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
May 26 16:01:24.657278 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.657260    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
May 26 16:01:24.664975 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.664951    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-797d988dd6-hvdzb"]
May 26 16:01:24.665188 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.665176    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8559875f4d-f4x8v"
May 26 16:01:24.668082 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.668030    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
May 26 16:01:24.668423 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.668404    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-5p2m5\""
May 26 16:01:24.677279 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.677259    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6nz6h"]
May 26 16:01:24.677409 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.677389    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:01:24.680123 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.680105    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 26 16:01:24.680222 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.680109    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 26 16:01:24.680514 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.680494    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 26 16:01:24.680702 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.680681    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-n6l5h\""
May 26 16:01:24.685342 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.685323    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-59589468bc-7p484"]
May 26 16:01:24.685435 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.685347    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-797d988dd6-hvdzb"]
May 26 16:01:24.685435 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.685359    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f5759674c-t6282"]
May 26 16:01:24.685435 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.685373    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-thjfg"]
May 26 16:01:24.685575 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.685473    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6nz6h"
May 26 16:01:24.686523 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.686504    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 26 16:01:24.687833 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.687816    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 26 16:01:24.687905 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.687833    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 26 16:01:24.688444 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.688428    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-w25cq\""
May 26 16:01:24.693993 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.693475    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8559875f4d-f4x8v"]
May 26 16:01:24.693993 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.693499    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6nz6h"]
May 26 16:01:24.693993 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.693511    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-thjfg"]
May 26 16:01:24.693993 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.693582    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-thjfg"
May 26 16:01:24.696430 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.696328    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-l7p5g\""
May 26 16:01:24.696430 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.696331    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 26 16:01:24.696571 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.696430    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 26 16:01:24.696571 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.696379    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 26 16:01:24.738564 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.738522    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ce89a932-1d2d-42a3-a647-782517d92dcd-ca\") pod \"cluster-proxy-proxy-agent-6f5759674c-t6282\" (UID: \"ce89a932-1d2d-42a3-a647-782517d92dcd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f5759674c-t6282"
May 26 16:01:24.738564 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.738559    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ce89a932-1d2d-42a3-a647-782517d92dcd-hub\") pod \"cluster-proxy-proxy-agent-6f5759674c-t6282\" (UID: \"ce89a932-1d2d-42a3-a647-782517d92dcd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f5759674c-t6282"
May 26 16:01:24.738764 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.738587    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ce89a932-1d2d-42a3-a647-782517d92dcd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6f5759674c-t6282\" (UID: \"ce89a932-1d2d-42a3-a647-782517d92dcd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f5759674c-t6282"
May 26 16:01:24.738764 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.738681    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/ce89a932-1d2d-42a3-a647-782517d92dcd-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-6f5759674c-t6282\" (UID: \"ce89a932-1d2d-42a3-a647-782517d92dcd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f5759674c-t6282"
May 26 16:01:24.738764 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.738713    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6drh\" (UniqueName: \"kubernetes.io/projected/ce89a932-1d2d-42a3-a647-782517d92dcd-kube-api-access-k6drh\") pod \"cluster-proxy-proxy-agent-6f5759674c-t6282\" (UID: \"ce89a932-1d2d-42a3-a647-782517d92dcd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f5759674c-t6282"
May 26 16:01:24.738764 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.738760    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ce89a932-1d2d-42a3-a647-782517d92dcd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6f5759674c-t6282\" (UID: \"ce89a932-1d2d-42a3-a647-782517d92dcd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f5759674c-t6282"
May 26 16:01:24.839771 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.839739    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ce89a932-1d2d-42a3-a647-782517d92dcd-ca\") pod \"cluster-proxy-proxy-agent-6f5759674c-t6282\" (UID: \"ce89a932-1d2d-42a3-a647-782517d92dcd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f5759674c-t6282"
May 26 16:01:24.839966 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.839792    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ce89a932-1d2d-42a3-a647-782517d92dcd-hub\") pod \"cluster-proxy-proxy-agent-6f5759674c-t6282\" (UID: \"ce89a932-1d2d-42a3-a647-782517d92dcd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f5759674c-t6282"
May 26 16:01:24.839966 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.839820    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ce89a932-1d2d-42a3-a647-782517d92dcd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6f5759674c-t6282\" (UID: \"ce89a932-1d2d-42a3-a647-782517d92dcd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f5759674c-t6282"
May 26 16:01:24.839966 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.839865    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/076454e2-eee8-47cb-a7e6-dc5e0a0270ae-config-volume\") pod \"dns-default-6nz6h\" (UID: \"076454e2-eee8-47cb-a7e6-dc5e0a0270ae\") " pod="openshift-dns/dns-default-6nz6h"
May 26 16:01:24.839966 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.839891    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/076454e2-eee8-47cb-a7e6-dc5e0a0270ae-metrics-tls\") pod \"dns-default-6nz6h\" (UID: \"076454e2-eee8-47cb-a7e6-dc5e0a0270ae\") " pod="openshift-dns/dns-default-6nz6h"
May 26 16:01:24.840174 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.840044    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dfdaec24-5a44-429c-bf06-3422d868294d-cert\") pod \"ingress-canary-thjfg\" (UID: \"dfdaec24-5a44-429c-bf06-3422d868294d\") " pod="openshift-ingress-canary/ingress-canary-thjfg"
May 26 16:01:24.840174 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.840092    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/076454e2-eee8-47cb-a7e6-dc5e0a0270ae-tmp-dir\") pod \"dns-default-6nz6h\" (UID: \"076454e2-eee8-47cb-a7e6-dc5e0a0270ae\") " pod="openshift-dns/dns-default-6nz6h"
May 26 16:01:24.840174 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.840120    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5995ba40-1aa4-444a-934b-abfe3e7d76ee-image-registry-private-configuration\") pod \"image-registry-797d988dd6-hvdzb\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") " pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:01:24.840174 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.840152    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd55k\" (UniqueName: \"kubernetes.io/projected/076454e2-eee8-47cb-a7e6-dc5e0a0270ae-kube-api-access-nd55k\") pod \"dns-default-6nz6h\" (UID: \"076454e2-eee8-47cb-a7e6-dc5e0a0270ae\") " pod="openshift-dns/dns-default-6nz6h"
May 26 16:01:24.840307 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.840175    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5995ba40-1aa4-444a-934b-abfe3e7d76ee-trusted-ca\") pod \"image-registry-797d988dd6-hvdzb\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") " pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:01:24.840307 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.840214    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/ce89a932-1d2d-42a3-a647-782517d92dcd-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-6f5759674c-t6282\" (UID: \"ce89a932-1d2d-42a3-a647-782517d92dcd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f5759674c-t6282"
May 26 16:01:24.840307 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.840239    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-k6drh\" (UniqueName: \"kubernetes.io/projected/ce89a932-1d2d-42a3-a647-782517d92dcd-kube-api-access-k6drh\") pod \"cluster-proxy-proxy-agent-6f5759674c-t6282\" (UID: \"ce89a932-1d2d-42a3-a647-782517d92dcd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f5759674c-t6282"
May 26 16:01:24.840307 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.840268    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6f7d7e6c-a91a-46c6-ac3c-317de74cfcb8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8559875f4d-f4x8v\" (UID: \"6f7d7e6c-a91a-46c6-ac3c-317de74cfcb8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8559875f4d-f4x8v"
May 26 16:01:24.840307 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.840294    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn5bb\" (UniqueName: \"kubernetes.io/projected/6f7d7e6c-a91a-46c6-ac3c-317de74cfcb8-kube-api-access-kn5bb\") pod \"managed-serviceaccount-addon-agent-8559875f4d-f4x8v\" (UID: \"6f7d7e6c-a91a-46c6-ac3c-317de74cfcb8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8559875f4d-f4x8v"
May 26 16:01:24.840531 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.840320    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5995ba40-1aa4-444a-934b-abfe3e7d76ee-ca-trust-extracted\") pod \"image-registry-797d988dd6-hvdzb\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") " pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:01:24.840531 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.840455    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztvvz\" (UniqueName: \"kubernetes.io/projected/33f9ccb8-9dde-4732-8c4e-ec44cf4a9b38-kube-api-access-ztvvz\") pod \"klusterlet-addon-workmgr-59589468bc-7p484\" (UID: \"33f9ccb8-9dde-4732-8c4e-ec44cf4a9b38\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59589468bc-7p484"
May 26 16:01:24.840531 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.840487    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5995ba40-1aa4-444a-934b-abfe3e7d76ee-registry-certificates\") pod \"image-registry-797d988dd6-hvdzb\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") " pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:01:24.840531 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.840513    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5995ba40-1aa4-444a-934b-abfe3e7d76ee-installation-pull-secrets\") pod \"image-registry-797d988dd6-hvdzb\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") " pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:01:24.840744 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.840540    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-bound-sa-token\") pod \"image-registry-797d988dd6-hvdzb\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") " pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:01:24.840744 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.840574    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ce89a932-1d2d-42a3-a647-782517d92dcd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6f5759674c-t6282\" (UID: \"ce89a932-1d2d-42a3-a647-782517d92dcd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f5759674c-t6282"
May 26 16:01:24.840744 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.840629    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33f9ccb8-9dde-4732-8c4e-ec44cf4a9b38-tmp\") pod \"klusterlet-addon-workmgr-59589468bc-7p484\" (UID: \"33f9ccb8-9dde-4732-8c4e-ec44cf4a9b38\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59589468bc-7p484"
May 26 16:01:24.840744 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.840666    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckz8h\" (UniqueName: \"kubernetes.io/projected/dfdaec24-5a44-429c-bf06-3422d868294d-kube-api-access-ckz8h\") pod \"ingress-canary-thjfg\" (UID: \"dfdaec24-5a44-429c-bf06-3422d868294d\") " pod="openshift-ingress-canary/ingress-canary-thjfg"
May 26 16:01:24.840744 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.840695    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-registry-tls\") pod \"image-registry-797d988dd6-hvdzb\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") " pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:01:24.840744 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.840731    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzgvd\" (UniqueName: \"kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-kube-api-access-zzgvd\") pod \"image-registry-797d988dd6-hvdzb\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") " pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:01:24.841144 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.840776    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/33f9ccb8-9dde-4732-8c4e-ec44cf4a9b38-klusterlet-config\") pod \"klusterlet-addon-workmgr-59589468bc-7p484\" (UID: \"33f9ccb8-9dde-4732-8c4e-ec44cf4a9b38\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59589468bc-7p484"
May 26 16:01:24.841198 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.841145    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/ce89a932-1d2d-42a3-a647-782517d92dcd-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-6f5759674c-t6282\" (UID: \"ce89a932-1d2d-42a3-a647-782517d92dcd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f5759674c-t6282"
May 26 16:01:24.844394 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.844372    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ce89a932-1d2d-42a3-a647-782517d92dcd-ca\") pod \"cluster-proxy-proxy-agent-6f5759674c-t6282\" (UID: \"ce89a932-1d2d-42a3-a647-782517d92dcd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f5759674c-t6282"
May 26 16:01:24.844511 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.844492    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ce89a932-1d2d-42a3-a647-782517d92dcd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6f5759674c-t6282\" (UID: \"ce89a932-1d2d-42a3-a647-782517d92dcd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f5759674c-t6282"
May 26 16:01:24.844552 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.844502    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ce89a932-1d2d-42a3-a647-782517d92dcd-hub\") pod \"cluster-proxy-proxy-agent-6f5759674c-t6282\" (UID: \"ce89a932-1d2d-42a3-a647-782517d92dcd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f5759674c-t6282"
May 26 16:01:24.844584 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.844557    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ce89a932-1d2d-42a3-a647-782517d92dcd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6f5759674c-t6282\" (UID: \"ce89a932-1d2d-42a3-a647-782517d92dcd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f5759674c-t6282"
May 26 16:01:24.852842 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.852797    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6drh\" (UniqueName: \"kubernetes.io/projected/ce89a932-1d2d-42a3-a647-782517d92dcd-kube-api-access-k6drh\") pod \"cluster-proxy-proxy-agent-6f5759674c-t6282\" (UID: \"ce89a932-1d2d-42a3-a647-782517d92dcd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f5759674c-t6282"
May 26 16:01:24.885838 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.885815    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5v7fc"
May 26 16:01:24.885838 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.885833    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gj94f"
May 26 16:01:24.886020 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.885817    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5zcf"
May 26 16:01:24.889857 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.888901    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nf4w7\""
May 26 16:01:24.889857 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.889333    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 26 16:01:24.889857 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.889550    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 26 16:01:24.889857 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.889715    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 26 16:01:24.889857 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.889785    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 26 16:01:24.890152 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.890035    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jsz4f\""
May 26 16:01:24.941486 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.941457    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33f9ccb8-9dde-4732-8c4e-ec44cf4a9b38-tmp\") pod \"klusterlet-addon-workmgr-59589468bc-7p484\" (UID: \"33f9ccb8-9dde-4732-8c4e-ec44cf4a9b38\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59589468bc-7p484"
May 26 16:01:24.941596 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.941495    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ckz8h\" (UniqueName: \"kubernetes.io/projected/dfdaec24-5a44-429c-bf06-3422d868294d-kube-api-access-ckz8h\") pod \"ingress-canary-thjfg\" (UID: \"dfdaec24-5a44-429c-bf06-3422d868294d\") " pod="openshift-ingress-canary/ingress-canary-thjfg"
May 26 16:01:24.941596 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.941520    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-registry-tls\") pod \"image-registry-797d988dd6-hvdzb\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") " pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:01:24.941596 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.941546    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zzgvd\" (UniqueName: \"kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-kube-api-access-zzgvd\") pod \"image-registry-797d988dd6-hvdzb\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") " pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:01:24.941596 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.941575    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/33f9ccb8-9dde-4732-8c4e-ec44cf4a9b38-klusterlet-config\") pod \"klusterlet-addon-workmgr-59589468bc-7p484\" (UID: \"33f9ccb8-9dde-4732-8c4e-ec44cf4a9b38\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59589468bc-7p484"
May 26 16:01:24.941801 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.941645    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/076454e2-eee8-47cb-a7e6-dc5e0a0270ae-config-volume\") pod \"dns-default-6nz6h\" (UID: \"076454e2-eee8-47cb-a7e6-dc5e0a0270ae\") " pod="openshift-dns/dns-default-6nz6h"
May 26 16:01:24.941801 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.941662    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/076454e2-eee8-47cb-a7e6-dc5e0a0270ae-metrics-tls\") pod \"dns-default-6nz6h\" (UID: \"076454e2-eee8-47cb-a7e6-dc5e0a0270ae\") " pod="openshift-dns/dns-default-6nz6h"
May 26 16:01:24.941801 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.941685    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dfdaec24-5a44-429c-bf06-3422d868294d-cert\") pod \"ingress-canary-thjfg\" (UID: \"dfdaec24-5a44-429c-bf06-3422d868294d\") " pod="openshift-ingress-canary/ingress-canary-thjfg"
May 26 16:01:24.941801 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.941711    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/076454e2-eee8-47cb-a7e6-dc5e0a0270ae-tmp-dir\") pod \"dns-default-6nz6h\" (UID: \"076454e2-eee8-47cb-a7e6-dc5e0a0270ae\") " pod="openshift-dns/dns-default-6nz6h"
May 26 16:01:24.941801 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.941728    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5995ba40-1aa4-444a-934b-abfe3e7d76ee-image-registry-private-configuration\") pod \"image-registry-797d988dd6-hvdzb\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") " pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:01:24.941801 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.941748    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nd55k\" (UniqueName: \"kubernetes.io/projected/076454e2-eee8-47cb-a7e6-dc5e0a0270ae-kube-api-access-nd55k\") pod \"dns-default-6nz6h\" (UID: \"076454e2-eee8-47cb-a7e6-dc5e0a0270ae\") " pod="openshift-dns/dns-default-6nz6h"
May 26 16:01:24.941801 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.941771    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5995ba40-1aa4-444a-934b-abfe3e7d76ee-trusted-ca\") pod \"image-registry-797d988dd6-hvdzb\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") " pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:01:24.942166 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.941805    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6f7d7e6c-a91a-46c6-ac3c-317de74cfcb8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8559875f4d-f4x8v\" (UID: \"6f7d7e6c-a91a-46c6-ac3c-317de74cfcb8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8559875f4d-f4x8v"
May 26 16:01:24.942166 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.941821    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kn5bb\" (UniqueName: \"kubernetes.io/projected/6f7d7e6c-a91a-46c6-ac3c-317de74cfcb8-kube-api-access-kn5bb\") pod \"managed-serviceaccount-addon-agent-8559875f4d-f4x8v\" (UID: \"6f7d7e6c-a91a-46c6-ac3c-317de74cfcb8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8559875f4d-f4x8v"
May 26 16:01:24.942166 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.941836    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5995ba40-1aa4-444a-934b-abfe3e7d76ee-ca-trust-extracted\") pod \"image-registry-797d988dd6-hvdzb\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") " pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:01:24.942166 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.941867    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33f9ccb8-9dde-4732-8c4e-ec44cf4a9b38-tmp\") pod \"klusterlet-addon-workmgr-59589468bc-7p484\" (UID: \"33f9ccb8-9dde-4732-8c4e-ec44cf4a9b38\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59589468bc-7p484"
May 26 16:01:24.942166 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.941873    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ztvvz\" (UniqueName: \"kubernetes.io/projected/33f9ccb8-9dde-4732-8c4e-ec44cf4a9b38-kube-api-access-ztvvz\") pod \"klusterlet-addon-workmgr-59589468bc-7p484\" (UID: \"33f9ccb8-9dde-4732-8c4e-ec44cf4a9b38\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59589468bc-7p484"
May 26 16:01:24.942166 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:24.941949    2552 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:01:24.942166 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:24.942016    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfdaec24-5a44-429c-bf06-3422d868294d-cert podName:dfdaec24-5a44-429c-bf06-3422d868294d nodeName:}" failed. No retries permitted until 2026-05-26 16:01:25.441999317 +0000 UTC m=+35.150454339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/dfdaec24-5a44-429c-bf06-3422d868294d-cert") pod "ingress-canary-thjfg" (UID: "dfdaec24-5a44-429c-bf06-3422d868294d") : secret "canary-serving-cert" not found
May 26 16:01:24.942166 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.942155    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/076454e2-eee8-47cb-a7e6-dc5e0a0270ae-tmp-dir\") pod \"dns-default-6nz6h\" (UID: \"076454e2-eee8-47cb-a7e6-dc5e0a0270ae\") " pod="openshift-dns/dns-default-6nz6h"
May 26 16:01:24.942542 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.941923    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5995ba40-1aa4-444a-934b-abfe3e7d76ee-registry-certificates\") pod \"image-registry-797d988dd6-hvdzb\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") " pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:01:24.942542 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.942235    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5995ba40-1aa4-444a-934b-abfe3e7d76ee-installation-pull-secrets\") pod \"image-registry-797d988dd6-hvdzb\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") " pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:01:24.942542 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.942259    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-bound-sa-token\") pod \"image-registry-797d988dd6-hvdzb\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") " pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:01:24.942542 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.942491    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5995ba40-1aa4-444a-934b-abfe3e7d76ee-registry-certificates\") pod \"image-registry-797d988dd6-hvdzb\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") " pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:01:24.942725 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:24.942678    2552 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 16:01:24.942725 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:24.942692    2552 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-797d988dd6-hvdzb: secret "image-registry-tls" not found
May 26 16:01:24.942820 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.942733    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5995ba40-1aa4-444a-934b-abfe3e7d76ee-ca-trust-extracted\") pod \"image-registry-797d988dd6-hvdzb\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") " pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:01:24.942820 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:24.942747    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-registry-tls podName:5995ba40-1aa4-444a-934b-abfe3e7d76ee nodeName:}" failed. No retries permitted until 2026-05-26 16:01:25.442731094 +0000 UTC m=+35.151186132 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-registry-tls") pod "image-registry-797d988dd6-hvdzb" (UID: "5995ba40-1aa4-444a-934b-abfe3e7d76ee") : secret "image-registry-tls" not found
May 26 16:01:24.942820 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:24.942813    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:01:24.942994 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:24.942847    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/076454e2-eee8-47cb-a7e6-dc5e0a0270ae-metrics-tls podName:076454e2-eee8-47cb-a7e6-dc5e0a0270ae nodeName:}" failed. No retries permitted until 2026-05-26 16:01:25.442835649 +0000 UTC m=+35.151290666 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/076454e2-eee8-47cb-a7e6-dc5e0a0270ae-metrics-tls") pod "dns-default-6nz6h" (UID: "076454e2-eee8-47cb-a7e6-dc5e0a0270ae") : secret "dns-default-metrics-tls" not found
May 26 16:01:24.942994 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.942809    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5995ba40-1aa4-444a-934b-abfe3e7d76ee-trusted-ca\") pod \"image-registry-797d988dd6-hvdzb\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") " pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:01:24.943326 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.943280    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/076454e2-eee8-47cb-a7e6-dc5e0a0270ae-config-volume\") pod \"dns-default-6nz6h\" (UID: \"076454e2-eee8-47cb-a7e6-dc5e0a0270ae\") " pod="openshift-dns/dns-default-6nz6h"
May 26 16:01:24.945022 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.944999    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5995ba40-1aa4-444a-934b-abfe3e7d76ee-image-registry-private-configuration\") pod \"image-registry-797d988dd6-hvdzb\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") " pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:01:24.945323 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.945303    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5995ba40-1aa4-444a-934b-abfe3e7d76ee-installation-pull-secrets\") pod \"image-registry-797d988dd6-hvdzb\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") " pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:01:24.945408 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.945311    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/33f9ccb8-9dde-4732-8c4e-ec44cf4a9b38-klusterlet-config\") pod \"klusterlet-addon-workmgr-59589468bc-7p484\" (UID: \"33f9ccb8-9dde-4732-8c4e-ec44cf4a9b38\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59589468bc-7p484"
May 26 16:01:24.945679 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.945662    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6f7d7e6c-a91a-46c6-ac3c-317de74cfcb8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8559875f4d-f4x8v\" (UID: \"6f7d7e6c-a91a-46c6-ac3c-317de74cfcb8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8559875f4d-f4x8v"
May 26 16:01:24.951466 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.951397    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-bound-sa-token\") pod \"image-registry-797d988dd6-hvdzb\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") " pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:01:24.957465 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.957440    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztvvz\" (UniqueName: \"kubernetes.io/projected/33f9ccb8-9dde-4732-8c4e-ec44cf4a9b38-kube-api-access-ztvvz\") pod \"klusterlet-addon-workmgr-59589468bc-7p484\" (UID: \"33f9ccb8-9dde-4732-8c4e-ec44cf4a9b38\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59589468bc-7p484"
May 26 16:01:24.958261 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.958238    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd55k\" (UniqueName: \"kubernetes.io/projected/076454e2-eee8-47cb-a7e6-dc5e0a0270ae-kube-api-access-nd55k\") pod \"dns-default-6nz6h\" (UID: \"076454e2-eee8-47cb-a7e6-dc5e0a0270ae\") " pod="openshift-dns/dns-default-6nz6h"
May 26 16:01:24.958622 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.958582    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzgvd\" (UniqueName: \"kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-kube-api-access-zzgvd\") pod \"image-registry-797d988dd6-hvdzb\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") " pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:01:24.959376 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.959359    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckz8h\" (UniqueName: \"kubernetes.io/projected/dfdaec24-5a44-429c-bf06-3422d868294d-kube-api-access-ckz8h\") pod \"ingress-canary-thjfg\" (UID: \"dfdaec24-5a44-429c-bf06-3422d868294d\") " pod="openshift-ingress-canary/ingress-canary-thjfg"
May 26 16:01:24.959599 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.959572    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn5bb\" (UniqueName: \"kubernetes.io/projected/6f7d7e6c-a91a-46c6-ac3c-317de74cfcb8-kube-api-access-kn5bb\") pod \"managed-serviceaccount-addon-agent-8559875f4d-f4x8v\" (UID: \"6f7d7e6c-a91a-46c6-ac3c-317de74cfcb8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8559875f4d-f4x8v"
May 26 16:01:24.963859 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.963836    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f5759674c-t6282"
May 26 16:01:24.974665 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.974647    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59589468bc-7p484"
May 26 16:01:24.980123 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:24.980106    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8559875f4d-f4x8v"
May 26 16:01:25.446179 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:25.446147    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-registry-tls\") pod \"image-registry-797d988dd6-hvdzb\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") " pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:01:25.446424 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:25.446225    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/076454e2-eee8-47cb-a7e6-dc5e0a0270ae-metrics-tls\") pod \"dns-default-6nz6h\" (UID: \"076454e2-eee8-47cb-a7e6-dc5e0a0270ae\") " pod="openshift-dns/dns-default-6nz6h"
May 26 16:01:25.446424 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:25.446289    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dfdaec24-5a44-429c-bf06-3422d868294d-cert\") pod \"ingress-canary-thjfg\" (UID: \"dfdaec24-5a44-429c-bf06-3422d868294d\") " pod="openshift-ingress-canary/ingress-canary-thjfg"
May 26 16:01:25.446424 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:25.446314    2552 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 16:01:25.446424 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:25.446330    2552 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-797d988dd6-hvdzb: secret "image-registry-tls" not found
May 26 16:01:25.446424 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:25.446403    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-registry-tls podName:5995ba40-1aa4-444a-934b-abfe3e7d76ee nodeName:}" failed. No retries permitted until 2026-05-26 16:01:26.446383574 +0000 UTC m=+36.154838608 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-registry-tls") pod "image-registry-797d988dd6-hvdzb" (UID: "5995ba40-1aa4-444a-934b-abfe3e7d76ee") : secret "image-registry-tls" not found
May 26 16:01:25.446424 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:25.446409    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:01:25.446672 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:25.446457    2552 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:01:25.446672 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:25.446484    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/076454e2-eee8-47cb-a7e6-dc5e0a0270ae-metrics-tls podName:076454e2-eee8-47cb-a7e6-dc5e0a0270ae nodeName:}" failed. No retries permitted until 2026-05-26 16:01:26.446465893 +0000 UTC m=+36.154920918 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/076454e2-eee8-47cb-a7e6-dc5e0a0270ae-metrics-tls") pod "dns-default-6nz6h" (UID: "076454e2-eee8-47cb-a7e6-dc5e0a0270ae") : secret "dns-default-metrics-tls" not found
May 26 16:01:25.446672 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:25.446519    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfdaec24-5a44-429c-bf06-3422d868294d-cert podName:dfdaec24-5a44-429c-bf06-3422d868294d nodeName:}" failed. No retries permitted until 2026-05-26 16:01:26.446504172 +0000 UTC m=+36.154959190 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/dfdaec24-5a44-429c-bf06-3422d868294d-cert") pod "ingress-canary-thjfg" (UID: "dfdaec24-5a44-429c-bf06-3422d868294d") : secret "canary-serving-cert" not found
May 26 16:01:26.455607 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:26.455568    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/076454e2-eee8-47cb-a7e6-dc5e0a0270ae-metrics-tls\") pod \"dns-default-6nz6h\" (UID: \"076454e2-eee8-47cb-a7e6-dc5e0a0270ae\") " pod="openshift-dns/dns-default-6nz6h"
May 26 16:01:26.455607 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:26.455608    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dfdaec24-5a44-429c-bf06-3422d868294d-cert\") pod \"ingress-canary-thjfg\" (UID: \"dfdaec24-5a44-429c-bf06-3422d868294d\") " pod="openshift-ingress-canary/ingress-canary-thjfg"
May 26 16:01:26.456235 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:26.455652    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-registry-tls\") pod \"image-registry-797d988dd6-hvdzb\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") " pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:01:26.456235 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:26.455735    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:01:26.456235 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:26.455755    2552 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:01:26.456235 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:26.455758    2552 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 16:01:26.456235 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:26.455806    2552 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-797d988dd6-hvdzb: secret "image-registry-tls" not found
May 26 16:01:26.456235 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:26.455811    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/076454e2-eee8-47cb-a7e6-dc5e0a0270ae-metrics-tls podName:076454e2-eee8-47cb-a7e6-dc5e0a0270ae nodeName:}" failed. No retries permitted until 2026-05-26 16:01:28.455791957 +0000 UTC m=+38.164246978 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/076454e2-eee8-47cb-a7e6-dc5e0a0270ae-metrics-tls") pod "dns-default-6nz6h" (UID: "076454e2-eee8-47cb-a7e6-dc5e0a0270ae") : secret "dns-default-metrics-tls" not found
May 26 16:01:26.456235 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:26.455839    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-registry-tls podName:5995ba40-1aa4-444a-934b-abfe3e7d76ee nodeName:}" failed. No retries permitted until 2026-05-26 16:01:28.455823594 +0000 UTC m=+38.164278627 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-registry-tls") pod "image-registry-797d988dd6-hvdzb" (UID: "5995ba40-1aa4-444a-934b-abfe3e7d76ee") : secret "image-registry-tls" not found
May 26 16:01:26.456235 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:26.455858    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfdaec24-5a44-429c-bf06-3422d868294d-cert podName:dfdaec24-5a44-429c-bf06-3422d868294d nodeName:}" failed. No retries permitted until 2026-05-26 16:01:28.455846615 +0000 UTC m=+38.164301644 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/dfdaec24-5a44-429c-bf06-3422d868294d-cert") pod "ingress-canary-thjfg" (UID: "dfdaec24-5a44-429c-bf06-3422d868294d") : secret "canary-serving-cert" not found
May 26 16:01:26.782703 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:26.782626    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f5759674c-t6282"]
May 26 16:01:26.784367 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:26.784309    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-59589468bc-7p484"]
May 26 16:01:26.800571 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:26.800546    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8559875f4d-f4x8v"]
May 26 16:01:26.800862 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:01:26.800843    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33f9ccb8_9dde_4732_8c4e_ec44cf4a9b38.slice/crio-e15f9b1f5319efccb11190ce44ca21b9eb12718843991c9cbf3b67fe0850a9e2 WatchSource:0}: Error finding container e15f9b1f5319efccb11190ce44ca21b9eb12718843991c9cbf3b67fe0850a9e2: Status 404 returned error can't find the container with id e15f9b1f5319efccb11190ce44ca21b9eb12718843991c9cbf3b67fe0850a9e2
May 26 16:01:26.801299 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:01:26.801282    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce89a932_1d2d_42a3_a647_782517d92dcd.slice/crio-0049fecd850858814884ad62c989043b7f28a38525c83d57b6d21ccb8e22de08 WatchSource:0}: Error finding container 0049fecd850858814884ad62c989043b7f28a38525c83d57b6d21ccb8e22de08: Status 404 returned error can't find the container with id 0049fecd850858814884ad62c989043b7f28a38525c83d57b6d21ccb8e22de08
May 26 16:01:26.804421 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:01:26.804396    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f7d7e6c_a91a_46c6_ac3c_317de74cfcb8.slice/crio-062cc2fc793a874fb4ae9048f48fee0270d2658d9af88ef91fb84d970314c41e WatchSource:0}: Error finding container 062cc2fc793a874fb4ae9048f48fee0270d2658d9af88ef91fb84d970314c41e: Status 404 returned error can't find the container with id 062cc2fc793a874fb4ae9048f48fee0270d2658d9af88ef91fb84d970314c41e
May 26 16:01:27.068588 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:27.068342    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59589468bc-7p484" event={"ID":"33f9ccb8-9dde-4732-8c4e-ec44cf4a9b38","Type":"ContainerStarted","Data":"e15f9b1f5319efccb11190ce44ca21b9eb12718843991c9cbf3b67fe0850a9e2"}
May 26 16:01:27.071202 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:27.071178    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bq8pz" event={"ID":"b90e9cb1-f2de-4919-ab89-8bc29c2476ea","Type":"ContainerStarted","Data":"a0b0444f637bb09f6c91bced55539c7a25dc698b73e86ecf63bce3d98651cd49"}
May 26 16:01:27.072246 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:27.072223    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f5759674c-t6282" event={"ID":"ce89a932-1d2d-42a3-a647-782517d92dcd","Type":"ContainerStarted","Data":"0049fecd850858814884ad62c989043b7f28a38525c83d57b6d21ccb8e22de08"}
May 26 16:01:27.073145 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:27.073126    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8559875f4d-f4x8v" event={"ID":"6f7d7e6c-a91a-46c6-ac3c-317de74cfcb8","Type":"ContainerStarted","Data":"062cc2fc793a874fb4ae9048f48fee0270d2658d9af88ef91fb84d970314c41e"}
May 26 16:01:28.080709 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:28.080670    2552 generic.go:358] "Generic (PLEG): container finished" podID="b90e9cb1-f2de-4919-ab89-8bc29c2476ea" containerID="a0b0444f637bb09f6c91bced55539c7a25dc698b73e86ecf63bce3d98651cd49" exitCode=0
May 26 16:01:28.081158 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:28.080719    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bq8pz" event={"ID":"b90e9cb1-f2de-4919-ab89-8bc29c2476ea","Type":"ContainerDied","Data":"a0b0444f637bb09f6c91bced55539c7a25dc698b73e86ecf63bce3d98651cd49"}
May 26 16:01:28.474806 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:28.474363    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/076454e2-eee8-47cb-a7e6-dc5e0a0270ae-metrics-tls\") pod \"dns-default-6nz6h\" (UID: \"076454e2-eee8-47cb-a7e6-dc5e0a0270ae\") " pod="openshift-dns/dns-default-6nz6h"
May 26 16:01:28.474806 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:28.474414    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dfdaec24-5a44-429c-bf06-3422d868294d-cert\") pod \"ingress-canary-thjfg\" (UID: \"dfdaec24-5a44-429c-bf06-3422d868294d\") " pod="openshift-ingress-canary/ingress-canary-thjfg"
May 26 16:01:28.474806 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:28.474482    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-registry-tls\") pod \"image-registry-797d988dd6-hvdzb\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") " pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:01:28.474806 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:28.474518    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:01:28.474806 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:28.474609    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/076454e2-eee8-47cb-a7e6-dc5e0a0270ae-metrics-tls podName:076454e2-eee8-47cb-a7e6-dc5e0a0270ae nodeName:}" failed. No retries permitted until 2026-05-26 16:01:32.474587252 +0000 UTC m=+42.183042287 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/076454e2-eee8-47cb-a7e6-dc5e0a0270ae-metrics-tls") pod "dns-default-6nz6h" (UID: "076454e2-eee8-47cb-a7e6-dc5e0a0270ae") : secret "dns-default-metrics-tls" not found
May 26 16:01:28.474806 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:28.474623    2552 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 16:01:28.474806 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:28.474638    2552 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-797d988dd6-hvdzb: secret "image-registry-tls" not found
May 26 16:01:28.474806 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:28.474694    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-registry-tls podName:5995ba40-1aa4-444a-934b-abfe3e7d76ee nodeName:}" failed. No retries permitted until 2026-05-26 16:01:32.474678917 +0000 UTC m=+42.183133938 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-registry-tls") pod "image-registry-797d988dd6-hvdzb" (UID: "5995ba40-1aa4-444a-934b-abfe3e7d76ee") : secret "image-registry-tls" not found
May 26 16:01:28.474806 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:28.474750    2552 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:01:28.474806 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:28.474780    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfdaec24-5a44-429c-bf06-3422d868294d-cert podName:dfdaec24-5a44-429c-bf06-3422d868294d nodeName:}" failed. No retries permitted until 2026-05-26 16:01:32.474769981 +0000 UTC m=+42.183225000 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/dfdaec24-5a44-429c-bf06-3422d868294d-cert") pod "ingress-canary-thjfg" (UID: "dfdaec24-5a44-429c-bf06-3422d868294d") : secret "canary-serving-cert" not found
May 26 16:01:29.087128 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:29.087092    2552 generic.go:358] "Generic (PLEG): container finished" podID="b90e9cb1-f2de-4919-ab89-8bc29c2476ea" containerID="5d665ba08b011bc6d75ce62f2e385ac6110e1384f4bceed51653759e859a1ba4" exitCode=0
May 26 16:01:29.087523 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:29.087152    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bq8pz" event={"ID":"b90e9cb1-f2de-4919-ab89-8bc29c2476ea","Type":"ContainerDied","Data":"5d665ba08b011bc6d75ce62f2e385ac6110e1384f4bceed51653759e859a1ba4"}
May 26 16:01:32.510116 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:32.510072    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-registry-tls\") pod \"image-registry-797d988dd6-hvdzb\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") " pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:01:32.510742 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:32.510153    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/076454e2-eee8-47cb-a7e6-dc5e0a0270ae-metrics-tls\") pod \"dns-default-6nz6h\" (UID: \"076454e2-eee8-47cb-a7e6-dc5e0a0270ae\") " pod="openshift-dns/dns-default-6nz6h"
May 26 16:01:32.510742 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:32.510173    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dfdaec24-5a44-429c-bf06-3422d868294d-cert\") pod \"ingress-canary-thjfg\" (UID: \"dfdaec24-5a44-429c-bf06-3422d868294d\") " pod="openshift-ingress-canary/ingress-canary-thjfg"
May 26 16:01:32.510742 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:32.510208    2552 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 16:01:32.510742 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:32.510228    2552 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-797d988dd6-hvdzb: secret "image-registry-tls" not found
May 26 16:01:32.510742 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:32.510278    2552 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:01:32.510742 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:32.510282    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:01:32.510742 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:32.510285    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-registry-tls podName:5995ba40-1aa4-444a-934b-abfe3e7d76ee nodeName:}" failed. No retries permitted until 2026-05-26 16:01:40.510268286 +0000 UTC m=+50.218723304 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-registry-tls") pod "image-registry-797d988dd6-hvdzb" (UID: "5995ba40-1aa4-444a-934b-abfe3e7d76ee") : secret "image-registry-tls" not found
May 26 16:01:32.510742 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:32.510328    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfdaec24-5a44-429c-bf06-3422d868294d-cert podName:dfdaec24-5a44-429c-bf06-3422d868294d nodeName:}" failed. No retries permitted until 2026-05-26 16:01:40.510316165 +0000 UTC m=+50.218771183 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/dfdaec24-5a44-429c-bf06-3422d868294d-cert") pod "ingress-canary-thjfg" (UID: "dfdaec24-5a44-429c-bf06-3422d868294d") : secret "canary-serving-cert" not found
May 26 16:01:32.510742 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:32.510338    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/076454e2-eee8-47cb-a7e6-dc5e0a0270ae-metrics-tls podName:076454e2-eee8-47cb-a7e6-dc5e0a0270ae nodeName:}" failed. No retries permitted until 2026-05-26 16:01:40.510332504 +0000 UTC m=+50.218787522 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/076454e2-eee8-47cb-a7e6-dc5e0a0270ae-metrics-tls") pod "dns-default-6nz6h" (UID: "076454e2-eee8-47cb-a7e6-dc5e0a0270ae") : secret "dns-default-metrics-tls" not found
May 26 16:01:35.103864 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:35.103824    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59589468bc-7p484" event={"ID":"33f9ccb8-9dde-4732-8c4e-ec44cf4a9b38","Type":"ContainerStarted","Data":"6c2f257b4519e4de4694d75f28d75aebdd83592a00f2176ee1eab0dc894d895a"}
May 26 16:01:35.104357 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:35.104067    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59589468bc-7p484"
May 26 16:01:35.106112 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:35.106089    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59589468bc-7p484"
May 26 16:01:35.107146 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:35.107120    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bq8pz" event={"ID":"b90e9cb1-f2de-4919-ab89-8bc29c2476ea","Type":"ContainerStarted","Data":"3a3e557279e35f6f7154fdb653648c100e2198234bc4a6d6d1582be5235672b5"}
May 26 16:01:35.108484 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:35.108456    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f5759674c-t6282" event={"ID":"ce89a932-1d2d-42a3-a647-782517d92dcd","Type":"ContainerStarted","Data":"b2a472f85d009c4f472bf45dc92884594b3a2218a21e96755437f7c2fa68a42b"}
May 26 16:01:35.109584 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:35.109562    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8559875f4d-f4x8v" event={"ID":"6f7d7e6c-a91a-46c6-ac3c-317de74cfcb8","Type":"ContainerStarted","Data":"266c07f551abc4a86ca007961a45c0ff392297a2a2b0beaaacc75c326271f33c"}
May 26 16:01:35.122630 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:35.122585    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59589468bc-7p484" podStartSLOduration=23.482682446 podStartE2EDuration="31.122570728s" podCreationTimestamp="2026-05-26 16:01:04 +0000 UTC" firstStartedPulling="2026-05-26 16:01:26.806323762 +0000 UTC m=+36.514778794" lastFinishedPulling="2026-05-26 16:01:34.446212051 +0000 UTC m=+44.154667076" observedRunningTime="2026-05-26 16:01:35.12198864 +0000 UTC m=+44.830443680" watchObservedRunningTime="2026-05-26 16:01:35.122570728 +0000 UTC m=+44.831025768"
May 26 16:01:35.166515 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:35.166447    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bq8pz" podStartSLOduration=11.831111167 podStartE2EDuration="45.166422699s" podCreationTimestamp="2026-05-26 16:00:50 +0000 UTC" firstStartedPulling="2026-05-26 16:00:53.49569438 +0000 UTC m=+3.204149411" lastFinishedPulling="2026-05-26 16:01:26.831005926 +0000 UTC m=+36.539460943" observedRunningTime="2026-05-26 16:01:35.164350753 +0000 UTC m=+44.872805793" watchObservedRunningTime="2026-05-26 16:01:35.166422699 +0000 UTC m=+44.874877741"
May 26 16:01:35.183820 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:35.183735    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8559875f4d-f4x8v" podStartSLOduration=23.562220268 podStartE2EDuration="31.183715927s" podCreationTimestamp="2026-05-26 16:01:04 +0000 UTC" firstStartedPulling="2026-05-26 16:01:26.80647356 +0000 UTC m=+36.514928588" lastFinishedPulling="2026-05-26 16:01:34.427969219 +0000 UTC m=+44.136424247" observedRunningTime="2026-05-26 16:01:35.183530446 +0000 UTC m=+44.891985485" watchObservedRunningTime="2026-05-26 16:01:35.183715927 +0000 UTC m=+44.892170946"
May 26 16:01:40.120377 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:40.120342    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f5759674c-t6282" event={"ID":"ce89a932-1d2d-42a3-a647-782517d92dcd","Type":"ContainerStarted","Data":"f1f4a2b19ce80eb8476007edca4a434c346427c70345f07a31792be4a72774e3"}
May 26 16:01:40.120377 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:40.120379    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f5759674c-t6282" event={"ID":"ce89a932-1d2d-42a3-a647-782517d92dcd","Type":"ContainerStarted","Data":"1c4807d3b907db4f56715b4058e031b008fd7125318cf198ba3c1ca237dc4918"}
May 26 16:01:40.159339 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:40.157468    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f5759674c-t6282" podStartSLOduration=23.305240266 podStartE2EDuration="36.157452551s" podCreationTimestamp="2026-05-26 16:01:04 +0000 UTC" firstStartedPulling="2026-05-26 16:01:26.806322902 +0000 UTC m=+36.514777933" lastFinishedPulling="2026-05-26 16:01:39.658535193 +0000 UTC m=+49.366990218" observedRunningTime="2026-05-26 16:01:40.155060711 +0000 UTC m=+49.863515752" watchObservedRunningTime="2026-05-26 16:01:40.157452551 +0000 UTC m=+49.865907591"
May 26 16:01:40.572521 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:40.572492    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-registry-tls\") pod \"image-registry-797d988dd6-hvdzb\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") " pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:01:40.572671 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:40.572566    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/076454e2-eee8-47cb-a7e6-dc5e0a0270ae-metrics-tls\") pod \"dns-default-6nz6h\" (UID: \"076454e2-eee8-47cb-a7e6-dc5e0a0270ae\") " pod="openshift-dns/dns-default-6nz6h"
May 26 16:01:40.572671 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:40.572592    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dfdaec24-5a44-429c-bf06-3422d868294d-cert\") pod \"ingress-canary-thjfg\" (UID: \"dfdaec24-5a44-429c-bf06-3422d868294d\") " pod="openshift-ingress-canary/ingress-canary-thjfg"
May 26 16:01:40.572671 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:40.572640    2552 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 16:01:40.572671 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:40.572656    2552 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-797d988dd6-hvdzb: secret "image-registry-tls" not found
May 26 16:01:40.572795 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:40.572699    2552 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:01:40.572795 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:40.572706    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:01:40.572795 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:40.572717    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-registry-tls podName:5995ba40-1aa4-444a-934b-abfe3e7d76ee nodeName:}" failed. No retries permitted until 2026-05-26 16:01:56.572698721 +0000 UTC m=+66.281153753 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-registry-tls") pod "image-registry-797d988dd6-hvdzb" (UID: "5995ba40-1aa4-444a-934b-abfe3e7d76ee") : secret "image-registry-tls" not found
May 26 16:01:40.572795 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:40.572786    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfdaec24-5a44-429c-bf06-3422d868294d-cert podName:dfdaec24-5a44-429c-bf06-3422d868294d nodeName:}" failed. No retries permitted until 2026-05-26 16:01:56.572772471 +0000 UTC m=+66.281227490 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/dfdaec24-5a44-429c-bf06-3422d868294d-cert") pod "ingress-canary-thjfg" (UID: "dfdaec24-5a44-429c-bf06-3422d868294d") : secret "canary-serving-cert" not found
May 26 16:01:40.572917 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:40.572797    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/076454e2-eee8-47cb-a7e6-dc5e0a0270ae-metrics-tls podName:076454e2-eee8-47cb-a7e6-dc5e0a0270ae nodeName:}" failed. No retries permitted until 2026-05-26 16:01:56.572791593 +0000 UTC m=+66.281246611 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/076454e2-eee8-47cb-a7e6-dc5e0a0270ae-metrics-tls") pod "dns-default-6nz6h" (UID: "076454e2-eee8-47cb-a7e6-dc5e0a0270ae") : secret "dns-default-metrics-tls" not found
May 26 16:01:52.066375 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:52.066345    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sj6db"
May 26 16:01:56.583461 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:56.583425    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/076454e2-eee8-47cb-a7e6-dc5e0a0270ae-metrics-tls\") pod \"dns-default-6nz6h\" (UID: \"076454e2-eee8-47cb-a7e6-dc5e0a0270ae\") " pod="openshift-dns/dns-default-6nz6h"
May 26 16:01:56.583461 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:56.583462    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dfdaec24-5a44-429c-bf06-3422d868294d-cert\") pod \"ingress-canary-thjfg\" (UID: \"dfdaec24-5a44-429c-bf06-3422d868294d\") " pod="openshift-ingress-canary/ingress-canary-thjfg"
May 26 16:01:56.584024 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:56.583510    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-registry-tls\") pod \"image-registry-797d988dd6-hvdzb\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") " pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:01:56.584024 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:56.583529    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/655a0641-f708-444d-91bc-fe7518694ed3-metrics-certs\") pod \"network-metrics-daemon-gj94f\" (UID: \"655a0641-f708-444d-91bc-fe7518694ed3\") " pod="openshift-multus/network-metrics-daemon-gj94f"
May 26 16:01:56.584024 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:56.583565    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:01:56.584024 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:56.583639    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/076454e2-eee8-47cb-a7e6-dc5e0a0270ae-metrics-tls podName:076454e2-eee8-47cb-a7e6-dc5e0a0270ae nodeName:}" failed. No retries permitted until 2026-05-26 16:02:28.583624024 +0000 UTC m=+98.292079047 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/076454e2-eee8-47cb-a7e6-dc5e0a0270ae-metrics-tls") pod "dns-default-6nz6h" (UID: "076454e2-eee8-47cb-a7e6-dc5e0a0270ae") : secret "dns-default-metrics-tls" not found
May 26 16:01:56.584024 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:56.583657    2552 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 16:01:56.584024 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:56.583685    2552 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-797d988dd6-hvdzb: secret "image-registry-tls" not found
May 26 16:01:56.584024 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:56.583692    2552 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:01:56.584024 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:56.583745    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-registry-tls podName:5995ba40-1aa4-444a-934b-abfe3e7d76ee nodeName:}" failed. No retries permitted until 2026-05-26 16:02:28.583727386 +0000 UTC m=+98.292182412 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-registry-tls") pod "image-registry-797d988dd6-hvdzb" (UID: "5995ba40-1aa4-444a-934b-abfe3e7d76ee") : secret "image-registry-tls" not found
May 26 16:01:56.584024 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:56.583766    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfdaec24-5a44-429c-bf06-3422d868294d-cert podName:dfdaec24-5a44-429c-bf06-3422d868294d nodeName:}" failed. No retries permitted until 2026-05-26 16:02:28.583755731 +0000 UTC m=+98.292210750 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/dfdaec24-5a44-429c-bf06-3422d868294d-cert") pod "ingress-canary-thjfg" (UID: "dfdaec24-5a44-429c-bf06-3422d868294d") : secret "canary-serving-cert" not found
May 26 16:01:56.586485 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:56.586470    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 26 16:01:56.593693 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:56.593680    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 26 16:01:56.593758 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:01:56.593716    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/655a0641-f708-444d-91bc-fe7518694ed3-metrics-certs podName:655a0641-f708-444d-91bc-fe7518694ed3 nodeName:}" failed. No retries permitted until 2026-05-26 16:03:00.593705795 +0000 UTC m=+130.302160812 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/655a0641-f708-444d-91bc-fe7518694ed3-metrics-certs") pod "network-metrics-daemon-gj94f" (UID: "655a0641-f708-444d-91bc-fe7518694ed3") : secret "metrics-daemon-secret" not found
May 26 16:01:56.684409 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:56.684376    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w6qxx\" (UniqueName: \"kubernetes.io/projected/c4c50203-4f83-413b-b73f-edc0a16e49ee-kube-api-access-w6qxx\") pod \"network-check-target-5v7fc\" (UID: \"c4c50203-4f83-413b-b73f-edc0a16e49ee\") " pod="openshift-network-diagnostics/network-check-target-5v7fc"
May 26 16:01:56.684551 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:56.684418    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3369da97-ee5f-42cb-a95f-e6066680f349-original-pull-secret\") pod \"global-pull-secret-syncer-p5zcf\" (UID: \"3369da97-ee5f-42cb-a95f-e6066680f349\") " pod="kube-system/global-pull-secret-syncer-p5zcf"
May 26 16:01:56.687392 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:56.687373    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 26 16:01:56.687446 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:56.687411    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 26 16:01:56.698224 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:56.698190    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 26 16:01:56.698457 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:56.698442    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3369da97-ee5f-42cb-a95f-e6066680f349-original-pull-secret\") pod \"global-pull-secret-syncer-p5zcf\" (UID: \"3369da97-ee5f-42cb-a95f-e6066680f349\") " pod="kube-system/global-pull-secret-syncer-p5zcf"
May 26 16:01:56.704665 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:56.704649    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5zcf"
May 26 16:01:56.708294 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:56.708272    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6qxx\" (UniqueName: \"kubernetes.io/projected/c4c50203-4f83-413b-b73f-edc0a16e49ee-kube-api-access-w6qxx\") pod \"network-check-target-5v7fc\" (UID: \"c4c50203-4f83-413b-b73f-edc0a16e49ee\") " pod="openshift-network-diagnostics/network-check-target-5v7fc"
May 26 16:01:56.822837 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:56.822801    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p5zcf"]
May 26 16:01:56.826129 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:01:56.826105    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3369da97_ee5f_42cb_a95f_e6066680f349.slice/crio-8d89a6ff5e44d58f90e36521bc0d8eb09d31c8fe2be8fbb11fb80fefc772232d WatchSource:0}: Error finding container 8d89a6ff5e44d58f90e36521bc0d8eb09d31c8fe2be8fbb11fb80fefc772232d: Status 404 returned error can't find the container with id 8d89a6ff5e44d58f90e36521bc0d8eb09d31c8fe2be8fbb11fb80fefc772232d
May 26 16:01:57.001171 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:57.001096    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nf4w7\""
May 26 16:01:57.009233 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:57.009216    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5v7fc"
May 26 16:01:57.129651 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:57.129621    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5v7fc"]
May 26 16:01:57.132282 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:01:57.132258    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4c50203_4f83_413b_b73f_edc0a16e49ee.slice/crio-d226e78101d12c153b0be2a5a1103713c4f8d7ec9a255b5f16777cceb38a44c5 WatchSource:0}: Error finding container d226e78101d12c153b0be2a5a1103713c4f8d7ec9a255b5f16777cceb38a44c5: Status 404 returned error can't find the container with id d226e78101d12c153b0be2a5a1103713c4f8d7ec9a255b5f16777cceb38a44c5
May 26 16:01:57.155703 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:57.155660    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5v7fc" event={"ID":"c4c50203-4f83-413b-b73f-edc0a16e49ee","Type":"ContainerStarted","Data":"d226e78101d12c153b0be2a5a1103713c4f8d7ec9a255b5f16777cceb38a44c5"}
May 26 16:01:57.156545 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:01:57.156522    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p5zcf" event={"ID":"3369da97-ee5f-42cb-a95f-e6066680f349","Type":"ContainerStarted","Data":"8d89a6ff5e44d58f90e36521bc0d8eb09d31c8fe2be8fbb11fb80fefc772232d"}
May 26 16:02:02.170487 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:02:02.170380    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5v7fc" event={"ID":"c4c50203-4f83-413b-b73f-edc0a16e49ee","Type":"ContainerStarted","Data":"42b775e0c91ac0d394f01446bd73d8d811e43e2193555d6a67d9280ef79da203"}
May 26 16:02:02.170487 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:02:02.170470    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5v7fc"
May 26 16:02:02.171798 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:02:02.171765    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p5zcf" event={"ID":"3369da97-ee5f-42cb-a95f-e6066680f349","Type":"ContainerStarted","Data":"69eac64df6ca7d79a2ce753527458c7e68ceafa71535a8c03519f1b8b969b67a"}
May 26 16:02:02.189616 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:02:02.189566    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5v7fc" podStartSLOduration=66.413351344 podStartE2EDuration="1m11.18955344s" podCreationTimestamp="2026-05-26 16:00:51 +0000 UTC" firstStartedPulling="2026-05-26 16:01:57.134047679 +0000 UTC m=+66.842502701" lastFinishedPulling="2026-05-26 16:02:01.91024978 +0000 UTC m=+71.618704797" observedRunningTime="2026-05-26 16:02:02.188757435 +0000 UTC m=+71.897212475" watchObservedRunningTime="2026-05-26 16:02:02.18955344 +0000 UTC m=+71.898008479"
May 26 16:02:02.205799 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:02:02.205753    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-p5zcf" podStartSLOduration=65.111795487 podStartE2EDuration="1m10.205741335s" podCreationTimestamp="2026-05-26 16:00:52 +0000 UTC" firstStartedPulling="2026-05-26 16:01:56.827822897 +0000 UTC m=+66.536277915" lastFinishedPulling="2026-05-26 16:02:01.921768742 +0000 UTC m=+71.630223763" observedRunningTime="2026-05-26 16:02:02.205349292 +0000 UTC m=+71.913804333" watchObservedRunningTime="2026-05-26 16:02:02.205741335 +0000 UTC m=+71.914196374"
May 26 16:02:28.623762 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:02:28.623627    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-registry-tls\") pod \"image-registry-797d988dd6-hvdzb\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") " pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:02:28.623762 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:02:28.623719    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/076454e2-eee8-47cb-a7e6-dc5e0a0270ae-metrics-tls\") pod \"dns-default-6nz6h\" (UID: \"076454e2-eee8-47cb-a7e6-dc5e0a0270ae\") " pod="openshift-dns/dns-default-6nz6h"
May 26 16:02:28.623762 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:02:28.623749    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dfdaec24-5a44-429c-bf06-3422d868294d-cert\") pod \"ingress-canary-thjfg\" (UID: \"dfdaec24-5a44-429c-bf06-3422d868294d\") " pod="openshift-ingress-canary/ingress-canary-thjfg"
May 26 16:02:28.624312 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:02:28.623796    2552 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 16:02:28.624312 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:02:28.623815    2552 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-797d988dd6-hvdzb: secret "image-registry-tls" not found
May 26 16:02:28.624312 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:02:28.623862    2552 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:02:28.624312 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:02:28.623861    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:02:28.624312 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:02:28.623880    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-registry-tls podName:5995ba40-1aa4-444a-934b-abfe3e7d76ee nodeName:}" failed. No retries permitted until 2026-05-26 16:03:32.623861119 +0000 UTC m=+162.332316140 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-registry-tls") pod "image-registry-797d988dd6-hvdzb" (UID: "5995ba40-1aa4-444a-934b-abfe3e7d76ee") : secret "image-registry-tls" not found
May 26 16:02:28.624312 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:02:28.623906    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfdaec24-5a44-429c-bf06-3422d868294d-cert podName:dfdaec24-5a44-429c-bf06-3422d868294d nodeName:}" failed. No retries permitted until 2026-05-26 16:03:32.623895241 +0000 UTC m=+162.332350269 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/dfdaec24-5a44-429c-bf06-3422d868294d-cert") pod "ingress-canary-thjfg" (UID: "dfdaec24-5a44-429c-bf06-3422d868294d") : secret "canary-serving-cert" not found
May 26 16:02:28.624312 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:02:28.623950    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/076454e2-eee8-47cb-a7e6-dc5e0a0270ae-metrics-tls podName:076454e2-eee8-47cb-a7e6-dc5e0a0270ae nodeName:}" failed. No retries permitted until 2026-05-26 16:03:32.623914652 +0000 UTC m=+162.332369670 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/076454e2-eee8-47cb-a7e6-dc5e0a0270ae-metrics-tls") pod "dns-default-6nz6h" (UID: "076454e2-eee8-47cb-a7e6-dc5e0a0270ae") : secret "dns-default-metrics-tls" not found
May 26 16:02:33.177066 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:02:33.177036    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5v7fc"
May 26 16:02:49.608950 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:02:49.608899    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j7hlf_4475d020-5281-4257-bf03-1a264049169d/dns-node-resolver/0.log"
May 26 16:02:50.395106 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:02:50.395077    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5xp7f_092e4858-1372-4cb9-b51b-dde9e50e0f9d/node-ca/0.log"
May 26 16:03:00.668604 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:00.668562    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/655a0641-f708-444d-91bc-fe7518694ed3-metrics-certs\") pod \"network-metrics-daemon-gj94f\" (UID: \"655a0641-f708-444d-91bc-fe7518694ed3\") " pod="openshift-multus/network-metrics-daemon-gj94f"
May 26 16:03:00.670830 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:00.670807    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/655a0641-f708-444d-91bc-fe7518694ed3-metrics-certs\") pod \"network-metrics-daemon-gj94f\" (UID: \"655a0641-f708-444d-91bc-fe7518694ed3\") " pod="openshift-multus/network-metrics-daemon-gj94f"
May 26 16:03:00.913843 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:00.913812    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jsz4f\""
May 26 16:03:00.921375 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:00.921313    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gj94f"
May 26 16:03:01.048033 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:01.048001    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gj94f"]
May 26 16:03:01.050819 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:03:01.050788    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod655a0641_f708_444d_91bc_fe7518694ed3.slice/crio-8f68db5f05fab7fb0efd9d4123f4041222778202c361820eac7e1a82e7eab5ab WatchSource:0}: Error finding container 8f68db5f05fab7fb0efd9d4123f4041222778202c361820eac7e1a82e7eab5ab: Status 404 returned error can't find the container with id 8f68db5f05fab7fb0efd9d4123f4041222778202c361820eac7e1a82e7eab5ab
May 26 16:03:01.315051 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:01.315010    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gj94f" event={"ID":"655a0641-f708-444d-91bc-fe7518694ed3","Type":"ContainerStarted","Data":"8f68db5f05fab7fb0efd9d4123f4041222778202c361820eac7e1a82e7eab5ab"}
May 26 16:03:02.319162 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:02.319129    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gj94f" event={"ID":"655a0641-f708-444d-91bc-fe7518694ed3","Type":"ContainerStarted","Data":"63d490104d39191daf8781b12448c786b383b1dd79348118946447b305aeee99"}
May 26 16:03:03.323141 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:03.323099    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gj94f" event={"ID":"655a0641-f708-444d-91bc-fe7518694ed3","Type":"ContainerStarted","Data":"722ddfcb07b9ed6a491e595bef028819aa400b80d375e761bf65418f2d2059b7"}
May 26 16:03:03.342292 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:03.342242    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gj94f" podStartSLOduration=132.217583442 podStartE2EDuration="2m13.342227044s" podCreationTimestamp="2026-05-26 16:00:50 +0000 UTC" firstStartedPulling="2026-05-26 16:03:01.052482507 +0000 UTC m=+130.760937525" lastFinishedPulling="2026-05-26 16:03:02.177126104 +0000 UTC m=+131.885581127" observedRunningTime="2026-05-26 16:03:03.341223201 +0000 UTC m=+133.049678241" watchObservedRunningTime="2026-05-26 16:03:03.342227044 +0000 UTC m=+133.050682083"
May 26 16:03:09.349301 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:09.349261    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2759w"]
May 26 16:03:09.351636 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:09.351613    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2759w"
May 26 16:03:09.354411 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:09.354391    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 26 16:03:09.355686 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:09.355659    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 26 16:03:09.355812 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:09.355664    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q4tcs\""
May 26 16:03:09.355812 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:09.355664    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 26 16:03:09.355812 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:09.355692    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 26 16:03:09.366291 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:09.366264    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2759w"]
May 26 16:03:09.437065 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:09.437029    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c844ea1b-4c3b-4273-a8b7-6667de9b3fd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2759w\" (UID: \"c844ea1b-4c3b-4273-a8b7-6667de9b3fd9\") " pod="openshift-insights/insights-runtime-extractor-2759w"
May 26 16:03:09.437065 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:09.437068    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c844ea1b-4c3b-4273-a8b7-6667de9b3fd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2759w\" (UID: \"c844ea1b-4c3b-4273-a8b7-6667de9b3fd9\") " pod="openshift-insights/insights-runtime-extractor-2759w"
May 26 16:03:09.437290 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:09.437092    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z546g\" (UniqueName: \"kubernetes.io/projected/c844ea1b-4c3b-4273-a8b7-6667de9b3fd9-kube-api-access-z546g\") pod \"insights-runtime-extractor-2759w\" (UID: \"c844ea1b-4c3b-4273-a8b7-6667de9b3fd9\") " pod="openshift-insights/insights-runtime-extractor-2759w"
May 26 16:03:09.437290 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:09.437184    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c844ea1b-4c3b-4273-a8b7-6667de9b3fd9-crio-socket\") pod \"insights-runtime-extractor-2759w\" (UID: \"c844ea1b-4c3b-4273-a8b7-6667de9b3fd9\") " pod="openshift-insights/insights-runtime-extractor-2759w"
May 26 16:03:09.437290 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:09.437208    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c844ea1b-4c3b-4273-a8b7-6667de9b3fd9-data-volume\") pod \"insights-runtime-extractor-2759w\" (UID: \"c844ea1b-4c3b-4273-a8b7-6667de9b3fd9\") " pod="openshift-insights/insights-runtime-extractor-2759w"
May 26 16:03:09.537719 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:09.537679    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c844ea1b-4c3b-4273-a8b7-6667de9b3fd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2759w\" (UID: \"c844ea1b-4c3b-4273-a8b7-6667de9b3fd9\") " pod="openshift-insights/insights-runtime-extractor-2759w"
May 26 16:03:09.537719 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:09.537715    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c844ea1b-4c3b-4273-a8b7-6667de9b3fd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2759w\" (UID: \"c844ea1b-4c3b-4273-a8b7-6667de9b3fd9\") " pod="openshift-insights/insights-runtime-extractor-2759w"
May 26 16:03:09.537993 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:09.537740    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-z546g\" (UniqueName: \"kubernetes.io/projected/c844ea1b-4c3b-4273-a8b7-6667de9b3fd9-kube-api-access-z546g\") pod \"insights-runtime-extractor-2759w\" (UID: \"c844ea1b-4c3b-4273-a8b7-6667de9b3fd9\") " pod="openshift-insights/insights-runtime-extractor-2759w"
May 26 16:03:09.537993 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:09.537904    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c844ea1b-4c3b-4273-a8b7-6667de9b3fd9-crio-socket\") pod \"insights-runtime-extractor-2759w\" (UID: \"c844ea1b-4c3b-4273-a8b7-6667de9b3fd9\") " pod="openshift-insights/insights-runtime-extractor-2759w"
May 26 16:03:09.537993 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:09.537964    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c844ea1b-4c3b-4273-a8b7-6667de9b3fd9-data-volume\") pod \"insights-runtime-extractor-2759w\" (UID: \"c844ea1b-4c3b-4273-a8b7-6667de9b3fd9\") " pod="openshift-insights/insights-runtime-extractor-2759w"
May 26 16:03:09.538147 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:09.538041    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c844ea1b-4c3b-4273-a8b7-6667de9b3fd9-crio-socket\") pod \"insights-runtime-extractor-2759w\" (UID: \"c844ea1b-4c3b-4273-a8b7-6667de9b3fd9\") " pod="openshift-insights/insights-runtime-extractor-2759w"
May 26 16:03:09.538239 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:09.538221    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c844ea1b-4c3b-4273-a8b7-6667de9b3fd9-data-volume\") pod \"insights-runtime-extractor-2759w\" (UID: \"c844ea1b-4c3b-4273-a8b7-6667de9b3fd9\") " pod="openshift-insights/insights-runtime-extractor-2759w"
May 26 16:03:09.538389 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:09.538367    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c844ea1b-4c3b-4273-a8b7-6667de9b3fd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2759w\" (UID: \"c844ea1b-4c3b-4273-a8b7-6667de9b3fd9\") " pod="openshift-insights/insights-runtime-extractor-2759w"
May 26 16:03:09.540127 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:09.540109    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c844ea1b-4c3b-4273-a8b7-6667de9b3fd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2759w\" (UID: \"c844ea1b-4c3b-4273-a8b7-6667de9b3fd9\") " pod="openshift-insights/insights-runtime-extractor-2759w"
May 26 16:03:09.550121 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:09.550100    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z546g\" (UniqueName: \"kubernetes.io/projected/c844ea1b-4c3b-4273-a8b7-6667de9b3fd9-kube-api-access-z546g\") pod \"insights-runtime-extractor-2759w\" (UID: \"c844ea1b-4c3b-4273-a8b7-6667de9b3fd9\") " pod="openshift-insights/insights-runtime-extractor-2759w"
May 26 16:03:09.660421 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:09.660335    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2759w"
May 26 16:03:09.792888 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:09.792855    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2759w"]
May 26 16:03:09.803704 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:03:09.803678    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc844ea1b_4c3b_4273_a8b7_6667de9b3fd9.slice/crio-045d7411aa0ded2c2e46550f0675826a1e4d805069c29d26fcb175b5479aabca WatchSource:0}: Error finding container 045d7411aa0ded2c2e46550f0675826a1e4d805069c29d26fcb175b5479aabca: Status 404 returned error can't find the container with id 045d7411aa0ded2c2e46550f0675826a1e4d805069c29d26fcb175b5479aabca
May 26 16:03:10.343681 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:10.343641    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2759w" event={"ID":"c844ea1b-4c3b-4273-a8b7-6667de9b3fd9","Type":"ContainerStarted","Data":"3b364c3ce603b6f02f3e5a0dc288d4c1918839e2ec8ea45ad3b8210bd0136aa9"}
May 26 16:03:10.343681 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:10.343681    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2759w" event={"ID":"c844ea1b-4c3b-4273-a8b7-6667de9b3fd9","Type":"ContainerStarted","Data":"045d7411aa0ded2c2e46550f0675826a1e4d805069c29d26fcb175b5479aabca"}
May 26 16:03:11.347165 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:11.347127    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2759w" event={"ID":"c844ea1b-4c3b-4273-a8b7-6667de9b3fd9","Type":"ContainerStarted","Data":"31c3c40d169f5bc82e249efcb4a3cca6f3189828003b723fd2617e577282bf77"}
May 26 16:03:12.351257 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:12.351222    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2759w" event={"ID":"c844ea1b-4c3b-4273-a8b7-6667de9b3fd9","Type":"ContainerStarted","Data":"77e4a69b5ee764a2f375472543e437bf1af382fc442a4fba2cbb1b69c0408954"}
May 26 16:03:12.374812 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:12.374751    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2759w" podStartSLOduration=1.032944756 podStartE2EDuration="3.374736089s" podCreationTimestamp="2026-05-26 16:03:09 +0000 UTC" firstStartedPulling="2026-05-26 16:03:09.857590716 +0000 UTC m=+139.566045734" lastFinishedPulling="2026-05-26 16:03:12.19938205 +0000 UTC m=+141.907837067" observedRunningTime="2026-05-26 16:03:12.373281225 +0000 UTC m=+142.081736265" watchObservedRunningTime="2026-05-26 16:03:12.374736089 +0000 UTC m=+142.083191129"
May 26 16:03:14.965042 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:14.964996    2552 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f5759674c-t6282" podUID="ce89a932-1d2d-42a3-a647-782517d92dcd" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 26 16:03:17.803104 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:17.803061    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-d7p8f"]
May 26 16:03:17.805284 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:17.805256    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d7p8f"
May 26 16:03:17.808396 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:17.808374    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 26 16:03:17.809213 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:17.809192    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 26 16:03:17.809335 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:17.809214    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 26 16:03:17.809335 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:17.809224    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cb4pg\""
May 26 16:03:17.809335 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:17.809237    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 26 16:03:17.809335 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:17.809225    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 26 16:03:17.809542 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:17.809526    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 26 16:03:17.904630 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:17.904585    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f008cf1-69a4-45fc-aca1-35a597b97653-metrics-client-ca\") pod \"node-exporter-d7p8f\" (UID: \"3f008cf1-69a4-45fc-aca1-35a597b97653\") " pod="openshift-monitoring/node-exporter-d7p8f"
May 26 16:03:17.904630 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:17.904630    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3f008cf1-69a4-45fc-aca1-35a597b97653-node-exporter-accelerators-collector-config\") pod \"node-exporter-d7p8f\" (UID: \"3f008cf1-69a4-45fc-aca1-35a597b97653\") " pod="openshift-monitoring/node-exporter-d7p8f"
May 26 16:03:17.904845 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:17.904752    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3f008cf1-69a4-45fc-aca1-35a597b97653-node-exporter-tls\") pod \"node-exporter-d7p8f\" (UID: \"3f008cf1-69a4-45fc-aca1-35a597b97653\") " pod="openshift-monitoring/node-exporter-d7p8f"
May 26 16:03:17.904845 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:17.904792    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77xrn\" (UniqueName: \"kubernetes.io/projected/3f008cf1-69a4-45fc-aca1-35a597b97653-kube-api-access-77xrn\") pod \"node-exporter-d7p8f\" (UID: \"3f008cf1-69a4-45fc-aca1-35a597b97653\") " pod="openshift-monitoring/node-exporter-d7p8f"
May 26 16:03:17.904845 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:17.904834    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3f008cf1-69a4-45fc-aca1-35a597b97653-node-exporter-textfile\") pod \"node-exporter-d7p8f\" (UID: \"3f008cf1-69a4-45fc-aca1-35a597b97653\") " pod="openshift-monitoring/node-exporter-d7p8f"
May 26 16:03:17.904975 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:17.904853    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3f008cf1-69a4-45fc-aca1-35a597b97653-root\") pod \"node-exporter-d7p8f\" (UID: \"3f008cf1-69a4-45fc-aca1-35a597b97653\") " pod="openshift-monitoring/node-exporter-d7p8f"
May 26 16:03:17.904975 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:17.904888    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f008cf1-69a4-45fc-aca1-35a597b97653-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d7p8f\" (UID: \"3f008cf1-69a4-45fc-aca1-35a597b97653\") " pod="openshift-monitoring/node-exporter-d7p8f"
May 26 16:03:17.904975 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:17.904917    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f008cf1-69a4-45fc-aca1-35a597b97653-sys\") pod \"node-exporter-d7p8f\" (UID: \"3f008cf1-69a4-45fc-aca1-35a597b97653\") " pod="openshift-monitoring/node-exporter-d7p8f"
May 26 16:03:17.904975 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:17.904966    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3f008cf1-69a4-45fc-aca1-35a597b97653-node-exporter-wtmp\") pod \"node-exporter-d7p8f\" (UID: \"3f008cf1-69a4-45fc-aca1-35a597b97653\") " pod="openshift-monitoring/node-exporter-d7p8f"
May 26 16:03:18.005415 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:18.005377    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3f008cf1-69a4-45fc-aca1-35a597b97653-node-exporter-tls\") pod \"node-exporter-d7p8f\" (UID: \"3f008cf1-69a4-45fc-aca1-35a597b97653\") " pod="openshift-monitoring/node-exporter-d7p8f"
May 26 16:03:18.005583 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:18.005420    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-77xrn\" (UniqueName: \"kubernetes.io/projected/3f008cf1-69a4-45fc-aca1-35a597b97653-kube-api-access-77xrn\") pod \"node-exporter-d7p8f\" (UID: \"3f008cf1-69a4-45fc-aca1-35a597b97653\") " pod="openshift-monitoring/node-exporter-d7p8f"
May 26 16:03:18.005583 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:18.005468    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3f008cf1-69a4-45fc-aca1-35a597b97653-node-exporter-textfile\") pod \"node-exporter-d7p8f\" (UID: \"3f008cf1-69a4-45fc-aca1-35a597b97653\") " pod="openshift-monitoring/node-exporter-d7p8f"
May 26 16:03:18.005583 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:18.005494    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3f008cf1-69a4-45fc-aca1-35a597b97653-root\") pod \"node-exporter-d7p8f\" (UID: \"3f008cf1-69a4-45fc-aca1-35a597b97653\") " pod="openshift-monitoring/node-exporter-d7p8f"
May 26 16:03:18.005583 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:18.005526    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f008cf1-69a4-45fc-aca1-35a597b97653-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d7p8f\" (UID: \"3f008cf1-69a4-45fc-aca1-35a597b97653\") " pod="openshift-monitoring/node-exporter-d7p8f"
May 26 16:03:18.005583 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:03:18.005543    2552 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
May 26 16:03:18.005824 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:03:18.005611    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f008cf1-69a4-45fc-aca1-35a597b97653-node-exporter-tls podName:3f008cf1-69a4-45fc-aca1-35a597b97653 nodeName:}" failed. No retries permitted until 2026-05-26 16:03:18.505591044 +0000 UTC m=+148.214046081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/3f008cf1-69a4-45fc-aca1-35a597b97653-node-exporter-tls") pod "node-exporter-d7p8f" (UID: "3f008cf1-69a4-45fc-aca1-35a597b97653") : secret "node-exporter-tls" not found
May 26 16:03:18.005824 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:18.005606    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3f008cf1-69a4-45fc-aca1-35a597b97653-root\") pod \"node-exporter-d7p8f\" (UID: \"3f008cf1-69a4-45fc-aca1-35a597b97653\") " pod="openshift-monitoring/node-exporter-d7p8f"
May 26 16:03:18.005824 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:18.005660    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f008cf1-69a4-45fc-aca1-35a597b97653-sys\") pod \"node-exporter-d7p8f\" (UID: \"3f008cf1-69a4-45fc-aca1-35a597b97653\") " pod="openshift-monitoring/node-exporter-d7p8f"
May 26 16:03:18.005824 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:18.005712    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3f008cf1-69a4-45fc-aca1-35a597b97653-node-exporter-wtmp\") pod \"node-exporter-d7p8f\" (UID: \"3f008cf1-69a4-45fc-aca1-35a597b97653\") " pod="openshift-monitoring/node-exporter-d7p8f"
May 26 16:03:18.005824 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:18.005742    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f008cf1-69a4-45fc-aca1-35a597b97653-metrics-client-ca\") pod \"node-exporter-d7p8f\" (UID: \"3f008cf1-69a4-45fc-aca1-35a597b97653\") " pod="openshift-monitoring/node-exporter-d7p8f"
May 26 16:03:18.005824 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:18.005772    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f008cf1-69a4-45fc-aca1-35a597b97653-sys\") pod \"node-exporter-d7p8f\" (UID: \"3f008cf1-69a4-45fc-aca1-35a597b97653\") " pod="openshift-monitoring/node-exporter-d7p8f"
May 26 16:03:18.005824 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:18.005808    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3f008cf1-69a4-45fc-aca1-35a597b97653-node-exporter-accelerators-collector-config\") pod \"node-exporter-d7p8f\" (UID: \"3f008cf1-69a4-45fc-aca1-35a597b97653\") " pod="openshift-monitoring/node-exporter-d7p8f"
May 26 16:03:18.006116 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:18.005849    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3f008cf1-69a4-45fc-aca1-35a597b97653-node-exporter-wtmp\") pod \"node-exporter-d7p8f\" (UID: \"3f008cf1-69a4-45fc-aca1-35a597b97653\") " pod="openshift-monitoring/node-exporter-d7p8f"
May 26 16:03:18.006116 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:18.005895    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3f008cf1-69a4-45fc-aca1-35a597b97653-node-exporter-textfile\") pod \"node-exporter-d7p8f\" (UID: \"3f008cf1-69a4-45fc-aca1-35a597b97653\") " pod="openshift-monitoring/node-exporter-d7p8f"
May 26 16:03:18.006285 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:18.006266    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f008cf1-69a4-45fc-aca1-35a597b97653-metrics-client-ca\") pod \"node-exporter-d7p8f\" (UID: \"3f008cf1-69a4-45fc-aca1-35a597b97653\") " pod="openshift-monitoring/node-exporter-d7p8f"
May 26 16:03:18.006329 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:18.006269    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3f008cf1-69a4-45fc-aca1-35a597b97653-node-exporter-accelerators-collector-config\") pod \"node-exporter-d7p8f\" (UID: \"3f008cf1-69a4-45fc-aca1-35a597b97653\") " pod="openshift-monitoring/node-exporter-d7p8f"
May 26 16:03:18.007830 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:18.007813    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f008cf1-69a4-45fc-aca1-35a597b97653-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d7p8f\" (UID: \"3f008cf1-69a4-45fc-aca1-35a597b97653\") " pod="openshift-monitoring/node-exporter-d7p8f"
May 26 16:03:18.018301 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:18.018276    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77xrn\" (UniqueName: \"kubernetes.io/projected/3f008cf1-69a4-45fc-aca1-35a597b97653-kube-api-access-77xrn\") pod \"node-exporter-d7p8f\" (UID: \"3f008cf1-69a4-45fc-aca1-35a597b97653\") " pod="openshift-monitoring/node-exporter-d7p8f"
May 26 16:03:18.511432 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:18.511388    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3f008cf1-69a4-45fc-aca1-35a597b97653-node-exporter-tls\") pod \"node-exporter-d7p8f\" (UID: \"3f008cf1-69a4-45fc-aca1-35a597b97653\") " pod="openshift-monitoring/node-exporter-d7p8f"
May 26 16:03:18.513667 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:18.513644    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3f008cf1-69a4-45fc-aca1-35a597b97653-node-exporter-tls\") pod \"node-exporter-d7p8f\" (UID: \"3f008cf1-69a4-45fc-aca1-35a597b97653\") " pod="openshift-monitoring/node-exporter-d7p8f"
May 26 16:03:18.717472 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:18.717421    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d7p8f"
May 26 16:03:18.725015 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:03:18.724988    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f008cf1_69a4_45fc_aca1_35a597b97653.slice/crio-a49a161b0fd128f2648d6b5d2668f0c75721da6b4580990b867c1ac52530e884 WatchSource:0}: Error finding container a49a161b0fd128f2648d6b5d2668f0c75721da6b4580990b867c1ac52530e884: Status 404 returned error can't find the container with id a49a161b0fd128f2648d6b5d2668f0c75721da6b4580990b867c1ac52530e884
May 26 16:03:19.368306 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:19.368270    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d7p8f" event={"ID":"3f008cf1-69a4-45fc-aca1-35a597b97653","Type":"ContainerStarted","Data":"a49a161b0fd128f2648d6b5d2668f0c75721da6b4580990b867c1ac52530e884"}
May 26 16:03:20.372967 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:20.372919    2552 generic.go:358] "Generic (PLEG): container finished" podID="3f008cf1-69a4-45fc-aca1-35a597b97653" containerID="ff640f533566ec0564c8be675dd592fc5937d2f6fc593906ef843563307d4980" exitCode=0
May 26 16:03:20.372967 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:20.372962    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d7p8f" event={"ID":"3f008cf1-69a4-45fc-aca1-35a597b97653","Type":"ContainerDied","Data":"ff640f533566ec0564c8be675dd592fc5937d2f6fc593906ef843563307d4980"}
May 26 16:03:21.376912 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:21.376875    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d7p8f" event={"ID":"3f008cf1-69a4-45fc-aca1-35a597b97653","Type":"ContainerStarted","Data":"81b8ef323e45715e45b311ea094d2669e432ea24baf089baa954290ff55a3924"}
May 26 16:03:21.376912 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:21.376914    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d7p8f" event={"ID":"3f008cf1-69a4-45fc-aca1-35a597b97653","Type":"ContainerStarted","Data":"226d3c1633b34c08fc736acab743e7a62ca91305bea115867dc63c982c1830aa"}
May 26 16:03:21.399996 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:21.399922    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-d7p8f" podStartSLOduration=3.612645024 podStartE2EDuration="4.399908186s" podCreationTimestamp="2026-05-26 16:03:17 +0000 UTC" firstStartedPulling="2026-05-26 16:03:18.7268412 +0000 UTC m=+148.435296218" lastFinishedPulling="2026-05-26 16:03:19.514104362 +0000 UTC m=+149.222559380" observedRunningTime="2026-05-26 16:03:21.399473051 +0000 UTC m=+151.107928092" watchObservedRunningTime="2026-05-26 16:03:21.399908186 +0000 UTC m=+151.108363226"
May 26 16:03:24.965511 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:24.965469    2552 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f5759674c-t6282" podUID="ce89a932-1d2d-42a3-a647-782517d92dcd" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 26 16:03:27.697534 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:03:27.697482    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-797d988dd6-hvdzb" podUID="5995ba40-1aa4-444a-934b-abfe3e7d76ee"
May 26 16:03:27.703656 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:03:27.703619    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-6nz6h" podUID="076454e2-eee8-47cb-a7e6-dc5e0a0270ae"
May 26 16:03:27.709810 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:03:27.709774    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-thjfg" podUID="dfdaec24-5a44-429c-bf06-3422d868294d"
May 26 16:03:28.395872 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:28.395840    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:03:28.396118 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:28.395840    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6nz6h"
May 26 16:03:32.624612 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:32.624558    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-registry-tls\") pod \"image-registry-797d988dd6-hvdzb\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") " pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:03:32.624612 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:32.624620    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/076454e2-eee8-47cb-a7e6-dc5e0a0270ae-metrics-tls\") pod \"dns-default-6nz6h\" (UID: \"076454e2-eee8-47cb-a7e6-dc5e0a0270ae\") " pod="openshift-dns/dns-default-6nz6h"
May 26 16:03:32.625082 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:32.624677    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dfdaec24-5a44-429c-bf06-3422d868294d-cert\") pod \"ingress-canary-thjfg\" (UID: \"dfdaec24-5a44-429c-bf06-3422d868294d\") " pod="openshift-ingress-canary/ingress-canary-thjfg"
May 26 16:03:32.626967 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:32.626923    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/076454e2-eee8-47cb-a7e6-dc5e0a0270ae-metrics-tls\") pod \"dns-default-6nz6h\" (UID: \"076454e2-eee8-47cb-a7e6-dc5e0a0270ae\") " pod="openshift-dns/dns-default-6nz6h"
May 26 16:03:32.627086 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:32.627049    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-registry-tls\") pod \"image-registry-797d988dd6-hvdzb\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") " pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:03:32.627086 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:32.627071    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dfdaec24-5a44-429c-bf06-3422d868294d-cert\") pod \"ingress-canary-thjfg\" (UID: \"dfdaec24-5a44-429c-bf06-3422d868294d\") " pod="openshift-ingress-canary/ingress-canary-thjfg"
May 26 16:03:32.900827 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:32.900746    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-w25cq\""
May 26 16:03:32.900827 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:32.900757    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-n6l5h\""
May 26 16:03:32.908035 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:32.908015    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:03:32.908187 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:32.908102    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6nz6h"
May 26 16:03:33.045910 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:33.045869    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-797d988dd6-hvdzb"]
May 26 16:03:33.049521 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:03:33.049492    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5995ba40_1aa4_444a_934b_abfe3e7d76ee.slice/crio-d92a3e26e16607960d26bd9f5164797ed0f2681ea64bb343aeb6eb0e8bf504ff WatchSource:0}: Error finding container d92a3e26e16607960d26bd9f5164797ed0f2681ea64bb343aeb6eb0e8bf504ff: Status 404 returned error can't find the container with id d92a3e26e16607960d26bd9f5164797ed0f2681ea64bb343aeb6eb0e8bf504ff
May 26 16:03:33.061522 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:33.061479    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6nz6h"]
May 26 16:03:33.063992 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:03:33.063959    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod076454e2_eee8_47cb_a7e6_dc5e0a0270ae.slice/crio-7505a5967e2f104a4bcab1525c6d45610fe3c65073b5c3b3215247501521b77f WatchSource:0}: Error finding container 7505a5967e2f104a4bcab1525c6d45610fe3c65073b5c3b3215247501521b77f: Status 404 returned error can't find the container with id 7505a5967e2f104a4bcab1525c6d45610fe3c65073b5c3b3215247501521b77f
May 26 16:03:33.409790 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:33.409755    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6nz6h" event={"ID":"076454e2-eee8-47cb-a7e6-dc5e0a0270ae","Type":"ContainerStarted","Data":"7505a5967e2f104a4bcab1525c6d45610fe3c65073b5c3b3215247501521b77f"}
May 26 16:03:33.411090 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:33.411065    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-797d988dd6-hvdzb" event={"ID":"5995ba40-1aa4-444a-934b-abfe3e7d76ee","Type":"ContainerStarted","Data":"ec45ce86faa53dc4183b8407b882760ef9ebf4ea1784f313efeb369676899be3"}
May 26 16:03:33.411213 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:33.411095    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-797d988dd6-hvdzb" event={"ID":"5995ba40-1aa4-444a-934b-abfe3e7d76ee","Type":"ContainerStarted","Data":"d92a3e26e16607960d26bd9f5164797ed0f2681ea64bb343aeb6eb0e8bf504ff"}
May 26 16:03:33.411260 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:33.411212    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:03:33.437027 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:33.436966    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-797d988dd6-hvdzb" podStartSLOduration=162.436945459 podStartE2EDuration="2m42.436945459s" podCreationTimestamp="2026-05-26 16:00:51 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:03:33.435219094 +0000 UTC m=+163.143674135" watchObservedRunningTime="2026-05-26 16:03:33.436945459 +0000 UTC m=+163.145400498"
May 26 16:03:34.414714 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:34.414687    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6nz6h" event={"ID":"076454e2-eee8-47cb-a7e6-dc5e0a0270ae","Type":"ContainerStarted","Data":"6ea24e3850078694447697f49288190ca61de12672fb855dd797862f4f322449"}
May 26 16:03:34.965893 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:34.965804    2552 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f5759674c-t6282" podUID="ce89a932-1d2d-42a3-a647-782517d92dcd" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 26 16:03:34.965893 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:34.965880    2552 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f5759674c-t6282"
May 26 16:03:34.966380 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:34.966351    2552 kuberuntime_manager.go:1107] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"f1f4a2b19ce80eb8476007edca4a434c346427c70345f07a31792be4a72774e3"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f5759674c-t6282" containerMessage="Container service-proxy failed liveness probe, will be restarted"
May 26 16:03:34.966427 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:34.966412    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f5759674c-t6282" podUID="ce89a932-1d2d-42a3-a647-782517d92dcd" containerName="service-proxy" containerID="cri-o://f1f4a2b19ce80eb8476007edca4a434c346427c70345f07a31792be4a72774e3" gracePeriod=30
May 26 16:03:34.975502 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:34.975466    2552 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59589468bc-7p484" podUID="33f9ccb8-9dde-4732-8c4e-ec44cf4a9b38" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.7:8000/healthz\": dial tcp 10.132.0.7:8000: connect: connection refused"
May 26 16:03:34.981126 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:34.981098    2552 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8559875f4d-f4x8v" podUID="6f7d7e6c-a91a-46c6-ac3c-317de74cfcb8" containerName="addon-agent" probeResult="failure" output="Get \"http://10.132.0.9:8000/healthz\": dial tcp 10.132.0.9:8000: connect: connection refused"
May 26 16:03:35.104523 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:35.104487    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59589468bc-7p484" podUID="33f9ccb8-9dde-4732-8c4e-ec44cf4a9b38" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.7:8000/readyz\": dial tcp 10.132.0.7:8000: connect: connection refused"
May 26 16:03:35.418347 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:35.418307    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6nz6h" event={"ID":"076454e2-eee8-47cb-a7e6-dc5e0a0270ae","Type":"ContainerStarted","Data":"f2d578c6aacf83affbb7d08357a7676e4528e95eb195fb3ed03dd06a6e710ee3"}
May 26 16:03:35.418794 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:35.418442    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6nz6h"
May 26 16:03:35.419540 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:35.419515    2552 generic.go:358] "Generic (PLEG): container finished" podID="33f9ccb8-9dde-4732-8c4e-ec44cf4a9b38" containerID="6c2f257b4519e4de4694d75f28d75aebdd83592a00f2176ee1eab0dc894d895a" exitCode=1
May 26 16:03:35.419659 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:35.419589    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59589468bc-7p484" event={"ID":"33f9ccb8-9dde-4732-8c4e-ec44cf4a9b38","Type":"ContainerDied","Data":"6c2f257b4519e4de4694d75f28d75aebdd83592a00f2176ee1eab0dc894d895a"}
May 26 16:03:35.419888 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:35.419871    2552 scope.go:117] "RemoveContainer" containerID="6c2f257b4519e4de4694d75f28d75aebdd83592a00f2176ee1eab0dc894d895a"
May 26 16:03:35.420825 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:35.420806    2552 generic.go:358] "Generic (PLEG): container finished" podID="6f7d7e6c-a91a-46c6-ac3c-317de74cfcb8" containerID="266c07f551abc4a86ca007961a45c0ff392297a2a2b0beaaacc75c326271f33c" exitCode=255
May 26 16:03:35.420922 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:35.420871    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8559875f4d-f4x8v" event={"ID":"6f7d7e6c-a91a-46c6-ac3c-317de74cfcb8","Type":"ContainerDied","Data":"266c07f551abc4a86ca007961a45c0ff392297a2a2b0beaaacc75c326271f33c"}
May 26 16:03:35.421210 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:35.421172    2552 scope.go:117] "RemoveContainer" containerID="266c07f551abc4a86ca007961a45c0ff392297a2a2b0beaaacc75c326271f33c"
May 26 16:03:35.423235 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:35.423216    2552 generic.go:358] "Generic (PLEG): container finished" podID="ce89a932-1d2d-42a3-a647-782517d92dcd" containerID="f1f4a2b19ce80eb8476007edca4a434c346427c70345f07a31792be4a72774e3" exitCode=2
May 26 16:03:35.423320 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:35.423246    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f5759674c-t6282" event={"ID":"ce89a932-1d2d-42a3-a647-782517d92dcd","Type":"ContainerDied","Data":"f1f4a2b19ce80eb8476007edca4a434c346427c70345f07a31792be4a72774e3"}
May 26 16:03:35.423320 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:35.423267    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f5759674c-t6282" event={"ID":"ce89a932-1d2d-42a3-a647-782517d92dcd","Type":"ContainerStarted","Data":"86801ba2bcf445af086c6950103bc8ce314bb500fe3e66b1b239af2fcf0d0263"}
May 26 16:03:35.437066 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:35.437011    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-6nz6h" podStartSLOduration=130.244962609 podStartE2EDuration="2m11.436992343s" podCreationTimestamp="2026-05-26 16:01:24 +0000 UTC" firstStartedPulling="2026-05-26 16:03:33.065774487 +0000 UTC m=+162.774229508" lastFinishedPulling="2026-05-26 16:03:34.257804221 +0000 UTC m=+163.966259242" observedRunningTime="2026-05-26 16:03:35.436334331 +0000 UTC m=+165.144789408" watchObservedRunningTime="2026-05-26 16:03:35.436992343 +0000 UTC m=+165.145447384"
May 26 16:03:36.430756 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:36.430717    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59589468bc-7p484" event={"ID":"33f9ccb8-9dde-4732-8c4e-ec44cf4a9b38","Type":"ContainerStarted","Data":"ef114ffc616474d61b8586a9e786e8938572674d0c3646ad05cd372ce1f65d03"}
May 26 16:03:36.431196 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:36.431043    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59589468bc-7p484"
May 26 16:03:36.432153 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:36.432132    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59589468bc-7p484"
May 26 16:03:36.432283 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:36.432262    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8559875f4d-f4x8v" event={"ID":"6f7d7e6c-a91a-46c6-ac3c-317de74cfcb8","Type":"ContainerStarted","Data":"87548c2ccbd9e804455617a102f61401e8e7ca02fd20beaffb405ae42ec1e20f"}
May 26 16:03:38.885787 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:38.885691    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-thjfg"
May 26 16:03:38.888943 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:38.888908    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-l7p5g\""
May 26 16:03:38.896307 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:38.896289    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-thjfg"
May 26 16:03:39.021652 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:39.021620    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-thjfg"]
May 26 16:03:39.024591 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:03:39.024561    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfdaec24_5a44_429c_bf06_3422d868294d.slice/crio-b30527096d35494cc047f5e9e207ab3b009c787ce06f12067b5c84f9329a834b WatchSource:0}: Error finding container b30527096d35494cc047f5e9e207ab3b009c787ce06f12067b5c84f9329a834b: Status 404 returned error can't find the container with id b30527096d35494cc047f5e9e207ab3b009c787ce06f12067b5c84f9329a834b
May 26 16:03:39.441553 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:39.441518    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-thjfg" event={"ID":"dfdaec24-5a44-429c-bf06-3422d868294d","Type":"ContainerStarted","Data":"b30527096d35494cc047f5e9e207ab3b009c787ce06f12067b5c84f9329a834b"}
May 26 16:03:41.449498 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:41.449459    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-thjfg" event={"ID":"dfdaec24-5a44-429c-bf06-3422d868294d","Type":"ContainerStarted","Data":"729cc370ccbbcaf307491f867945ed0b16bf4a1848b4e906b56ee51e369b522f"}
May 26 16:03:41.465516 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:41.465475    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-thjfg" podStartSLOduration=135.87978593 podStartE2EDuration="2m17.465460428s" podCreationTimestamp="2026-05-26 16:01:24 +0000 UTC" firstStartedPulling="2026-05-26 16:03:39.026483557 +0000 UTC m=+168.734938579" lastFinishedPulling="2026-05-26 16:03:40.612158059 +0000 UTC m=+170.320613077" observedRunningTime="2026-05-26 16:03:41.464483106 +0000 UTC m=+171.172938146" watchObservedRunningTime="2026-05-26 16:03:41.465460428 +0000 UTC m=+171.173915514"
May 26 16:03:45.434901 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:45.434862    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6nz6h"
May 26 16:03:52.912675 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:52.912639    2552 patch_prober.go:28] interesting pod/image-registry-797d988dd6-hvdzb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 26 16:03:52.913090 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:52.912688    2552 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-797d988dd6-hvdzb" podUID="5995ba40-1aa4-444a-934b-abfe3e7d76ee" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:03:54.419244 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:54.419211    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:03:59.724857 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:03:59.724825    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-797d988dd6-hvdzb"]
May 26 16:04:18.412433 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:18.412404    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-797d988dd6-hvdzb_5995ba40-1aa4-444a-934b-abfe3e7d76ee/registry/0.log"
May 26 16:04:18.418142 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:18.418118    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5xp7f_092e4858-1372-4cb9-b51b-dde9e50e0f9d/node-ca/0.log"
May 26 16:04:24.743612 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:24.743571    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-797d988dd6-hvdzb" podUID="5995ba40-1aa4-444a-934b-abfe3e7d76ee" containerName="registry" containerID="cri-o://ec45ce86faa53dc4183b8407b882760ef9ebf4ea1784f313efeb369676899be3" gracePeriod=30
May 26 16:04:25.969940 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:25.969903    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:04:26.117170 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:26.117077    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5995ba40-1aa4-444a-934b-abfe3e7d76ee-ca-trust-extracted\") pod \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") "
May 26 16:04:26.117170 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:26.117128    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzgvd\" (UniqueName: \"kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-kube-api-access-zzgvd\") pod \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") "
May 26 16:04:26.117170 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:26.117164    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5995ba40-1aa4-444a-934b-abfe3e7d76ee-registry-certificates\") pod \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") "
May 26 16:04:26.117443 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:26.117188    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-registry-tls\") pod \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") "
May 26 16:04:26.117443 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:26.117216    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5995ba40-1aa4-444a-934b-abfe3e7d76ee-trusted-ca\") pod \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") "
May 26 16:04:26.117443 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:26.117238    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5995ba40-1aa4-444a-934b-abfe3e7d76ee-image-registry-private-configuration\") pod \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") "
May 26 16:04:26.117443 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:26.117269    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-bound-sa-token\") pod \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") "
May 26 16:04:26.117443 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:26.117298    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5995ba40-1aa4-444a-934b-abfe3e7d76ee-installation-pull-secrets\") pod \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\" (UID: \"5995ba40-1aa4-444a-934b-abfe3e7d76ee\") "
May 26 16:04:26.117686 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:26.117600    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5995ba40-1aa4-444a-934b-abfe3e7d76ee-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "5995ba40-1aa4-444a-934b-abfe3e7d76ee" (UID: "5995ba40-1aa4-444a-934b-abfe3e7d76ee"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:04:26.117968 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:26.117767    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5995ba40-1aa4-444a-934b-abfe3e7d76ee-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "5995ba40-1aa4-444a-934b-abfe3e7d76ee" (UID: "5995ba40-1aa4-444a-934b-abfe3e7d76ee"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:04:26.119956 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:26.119896    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5995ba40-1aa4-444a-934b-abfe3e7d76ee-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "5995ba40-1aa4-444a-934b-abfe3e7d76ee" (UID: "5995ba40-1aa4-444a-934b-abfe3e7d76ee"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:04:26.119956 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:26.119947    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-kube-api-access-zzgvd" (OuterVolumeSpecName: "kube-api-access-zzgvd") pod "5995ba40-1aa4-444a-934b-abfe3e7d76ee" (UID: "5995ba40-1aa4-444a-934b-abfe3e7d76ee"). InnerVolumeSpecName "kube-api-access-zzgvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:04:26.120095 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:26.119974    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5995ba40-1aa4-444a-934b-abfe3e7d76ee" (UID: "5995ba40-1aa4-444a-934b-abfe3e7d76ee"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:04:26.120095 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:26.119971    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5995ba40-1aa4-444a-934b-abfe3e7d76ee-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "5995ba40-1aa4-444a-934b-abfe3e7d76ee" (UID: "5995ba40-1aa4-444a-934b-abfe3e7d76ee"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:04:26.120184 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:26.120161    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "5995ba40-1aa4-444a-934b-abfe3e7d76ee" (UID: "5995ba40-1aa4-444a-934b-abfe3e7d76ee"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:04:26.128604 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:26.128576    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5995ba40-1aa4-444a-934b-abfe3e7d76ee-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "5995ba40-1aa4-444a-934b-abfe3e7d76ee" (UID: "5995ba40-1aa4-444a-934b-abfe3e7d76ee"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:04:26.218256 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:26.218221    2552 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5995ba40-1aa4-444a-934b-abfe3e7d76ee-trusted-ca\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:04:26.218256 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:26.218252    2552 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5995ba40-1aa4-444a-934b-abfe3e7d76ee-image-registry-private-configuration\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:04:26.218256 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:26.218264    2552 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-bound-sa-token\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:04:26.218492 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:26.218276    2552 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5995ba40-1aa4-444a-934b-abfe3e7d76ee-installation-pull-secrets\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:04:26.218492 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:26.218288    2552 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5995ba40-1aa4-444a-934b-abfe3e7d76ee-ca-trust-extracted\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:04:26.218492 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:26.218296    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzgvd\" (UniqueName: \"kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-kube-api-access-zzgvd\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:04:26.218492 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:26.218304    2552 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5995ba40-1aa4-444a-934b-abfe3e7d76ee-registry-certificates\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:04:26.218492 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:26.218314    2552 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5995ba40-1aa4-444a-934b-abfe3e7d76ee-registry-tls\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:04:26.564052 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:26.564019    2552 generic.go:358] "Generic (PLEG): container finished" podID="5995ba40-1aa4-444a-934b-abfe3e7d76ee" containerID="ec45ce86faa53dc4183b8407b882760ef9ebf4ea1784f313efeb369676899be3" exitCode=0
May 26 16:04:26.564225 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:26.564078    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-797d988dd6-hvdzb"
May 26 16:04:26.564225 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:26.564086    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-797d988dd6-hvdzb" event={"ID":"5995ba40-1aa4-444a-934b-abfe3e7d76ee","Type":"ContainerDied","Data":"ec45ce86faa53dc4183b8407b882760ef9ebf4ea1784f313efeb369676899be3"}
May 26 16:04:26.564225 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:26.564112    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-797d988dd6-hvdzb" event={"ID":"5995ba40-1aa4-444a-934b-abfe3e7d76ee","Type":"ContainerDied","Data":"d92a3e26e16607960d26bd9f5164797ed0f2681ea64bb343aeb6eb0e8bf504ff"}
May 26 16:04:26.564225 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:26.564126    2552 scope.go:117] "RemoveContainer" containerID="ec45ce86faa53dc4183b8407b882760ef9ebf4ea1784f313efeb369676899be3"
May 26 16:04:26.574752 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:26.574727    2552 scope.go:117] "RemoveContainer" containerID="ec45ce86faa53dc4183b8407b882760ef9ebf4ea1784f313efeb369676899be3"
May 26 16:04:26.575424 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:04:26.575093    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec45ce86faa53dc4183b8407b882760ef9ebf4ea1784f313efeb369676899be3\": container with ID starting with ec45ce86faa53dc4183b8407b882760ef9ebf4ea1784f313efeb369676899be3 not found: ID does not exist" containerID="ec45ce86faa53dc4183b8407b882760ef9ebf4ea1784f313efeb369676899be3"
May 26 16:04:26.575509 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:26.575434    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec45ce86faa53dc4183b8407b882760ef9ebf4ea1784f313efeb369676899be3"} err="failed to get container status \"ec45ce86faa53dc4183b8407b882760ef9ebf4ea1784f313efeb369676899be3\": rpc error: code = NotFound desc = could not find container \"ec45ce86faa53dc4183b8407b882760ef9ebf4ea1784f313efeb369676899be3\": container with ID starting with ec45ce86faa53dc4183b8407b882760ef9ebf4ea1784f313efeb369676899be3 not found: ID does not exist"
May 26 16:04:26.592940 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:26.592902    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-797d988dd6-hvdzb"]
May 26 16:04:26.594725 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:26.594705    2552 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-797d988dd6-hvdzb"]
May 26 16:04:26.889671 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:04:26.889596    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5995ba40-1aa4-444a-934b-abfe3e7d76ee" path="/var/lib/kubelet/pods/5995ba40-1aa4-444a-934b-abfe3e7d76ee/volumes"
May 26 16:05:50.771520 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:05:50.771493    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sj6db_5c0f9fc6-2bd6-4370-96d0-5ae315363c7f/ovn-acl-logging/0.log"
May 26 16:05:50.772978 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:05:50.772955    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sj6db_5c0f9fc6-2bd6-4370-96d0-5ae315363c7f/ovn-acl-logging/0.log"
May 26 16:05:50.774333 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:05:50.774310    2552 kubelet.go:1628] "Image garbage collection succeeded"
May 26 16:07:45.122848 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:07:45.122815    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-fzsbl"]
May 26 16:07:45.123257 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:07:45.123042    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5995ba40-1aa4-444a-934b-abfe3e7d76ee" containerName="registry"
May 26 16:07:45.123257 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:07:45.123053    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="5995ba40-1aa4-444a-934b-abfe3e7d76ee" containerName="registry"
May 26 16:07:45.123257 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:07:45.123098    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="5995ba40-1aa4-444a-934b-abfe3e7d76ee" containerName="registry"
May 26 16:07:45.124746 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:07:45.124730    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-fzsbl"
May 26 16:07:45.127288 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:07:45.127260    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
May 26 16:07:45.127401 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:07:45.127310    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
May 26 16:07:45.127401 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:07:45.127313    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
May 26 16:07:45.127639 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:07:45.127626    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
May 26 16:07:45.128273 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:07:45.128241    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-hcztz\""
May 26 16:07:45.128348 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:07:45.128332    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
May 26 16:07:45.134576 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:07:45.134556    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-fzsbl"]
May 26 16:07:45.238116 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:07:45.238082    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c82643b8-9575-4765-b614-71018ec384b4-certificates\") pod \"keda-metrics-apiserver-7c9f485588-fzsbl\" (UID: \"c82643b8-9575-4765-b614-71018ec384b4\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-fzsbl"
May 26 16:07:45.238270 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:07:45.238167    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c82643b8-9575-4765-b614-71018ec384b4-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-fzsbl\" (UID: \"c82643b8-9575-4765-b614-71018ec384b4\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-fzsbl"
May 26 16:07:45.238270 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:07:45.238198    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prj4d\" (UniqueName: \"kubernetes.io/projected/c82643b8-9575-4765-b614-71018ec384b4-kube-api-access-prj4d\") pod \"keda-metrics-apiserver-7c9f485588-fzsbl\" (UID: \"c82643b8-9575-4765-b614-71018ec384b4\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-fzsbl"
May 26 16:07:45.338659 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:07:45.338631    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c82643b8-9575-4765-b614-71018ec384b4-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-fzsbl\" (UID: \"c82643b8-9575-4765-b614-71018ec384b4\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-fzsbl"
May 26 16:07:45.338659 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:07:45.338662    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-prj4d\" (UniqueName: \"kubernetes.io/projected/c82643b8-9575-4765-b614-71018ec384b4-kube-api-access-prj4d\") pod \"keda-metrics-apiserver-7c9f485588-fzsbl\" (UID: \"c82643b8-9575-4765-b614-71018ec384b4\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-fzsbl"
May 26 16:07:45.338837 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:07:45.338706    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c82643b8-9575-4765-b614-71018ec384b4-certificates\") pod \"keda-metrics-apiserver-7c9f485588-fzsbl\" (UID: \"c82643b8-9575-4765-b614-71018ec384b4\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-fzsbl"
May 26 16:07:45.338837 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:07:45.338793    2552 secret.go:281] references non-existent secret key: tls.crt
May 26 16:07:45.338837 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:07:45.338804    2552 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 26 16:07:45.338837 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:07:45.338817    2552 projected.go:264] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
May 26 16:07:45.338837 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:07:45.338833    2552 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-fzsbl: [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
May 26 16:07:45.339028 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:07:45.338894    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c82643b8-9575-4765-b614-71018ec384b4-certificates podName:c82643b8-9575-4765-b614-71018ec384b4 nodeName:}" failed. No retries permitted until 2026-05-26 16:07:45.838876727 +0000 UTC m=+415.547331764 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/c82643b8-9575-4765-b614-71018ec384b4-certificates") pod "keda-metrics-apiserver-7c9f485588-fzsbl" (UID: "c82643b8-9575-4765-b614-71018ec384b4") : [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
May 26 16:07:45.339028 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:07:45.338984    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c82643b8-9575-4765-b614-71018ec384b4-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-fzsbl\" (UID: \"c82643b8-9575-4765-b614-71018ec384b4\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-fzsbl"
May 26 16:07:45.350750 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:07:45.350719    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prj4d\" (UniqueName: \"kubernetes.io/projected/c82643b8-9575-4765-b614-71018ec384b4-kube-api-access-prj4d\") pod \"keda-metrics-apiserver-7c9f485588-fzsbl\" (UID: \"c82643b8-9575-4765-b614-71018ec384b4\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-fzsbl"
May 26 16:07:45.842909 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:07:45.842870    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c82643b8-9575-4765-b614-71018ec384b4-certificates\") pod \"keda-metrics-apiserver-7c9f485588-fzsbl\" (UID: \"c82643b8-9575-4765-b614-71018ec384b4\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-fzsbl"
May 26 16:07:45.843098 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:07:45.843021    2552 secret.go:281] references non-existent secret key: tls.crt
May 26 16:07:45.843098 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:07:45.843036    2552 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 26 16:07:45.843098 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:07:45.843052    2552 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-fzsbl: references non-existent secret key: tls.crt
May 26 16:07:45.843210 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:07:45.843114    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c82643b8-9575-4765-b614-71018ec384b4-certificates podName:c82643b8-9575-4765-b614-71018ec384b4 nodeName:}" failed. No retries permitted until 2026-05-26 16:07:46.843096647 +0000 UTC m=+416.551551673 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/c82643b8-9575-4765-b614-71018ec384b4-certificates") pod "keda-metrics-apiserver-7c9f485588-fzsbl" (UID: "c82643b8-9575-4765-b614-71018ec384b4") : references non-existent secret key: tls.crt
May 26 16:07:46.851431 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:07:46.851395    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c82643b8-9575-4765-b614-71018ec384b4-certificates\") pod \"keda-metrics-apiserver-7c9f485588-fzsbl\" (UID: \"c82643b8-9575-4765-b614-71018ec384b4\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-fzsbl"
May 26 16:07:46.851831 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:07:46.851513    2552 secret.go:281] references non-existent secret key: tls.crt
May 26 16:07:46.851831 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:07:46.851526    2552 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 26 16:07:46.851831 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:07:46.851543    2552 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-fzsbl: references non-existent secret key: tls.crt
May 26 16:07:46.851831 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:07:46.851602    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c82643b8-9575-4765-b614-71018ec384b4-certificates podName:c82643b8-9575-4765-b614-71018ec384b4 nodeName:}" failed. No retries permitted until 2026-05-26 16:07:48.851585813 +0000 UTC m=+418.560040836 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/c82643b8-9575-4765-b614-71018ec384b4-certificates") pod "keda-metrics-apiserver-7c9f485588-fzsbl" (UID: "c82643b8-9575-4765-b614-71018ec384b4") : references non-existent secret key: tls.crt
May 26 16:07:48.867291 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:07:48.867247    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c82643b8-9575-4765-b614-71018ec384b4-certificates\") pod \"keda-metrics-apiserver-7c9f485588-fzsbl\" (UID: \"c82643b8-9575-4765-b614-71018ec384b4\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-fzsbl"
May 26 16:07:48.867768 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:07:48.867380    2552 secret.go:281] references non-existent secret key: tls.crt
May 26 16:07:48.867768 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:07:48.867400    2552 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 26 16:07:48.867768 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:07:48.867421    2552 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-fzsbl: references non-existent secret key: tls.crt
May 26 16:07:48.867768 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:07:48.867491    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c82643b8-9575-4765-b614-71018ec384b4-certificates podName:c82643b8-9575-4765-b614-71018ec384b4 nodeName:}" failed. No retries permitted until 2026-05-26 16:07:52.867471891 +0000 UTC m=+422.575926923 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/c82643b8-9575-4765-b614-71018ec384b4-certificates") pod "keda-metrics-apiserver-7c9f485588-fzsbl" (UID: "c82643b8-9575-4765-b614-71018ec384b4") : references non-existent secret key: tls.crt
May 26 16:07:52.895254 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:07:52.895219    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c82643b8-9575-4765-b614-71018ec384b4-certificates\") pod \"keda-metrics-apiserver-7c9f485588-fzsbl\" (UID: \"c82643b8-9575-4765-b614-71018ec384b4\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-fzsbl"
May 26 16:07:52.897739 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:07:52.897714    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c82643b8-9575-4765-b614-71018ec384b4-certificates\") pod \"keda-metrics-apiserver-7c9f485588-fzsbl\" (UID: \"c82643b8-9575-4765-b614-71018ec384b4\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-fzsbl"
May 26 16:07:52.937569 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:07:52.937538    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-hcztz\""
May 26 16:07:52.945554 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:07:52.945534    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-fzsbl"
May 26 16:07:53.064235 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:07:53.064206    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-fzsbl"]
May 26 16:07:53.067163 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:07:53.067132    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc82643b8_9575_4765_b614_71018ec384b4.slice/crio-f73ef24c7ed1647bc5c3023b43b9952d05c56359df5d21492569c6be9fb209ed WatchSource:0}: Error finding container f73ef24c7ed1647bc5c3023b43b9952d05c56359df5d21492569c6be9fb209ed: Status 404 returned error can't find the container with id f73ef24c7ed1647bc5c3023b43b9952d05c56359df5d21492569c6be9fb209ed
May 26 16:07:53.068282 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:07:53.068263    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:07:53.085481 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:07:53.085457    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-fzsbl" event={"ID":"c82643b8-9575-4765-b614-71018ec384b4","Type":"ContainerStarted","Data":"f73ef24c7ed1647bc5c3023b43b9952d05c56359df5d21492569c6be9fb209ed"}
May 26 16:07:57.099562 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:07:57.099527    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-fzsbl" event={"ID":"c82643b8-9575-4765-b614-71018ec384b4","Type":"ContainerStarted","Data":"a8c23956a41758643622d993c6e853fa76c6c3982461a220ad05e879ac554c60"}
May 26 16:07:57.099967 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:07:57.099648    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-fzsbl"
May 26 16:07:57.116471 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:07:57.116427    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-fzsbl" podStartSLOduration=9.071466736 podStartE2EDuration="12.116411985s" podCreationTimestamp="2026-05-26 16:07:45 +0000 UTC" firstStartedPulling="2026-05-26 16:07:53.068387928 +0000 UTC m=+422.776842945" lastFinishedPulling="2026-05-26 16:07:56.113333172 +0000 UTC m=+425.821788194" observedRunningTime="2026-05-26 16:07:57.115675732 +0000 UTC m=+426.824130753" watchObservedRunningTime="2026-05-26 16:07:57.116411985 +0000 UTC m=+426.824867026"
May 26 16:08:08.108600 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:08:08.108520    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-fzsbl"
May 26 16:08:49.689943 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:08:49.689889    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-g4tjd"]
May 26 16:08:49.691803 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:08:49.691785    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-g4tjd"
May 26 16:08:49.694288 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:08:49.694268    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
May 26 16:08:49.694682 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:08:49.694668    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
May 26 16:08:49.695063 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:08:49.695040    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-hmdx4\""
May 26 16:08:49.705853 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:08:49.705829    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-g4tjd"]
May 26 16:08:49.792564 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:08:49.792535    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ntpt\" (UniqueName: \"kubernetes.io/projected/abdabed7-efae-4101-8c17-848e09a131bf-kube-api-access-4ntpt\") pod \"cert-manager-operator-controller-manager-54b9655956-g4tjd\" (UID: \"abdabed7-efae-4101-8c17-848e09a131bf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-g4tjd"
May 26 16:08:49.792714 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:08:49.792631    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/abdabed7-efae-4101-8c17-848e09a131bf-tmp\") pod \"cert-manager-operator-controller-manager-54b9655956-g4tjd\" (UID: \"abdabed7-efae-4101-8c17-848e09a131bf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-g4tjd"
May 26 16:08:49.893288 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:08:49.893249    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/abdabed7-efae-4101-8c17-848e09a131bf-tmp\") pod \"cert-manager-operator-controller-manager-54b9655956-g4tjd\" (UID: \"abdabed7-efae-4101-8c17-848e09a131bf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-g4tjd"
May 26 16:08:49.893456 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:08:49.893295    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4ntpt\" (UniqueName: \"kubernetes.io/projected/abdabed7-efae-4101-8c17-848e09a131bf-kube-api-access-4ntpt\") pod \"cert-manager-operator-controller-manager-54b9655956-g4tjd\" (UID: \"abdabed7-efae-4101-8c17-848e09a131bf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-g4tjd"
May 26 16:08:49.893679 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:08:49.893656    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/abdabed7-efae-4101-8c17-848e09a131bf-tmp\") pod \"cert-manager-operator-controller-manager-54b9655956-g4tjd\" (UID: \"abdabed7-efae-4101-8c17-848e09a131bf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-g4tjd"
May 26 16:08:49.914783 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:08:49.914761    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ntpt\" (UniqueName: \"kubernetes.io/projected/abdabed7-efae-4101-8c17-848e09a131bf-kube-api-access-4ntpt\") pod \"cert-manager-operator-controller-manager-54b9655956-g4tjd\" (UID: \"abdabed7-efae-4101-8c17-848e09a131bf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-g4tjd"
May 26 16:08:50.000876 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:08:50.000799    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-g4tjd"
May 26 16:08:50.129861 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:08:50.129749    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-g4tjd"]
May 26 16:08:50.132727 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:08:50.132696    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabdabed7_efae_4101_8c17_848e09a131bf.slice/crio-2ce2329f768c8901eb0fa9ad0cebe4a600e6f5e0981b13092f07b7eee73535c3 WatchSource:0}: Error finding container 2ce2329f768c8901eb0fa9ad0cebe4a600e6f5e0981b13092f07b7eee73535c3: Status 404 returned error can't find the container with id 2ce2329f768c8901eb0fa9ad0cebe4a600e6f5e0981b13092f07b7eee73535c3
May 26 16:08:50.236988 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:08:50.236956    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-g4tjd" event={"ID":"abdabed7-efae-4101-8c17-848e09a131bf","Type":"ContainerStarted","Data":"2ce2329f768c8901eb0fa9ad0cebe4a600e6f5e0981b13092f07b7eee73535c3"}
May 26 16:08:52.244259 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:08:52.244214    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-g4tjd" event={"ID":"abdabed7-efae-4101-8c17-848e09a131bf","Type":"ContainerStarted","Data":"0f81076dffd7d61c0ebc44743e535a40eafe211817b0f1f5617c5bbc2a3ffd20"}
May 26 16:08:52.272476 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:08:52.272423    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-g4tjd" podStartSLOduration=1.742798225 podStartE2EDuration="3.272406246s" podCreationTimestamp="2026-05-26 16:08:49 +0000 UTC" firstStartedPulling="2026-05-26 16:08:50.135145703 +0000 UTC m=+479.843600723" lastFinishedPulling="2026-05-26 16:08:51.664753712 +0000 UTC m=+481.373208744" observedRunningTime="2026-05-26 16:08:52.270334092 +0000 UTC m=+481.978789133" watchObservedRunningTime="2026-05-26 16:08:52.272406246 +0000 UTC m=+481.980861288"
May 26 16:09:00.208865 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:00.208823    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-587ccfb98-98mtk"]
May 26 16:09:00.211856 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:00.211838    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-587ccfb98-98mtk"
May 26 16:09:00.214700 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:00.214673    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
May 26 16:09:00.215671 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:00.215651    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-t8k9z\""
May 26 16:09:00.215757 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:00.215696    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
May 26 16:09:00.222655 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:00.222630    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-587ccfb98-98mtk"]
May 26 16:09:00.263978 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:00.263919    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9e94e07e-a768-4563-814e-ffc226ab9037-bound-sa-token\") pod \"cert-manager-webhook-587ccfb98-98mtk\" (UID: \"9e94e07e-a768-4563-814e-ffc226ab9037\") " pod="cert-manager/cert-manager-webhook-587ccfb98-98mtk"
May 26 16:09:00.264139 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:00.263997    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgkh8\" (UniqueName: \"kubernetes.io/projected/9e94e07e-a768-4563-814e-ffc226ab9037-kube-api-access-hgkh8\") pod \"cert-manager-webhook-587ccfb98-98mtk\" (UID: \"9e94e07e-a768-4563-814e-ffc226ab9037\") " pod="cert-manager/cert-manager-webhook-587ccfb98-98mtk"
May 26 16:09:00.364482 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:00.364449    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9e94e07e-a768-4563-814e-ffc226ab9037-bound-sa-token\") pod \"cert-manager-webhook-587ccfb98-98mtk\" (UID: \"9e94e07e-a768-4563-814e-ffc226ab9037\") " pod="cert-manager/cert-manager-webhook-587ccfb98-98mtk"
May 26 16:09:00.364650 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:00.364503    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hgkh8\" (UniqueName: \"kubernetes.io/projected/9e94e07e-a768-4563-814e-ffc226ab9037-kube-api-access-hgkh8\") pod \"cert-manager-webhook-587ccfb98-98mtk\" (UID: \"9e94e07e-a768-4563-814e-ffc226ab9037\") " pod="cert-manager/cert-manager-webhook-587ccfb98-98mtk"
May 26 16:09:00.372554 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:00.372527    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9e94e07e-a768-4563-814e-ffc226ab9037-bound-sa-token\") pod \"cert-manager-webhook-587ccfb98-98mtk\" (UID: \"9e94e07e-a768-4563-814e-ffc226ab9037\") " pod="cert-manager/cert-manager-webhook-587ccfb98-98mtk"
May 26 16:09:00.372676 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:00.372602    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgkh8\" (UniqueName: \"kubernetes.io/projected/9e94e07e-a768-4563-814e-ffc226ab9037-kube-api-access-hgkh8\") pod \"cert-manager-webhook-587ccfb98-98mtk\" (UID: \"9e94e07e-a768-4563-814e-ffc226ab9037\") " pod="cert-manager/cert-manager-webhook-587ccfb98-98mtk"
May 26 16:09:00.521131 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:00.521105    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-587ccfb98-98mtk"
May 26 16:09:00.645730 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:00.645698    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-587ccfb98-98mtk"]
May 26 16:09:00.648702 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:09:00.648670    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e94e07e_a768_4563_814e_ffc226ab9037.slice/crio-2bd84cd8c5b85c135d2f00a2376d36c704175c3aa16015d249d82aec12436dd1 WatchSource:0}: Error finding container 2bd84cd8c5b85c135d2f00a2376d36c704175c3aa16015d249d82aec12436dd1: Status 404 returned error can't find the container with id 2bd84cd8c5b85c135d2f00a2376d36c704175c3aa16015d249d82aec12436dd1
May 26 16:09:01.272390 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:01.272354    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-587ccfb98-98mtk" event={"ID":"9e94e07e-a768-4563-814e-ffc226ab9037","Type":"ContainerStarted","Data":"2bd84cd8c5b85c135d2f00a2376d36c704175c3aa16015d249d82aec12436dd1"}
May 26 16:09:03.277985 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:03.277953    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-587ccfb98-98mtk" event={"ID":"9e94e07e-a768-4563-814e-ffc226ab9037","Type":"ContainerStarted","Data":"c1ff79d95ff14e6a3fde2080d5f7a8c1b8dd92f2862e0755b1cc9c525c075107"}
May 26 16:09:03.278338 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:03.278007    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-587ccfb98-98mtk"
May 26 16:09:03.295386 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:03.295319    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-587ccfb98-98mtk" podStartSLOduration=1.142304137 podStartE2EDuration="3.295306008s" podCreationTimestamp="2026-05-26 16:09:00 +0000 UTC" firstStartedPulling="2026-05-26 16:09:00.650467739 +0000 UTC m=+490.358922757" lastFinishedPulling="2026-05-26 16:09:02.803469609 +0000 UTC m=+492.511924628" observedRunningTime="2026-05-26 16:09:03.293196301 +0000 UTC m=+493.001651341" watchObservedRunningTime="2026-05-26 16:09:03.295306008 +0000 UTC m=+493.003761047"
May 26 16:09:09.282470 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:09.282440    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-587ccfb98-98mtk"
May 26 16:09:50.099598 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:50.099523    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm"]
May 26 16:09:50.101482 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:50.101467    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm"
May 26 16:09:50.107630 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:50.107561    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
May 26 16:09:50.107812 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:50.107787    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
May 26 16:09:50.108407 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:50.108386    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
May 26 16:09:50.109577 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:50.109561    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-djwc4\""
May 26 16:09:50.109662 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:50.109620    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
May 26 16:09:50.109721 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:50.109671    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
May 26 16:09:50.109721 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:50.109688    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
May 26 16:09:50.117047 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:50.117021    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm"]
May 26 16:09:50.206239 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:50.206213    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/10c3c960-7994-4f89-b867-4d41ba86d9b2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-767cf5cf55-46kkm\" (UID: \"10c3c960-7994-4f89-b867-4d41ba86d9b2\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm"
May 26 16:09:50.206401 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:50.206247    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/10c3c960-7994-4f89-b867-4d41ba86d9b2-cacerts\") pod \"istiod-openshift-gateway-767cf5cf55-46kkm\" (UID: \"10c3c960-7994-4f89-b867-4d41ba86d9b2\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm"
May 26 16:09:50.206401 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:50.206271    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10c3c960-7994-4f89-b867-4d41ba86d9b2-istio-token\") pod \"istiod-openshift-gateway-767cf5cf55-46kkm\" (UID: \"10c3c960-7994-4f89-b867-4d41ba86d9b2\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm"
May 26 16:09:50.206401 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:50.206297    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/10c3c960-7994-4f89-b867-4d41ba86d9b2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-767cf5cf55-46kkm\" (UID: \"10c3c960-7994-4f89-b867-4d41ba86d9b2\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm"
May 26 16:09:50.206401 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:50.206331    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmrqz\" (UniqueName: \"kubernetes.io/projected/10c3c960-7994-4f89-b867-4d41ba86d9b2-kube-api-access-pmrqz\") pod \"istiod-openshift-gateway-767cf5cf55-46kkm\" (UID: \"10c3c960-7994-4f89-b867-4d41ba86d9b2\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm"
May 26 16:09:50.206401 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:50.206347    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/10c3c960-7994-4f89-b867-4d41ba86d9b2-istio-kubeconfig\") pod \"istiod-openshift-gateway-767cf5cf55-46kkm\" (UID: \"10c3c960-7994-4f89-b867-4d41ba86d9b2\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm"
May 26 16:09:50.206401 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:50.206391    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/10c3c960-7994-4f89-b867-4d41ba86d9b2-local-certs\") pod \"istiod-openshift-gateway-767cf5cf55-46kkm\" (UID: \"10c3c960-7994-4f89-b867-4d41ba86d9b2\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm"
May 26 16:09:50.307035 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:50.307008    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/10c3c960-7994-4f89-b867-4d41ba86d9b2-local-certs\") pod \"istiod-openshift-gateway-767cf5cf55-46kkm\" (UID: \"10c3c960-7994-4f89-b867-4d41ba86d9b2\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm"
May 26 16:09:50.307195 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:50.307051    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/10c3c960-7994-4f89-b867-4d41ba86d9b2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-767cf5cf55-46kkm\" (UID: \"10c3c960-7994-4f89-b867-4d41ba86d9b2\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm"
May 26 16:09:50.307195 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:50.307071    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/10c3c960-7994-4f89-b867-4d41ba86d9b2-cacerts\") pod \"istiod-openshift-gateway-767cf5cf55-46kkm\" (UID: \"10c3c960-7994-4f89-b867-4d41ba86d9b2\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm"
May 26 16:09:50.307195 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:50.307093    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10c3c960-7994-4f89-b867-4d41ba86d9b2-istio-token\") pod \"istiod-openshift-gateway-767cf5cf55-46kkm\" (UID: \"10c3c960-7994-4f89-b867-4d41ba86d9b2\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm"
May 26 16:09:50.307352 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:50.307209    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/10c3c960-7994-4f89-b867-4d41ba86d9b2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-767cf5cf55-46kkm\" (UID: \"10c3c960-7994-4f89-b867-4d41ba86d9b2\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm"
May 26 16:09:50.307352 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:50.307268    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pmrqz\" (UniqueName: \"kubernetes.io/projected/10c3c960-7994-4f89-b867-4d41ba86d9b2-kube-api-access-pmrqz\") pod \"istiod-openshift-gateway-767cf5cf55-46kkm\" (UID: \"10c3c960-7994-4f89-b867-4d41ba86d9b2\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm"
May 26 16:09:50.307352 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:50.307302    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/10c3c960-7994-4f89-b867-4d41ba86d9b2-istio-kubeconfig\") pod \"istiod-openshift-gateway-767cf5cf55-46kkm\" (UID: \"10c3c960-7994-4f89-b867-4d41ba86d9b2\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm"
May 26 16:09:50.307789 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:50.307758    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/10c3c960-7994-4f89-b867-4d41ba86d9b2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-767cf5cf55-46kkm\" (UID: \"10c3c960-7994-4f89-b867-4d41ba86d9b2\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm"
May 26 16:09:50.309323 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:50.309286    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/10c3c960-7994-4f89-b867-4d41ba86d9b2-local-certs\") pod \"istiod-openshift-gateway-767cf5cf55-46kkm\" (UID: \"10c3c960-7994-4f89-b867-4d41ba86d9b2\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm"
May 26 16:09:50.309493 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:50.309475    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/10c3c960-7994-4f89-b867-4d41ba86d9b2-istio-kubeconfig\") pod \"istiod-openshift-gateway-767cf5cf55-46kkm\" (UID: \"10c3c960-7994-4f89-b867-4d41ba86d9b2\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm"
May 26 16:09:50.309555 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:50.309474    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/10c3c960-7994-4f89-b867-4d41ba86d9b2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-767cf5cf55-46kkm\" (UID: \"10c3c960-7994-4f89-b867-4d41ba86d9b2\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm"
May 26 16:09:50.309555 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:50.309538    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/10c3c960-7994-4f89-b867-4d41ba86d9b2-cacerts\") pod \"istiod-openshift-gateway-767cf5cf55-46kkm\" (UID: \"10c3c960-7994-4f89-b867-4d41ba86d9b2\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm"
May 26 16:09:50.316241 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:50.316197    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmrqz\" (UniqueName: \"kubernetes.io/projected/10c3c960-7994-4f89-b867-4d41ba86d9b2-kube-api-access-pmrqz\") pod \"istiod-openshift-gateway-767cf5cf55-46kkm\" (UID: \"10c3c960-7994-4f89-b867-4d41ba86d9b2\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm"
May 26 16:09:50.316534 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:50.316512    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10c3c960-7994-4f89-b867-4d41ba86d9b2-istio-token\") pod \"istiod-openshift-gateway-767cf5cf55-46kkm\" (UID: \"10c3c960-7994-4f89-b867-4d41ba86d9b2\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm"
May 26 16:09:50.410305 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:50.410227    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm"
May 26 16:09:50.536122 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:50.536068    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm"]
May 26 16:09:50.538638 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:09:50.538606    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c3c960_7994_4f89_b867_4d41ba86d9b2.slice/crio-6842788b9090c97e59dc1b82f67fbf4d37ecd159f81a22346f136237fd66416a WatchSource:0}: Error finding container 6842788b9090c97e59dc1b82f67fbf4d37ecd159f81a22346f136237fd66416a: Status 404 returned error can't find the container with id 6842788b9090c97e59dc1b82f67fbf4d37ecd159f81a22346f136237fd66416a
May 26 16:09:51.401223 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:51.401180    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm" event={"ID":"10c3c960-7994-4f89-b867-4d41ba86d9b2","Type":"ContainerStarted","Data":"6842788b9090c97e59dc1b82f67fbf4d37ecd159f81a22346f136237fd66416a"}
May 26 16:09:53.213412 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:53.213370    2552 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
May 26 16:09:53.213719 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:53.213441    2552 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
May 26 16:09:53.407925 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:53.407891    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm" event={"ID":"10c3c960-7994-4f89-b867-4d41ba86d9b2","Type":"ContainerStarted","Data":"d49c781fd379d1dc370239b1bf2349be377648f5d951db0d4413cb943a0786a8"}
May 26 16:09:53.408115 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:53.408006    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm"
May 26 16:09:53.428636 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:53.428588    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm" podStartSLOduration=0.755778806 podStartE2EDuration="3.428576564s" podCreationTimestamp="2026-05-26 16:09:50 +0000 UTC" firstStartedPulling="2026-05-26 16:09:50.540331407 +0000 UTC m=+540.248786424" lastFinishedPulling="2026-05-26 16:09:53.213129164 +0000 UTC m=+542.921584182" observedRunningTime="2026-05-26 16:09:53.427612237 +0000 UTC m=+543.136067288" watchObservedRunningTime="2026-05-26 16:09:53.428576564 +0000 UTC m=+543.137031605"
May 26 16:09:54.412792 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:54.412765    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm"
May 26 16:09:55.852575 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:55.852542    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt"]
May 26 16:09:55.854757 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:55.854741    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt"
May 26 16:09:55.857647 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:55.857623    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-84f9n\""
May 26 16:09:55.866793 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:55.866763    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt"]
May 26 16:09:55.953789 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:55.953756    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n5v8\" (UniqueName: \"kubernetes.io/projected/8310f0a3-b09d-4746-8561-11c489116cf8-kube-api-access-5n5v8\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-m95mt\" (UID: \"8310f0a3-b09d-4746-8561-11c489116cf8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt"
May 26 16:09:55.953789 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:55.953787    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8310f0a3-b09d-4746-8561-11c489116cf8-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-m95mt\" (UID: \"8310f0a3-b09d-4746-8561-11c489116cf8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt"
May 26 16:09:55.954063 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:55.953812    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8310f0a3-b09d-4746-8561-11c489116cf8-istio-data\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-m95mt\" (UID: \"8310f0a3-b09d-4746-8561-11c489116cf8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt"
May 26 16:09:55.954063 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:55.953861    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8310f0a3-b09d-4746-8561-11c489116cf8-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-m95mt\" (UID: \"8310f0a3-b09d-4746-8561-11c489116cf8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt"
May 26 16:09:55.954063 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:55.953921    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8310f0a3-b09d-4746-8561-11c489116cf8-credential-socket\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-m95mt\" (UID: \"8310f0a3-b09d-4746-8561-11c489116cf8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt"
May 26 16:09:55.954063 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:55.953968    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8310f0a3-b09d-4746-8561-11c489116cf8-istio-token\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-m95mt\" (UID: \"8310f0a3-b09d-4746-8561-11c489116cf8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt"
May 26 16:09:55.954063 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:55.954007    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8310f0a3-b09d-4746-8561-11c489116cf8-workload-socket\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-m95mt\" (UID: \"8310f0a3-b09d-4746-8561-11c489116cf8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt"
May 26 16:09:55.954063 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:55.954057    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8310f0a3-b09d-4746-8561-11c489116cf8-workload-certs\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-m95mt\" (UID: \"8310f0a3-b09d-4746-8561-11c489116cf8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt"
May 26 16:09:55.954290 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:55.954107    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8310f0a3-b09d-4746-8561-11c489116cf8-istio-envoy\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-m95mt\" (UID: \"8310f0a3-b09d-4746-8561-11c489116cf8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt"
May 26 16:09:56.055194 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:56.055158    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5n5v8\" (UniqueName: \"kubernetes.io/projected/8310f0a3-b09d-4746-8561-11c489116cf8-kube-api-access-5n5v8\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-m95mt\" (UID: \"8310f0a3-b09d-4746-8561-11c489116cf8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt"
May 26 16:09:56.055367 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:56.055203    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8310f0a3-b09d-4746-8561-11c489116cf8-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-m95mt\" (UID: \"8310f0a3-b09d-4746-8561-11c489116cf8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt"
May 26 16:09:56.055367 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:56.055229    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8310f0a3-b09d-4746-8561-11c489116cf8-istio-data\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-m95mt\" (UID: \"8310f0a3-b09d-4746-8561-11c489116cf8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt"
May 26 16:09:56.055367 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:56.055255    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8310f0a3-b09d-4746-8561-11c489116cf8-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-m95mt\" (UID: \"8310f0a3-b09d-4746-8561-11c489116cf8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt"
May 26 16:09:56.055367 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:56.055282    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8310f0a3-b09d-4746-8561-11c489116cf8-credential-socket\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-m95mt\" (UID: \"8310f0a3-b09d-4746-8561-11c489116cf8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt"
May 26 16:09:56.055367 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:56.055306    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8310f0a3-b09d-4746-8561-11c489116cf8-istio-token\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-m95mt\" (UID: \"8310f0a3-b09d-4746-8561-11c489116cf8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt"
May 26 16:09:56.055367 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:56.055352    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8310f0a3-b09d-4746-8561-11c489116cf8-workload-socket\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-m95mt\" (UID: \"8310f0a3-b09d-4746-8561-11c489116cf8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt"
May 26 16:09:56.055803 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:56.055407    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8310f0a3-b09d-4746-8561-11c489116cf8-workload-certs\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-m95mt\" (UID: \"8310f0a3-b09d-4746-8561-11c489116cf8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt"
May 26 16:09:56.055803 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:56.055509    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8310f0a3-b09d-4746-8561-11c489116cf8-istio-envoy\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-m95mt\" (UID: \"8310f0a3-b09d-4746-8561-11c489116cf8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt"
May 26 16:09:56.055803 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:56.055719    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8310f0a3-b09d-4746-8561-11c489116cf8-istio-data\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-m95mt\" (UID: \"8310f0a3-b09d-4746-8561-11c489116cf8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt"
May 26 16:09:56.055803 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:56.055768    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8310f0a3-b09d-4746-8561-11c489116cf8-credential-socket\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-m95mt\" (UID: \"8310f0a3-b09d-4746-8561-11c489116cf8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt"
May 26 16:09:56.056046 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:56.055843    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8310f0a3-b09d-4746-8561-11c489116cf8-workload-certs\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-m95mt\" (UID: \"8310f0a3-b09d-4746-8561-11c489116cf8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt"
May 26 16:09:56.056046 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:56.055841    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8310f0a3-b09d-4746-8561-11c489116cf8-workload-socket\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-m95mt\" (UID: \"8310f0a3-b09d-4746-8561-11c489116cf8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt"
May 26 16:09:56.056156 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:56.056127    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8310f0a3-b09d-4746-8561-11c489116cf8-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-m95mt\" (UID: \"8310f0a3-b09d-4746-8561-11c489116cf8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt"
May 26 16:09:56.057740 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:56.057715    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8310f0a3-b09d-4746-8561-11c489116cf8-istio-envoy\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-m95mt\" (UID: \"8310f0a3-b09d-4746-8561-11c489116cf8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt"
May 26 16:09:56.057842 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:56.057823    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8310f0a3-b09d-4746-8561-11c489116cf8-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-m95mt\" (UID: \"8310f0a3-b09d-4746-8561-11c489116cf8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt"
May 26 16:09:56.062993 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:56.062971    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8310f0a3-b09d-4746-8561-11c489116cf8-istio-token\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-m95mt\" (UID: \"8310f0a3-b09d-4746-8561-11c489116cf8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt"
May 26 16:09:56.063429 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:56.063404    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n5v8\" (UniqueName: \"kubernetes.io/projected/8310f0a3-b09d-4746-8561-11c489116cf8-kube-api-access-5n5v8\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-m95mt\" (UID: \"8310f0a3-b09d-4746-8561-11c489116cf8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt"
May 26 16:09:56.168140 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:56.168075    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt"
May 26 16:09:56.291152 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:56.291119    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt"]
May 26 16:09:56.294111 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:09:56.294085    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8310f0a3_b09d_4746_8561_11c489116cf8.slice/crio-51aa900dca2f622d46fdab1009fa7ac056e3d8ebbb079b881e16196c24fa9d72 WatchSource:0}: Error finding container 51aa900dca2f622d46fdab1009fa7ac056e3d8ebbb079b881e16196c24fa9d72: Status 404 returned error can't find the container with id 51aa900dca2f622d46fdab1009fa7ac056e3d8ebbb079b881e16196c24fa9d72
May 26 16:09:56.417920 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:56.417888    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt" event={"ID":"8310f0a3-b09d-4746-8561-11c489116cf8","Type":"ContainerStarted","Data":"51aa900dca2f622d46fdab1009fa7ac056e3d8ebbb079b881e16196c24fa9d72"}
May 26 16:09:58.875229 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:58.875189    2552 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
May 26 16:09:58.875501 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:58.875276    2552 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
May 26 16:09:58.875501 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:58.875313    2552 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
May 26 16:09:59.433127 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:59.433096    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt" event={"ID":"8310f0a3-b09d-4746-8561-11c489116cf8","Type":"ContainerStarted","Data":"33651ebef425f6183ff0b06dc46bcf9dab8500616fa73e29d25b6913069bfa86"}
May 26 16:09:59.454624 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:09:59.454579    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt" podStartSLOduration=1.875756773 podStartE2EDuration="4.454565398s" podCreationTimestamp="2026-05-26 16:09:55 +0000 UTC" firstStartedPulling="2026-05-26 16:09:56.29611781 +0000 UTC m=+546.004572827" lastFinishedPulling="2026-05-26 16:09:58.874926434 +0000 UTC m=+548.583381452" observedRunningTime="2026-05-26 16:09:59.452571006 +0000 UTC m=+549.161026046" watchObservedRunningTime="2026-05-26 16:09:59.454565398 +0000 UTC m=+549.163020437"
May 26 16:10:00.168812 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:10:00.168767    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt"
May 26 16:10:00.173494 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:10:00.173468    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt"
May 26 16:10:00.436904 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:10:00.436824    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt"
May 26 16:10:00.437778 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:10:00.437760    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-m95mt"
May 26 16:10:16.715660 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:10:16.715627    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7587b89b76-ntp27"]
May 26 16:10:16.720176 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:10:16.720136    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7587b89b76-ntp27"
May 26 16:10:16.723061 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:10:16.723039    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
May 26 16:10:16.723183 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:10:16.723042    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
May 26 16:10:16.723183 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:10:16.723042    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-hdrmt\""
May 26 16:10:16.728169 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:10:16.728143    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7587b89b76-ntp27"]
May 26 16:10:16.809990 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:10:16.809954    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcljh\" (UniqueName: \"kubernetes.io/projected/f8572d1e-9ffd-4442-bd73-d63b4a458cce-kube-api-access-lcljh\") pod \"authorino-operator-7587b89b76-ntp27\" (UID: \"f8572d1e-9ffd-4442-bd73-d63b4a458cce\") " pod="kuadrant-system/authorino-operator-7587b89b76-ntp27"
May 26 16:10:16.910590 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:10:16.910547    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lcljh\" (UniqueName: \"kubernetes.io/projected/f8572d1e-9ffd-4442-bd73-d63b4a458cce-kube-api-access-lcljh\") pod \"authorino-operator-7587b89b76-ntp27\" (UID: \"f8572d1e-9ffd-4442-bd73-d63b4a458cce\") " pod="kuadrant-system/authorino-operator-7587b89b76-ntp27"
May 26 16:10:16.920261 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:10:16.920231    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcljh\" (UniqueName: \"kubernetes.io/projected/f8572d1e-9ffd-4442-bd73-d63b4a458cce-kube-api-access-lcljh\") pod \"authorino-operator-7587b89b76-ntp27\" (UID: \"f8572d1e-9ffd-4442-bd73-d63b4a458cce\") " pod="kuadrant-system/authorino-operator-7587b89b76-ntp27"
May 26 16:10:17.031350 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:10:17.031317    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7587b89b76-ntp27"
May 26 16:10:17.161002 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:10:17.160979    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7587b89b76-ntp27"]
May 26 16:10:17.163598 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:10:17.163570    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8572d1e_9ffd_4442_bd73_d63b4a458cce.slice/crio-a67df9af300d625a3b2075dc2be9e7a4727d7a36fe1a9157cd385d3ddb8a3bbc WatchSource:0}: Error finding container a67df9af300d625a3b2075dc2be9e7a4727d7a36fe1a9157cd385d3ddb8a3bbc: Status 404 returned error can't find the container with id a67df9af300d625a3b2075dc2be9e7a4727d7a36fe1a9157cd385d3ddb8a3bbc
May 26 16:10:17.489315 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:10:17.489222    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7587b89b76-ntp27" event={"ID":"f8572d1e-9ffd-4442-bd73-d63b4a458cce","Type":"ContainerStarted","Data":"a67df9af300d625a3b2075dc2be9e7a4727d7a36fe1a9157cd385d3ddb8a3bbc"}
May 26 16:10:19.036545 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:10:19.036512    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-rbb7k"]
May 26 16:10:19.043287 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:10:19.043256    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-rbb7k"
May 26 16:10:19.046559 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:10:19.046537    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-ndblk\""
May 26 16:10:19.051947 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:10:19.051597    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-rbb7k"]
May 26 16:10:19.129111 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:10:19.129069    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng7f6\" (UniqueName: \"kubernetes.io/projected/a07ead44-5922-4db0-aea0-2f827a0c0f4c-kube-api-access-ng7f6\") pod \"limitador-operator-controller-manager-c7fb4c8d5-rbb7k\" (UID: \"a07ead44-5922-4db0-aea0-2f827a0c0f4c\") " pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-rbb7k"
May 26 16:10:19.229947 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:10:19.229887    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ng7f6\" (UniqueName: \"kubernetes.io/projected/a07ead44-5922-4db0-aea0-2f827a0c0f4c-kube-api-access-ng7f6\") pod \"limitador-operator-controller-manager-c7fb4c8d5-rbb7k\" (UID: \"a07ead44-5922-4db0-aea0-2f827a0c0f4c\") " pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-rbb7k"
May 26 16:10:19.244959 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:10:19.244915    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng7f6\" (UniqueName: \"kubernetes.io/projected/a07ead44-5922-4db0-aea0-2f827a0c0f4c-kube-api-access-ng7f6\") pod \"limitador-operator-controller-manager-c7fb4c8d5-rbb7k\" (UID: \"a07ead44-5922-4db0-aea0-2f827a0c0f4c\") " pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-rbb7k"
May 26 16:10:19.356414 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:10:19.356318    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-rbb7k"
May 26 16:10:19.833963 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:10:19.833914    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-rbb7k"]
May 26 16:10:19.837132 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:10:19.837100    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda07ead44_5922_4db0_aea0_2f827a0c0f4c.slice/crio-68849ecca602a195c7d516b888e4d97d60cfe9d1968750b3c2360b6ee03ae632 WatchSource:0}: Error finding container 68849ecca602a195c7d516b888e4d97d60cfe9d1968750b3c2360b6ee03ae632: Status 404 returned error can't find the container with id 68849ecca602a195c7d516b888e4d97d60cfe9d1968750b3c2360b6ee03ae632
May 26 16:10:20.500152 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:10:20.500112    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7587b89b76-ntp27" event={"ID":"f8572d1e-9ffd-4442-bd73-d63b4a458cce","Type":"ContainerStarted","Data":"61478c2acfd9a7c3935938cc141e4bbee52affb6fab7450f198d1b3218dfbec6"}
May 26 16:10:20.500605 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:10:20.500209    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7587b89b76-ntp27"
May 26 16:10:20.501283 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:10:20.501259    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-rbb7k" event={"ID":"a07ead44-5922-4db0-aea0-2f827a0c0f4c","Type":"ContainerStarted","Data":"68849ecca602a195c7d516b888e4d97d60cfe9d1968750b3c2360b6ee03ae632"}
May 26 16:10:20.520679 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:10:20.520631    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7587b89b76-ntp27" podStartSLOduration=1.959560993 podStartE2EDuration="4.520614702s" podCreationTimestamp="2026-05-26 16:10:16 +0000 UTC" firstStartedPulling="2026-05-26 16:10:17.165509828 +0000 UTC m=+566.873964846" lastFinishedPulling="2026-05-26 16:10:19.726563534 +0000 UTC m=+569.435018555" observedRunningTime="2026-05-26 16:10:20.520318135 +0000 UTC m=+570.228773175" watchObservedRunningTime="2026-05-26 16:10:20.520614702 +0000 UTC m=+570.229069742"
May 26 16:10:23.514975 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:10:23.514916    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-rbb7k" event={"ID":"a07ead44-5922-4db0-aea0-2f827a0c0f4c","Type":"ContainerStarted","Data":"d725335ea4e4a2fc593dde37558158d609c5bb33e3b3286cd52460658a899ef9"}
May 26 16:10:23.515377 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:10:23.515156    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-rbb7k"
May 26 16:10:23.534579 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:10:23.534526    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-rbb7k" podStartSLOduration=1.8633397839999999 podStartE2EDuration="4.534511189s" podCreationTimestamp="2026-05-26 16:10:19 +0000 UTC" firstStartedPulling="2026-05-26 16:10:19.839462964 +0000 UTC m=+569.547917996" lastFinishedPulling="2026-05-26 16:10:22.510634381 +0000 UTC m=+572.219089401" observedRunningTime="2026-05-26 16:10:23.533466569 +0000 UTC m=+573.241921610" watchObservedRunningTime="2026-05-26 16:10:23.534511189 +0000 UTC m=+573.242966233"
May 26 16:10:31.506822 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:10:31.506789    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7587b89b76-ntp27"
May 26 16:10:34.520745 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:10:34.520713    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-rbb7k"
May 26 16:10:50.795446 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:10:50.795413    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sj6db_5c0f9fc6-2bd6-4370-96d0-5ae315363c7f/ovn-acl-logging/0.log"
May 26 16:10:50.796048 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:10:50.795457    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sj6db_5c0f9fc6-2bd6-4370-96d0-5ae315363c7f/ovn-acl-logging/0.log"
May 26 16:11:14.063237 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:14.063149    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-6zn8m"]
May 26 16:11:14.065213 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:14.065196    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c8f475fb-6zn8m"
May 26 16:11:14.068451 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:14.068427    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
May 26 16:11:14.068680 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:14.068665    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7vtq7\""
May 26 16:11:14.079254 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:14.079220    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-6zn8m"]
May 26 16:11:14.148994 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:14.148953    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2be0e331-ed9b-4b9f-ba7c-adc0e226e929-config-file\") pod \"limitador-limitador-64c8f475fb-6zn8m\" (UID: \"2be0e331-ed9b-4b9f-ba7c-adc0e226e929\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-6zn8m"
May 26 16:11:14.149183 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:14.149119    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs9vl\" (UniqueName: \"kubernetes.io/projected/2be0e331-ed9b-4b9f-ba7c-adc0e226e929-kube-api-access-qs9vl\") pod \"limitador-limitador-64c8f475fb-6zn8m\" (UID: \"2be0e331-ed9b-4b9f-ba7c-adc0e226e929\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-6zn8m"
May 26 16:11:14.164519 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:14.164477    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-6zn8m"]
May 26 16:11:14.250443 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:14.250405    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2be0e331-ed9b-4b9f-ba7c-adc0e226e929-config-file\") pod \"limitador-limitador-64c8f475fb-6zn8m\" (UID: \"2be0e331-ed9b-4b9f-ba7c-adc0e226e929\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-6zn8m"
May 26 16:11:14.250645 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:14.250474    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qs9vl\" (UniqueName: \"kubernetes.io/projected/2be0e331-ed9b-4b9f-ba7c-adc0e226e929-kube-api-access-qs9vl\") pod \"limitador-limitador-64c8f475fb-6zn8m\" (UID: \"2be0e331-ed9b-4b9f-ba7c-adc0e226e929\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-6zn8m"
May 26 16:11:14.251124 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:14.251100    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2be0e331-ed9b-4b9f-ba7c-adc0e226e929-config-file\") pod \"limitador-limitador-64c8f475fb-6zn8m\" (UID: \"2be0e331-ed9b-4b9f-ba7c-adc0e226e929\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-6zn8m"
May 26 16:11:14.262315 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:14.262280    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs9vl\" (UniqueName: \"kubernetes.io/projected/2be0e331-ed9b-4b9f-ba7c-adc0e226e929-kube-api-access-qs9vl\") pod \"limitador-limitador-64c8f475fb-6zn8m\" (UID: \"2be0e331-ed9b-4b9f-ba7c-adc0e226e929\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-6zn8m"
May 26 16:11:14.374900 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:14.374809    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c8f475fb-6zn8m"
May 26 16:11:14.529582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:14.529552    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-6zn8m"]
May 26 16:11:14.532463 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:11:14.532435    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2be0e331_ed9b_4b9f_ba7c_adc0e226e929.slice/crio-192bcd39e8cbc32b56b2d0032066adb4cde42962251af8c9e98c2d28ce244d59 WatchSource:0}: Error finding container 192bcd39e8cbc32b56b2d0032066adb4cde42962251af8c9e98c2d28ce244d59: Status 404 returned error can't find the container with id 192bcd39e8cbc32b56b2d0032066adb4cde42962251af8c9e98c2d28ce244d59
May 26 16:11:14.675843 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:14.675744    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c8f475fb-6zn8m" event={"ID":"2be0e331-ed9b-4b9f-ba7c-adc0e226e929","Type":"ContainerStarted","Data":"192bcd39e8cbc32b56b2d0032066adb4cde42962251af8c9e98c2d28ce244d59"}
May 26 16:11:18.691540 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:18.691506    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c8f475fb-6zn8m" event={"ID":"2be0e331-ed9b-4b9f-ba7c-adc0e226e929","Type":"ContainerStarted","Data":"791149e70f1e72aa267c17bdc94748494bdbaffe7f4ff8647b21de945c80d276"}
May 26 16:11:18.691965 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:18.691565    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c8f475fb-6zn8m"
May 26 16:11:18.717472 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:18.717412    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c8f475fb-6zn8m" podStartSLOduration=1.030219802 podStartE2EDuration="4.717390176s" podCreationTimestamp="2026-05-26 16:11:14 +0000 UTC" firstStartedPulling="2026-05-26 16:11:14.534579449 +0000 UTC m=+624.243034466" lastFinishedPulling="2026-05-26 16:11:18.221749808 +0000 UTC m=+627.930204840" observedRunningTime="2026-05-26 16:11:18.715906314 +0000 UTC m=+628.424361355" watchObservedRunningTime="2026-05-26 16:11:18.717390176 +0000 UTC m=+628.425845216"
May 26 16:11:27.267970 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:27.267926    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-6zn8m"]
May 26 16:11:27.268499 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:27.268222    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c8f475fb-6zn8m" podUID="2be0e331-ed9b-4b9f-ba7c-adc0e226e929" containerName="limitador" containerID="cri-o://791149e70f1e72aa267c17bdc94748494bdbaffe7f4ff8647b21de945c80d276" gracePeriod=30
May 26 16:11:27.271109 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:27.271081    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c8f475fb-6zn8m"
May 26 16:11:27.721371 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:27.721337    2552 generic.go:358] "Generic (PLEG): container finished" podID="2be0e331-ed9b-4b9f-ba7c-adc0e226e929" containerID="791149e70f1e72aa267c17bdc94748494bdbaffe7f4ff8647b21de945c80d276" exitCode=0
May 26 16:11:27.721519 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:27.721402    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c8f475fb-6zn8m" event={"ID":"2be0e331-ed9b-4b9f-ba7c-adc0e226e929","Type":"ContainerDied","Data":"791149e70f1e72aa267c17bdc94748494bdbaffe7f4ff8647b21de945c80d276"}
May 26 16:11:27.808307 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:27.808284    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c8f475fb-6zn8m"
May 26 16:11:27.978762 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:27.978669    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2be0e331-ed9b-4b9f-ba7c-adc0e226e929-config-file\") pod \"2be0e331-ed9b-4b9f-ba7c-adc0e226e929\" (UID: \"2be0e331-ed9b-4b9f-ba7c-adc0e226e929\") "
May 26 16:11:27.978762 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:27.978729    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qs9vl\" (UniqueName: \"kubernetes.io/projected/2be0e331-ed9b-4b9f-ba7c-adc0e226e929-kube-api-access-qs9vl\") pod \"2be0e331-ed9b-4b9f-ba7c-adc0e226e929\" (UID: \"2be0e331-ed9b-4b9f-ba7c-adc0e226e929\") "
May 26 16:11:27.979101 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:27.979076    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2be0e331-ed9b-4b9f-ba7c-adc0e226e929-config-file" (OuterVolumeSpecName: "config-file") pod "2be0e331-ed9b-4b9f-ba7c-adc0e226e929" (UID: "2be0e331-ed9b-4b9f-ba7c-adc0e226e929"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:11:27.980883 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:27.980862    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2be0e331-ed9b-4b9f-ba7c-adc0e226e929-kube-api-access-qs9vl" (OuterVolumeSpecName: "kube-api-access-qs9vl") pod "2be0e331-ed9b-4b9f-ba7c-adc0e226e929" (UID: "2be0e331-ed9b-4b9f-ba7c-adc0e226e929"). InnerVolumeSpecName "kube-api-access-qs9vl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:11:28.079660 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:28.079618    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs9vl\" (UniqueName: \"kubernetes.io/projected/2be0e331-ed9b-4b9f-ba7c-adc0e226e929-kube-api-access-qs9vl\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:11:28.079660 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:28.079656    2552 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2be0e331-ed9b-4b9f-ba7c-adc0e226e929-config-file\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:11:28.726618 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:28.726591    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c8f475fb-6zn8m"
May 26 16:11:28.727062 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:28.726585    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c8f475fb-6zn8m" event={"ID":"2be0e331-ed9b-4b9f-ba7c-adc0e226e929","Type":"ContainerDied","Data":"192bcd39e8cbc32b56b2d0032066adb4cde42962251af8c9e98c2d28ce244d59"}
May 26 16:11:28.727062 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:28.726712    2552 scope.go:117] "RemoveContainer" containerID="791149e70f1e72aa267c17bdc94748494bdbaffe7f4ff8647b21de945c80d276"
May 26 16:11:28.754731 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:28.754694    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-6zn8m"]
May 26 16:11:28.763912 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:28.763877    2552 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-6zn8m"]
May 26 16:11:28.889946 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:28.889903    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2be0e331-ed9b-4b9f-ba7c-adc0e226e929" path="/var/lib/kubelet/pods/2be0e331-ed9b-4b9f-ba7c-adc0e226e929/volumes"
May 26 16:11:48.094611 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:48.092669    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-798958bb55-mm958"]
May 26 16:11:48.094611 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:48.093267    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2be0e331-ed9b-4b9f-ba7c-adc0e226e929" containerName="limitador"
May 26 16:11:48.094611 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:48.093285    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="2be0e331-ed9b-4b9f-ba7c-adc0e226e929" containerName="limitador"
May 26 16:11:48.094611 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:48.093428    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="2be0e331-ed9b-4b9f-ba7c-adc0e226e929" containerName="limitador"
May 26 16:11:48.096817 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:48.096786    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mm958"
May 26 16:11:48.114385 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:48.114352    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-798958bb55-mm958"]
May 26 16:11:48.132489 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:48.132449    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvddr\" (UniqueName: \"kubernetes.io/projected/ce1e9de0-8ee7-43fb-9091-553fe6cfe394-kube-api-access-hvddr\") pod \"istiod-openshift-gateway-798958bb55-mm958\" (UID: \"ce1e9de0-8ee7-43fb-9091-553fe6cfe394\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mm958"
May 26 16:11:48.132489 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:48.132489    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ce1e9de0-8ee7-43fb-9091-553fe6cfe394-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-798958bb55-mm958\" (UID: \"ce1e9de0-8ee7-43fb-9091-553fe6cfe394\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mm958"
May 26 16:11:48.132702 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:48.132514    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ce1e9de0-8ee7-43fb-9091-553fe6cfe394-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-798958bb55-mm958\" (UID: \"ce1e9de0-8ee7-43fb-9091-553fe6cfe394\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mm958"
May 26 16:11:48.132702 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:48.132589    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ce1e9de0-8ee7-43fb-9091-553fe6cfe394-istio-kubeconfig\") pod \"istiod-openshift-gateway-798958bb55-mm958\" (UID: \"ce1e9de0-8ee7-43fb-9091-553fe6cfe394\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mm958"
May 26 16:11:48.132702 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:48.132621    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ce1e9de0-8ee7-43fb-9091-553fe6cfe394-cacerts\") pod \"istiod-openshift-gateway-798958bb55-mm958\" (UID: \"ce1e9de0-8ee7-43fb-9091-553fe6cfe394\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mm958"
May 26 16:11:48.132702 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:48.132650    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1e9de0-8ee7-43fb-9091-553fe6cfe394-local-certs\") pod \"istiod-openshift-gateway-798958bb55-mm958\" (UID: \"ce1e9de0-8ee7-43fb-9091-553fe6cfe394\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mm958"
May 26 16:11:48.132702 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:48.132669    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1e9de0-8ee7-43fb-9091-553fe6cfe394-istio-token\") pod \"istiod-openshift-gateway-798958bb55-mm958\" (UID: \"ce1e9de0-8ee7-43fb-9091-553fe6cfe394\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mm958"
May 26 16:11:48.233136 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:48.233095    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hvddr\" (UniqueName: \"kubernetes.io/projected/ce1e9de0-8ee7-43fb-9091-553fe6cfe394-kube-api-access-hvddr\") pod \"istiod-openshift-gateway-798958bb55-mm958\" (UID: \"ce1e9de0-8ee7-43fb-9091-553fe6cfe394\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mm958"
May 26 16:11:48.233330 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:48.233144    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ce1e9de0-8ee7-43fb-9091-553fe6cfe394-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-798958bb55-mm958\" (UID: \"ce1e9de0-8ee7-43fb-9091-553fe6cfe394\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mm958"
May 26 16:11:48.233330 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:48.233189    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ce1e9de0-8ee7-43fb-9091-553fe6cfe394-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-798958bb55-mm958\" (UID: \"ce1e9de0-8ee7-43fb-9091-553fe6cfe394\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mm958"
May 26 16:11:48.233330 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:48.233224    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ce1e9de0-8ee7-43fb-9091-553fe6cfe394-istio-kubeconfig\") pod \"istiod-openshift-gateway-798958bb55-mm958\" (UID: \"ce1e9de0-8ee7-43fb-9091-553fe6cfe394\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mm958"
May 26 16:11:48.233491 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:48.233368    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ce1e9de0-8ee7-43fb-9091-553fe6cfe394-cacerts\") pod \"istiod-openshift-gateway-798958bb55-mm958\" (UID: \"ce1e9de0-8ee7-43fb-9091-553fe6cfe394\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mm958"
May 26 16:11:48.233491 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:48.233411    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1e9de0-8ee7-43fb-9091-553fe6cfe394-local-certs\") pod \"istiod-openshift-gateway-798958bb55-mm958\" (UID: \"ce1e9de0-8ee7-43fb-9091-553fe6cfe394\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mm958"
May 26 16:11:48.233491 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:48.233445    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1e9de0-8ee7-43fb-9091-553fe6cfe394-istio-token\") pod \"istiod-openshift-gateway-798958bb55-mm958\" (UID: \"ce1e9de0-8ee7-43fb-9091-553fe6cfe394\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mm958"
May 26 16:11:48.234021 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:48.233993    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ce1e9de0-8ee7-43fb-9091-553fe6cfe394-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-798958bb55-mm958\" (UID: \"ce1e9de0-8ee7-43fb-9091-553fe6cfe394\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mm958"
May 26 16:11:48.235640 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:48.235617    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1e9de0-8ee7-43fb-9091-553fe6cfe394-local-certs\") pod \"istiod-openshift-gateway-798958bb55-mm958\" (UID: \"ce1e9de0-8ee7-43fb-9091-553fe6cfe394\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mm958"
May 26 16:11:48.235810 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:48.235792    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ce1e9de0-8ee7-43fb-9091-553fe6cfe394-istio-kubeconfig\") pod \"istiod-openshift-gateway-798958bb55-mm958\" (UID: \"ce1e9de0-8ee7-43fb-9091-553fe6cfe394\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mm958"
May 26 16:11:48.235810 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:48.235797    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ce1e9de0-8ee7-43fb-9091-553fe6cfe394-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-798958bb55-mm958\" (UID: \"ce1e9de0-8ee7-43fb-9091-553fe6cfe394\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mm958"
May 26 16:11:48.235922 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:48.235876    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ce1e9de0-8ee7-43fb-9091-553fe6cfe394-cacerts\") pod \"istiod-openshift-gateway-798958bb55-mm958\" (UID: \"ce1e9de0-8ee7-43fb-9091-553fe6cfe394\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mm958"
May 26 16:11:48.243995 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:48.243960    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvddr\" (UniqueName: \"kubernetes.io/projected/ce1e9de0-8ee7-43fb-9091-553fe6cfe394-kube-api-access-hvddr\") pod \"istiod-openshift-gateway-798958bb55-mm958\" (UID: \"ce1e9de0-8ee7-43fb-9091-553fe6cfe394\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mm958"
May 26 16:11:48.245298 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:48.245281    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1e9de0-8ee7-43fb-9091-553fe6cfe394-istio-token\") pod \"istiod-openshift-gateway-798958bb55-mm958\" (UID: \"ce1e9de0-8ee7-43fb-9091-553fe6cfe394\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mm958"
May 26 16:11:48.406414 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:48.406311    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mm958"
May 26 16:11:48.591253 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:48.591225    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-798958bb55-mm958"]
May 26 16:11:48.592573 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:11:48.592544    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce1e9de0_8ee7_43fb_9091_553fe6cfe394.slice/crio-d80d0ad88a52850facc6b27fcde610302e482010c47432ec8b694e1916c33055 WatchSource:0}: Error finding container d80d0ad88a52850facc6b27fcde610302e482010c47432ec8b694e1916c33055: Status 404 returned error can't find the container with id d80d0ad88a52850facc6b27fcde610302e482010c47432ec8b694e1916c33055
May 26 16:11:48.594520 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:48.594489    2552 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
May 26 16:11:48.594605 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:48.594568    2552 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
May 26 16:11:48.789136 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:48.789099    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mm958" event={"ID":"ce1e9de0-8ee7-43fb-9091-553fe6cfe394","Type":"ContainerStarted","Data":"69fa66112c20272b9119fe92423c51775c4f946830d29107d9b95f256ace3ab5"}
May 26 16:11:48.789136 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:48.789136    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mm958" event={"ID":"ce1e9de0-8ee7-43fb-9091-553fe6cfe394","Type":"ContainerStarted","Data":"d80d0ad88a52850facc6b27fcde610302e482010c47432ec8b694e1916c33055"}
May 26 16:11:48.789331 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:48.789162    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mm958"
May 26 16:11:48.825282 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:48.825211    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mm958" podStartSLOduration=0.825195963 podStartE2EDuration="825.195963ms" podCreationTimestamp="2026-05-26 16:11:48 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:11:48.824359965 +0000 UTC m=+658.532815004" watchObservedRunningTime="2026-05-26 16:11:48.825195963 +0000 UTC m=+658.533651003"
May 26 16:11:49.794171 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:49.794141    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mm958"
May 26 16:11:49.905160 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:49.905130    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm"]
May 26 16:11:49.905385 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:49.905362    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm" podUID="10c3c960-7994-4f89-b867-4d41ba86d9b2" containerName="discovery" containerID="cri-o://d49c781fd379d1dc370239b1bf2349be377648f5d951db0d4413cb943a0786a8" gracePeriod=30
May 26 16:11:50.153440 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:50.153410    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm"
May 26 16:11:50.248475 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:50.248440    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/10c3c960-7994-4f89-b867-4d41ba86d9b2-cacerts\") pod \"10c3c960-7994-4f89-b867-4d41ba86d9b2\" (UID: \"10c3c960-7994-4f89-b867-4d41ba86d9b2\") "
May 26 16:11:50.248475 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:50.248482    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/10c3c960-7994-4f89-b867-4d41ba86d9b2-local-certs\") pod \"10c3c960-7994-4f89-b867-4d41ba86d9b2\" (UID: \"10c3c960-7994-4f89-b867-4d41ba86d9b2\") "
May 26 16:11:50.248720 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:50.248523    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmrqz\" (UniqueName: \"kubernetes.io/projected/10c3c960-7994-4f89-b867-4d41ba86d9b2-kube-api-access-pmrqz\") pod \"10c3c960-7994-4f89-b867-4d41ba86d9b2\" (UID: \"10c3c960-7994-4f89-b867-4d41ba86d9b2\") "
May 26 16:11:50.248720 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:50.248551    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/10c3c960-7994-4f89-b867-4d41ba86d9b2-istio-csr-ca-configmap\") pod \"10c3c960-7994-4f89-b867-4d41ba86d9b2\" (UID: \"10c3c960-7994-4f89-b867-4d41ba86d9b2\") "
May 26 16:11:50.248720 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:50.248575    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/10c3c960-7994-4f89-b867-4d41ba86d9b2-istio-csr-dns-cert\") pod \"10c3c960-7994-4f89-b867-4d41ba86d9b2\" (UID: \"10c3c960-7994-4f89-b867-4d41ba86d9b2\") "
May 26 16:11:50.248720 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:50.248605    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/10c3c960-7994-4f89-b867-4d41ba86d9b2-istio-kubeconfig\") pod \"10c3c960-7994-4f89-b867-4d41ba86d9b2\" (UID: \"10c3c960-7994-4f89-b867-4d41ba86d9b2\") "
May 26 16:11:50.248720 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:50.248673    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10c3c960-7994-4f89-b867-4d41ba86d9b2-istio-token\") pod \"10c3c960-7994-4f89-b867-4d41ba86d9b2\" (UID: \"10c3c960-7994-4f89-b867-4d41ba86d9b2\") "
May 26 16:11:50.249062 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:50.249018    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10c3c960-7994-4f89-b867-4d41ba86d9b2-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "10c3c960-7994-4f89-b867-4d41ba86d9b2" (UID: "10c3c960-7994-4f89-b867-4d41ba86d9b2"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:11:50.251139 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:50.251100    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10c3c960-7994-4f89-b867-4d41ba86d9b2-cacerts" (OuterVolumeSpecName: "cacerts") pod "10c3c960-7994-4f89-b867-4d41ba86d9b2" (UID: "10c3c960-7994-4f89-b867-4d41ba86d9b2"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:11:50.251139 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:50.251120    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10c3c960-7994-4f89-b867-4d41ba86d9b2-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "10c3c960-7994-4f89-b867-4d41ba86d9b2" (UID: "10c3c960-7994-4f89-b867-4d41ba86d9b2"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:11:50.251301 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:50.251216    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c3c960-7994-4f89-b867-4d41ba86d9b2-istio-token" (OuterVolumeSpecName: "istio-token") pod "10c3c960-7994-4f89-b867-4d41ba86d9b2" (UID: "10c3c960-7994-4f89-b867-4d41ba86d9b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:11:50.251301 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:50.251229    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c3c960-7994-4f89-b867-4d41ba86d9b2-local-certs" (OuterVolumeSpecName: "local-certs") pod "10c3c960-7994-4f89-b867-4d41ba86d9b2" (UID: "10c3c960-7994-4f89-b867-4d41ba86d9b2"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:11:50.251369 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:50.251317    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c3c960-7994-4f89-b867-4d41ba86d9b2-kube-api-access-pmrqz" (OuterVolumeSpecName: "kube-api-access-pmrqz") pod "10c3c960-7994-4f89-b867-4d41ba86d9b2" (UID: "10c3c960-7994-4f89-b867-4d41ba86d9b2"). InnerVolumeSpecName "kube-api-access-pmrqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:11:50.251369 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:50.251318    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10c3c960-7994-4f89-b867-4d41ba86d9b2-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "10c3c960-7994-4f89-b867-4d41ba86d9b2" (UID: "10c3c960-7994-4f89-b867-4d41ba86d9b2"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:11:50.350256 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:50.350166    2552 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/10c3c960-7994-4f89-b867-4d41ba86d9b2-istio-csr-ca-configmap\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:11:50.350256 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:50.350205    2552 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/10c3c960-7994-4f89-b867-4d41ba86d9b2-istio-csr-dns-cert\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:11:50.350256 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:50.350215    2552 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/10c3c960-7994-4f89-b867-4d41ba86d9b2-istio-kubeconfig\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:11:50.350256 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:50.350224    2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10c3c960-7994-4f89-b867-4d41ba86d9b2-istio-token\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:11:50.350256 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:50.350233    2552 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/10c3c960-7994-4f89-b867-4d41ba86d9b2-cacerts\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:11:50.350256 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:50.350240    2552 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/10c3c960-7994-4f89-b867-4d41ba86d9b2-local-certs\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:11:50.350256 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:50.350249    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmrqz\" (UniqueName: \"kubernetes.io/projected/10c3c960-7994-4f89-b867-4d41ba86d9b2-kube-api-access-pmrqz\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:11:50.796426 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:50.796390    2552 generic.go:358] "Generic (PLEG): container finished" podID="10c3c960-7994-4f89-b867-4d41ba86d9b2" containerID="d49c781fd379d1dc370239b1bf2349be377648f5d951db0d4413cb943a0786a8" exitCode=0
May 26 16:11:50.796890 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:50.796450    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm"
May 26 16:11:50.796890 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:50.796469    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm" event={"ID":"10c3c960-7994-4f89-b867-4d41ba86d9b2","Type":"ContainerDied","Data":"d49c781fd379d1dc370239b1bf2349be377648f5d951db0d4413cb943a0786a8"}
May 26 16:11:50.796890 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:50.796507    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm" event={"ID":"10c3c960-7994-4f89-b867-4d41ba86d9b2","Type":"ContainerDied","Data":"6842788b9090c97e59dc1b82f67fbf4d37ecd159f81a22346f136237fd66416a"}
May 26 16:11:50.796890 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:50.796527    2552 scope.go:117] "RemoveContainer" containerID="d49c781fd379d1dc370239b1bf2349be377648f5d951db0d4413cb943a0786a8"
May 26 16:11:50.808016 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:50.807995    2552 scope.go:117] "RemoveContainer" containerID="d49c781fd379d1dc370239b1bf2349be377648f5d951db0d4413cb943a0786a8"
May 26 16:11:50.808269 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:11:50.808247    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d49c781fd379d1dc370239b1bf2349be377648f5d951db0d4413cb943a0786a8\": container with ID starting with d49c781fd379d1dc370239b1bf2349be377648f5d951db0d4413cb943a0786a8 not found: ID does not exist" containerID="d49c781fd379d1dc370239b1bf2349be377648f5d951db0d4413cb943a0786a8"
May 26 16:11:50.808343 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:50.808276    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d49c781fd379d1dc370239b1bf2349be377648f5d951db0d4413cb943a0786a8"} err="failed to get container status \"d49c781fd379d1dc370239b1bf2349be377648f5d951db0d4413cb943a0786a8\": rpc error: code = NotFound desc = could not find container \"d49c781fd379d1dc370239b1bf2349be377648f5d951db0d4413cb943a0786a8\": container with ID starting with d49c781fd379d1dc370239b1bf2349be377648f5d951db0d4413cb943a0786a8 not found: ID does not exist"
May 26 16:11:50.851563 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:50.851529    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm"]
May 26 16:11:50.868006 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:50.867971    2552 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-767cf5cf55-46kkm"]
May 26 16:11:50.894268 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:11:50.894238    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10c3c960-7994-4f89-b867-4d41ba86d9b2" path="/var/lib/kubelet/pods/10c3c960-7994-4f89-b867-4d41ba86d9b2/volumes"
May 26 16:12:04.644335 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.644299    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6f7545b5b9-nvv7f"]
May 26 16:12:04.645039 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.645016    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10c3c960-7994-4f89-b867-4d41ba86d9b2" containerName="discovery"
May 26 16:12:04.645151 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.645043    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="10c3c960-7994-4f89-b867-4d41ba86d9b2" containerName="discovery"
May 26 16:12:04.650244 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.650203    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="10c3c960-7994-4f89-b867-4d41ba86d9b2" containerName="discovery"
May 26 16:12:04.669234 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.669198    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6f7545b5b9-nvv7f"]
May 26 16:12:04.669383 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.669332    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6f7545b5b9-nvv7f"
May 26 16:12:04.672731 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.672705    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 26 16:12:04.677110 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.677085    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
May 26 16:12:04.677226 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.677169    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-rh6cp\""
May 26 16:12:04.677226 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.677191    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 26 16:12:04.691995 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.691964    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-7f6dfc774f-znr6c"]
May 26 16:12:04.695087 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.695071    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-7f6dfc774f-znr6c"
May 26 16:12:04.700314 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.700285    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-djnd8\""
May 26 16:12:04.700713 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.700691    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
May 26 16:12:04.712251 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.712222    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-7f6dfc774f-znr6c"]
May 26 16:12:04.767233 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.767203    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/abc48fc2-b76c-4cdb-bdde-7239f5f9ab03-cert\") pod \"kserve-controller-manager-6f7545b5b9-nvv7f\" (UID: \"abc48fc2-b76c-4cdb-bdde-7239f5f9ab03\") " pod="kserve/kserve-controller-manager-6f7545b5b9-nvv7f"
May 26 16:12:04.767233 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.767242    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4prp\" (UniqueName: \"kubernetes.io/projected/880bb567-ef75-4853-ad67-794058f47650-kube-api-access-t4prp\") pod \"llmisvc-controller-manager-7f6dfc774f-znr6c\" (UID: \"880bb567-ef75-4853-ad67-794058f47650\") " pod="kserve/llmisvc-controller-manager-7f6dfc774f-znr6c"
May 26 16:12:04.767458 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.767276    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rvxg\" (UniqueName: \"kubernetes.io/projected/abc48fc2-b76c-4cdb-bdde-7239f5f9ab03-kube-api-access-2rvxg\") pod \"kserve-controller-manager-6f7545b5b9-nvv7f\" (UID: \"abc48fc2-b76c-4cdb-bdde-7239f5f9ab03\") " pod="kserve/kserve-controller-manager-6f7545b5b9-nvv7f"
May 26 16:12:04.767458 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.767337    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/880bb567-ef75-4853-ad67-794058f47650-cert\") pod \"llmisvc-controller-manager-7f6dfc774f-znr6c\" (UID: \"880bb567-ef75-4853-ad67-794058f47650\") " pod="kserve/llmisvc-controller-manager-7f6dfc774f-znr6c"
May 26 16:12:04.768481 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.768461    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-79f457656b-xwhcb"]
May 26 16:12:04.771570 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.771543    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-xwhcb"
May 26 16:12:04.774959 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.774924    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
May 26 16:12:04.777966 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.777816    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-8x2dr\""
May 26 16:12:04.790051 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.790023    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-xwhcb"]
May 26 16:12:04.868707 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.868664    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtj4t\" (UniqueName: \"kubernetes.io/projected/7232bab9-5261-4dc4-8fe6-c0bcbc19eec0-kube-api-access-qtj4t\") pod \"seaweedfs-79f457656b-xwhcb\" (UID: \"7232bab9-5261-4dc4-8fe6-c0bcbc19eec0\") " pod="kserve/seaweedfs-79f457656b-xwhcb"
May 26 16:12:04.868707 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.868711    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/abc48fc2-b76c-4cdb-bdde-7239f5f9ab03-cert\") pod \"kserve-controller-manager-6f7545b5b9-nvv7f\" (UID: \"abc48fc2-b76c-4cdb-bdde-7239f5f9ab03\") " pod="kserve/kserve-controller-manager-6f7545b5b9-nvv7f"
May 26 16:12:04.869009 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.868734    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t4prp\" (UniqueName: \"kubernetes.io/projected/880bb567-ef75-4853-ad67-794058f47650-kube-api-access-t4prp\") pod \"llmisvc-controller-manager-7f6dfc774f-znr6c\" (UID: \"880bb567-ef75-4853-ad67-794058f47650\") " pod="kserve/llmisvc-controller-manager-7f6dfc774f-znr6c"
May 26 16:12:04.869009 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.868756    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2rvxg\" (UniqueName: \"kubernetes.io/projected/abc48fc2-b76c-4cdb-bdde-7239f5f9ab03-kube-api-access-2rvxg\") pod \"kserve-controller-manager-6f7545b5b9-nvv7f\" (UID: \"abc48fc2-b76c-4cdb-bdde-7239f5f9ab03\") " pod="kserve/kserve-controller-manager-6f7545b5b9-nvv7f"
May 26 16:12:04.869009 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.868773    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7232bab9-5261-4dc4-8fe6-c0bcbc19eec0-data\") pod \"seaweedfs-79f457656b-xwhcb\" (UID: \"7232bab9-5261-4dc4-8fe6-c0bcbc19eec0\") " pod="kserve/seaweedfs-79f457656b-xwhcb"
May 26 16:12:04.869009 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.868792    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/880bb567-ef75-4853-ad67-794058f47650-cert\") pod \"llmisvc-controller-manager-7f6dfc774f-znr6c\" (UID: \"880bb567-ef75-4853-ad67-794058f47650\") " pod="kserve/llmisvc-controller-manager-7f6dfc774f-znr6c"
May 26 16:12:04.869009 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:12:04.868897    2552 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
May 26 16:12:04.869009 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:12:04.868985    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/880bb567-ef75-4853-ad67-794058f47650-cert podName:880bb567-ef75-4853-ad67-794058f47650 nodeName:}" failed. No retries permitted until 2026-05-26 16:12:05.36896787 +0000 UTC m=+675.077422888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/880bb567-ef75-4853-ad67-794058f47650-cert") pod "llmisvc-controller-manager-7f6dfc774f-znr6c" (UID: "880bb567-ef75-4853-ad67-794058f47650") : secret "llmisvc-webhook-server-cert" not found
May 26 16:12:04.871342 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.871314    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/abc48fc2-b76c-4cdb-bdde-7239f5f9ab03-cert\") pod \"kserve-controller-manager-6f7545b5b9-nvv7f\" (UID: \"abc48fc2-b76c-4cdb-bdde-7239f5f9ab03\") " pod="kserve/kserve-controller-manager-6f7545b5b9-nvv7f"
May 26 16:12:04.887966 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.887911    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rvxg\" (UniqueName: \"kubernetes.io/projected/abc48fc2-b76c-4cdb-bdde-7239f5f9ab03-kube-api-access-2rvxg\") pod \"kserve-controller-manager-6f7545b5b9-nvv7f\" (UID: \"abc48fc2-b76c-4cdb-bdde-7239f5f9ab03\") " pod="kserve/kserve-controller-manager-6f7545b5b9-nvv7f"
May 26 16:12:04.889487 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.889464    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4prp\" (UniqueName: \"kubernetes.io/projected/880bb567-ef75-4853-ad67-794058f47650-kube-api-access-t4prp\") pod \"llmisvc-controller-manager-7f6dfc774f-znr6c\" (UID: \"880bb567-ef75-4853-ad67-794058f47650\") " pod="kserve/llmisvc-controller-manager-7f6dfc774f-znr6c"
May 26 16:12:04.970051 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.969970    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qtj4t\" (UniqueName: \"kubernetes.io/projected/7232bab9-5261-4dc4-8fe6-c0bcbc19eec0-kube-api-access-qtj4t\") pod \"seaweedfs-79f457656b-xwhcb\" (UID: \"7232bab9-5261-4dc4-8fe6-c0bcbc19eec0\") " pod="kserve/seaweedfs-79f457656b-xwhcb"
May 26 16:12:04.970051 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.970019    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7232bab9-5261-4dc4-8fe6-c0bcbc19eec0-data\") pod \"seaweedfs-79f457656b-xwhcb\" (UID: \"7232bab9-5261-4dc4-8fe6-c0bcbc19eec0\") " pod="kserve/seaweedfs-79f457656b-xwhcb"
May 26 16:12:04.970363 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.970348    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7232bab9-5261-4dc4-8fe6-c0bcbc19eec0-data\") pod \"seaweedfs-79f457656b-xwhcb\" (UID: \"7232bab9-5261-4dc4-8fe6-c0bcbc19eec0\") " pod="kserve/seaweedfs-79f457656b-xwhcb"
May 26 16:12:04.980050 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.980030    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6f7545b5b9-nvv7f"
May 26 16:12:04.992977 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:04.992956    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtj4t\" (UniqueName: \"kubernetes.io/projected/7232bab9-5261-4dc4-8fe6-c0bcbc19eec0-kube-api-access-qtj4t\") pod \"seaweedfs-79f457656b-xwhcb\" (UID: \"7232bab9-5261-4dc4-8fe6-c0bcbc19eec0\") " pod="kserve/seaweedfs-79f457656b-xwhcb"
May 26 16:12:05.080339 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:05.080302    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-xwhcb"
May 26 16:12:05.130301 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:05.130236    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6f7545b5b9-nvv7f"]
May 26 16:12:05.132843 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:12:05.132816    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabc48fc2_b76c_4cdb_bdde_7239f5f9ab03.slice/crio-e9cecfacda1ae732a5711490c222d4a3043209c74edfd1d3959399308decf37f WatchSource:0}: Error finding container e9cecfacda1ae732a5711490c222d4a3043209c74edfd1d3959399308decf37f: Status 404 returned error can't find the container with id e9cecfacda1ae732a5711490c222d4a3043209c74edfd1d3959399308decf37f
May 26 16:12:05.229092 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:05.229063    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-xwhcb"]
May 26 16:12:05.229634 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:12:05.229604    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7232bab9_5261_4dc4_8fe6_c0bcbc19eec0.slice/crio-69e47999cfe97279fbc5e749ec737bc1e30d75a544b9968864cf2e05e709a006 WatchSource:0}: Error finding container 69e47999cfe97279fbc5e749ec737bc1e30d75a544b9968864cf2e05e709a006: Status 404 returned error can't find the container with id 69e47999cfe97279fbc5e749ec737bc1e30d75a544b9968864cf2e05e709a006
May 26 16:12:05.372999 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:05.372957    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/880bb567-ef75-4853-ad67-794058f47650-cert\") pod \"llmisvc-controller-manager-7f6dfc774f-znr6c\" (UID: \"880bb567-ef75-4853-ad67-794058f47650\") " pod="kserve/llmisvc-controller-manager-7f6dfc774f-znr6c"
May 26 16:12:05.375454 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:05.375435    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/880bb567-ef75-4853-ad67-794058f47650-cert\") pod \"llmisvc-controller-manager-7f6dfc774f-znr6c\" (UID: \"880bb567-ef75-4853-ad67-794058f47650\") " pod="kserve/llmisvc-controller-manager-7f6dfc774f-znr6c"
May 26 16:12:05.604783 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:05.604741    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-7f6dfc774f-znr6c"
May 26 16:12:05.748871 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:05.748800    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-7f6dfc774f-znr6c"]
May 26 16:12:05.753349 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:12:05.753318    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod880bb567_ef75_4853_ad67_794058f47650.slice/crio-d04c1a6805a2d4baf2acb971c8ea8a393285b5ad0cd03fb0e199c526d00802ff WatchSource:0}: Error finding container d04c1a6805a2d4baf2acb971c8ea8a393285b5ad0cd03fb0e199c526d00802ff: Status 404 returned error can't find the container with id d04c1a6805a2d4baf2acb971c8ea8a393285b5ad0cd03fb0e199c526d00802ff
May 26 16:12:05.845075 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:05.845023    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6f7545b5b9-nvv7f" event={"ID":"abc48fc2-b76c-4cdb-bdde-7239f5f9ab03","Type":"ContainerStarted","Data":"e9cecfacda1ae732a5711490c222d4a3043209c74edfd1d3959399308decf37f"}
May 26 16:12:05.846522 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:05.846494    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-7f6dfc774f-znr6c" event={"ID":"880bb567-ef75-4853-ad67-794058f47650","Type":"ContainerStarted","Data":"d04c1a6805a2d4baf2acb971c8ea8a393285b5ad0cd03fb0e199c526d00802ff"}
May 26 16:12:05.847834 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:05.847782    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-xwhcb" event={"ID":"7232bab9-5261-4dc4-8fe6-c0bcbc19eec0","Type":"ContainerStarted","Data":"69e47999cfe97279fbc5e749ec737bc1e30d75a544b9968864cf2e05e709a006"}
May 26 16:12:10.870115 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:10.870078    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6f7545b5b9-nvv7f" event={"ID":"abc48fc2-b76c-4cdb-bdde-7239f5f9ab03","Type":"ContainerStarted","Data":"51fa49880547b0c5971a29a12830c51cd38bee3ab4f349cbb20052ab23d18cc3"}
May 26 16:12:10.870578 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:10.870139    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6f7545b5b9-nvv7f"
May 26 16:12:10.871358 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:10.871337    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-7f6dfc774f-znr6c" event={"ID":"880bb567-ef75-4853-ad67-794058f47650","Type":"ContainerStarted","Data":"c2089bf3dcdd62483b5f929e34d9a378085ef4e1a336d58121490fc7aa1496b5"}
May 26 16:12:10.871488 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:10.871450    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-7f6dfc774f-znr6c"
May 26 16:12:10.872581 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:10.872561    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-xwhcb" event={"ID":"7232bab9-5261-4dc4-8fe6-c0bcbc19eec0","Type":"ContainerStarted","Data":"d7c772ce84ce44d33201395a998b41d1205943e810db4b9f252fd6616eeb2c2f"}
May 26 16:12:10.872687 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:10.872668    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-79f457656b-xwhcb"
May 26 16:12:10.895519 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:10.895452    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6f7545b5b9-nvv7f" podStartSLOduration=2.914481672 podStartE2EDuration="6.895437367s" podCreationTimestamp="2026-05-26 16:12:04 +0000 UTC" firstStartedPulling="2026-05-26 16:12:05.134351228 +0000 UTC m=+674.842806246" lastFinishedPulling="2026-05-26 16:12:09.115306909 +0000 UTC m=+678.823761941" observedRunningTime="2026-05-26 16:12:10.893221111 +0000 UTC m=+680.601676151" watchObservedRunningTime="2026-05-26 16:12:10.895437367 +0000 UTC m=+680.603892425"
May 26 16:12:10.913396 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:10.913346    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-7f6dfc774f-znr6c" podStartSLOduration=2.529075166 podStartE2EDuration="6.913330107s" podCreationTimestamp="2026-05-26 16:12:04 +0000 UTC" firstStartedPulling="2026-05-26 16:12:05.755314782 +0000 UTC m=+675.463769801" lastFinishedPulling="2026-05-26 16:12:10.13956972 +0000 UTC m=+679.848024742" observedRunningTime="2026-05-26 16:12:10.912141684 +0000 UTC m=+680.620596724" watchObservedRunningTime="2026-05-26 16:12:10.913330107 +0000 UTC m=+680.621785147"
May 26 16:12:10.934317 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:10.934269    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-79f457656b-xwhcb" podStartSLOduration=2.080618904 podStartE2EDuration="6.934255023s" podCreationTimestamp="2026-05-26 16:12:04 +0000 UTC" firstStartedPulling="2026-05-26 16:12:05.232715557 +0000 UTC m=+674.941170575" lastFinishedPulling="2026-05-26 16:12:10.086351656 +0000 UTC m=+679.794806694" observedRunningTime="2026-05-26 16:12:10.933457169 +0000 UTC m=+680.641912208" watchObservedRunningTime="2026-05-26 16:12:10.934255023 +0000 UTC m=+680.642710041"
May 26 16:12:16.878915 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:16.878882    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-79f457656b-xwhcb"
May 26 16:12:41.878271 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:41.878194    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-7f6dfc774f-znr6c"
May 26 16:12:41.881308 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:41.881287    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6f7545b5b9-nvv7f"
May 26 16:12:43.308115 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:43.308079    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6f7545b5b9-nvv7f"]
May 26 16:12:43.308504 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:43.308338    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve/kserve-controller-manager-6f7545b5b9-nvv7f" podUID="abc48fc2-b76c-4cdb-bdde-7239f5f9ab03" containerName="manager" containerID="cri-o://51fa49880547b0c5971a29a12830c51cd38bee3ab4f349cbb20052ab23d18cc3" gracePeriod=10
May 26 16:12:43.333842 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:43.333815    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6f7545b5b9-grtw8"]
May 26 16:12:43.335915 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:43.335900    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6f7545b5b9-grtw8"
May 26 16:12:43.345319 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:43.345289    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6f7545b5b9-grtw8"]
May 26 16:12:43.484622 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:43.484584    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9ef56ea-aca9-4eea-9cd9-7426c3a1c87c-cert\") pod \"kserve-controller-manager-6f7545b5b9-grtw8\" (UID: \"a9ef56ea-aca9-4eea-9cd9-7426c3a1c87c\") " pod="kserve/kserve-controller-manager-6f7545b5b9-grtw8"
May 26 16:12:43.484771 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:43.484639    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c999g\" (UniqueName: \"kubernetes.io/projected/a9ef56ea-aca9-4eea-9cd9-7426c3a1c87c-kube-api-access-c999g\") pod \"kserve-controller-manager-6f7545b5b9-grtw8\" (UID: \"a9ef56ea-aca9-4eea-9cd9-7426c3a1c87c\") " pod="kserve/kserve-controller-manager-6f7545b5b9-grtw8"
May 26 16:12:43.545084 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:43.545060    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6f7545b5b9-nvv7f"
May 26 16:12:43.585973 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:43.585856    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9ef56ea-aca9-4eea-9cd9-7426c3a1c87c-cert\") pod \"kserve-controller-manager-6f7545b5b9-grtw8\" (UID: \"a9ef56ea-aca9-4eea-9cd9-7426c3a1c87c\") " pod="kserve/kserve-controller-manager-6f7545b5b9-grtw8"
May 26 16:12:43.585973 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:43.585909    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c999g\" (UniqueName: \"kubernetes.io/projected/a9ef56ea-aca9-4eea-9cd9-7426c3a1c87c-kube-api-access-c999g\") pod \"kserve-controller-manager-6f7545b5b9-grtw8\" (UID: \"a9ef56ea-aca9-4eea-9cd9-7426c3a1c87c\") " pod="kserve/kserve-controller-manager-6f7545b5b9-grtw8"
May 26 16:12:43.588350 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:43.588320    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9ef56ea-aca9-4eea-9cd9-7426c3a1c87c-cert\") pod \"kserve-controller-manager-6f7545b5b9-grtw8\" (UID: \"a9ef56ea-aca9-4eea-9cd9-7426c3a1c87c\") " pod="kserve/kserve-controller-manager-6f7545b5b9-grtw8"
May 26 16:12:43.595383 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:43.595353    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c999g\" (UniqueName: \"kubernetes.io/projected/a9ef56ea-aca9-4eea-9cd9-7426c3a1c87c-kube-api-access-c999g\") pod \"kserve-controller-manager-6f7545b5b9-grtw8\" (UID: \"a9ef56ea-aca9-4eea-9cd9-7426c3a1c87c\") " pod="kserve/kserve-controller-manager-6f7545b5b9-grtw8"
May 26 16:12:43.686539 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:43.686498    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/abc48fc2-b76c-4cdb-bdde-7239f5f9ab03-cert\") pod \"abc48fc2-b76c-4cdb-bdde-7239f5f9ab03\" (UID: \"abc48fc2-b76c-4cdb-bdde-7239f5f9ab03\") "
May 26 16:12:43.686711 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:43.686581    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rvxg\" (UniqueName: \"kubernetes.io/projected/abc48fc2-b76c-4cdb-bdde-7239f5f9ab03-kube-api-access-2rvxg\") pod \"abc48fc2-b76c-4cdb-bdde-7239f5f9ab03\" (UID: \"abc48fc2-b76c-4cdb-bdde-7239f5f9ab03\") "
May 26 16:12:43.688731 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:43.688697    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abc48fc2-b76c-4cdb-bdde-7239f5f9ab03-cert" (OuterVolumeSpecName: "cert") pod "abc48fc2-b76c-4cdb-bdde-7239f5f9ab03" (UID: "abc48fc2-b76c-4cdb-bdde-7239f5f9ab03"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:12:43.688731 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:43.688697    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc48fc2-b76c-4cdb-bdde-7239f5f9ab03-kube-api-access-2rvxg" (OuterVolumeSpecName: "kube-api-access-2rvxg") pod "abc48fc2-b76c-4cdb-bdde-7239f5f9ab03" (UID: "abc48fc2-b76c-4cdb-bdde-7239f5f9ab03"). InnerVolumeSpecName "kube-api-access-2rvxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:12:43.692884 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:43.692866    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6f7545b5b9-grtw8"
May 26 16:12:43.787467 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:43.787436    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rvxg\" (UniqueName: \"kubernetes.io/projected/abc48fc2-b76c-4cdb-bdde-7239f5f9ab03-kube-api-access-2rvxg\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:12:43.787467 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:43.787463    2552 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/abc48fc2-b76c-4cdb-bdde-7239f5f9ab03-cert\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:12:43.821897 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:43.821871    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6f7545b5b9-grtw8"]
May 26 16:12:43.824535 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:12:43.824504    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9ef56ea_aca9_4eea_9cd9_7426c3a1c87c.slice/crio-d095e96c280c9872871f0a4cd0fc4bdfcd75a535a2575d725fc44ae3ef2f64b0 WatchSource:0}: Error finding container d095e96c280c9872871f0a4cd0fc4bdfcd75a535a2575d725fc44ae3ef2f64b0: Status 404 returned error can't find the container with id d095e96c280c9872871f0a4cd0fc4bdfcd75a535a2575d725fc44ae3ef2f64b0
May 26 16:12:43.974073 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:43.973987    2552 generic.go:358] "Generic (PLEG): container finished" podID="abc48fc2-b76c-4cdb-bdde-7239f5f9ab03" containerID="51fa49880547b0c5971a29a12830c51cd38bee3ab4f349cbb20052ab23d18cc3" exitCode=0
May 26 16:12:43.974073 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:43.974052    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6f7545b5b9-nvv7f"
May 26 16:12:43.974280 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:43.974070    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6f7545b5b9-nvv7f" event={"ID":"abc48fc2-b76c-4cdb-bdde-7239f5f9ab03","Type":"ContainerDied","Data":"51fa49880547b0c5971a29a12830c51cd38bee3ab4f349cbb20052ab23d18cc3"}
May 26 16:12:43.974280 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:43.974107    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6f7545b5b9-nvv7f" event={"ID":"abc48fc2-b76c-4cdb-bdde-7239f5f9ab03","Type":"ContainerDied","Data":"e9cecfacda1ae732a5711490c222d4a3043209c74edfd1d3959399308decf37f"}
May 26 16:12:43.974280 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:43.974126    2552 scope.go:117] "RemoveContainer" containerID="51fa49880547b0c5971a29a12830c51cd38bee3ab4f349cbb20052ab23d18cc3"
May 26 16:12:43.975257 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:43.975226    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6f7545b5b9-grtw8" event={"ID":"a9ef56ea-aca9-4eea-9cd9-7426c3a1c87c","Type":"ContainerStarted","Data":"d095e96c280c9872871f0a4cd0fc4bdfcd75a535a2575d725fc44ae3ef2f64b0"}
May 26 16:12:43.982134 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:43.982115    2552 scope.go:117] "RemoveContainer" containerID="51fa49880547b0c5971a29a12830c51cd38bee3ab4f349cbb20052ab23d18cc3"
May 26 16:12:43.982418 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:12:43.982396    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51fa49880547b0c5971a29a12830c51cd38bee3ab4f349cbb20052ab23d18cc3\": container with ID starting with 51fa49880547b0c5971a29a12830c51cd38bee3ab4f349cbb20052ab23d18cc3 not found: ID does not exist" containerID="51fa49880547b0c5971a29a12830c51cd38bee3ab4f349cbb20052ab23d18cc3"
May 26 16:12:43.982491 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:43.982427    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51fa49880547b0c5971a29a12830c51cd38bee3ab4f349cbb20052ab23d18cc3"} err="failed to get container status \"51fa49880547b0c5971a29a12830c51cd38bee3ab4f349cbb20052ab23d18cc3\": rpc error: code = NotFound desc = could not find container \"51fa49880547b0c5971a29a12830c51cd38bee3ab4f349cbb20052ab23d18cc3\": container with ID starting with 51fa49880547b0c5971a29a12830c51cd38bee3ab4f349cbb20052ab23d18cc3 not found: ID does not exist"
May 26 16:12:43.998022 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:43.997992    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6f7545b5b9-nvv7f"]
May 26 16:12:44.001965 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:44.001939    2552 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-6f7545b5b9-nvv7f"]
May 26 16:12:44.890024 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:44.889991    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abc48fc2-b76c-4cdb-bdde-7239f5f9ab03" path="/var/lib/kubelet/pods/abc48fc2-b76c-4cdb-bdde-7239f5f9ab03/volumes"
May 26 16:12:44.981862 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:44.981827    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6f7545b5b9-grtw8" event={"ID":"a9ef56ea-aca9-4eea-9cd9-7426c3a1c87c","Type":"ContainerStarted","Data":"0a8338713575367a108a89ace172e93845fea72ed895934ee6821e9b2a197727"}
May 26 16:12:44.982069 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:44.981906    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6f7545b5b9-grtw8"
May 26 16:12:45.001082 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:12:45.001023    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6f7545b5b9-grtw8" podStartSLOduration=1.606626766 podStartE2EDuration="2.001007083s" podCreationTimestamp="2026-05-26 16:12:43 +0000 UTC" firstStartedPulling="2026-05-26 16:12:43.825789064 +0000 UTC m=+713.534244085" lastFinishedPulling="2026-05-26 16:12:44.22016938 +0000 UTC m=+713.928624402" observedRunningTime="2026-05-26 16:12:44.999227022 +0000 UTC m=+714.707682064" watchObservedRunningTime="2026-05-26 16:12:45.001007083 +0000 UTC m=+714.709462131"
May 26 16:13:15.990876 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:15.990845    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6f7545b5b9-grtw8"
May 26 16:13:17.491733 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:17.491697    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-86f7b4b499-7pzn4"]
May 26 16:13:17.492198 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:17.492027    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abc48fc2-b76c-4cdb-bdde-7239f5f9ab03" containerName="manager"
May 26 16:13:17.492198 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:17.492040    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="abc48fc2-b76c-4cdb-bdde-7239f5f9ab03" containerName="manager"
May 26 16:13:17.492198 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:17.492108    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="abc48fc2-b76c-4cdb-bdde-7239f5f9ab03" containerName="manager"
May 26 16:13:17.495123 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:17.495102    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-7pzn4"
May 26 16:13:17.497602 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:17.497574    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
May 26 16:13:17.497730 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:17.497711    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-jbx9z\""
May 26 16:13:17.504943 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:17.504901    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-7pzn4"]
May 26 16:13:17.550323 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:17.550279    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42bvz\" (UniqueName: \"kubernetes.io/projected/e5076f02-1823-47b1-9c24-af72ea72ec6a-kube-api-access-42bvz\") pod \"model-serving-api-86f7b4b499-7pzn4\" (UID: \"e5076f02-1823-47b1-9c24-af72ea72ec6a\") " pod="kserve/model-serving-api-86f7b4b499-7pzn4"
May 26 16:13:17.550489 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:17.550382    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5076f02-1823-47b1-9c24-af72ea72ec6a-tls-certs\") pod \"model-serving-api-86f7b4b499-7pzn4\" (UID: \"e5076f02-1823-47b1-9c24-af72ea72ec6a\") " pod="kserve/model-serving-api-86f7b4b499-7pzn4"
May 26 16:13:17.651645 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:17.651604    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5076f02-1823-47b1-9c24-af72ea72ec6a-tls-certs\") pod \"model-serving-api-86f7b4b499-7pzn4\" (UID: \"e5076f02-1823-47b1-9c24-af72ea72ec6a\") " pod="kserve/model-serving-api-86f7b4b499-7pzn4"
May 26 16:13:17.651866 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:17.651682    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-42bvz\" (UniqueName: \"kubernetes.io/projected/e5076f02-1823-47b1-9c24-af72ea72ec6a-kube-api-access-42bvz\") pod \"model-serving-api-86f7b4b499-7pzn4\" (UID: \"e5076f02-1823-47b1-9c24-af72ea72ec6a\") " pod="kserve/model-serving-api-86f7b4b499-7pzn4"
May 26 16:13:17.651866 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:13:17.651775    2552 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
May 26 16:13:17.652016 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:13:17.651876    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5076f02-1823-47b1-9c24-af72ea72ec6a-tls-certs podName:e5076f02-1823-47b1-9c24-af72ea72ec6a nodeName:}" failed. No retries permitted until 2026-05-26 16:13:18.151853926 +0000 UTC m=+747.860308944 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e5076f02-1823-47b1-9c24-af72ea72ec6a-tls-certs") pod "model-serving-api-86f7b4b499-7pzn4" (UID: "e5076f02-1823-47b1-9c24-af72ea72ec6a") : secret "model-serving-api-tls" not found
May 26 16:13:17.661754 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:17.661724    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42bvz\" (UniqueName: \"kubernetes.io/projected/e5076f02-1823-47b1-9c24-af72ea72ec6a-kube-api-access-42bvz\") pod \"model-serving-api-86f7b4b499-7pzn4\" (UID: \"e5076f02-1823-47b1-9c24-af72ea72ec6a\") " pod="kserve/model-serving-api-86f7b4b499-7pzn4"
May 26 16:13:18.155545 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:18.155505    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5076f02-1823-47b1-9c24-af72ea72ec6a-tls-certs\") pod \"model-serving-api-86f7b4b499-7pzn4\" (UID: \"e5076f02-1823-47b1-9c24-af72ea72ec6a\") " pod="kserve/model-serving-api-86f7b4b499-7pzn4"
May 26 16:13:18.157914 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:18.157890    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5076f02-1823-47b1-9c24-af72ea72ec6a-tls-certs\") pod \"model-serving-api-86f7b4b499-7pzn4\" (UID: \"e5076f02-1823-47b1-9c24-af72ea72ec6a\") " pod="kserve/model-serving-api-86f7b4b499-7pzn4"
May 26 16:13:18.406827 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:18.406724    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-7pzn4"
May 26 16:13:18.540328 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:18.540271    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-7pzn4"]
May 26 16:13:18.543151 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:13:18.543120    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5076f02_1823_47b1_9c24_af72ea72ec6a.slice/crio-6c32fc72a9e97be40ac3effeb40ceb1c42facc1137e9d0e4dd9d73c74bc5a2a7 WatchSource:0}: Error finding container 6c32fc72a9e97be40ac3effeb40ceb1c42facc1137e9d0e4dd9d73c74bc5a2a7: Status 404 returned error can't find the container with id 6c32fc72a9e97be40ac3effeb40ceb1c42facc1137e9d0e4dd9d73c74bc5a2a7
May 26 16:13:18.544958 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:18.544945    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:13:19.092317 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:19.092281    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-7pzn4" event={"ID":"e5076f02-1823-47b1-9c24-af72ea72ec6a","Type":"ContainerStarted","Data":"6c32fc72a9e97be40ac3effeb40ceb1c42facc1137e9d0e4dd9d73c74bc5a2a7"}
May 26 16:13:21.100800 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:21.100760    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-7pzn4" event={"ID":"e5076f02-1823-47b1-9c24-af72ea72ec6a","Type":"ContainerStarted","Data":"c8c5e82c64310e993d23c3bc354ac5922f13e59381e7d0d3f627905e8dc3e050"}
May 26 16:13:21.101261 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:21.100891    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-86f7b4b499-7pzn4"
May 26 16:13:21.118612 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:21.118554    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/model-serving-api-86f7b4b499-7pzn4" podStartSLOduration=1.782285081 podStartE2EDuration="4.118538783s" podCreationTimestamp="2026-05-26 16:13:17 +0000 UTC" firstStartedPulling="2026-05-26 16:13:18.545076731 +0000 UTC m=+748.253531748" lastFinishedPulling="2026-05-26 16:13:20.881330418 +0000 UTC m=+750.589785450" observedRunningTime="2026-05-26 16:13:21.117447885 +0000 UTC m=+750.825902940" watchObservedRunningTime="2026-05-26 16:13:21.118538783 +0000 UTC m=+750.826993823"
May 26 16:13:32.107996 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:32.107959    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-86f7b4b499-7pzn4"
May 26 16:13:53.990897 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:53.990853    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw"]
May 26 16:13:53.994448 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:53.994424    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw"
May 26 16:13:53.997065 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:53.997037    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
May 26 16:13:53.997234 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:53.997037    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-rdp2t\""
May 26 16:13:53.997234 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:53.997050    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 26 16:13:53.997234 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:53.997072    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 26 16:13:54.004322 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:54.004291    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw"]
May 26 16:13:54.150483 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:54.150446    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1f769c9-1a3d-4212-8317-9b4b357feb0e-workload-certs\") pod \"router-gateway-1-openshift-default-7b9cf87876-6gstw\" (UID: \"d1f769c9-1a3d-4212-8317-9b4b357feb0e\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw"
May 26 16:13:54.150483 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:54.150487    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2nxp\" (UniqueName: \"kubernetes.io/projected/d1f769c9-1a3d-4212-8317-9b4b357feb0e-kube-api-access-j2nxp\") pod \"router-gateway-1-openshift-default-7b9cf87876-6gstw\" (UID: \"d1f769c9-1a3d-4212-8317-9b4b357feb0e\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw"
May 26 16:13:54.150758 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:54.150514    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f769c9-1a3d-4212-8317-9b4b357feb0e-workload-socket\") pod \"router-gateway-1-openshift-default-7b9cf87876-6gstw\" (UID: \"d1f769c9-1a3d-4212-8317-9b4b357feb0e\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw"
May 26 16:13:54.150758 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:54.150631    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f769c9-1a3d-4212-8317-9b4b357feb0e-credential-socket\") pod \"router-gateway-1-openshift-default-7b9cf87876-6gstw\" (UID: \"d1f769c9-1a3d-4212-8317-9b4b357feb0e\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw"
May 26 16:13:54.150758 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:54.150688    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1f769c9-1a3d-4212-8317-9b4b357feb0e-istio-envoy\") pod \"router-gateway-1-openshift-default-7b9cf87876-6gstw\" (UID: \"d1f769c9-1a3d-4212-8317-9b4b357feb0e\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw"
May 26 16:13:54.150758 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:54.150718    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1f769c9-1a3d-4212-8317-9b4b357feb0e-istio-token\") pod \"router-gateway-1-openshift-default-7b9cf87876-6gstw\" (UID: \"d1f769c9-1a3d-4212-8317-9b4b357feb0e\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw"
May 26 16:13:54.150964 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:54.150762    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1f769c9-1a3d-4212-8317-9b4b357feb0e-istio-data\") pod \"router-gateway-1-openshift-default-7b9cf87876-6gstw\" (UID: \"d1f769c9-1a3d-4212-8317-9b4b357feb0e\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw"
May 26 16:13:54.150964 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:54.150797    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1f769c9-1a3d-4212-8317-9b4b357feb0e-istio-podinfo\") pod \"router-gateway-1-openshift-default-7b9cf87876-6gstw\" (UID: \"d1f769c9-1a3d-4212-8317-9b4b357feb0e\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw"
May 26 16:13:54.150964 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:54.150848    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1f769c9-1a3d-4212-8317-9b4b357feb0e-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7b9cf87876-6gstw\" (UID: \"d1f769c9-1a3d-4212-8317-9b4b357feb0e\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw"
May 26 16:13:54.252328 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:54.252230    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1f769c9-1a3d-4212-8317-9b4b357feb0e-istio-envoy\") pod \"router-gateway-1-openshift-default-7b9cf87876-6gstw\" (UID: \"d1f769c9-1a3d-4212-8317-9b4b357feb0e\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw"
May 26 16:13:54.252328 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:54.252275    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1f769c9-1a3d-4212-8317-9b4b357feb0e-istio-token\") pod \"router-gateway-1-openshift-default-7b9cf87876-6gstw\" (UID: \"d1f769c9-1a3d-4212-8317-9b4b357feb0e\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw"
May 26 16:13:54.252553 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:54.252332    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1f769c9-1a3d-4212-8317-9b4b357feb0e-istio-data\") pod \"router-gateway-1-openshift-default-7b9cf87876-6gstw\" (UID: \"d1f769c9-1a3d-4212-8317-9b4b357feb0e\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw"
May 26 16:13:54.252553 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:54.252366    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1f769c9-1a3d-4212-8317-9b4b357feb0e-istio-podinfo\") pod \"router-gateway-1-openshift-default-7b9cf87876-6gstw\" (UID: \"d1f769c9-1a3d-4212-8317-9b4b357feb0e\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw"
May 26 16:13:54.252553 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:54.252395    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1f769c9-1a3d-4212-8317-9b4b357feb0e-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7b9cf87876-6gstw\" (UID: \"d1f769c9-1a3d-4212-8317-9b4b357feb0e\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw"
May 26 16:13:54.252553 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:54.252444    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1f769c9-1a3d-4212-8317-9b4b357feb0e-workload-certs\") pod \"router-gateway-1-openshift-default-7b9cf87876-6gstw\" (UID: \"d1f769c9-1a3d-4212-8317-9b4b357feb0e\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw"
May 26 16:13:54.252553 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:54.252469    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j2nxp\" (UniqueName: \"kubernetes.io/projected/d1f769c9-1a3d-4212-8317-9b4b357feb0e-kube-api-access-j2nxp\") pod \"router-gateway-1-openshift-default-7b9cf87876-6gstw\" (UID: \"d1f769c9-1a3d-4212-8317-9b4b357feb0e\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw"
May 26 16:13:54.252553 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:54.252498    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f769c9-1a3d-4212-8317-9b4b357feb0e-workload-socket\") pod \"router-gateway-1-openshift-default-7b9cf87876-6gstw\" (UID: \"d1f769c9-1a3d-4212-8317-9b4b357feb0e\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw"
May 26 16:13:54.252553 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:54.252547    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f769c9-1a3d-4212-8317-9b4b357feb0e-credential-socket\") pod \"router-gateway-1-openshift-default-7b9cf87876-6gstw\" (UID: \"d1f769c9-1a3d-4212-8317-9b4b357feb0e\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw"
May 26 16:13:54.252904 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:54.252879    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1f769c9-1a3d-4212-8317-9b4b357feb0e-workload-certs\") pod \"router-gateway-1-openshift-default-7b9cf87876-6gstw\" (UID: \"d1f769c9-1a3d-4212-8317-9b4b357feb0e\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw"
May 26 16:13:54.253006 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:54.252975    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f769c9-1a3d-4212-8317-9b4b357feb0e-credential-socket\") pod \"router-gateway-1-openshift-default-7b9cf87876-6gstw\" (UID: \"d1f769c9-1a3d-4212-8317-9b4b357feb0e\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw"
May 26 16:13:54.253065 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:54.253031    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f769c9-1a3d-4212-8317-9b4b357feb0e-workload-socket\") pod \"router-gateway-1-openshift-default-7b9cf87876-6gstw\" (UID: \"d1f769c9-1a3d-4212-8317-9b4b357feb0e\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw"
May 26 16:13:54.253111 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:54.253076    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1f769c9-1a3d-4212-8317-9b4b357feb0e-istio-data\") pod \"router-gateway-1-openshift-default-7b9cf87876-6gstw\" (UID: \"d1f769c9-1a3d-4212-8317-9b4b357feb0e\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw"
May 26 16:13:54.253402 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:54.253380    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1f769c9-1a3d-4212-8317-9b4b357feb0e-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7b9cf87876-6gstw\" (UID: \"d1f769c9-1a3d-4212-8317-9b4b357feb0e\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw"
May 26 16:13:54.255070 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:54.255051    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1f769c9-1a3d-4212-8317-9b4b357feb0e-istio-envoy\") pod \"router-gateway-1-openshift-default-7b9cf87876-6gstw\" (UID: \"d1f769c9-1a3d-4212-8317-9b4b357feb0e\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw"
May 26 16:13:54.255396 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:54.255374    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1f769c9-1a3d-4212-8317-9b4b357feb0e-istio-podinfo\") pod \"router-gateway-1-openshift-default-7b9cf87876-6gstw\" (UID: \"d1f769c9-1a3d-4212-8317-9b4b357feb0e\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw"
May 26 16:13:54.261520 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:54.261494    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1f769c9-1a3d-4212-8317-9b4b357feb0e-istio-token\") pod \"router-gateway-1-openshift-default-7b9cf87876-6gstw\" (UID: \"d1f769c9-1a3d-4212-8317-9b4b357feb0e\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw"
May 26 16:13:54.261676 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:54.261657    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2nxp\" (UniqueName: \"kubernetes.io/projected/d1f769c9-1a3d-4212-8317-9b4b357feb0e-kube-api-access-j2nxp\") pod \"router-gateway-1-openshift-default-7b9cf87876-6gstw\" (UID: \"d1f769c9-1a3d-4212-8317-9b4b357feb0e\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw"
May 26 16:13:54.307348 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:54.307306    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw"
May 26 16:13:54.443090 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:54.442916    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw"]
May 26 16:13:54.445657 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:13:54.445622    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1f769c9_1a3d_4212_8317_9b4b357feb0e.slice/crio-7703c0eab91635ae2a609c24e26530d1975c6bad26851d57cccb0a04bee94658 WatchSource:0}: Error finding container 7703c0eab91635ae2a609c24e26530d1975c6bad26851d57cccb0a04bee94658: Status 404 returned error can't find the container with id 7703c0eab91635ae2a609c24e26530d1975c6bad26851d57cccb0a04bee94658
May 26 16:13:54.447878 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:54.447840    2552 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
May 26 16:13:54.447980 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:54.447915    2552 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
May 26 16:13:54.447980 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:54.447965    2552 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
May 26 16:13:55.214307 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:55.214267    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw" event={"ID":"d1f769c9-1a3d-4212-8317-9b4b357feb0e","Type":"ContainerStarted","Data":"4a2bf2d75d690325704a21f93aa55a6d8de82a77be69349fb8e688503ac8934d"}
May 26 16:13:55.214307 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:55.214311    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw" event={"ID":"d1f769c9-1a3d-4212-8317-9b4b357feb0e","Type":"ContainerStarted","Data":"7703c0eab91635ae2a609c24e26530d1975c6bad26851d57cccb0a04bee94658"}
May 26 16:13:55.235389 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:55.235336    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw" podStartSLOduration=2.235318099 podStartE2EDuration="2.235318099s" podCreationTimestamp="2026-05-26 16:13:53 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:13:55.233373009 +0000 UTC m=+784.941828048" watchObservedRunningTime="2026-05-26 16:13:55.235318099 +0000 UTC m=+784.943773140"
May 26 16:13:55.308209 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:55.308168    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw"
May 26 16:13:55.313108 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:55.313081    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw"
May 26 16:13:56.217738 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:56.217701    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw"
May 26 16:13:56.218809 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:13:56.218789    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-6gstw"
May 26 16:14:23.086892 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:23.086806    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg"]
May 26 16:14:23.090943 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:23.090904    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg"
May 26 16:14:23.094482 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:23.094450    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
May 26 16:14:23.094644 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:23.094461    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8m4rr\""
May 26 16:14:23.105098 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:23.105068    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg"]
May 26 16:14:23.197702 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:23.197664    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5890149-4981-4a42-85c8-a3cf3d293f29-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg\" (UID: \"e5890149-4981-4a42-85c8-a3cf3d293f29\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg"
May 26 16:14:23.197702 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:23.197704    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5890149-4981-4a42-85c8-a3cf3d293f29-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg\" (UID: \"e5890149-4981-4a42-85c8-a3cf3d293f29\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg"
May 26 16:14:23.197951 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:23.197725    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5890149-4981-4a42-85c8-a3cf3d293f29-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg\" (UID: \"e5890149-4981-4a42-85c8-a3cf3d293f29\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg"
May 26 16:14:23.197951 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:23.197808    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5890149-4981-4a42-85c8-a3cf3d293f29-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg\" (UID: \"e5890149-4981-4a42-85c8-a3cf3d293f29\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg"
May 26 16:14:23.197951 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:23.197844    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5890149-4981-4a42-85c8-a3cf3d293f29-kserve-provision-location\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg\" (UID: \"e5890149-4981-4a42-85c8-a3cf3d293f29\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg"
May 26 16:14:23.197951 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:23.197906    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bst9\" (UniqueName: \"kubernetes.io/projected/e5890149-4981-4a42-85c8-a3cf3d293f29-kube-api-access-6bst9\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg\" (UID: \"e5890149-4981-4a42-85c8-a3cf3d293f29\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg"
May 26 16:14:23.198101 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:23.197970    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5890149-4981-4a42-85c8-a3cf3d293f29-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg\" (UID: \"e5890149-4981-4a42-85c8-a3cf3d293f29\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg"
May 26 16:14:23.298984 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:23.298916    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6bst9\" (UniqueName: \"kubernetes.io/projected/e5890149-4981-4a42-85c8-a3cf3d293f29-kube-api-access-6bst9\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg\" (UID: \"e5890149-4981-4a42-85c8-a3cf3d293f29\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg"
May 26 16:14:23.299161 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:23.299014    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5890149-4981-4a42-85c8-a3cf3d293f29-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg\" (UID: \"e5890149-4981-4a42-85c8-a3cf3d293f29\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg"
May 26 16:14:23.299161 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:23.299069    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5890149-4981-4a42-85c8-a3cf3d293f29-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg\" (UID: \"e5890149-4981-4a42-85c8-a3cf3d293f29\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg"
May 26 16:14:23.299161 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:23.299117    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5890149-4981-4a42-85c8-a3cf3d293f29-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg\" (UID: \"e5890149-4981-4a42-85c8-a3cf3d293f29\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg"
May 26 16:14:23.299161 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:23.299159    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5890149-4981-4a42-85c8-a3cf3d293f29-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg\" (UID: \"e5890149-4981-4a42-85c8-a3cf3d293f29\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg"
May 26 16:14:23.299356 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:23.299186    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5890149-4981-4a42-85c8-a3cf3d293f29-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg\" (UID: \"e5890149-4981-4a42-85c8-a3cf3d293f29\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg"
May 26 16:14:23.299356 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:23.299223    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5890149-4981-4a42-85c8-a3cf3d293f29-kserve-provision-location\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg\" (UID: \"e5890149-4981-4a42-85c8-a3cf3d293f29\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg"
May 26 16:14:23.299641 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:23.299609    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5890149-4981-4a42-85c8-a3cf3d293f29-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg\" (UID: \"e5890149-4981-4a42-85c8-a3cf3d293f29\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg"
May 26 16:14:23.299641 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:23.299630    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5890149-4981-4a42-85c8-a3cf3d293f29-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg\" (UID: \"e5890149-4981-4a42-85c8-a3cf3d293f29\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg"
May 26 16:14:23.299890 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:23.299688    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5890149-4981-4a42-85c8-a3cf3d293f29-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg\" (UID: \"e5890149-4981-4a42-85c8-a3cf3d293f29\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg"
May 26 16:14:23.299890 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:23.299726    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5890149-4981-4a42-85c8-a3cf3d293f29-kserve-provision-location\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg\" (UID: \"e5890149-4981-4a42-85c8-a3cf3d293f29\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg"
May 26 16:14:23.301544 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:23.301519    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5890149-4981-4a42-85c8-a3cf3d293f29-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg\" (UID: \"e5890149-4981-4a42-85c8-a3cf3d293f29\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg"
May 26 16:14:23.301872 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:23.301850    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5890149-4981-4a42-85c8-a3cf3d293f29-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg\" (UID: \"e5890149-4981-4a42-85c8-a3cf3d293f29\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg"
May 26 16:14:23.306824 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:23.306802    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bst9\" (UniqueName: \"kubernetes.io/projected/e5890149-4981-4a42-85c8-a3cf3d293f29-kube-api-access-6bst9\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg\" (UID: \"e5890149-4981-4a42-85c8-a3cf3d293f29\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg"
May 26 16:14:23.402075 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:23.401984    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg"
May 26 16:14:23.537571 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:23.537515    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg"]
May 26 16:14:23.540474 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:14:23.540444    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5890149_4981_4a42_85c8_a3cf3d293f29.slice/crio-7831ad3fe8003df4a01a78512accc269778d7be1844eff2d90f2bfd0f0c50419 WatchSource:0}: Error finding container 7831ad3fe8003df4a01a78512accc269778d7be1844eff2d90f2bfd0f0c50419: Status 404 returned error can't find the container with id 7831ad3fe8003df4a01a78512accc269778d7be1844eff2d90f2bfd0f0c50419
May 26 16:14:24.308695 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:24.308651    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg" event={"ID":"e5890149-4981-4a42-85c8-a3cf3d293f29","Type":"ContainerStarted","Data":"7831ad3fe8003df4a01a78512accc269778d7be1844eff2d90f2bfd0f0c50419"}
May 26 16:14:27.321070 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:27.321029    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg" event={"ID":"e5890149-4981-4a42-85c8-a3cf3d293f29","Type":"ContainerStarted","Data":"97ab06796c1cbca1ac6f92554033f6c4f9ec18b1b92f53557f7b9149b1d95033"}
May 26 16:14:40.367217 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:40.367179    2552 generic.go:358] "Generic (PLEG): container finished" podID="e5890149-4981-4a42-85c8-a3cf3d293f29" containerID="97ab06796c1cbca1ac6f92554033f6c4f9ec18b1b92f53557f7b9149b1d95033" exitCode=0
May 26 16:14:40.367595 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:40.367233    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg" event={"ID":"e5890149-4981-4a42-85c8-a3cf3d293f29","Type":"ContainerDied","Data":"97ab06796c1cbca1ac6f92554033f6c4f9ec18b1b92f53557f7b9149b1d95033"}
May 26 16:14:42.375219 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:42.375181    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg" event={"ID":"e5890149-4981-4a42-85c8-a3cf3d293f29","Type":"ContainerStarted","Data":"cafd4bcf55255e0e00c768f9b10fd7530071a0c4b7c0c3111bf94d9c3d598588"}
May 26 16:14:42.395656 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:42.395596    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg" podStartSLOduration=1.076112139 podStartE2EDuration="19.395580347s" podCreationTimestamp="2026-05-26 16:14:23 +0000 UTC" firstStartedPulling="2026-05-26 16:14:23.542207412 +0000 UTC m=+813.250662430" lastFinishedPulling="2026-05-26 16:14:41.861675616 +0000 UTC m=+831.570130638" observedRunningTime="2026-05-26 16:14:42.394565779 +0000 UTC m=+832.103020819" watchObservedRunningTime="2026-05-26 16:14:42.395580347 +0000 UTC m=+832.104035387"
May 26 16:14:43.402725 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:43.402682    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg"
May 26 16:14:43.402725 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:43.402731    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg"
May 26 16:14:43.415149 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:43.415116    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg"
May 26 16:14:44.392089 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:44.392061    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg"
May 26 16:14:48.741444 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:48.741410    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg"]
May 26 16:14:48.741888 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:48.741683    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg" podUID="e5890149-4981-4a42-85c8-a3cf3d293f29" containerName="main" containerID="cri-o://cafd4bcf55255e0e00c768f9b10fd7530071a0c4b7c0c3111bf94d9c3d598588" gracePeriod=30
May 26 16:14:49.007450 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:49.007425    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg"
May 26 16:14:49.129788 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:49.129745    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5890149-4981-4a42-85c8-a3cf3d293f29-kserve-provision-location\") pod \"e5890149-4981-4a42-85c8-a3cf3d293f29\" (UID: \"e5890149-4981-4a42-85c8-a3cf3d293f29\") "
May 26 16:14:49.129788 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:49.129791    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bst9\" (UniqueName: \"kubernetes.io/projected/e5890149-4981-4a42-85c8-a3cf3d293f29-kube-api-access-6bst9\") pod \"e5890149-4981-4a42-85c8-a3cf3d293f29\" (UID: \"e5890149-4981-4a42-85c8-a3cf3d293f29\") "
May 26 16:14:49.130053 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:49.129814    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5890149-4981-4a42-85c8-a3cf3d293f29-model-cache\") pod \"e5890149-4981-4a42-85c8-a3cf3d293f29\" (UID: \"e5890149-4981-4a42-85c8-a3cf3d293f29\") "
May 26 16:14:49.130053 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:49.129853    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5890149-4981-4a42-85c8-a3cf3d293f29-dshm\") pod \"e5890149-4981-4a42-85c8-a3cf3d293f29\" (UID: \"e5890149-4981-4a42-85c8-a3cf3d293f29\") "
May 26 16:14:49.130053 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:49.129873    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5890149-4981-4a42-85c8-a3cf3d293f29-home\") pod \"e5890149-4981-4a42-85c8-a3cf3d293f29\" (UID: \"e5890149-4981-4a42-85c8-a3cf3d293f29\") "
May 26 16:14:49.130053 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:49.129893    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5890149-4981-4a42-85c8-a3cf3d293f29-tmp-dir\") pod \"e5890149-4981-4a42-85c8-a3cf3d293f29\" (UID: \"e5890149-4981-4a42-85c8-a3cf3d293f29\") "
May 26 16:14:49.130053 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:49.129953    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5890149-4981-4a42-85c8-a3cf3d293f29-tls-certs\") pod \"e5890149-4981-4a42-85c8-a3cf3d293f29\" (UID: \"e5890149-4981-4a42-85c8-a3cf3d293f29\") "
May 26 16:14:49.130269 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:49.130103    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5890149-4981-4a42-85c8-a3cf3d293f29-model-cache" (OuterVolumeSpecName: "model-cache") pod "e5890149-4981-4a42-85c8-a3cf3d293f29" (UID: "e5890149-4981-4a42-85c8-a3cf3d293f29"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:14:49.130269 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:49.130247    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5890149-4981-4a42-85c8-a3cf3d293f29-model-cache\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:14:49.130624 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:49.130490    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5890149-4981-4a42-85c8-a3cf3d293f29-home" (OuterVolumeSpecName: "home") pod "e5890149-4981-4a42-85c8-a3cf3d293f29" (UID: "e5890149-4981-4a42-85c8-a3cf3d293f29"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:14:49.130624 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:49.130582    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5890149-4981-4a42-85c8-a3cf3d293f29-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e5890149-4981-4a42-85c8-a3cf3d293f29" (UID: "e5890149-4981-4a42-85c8-a3cf3d293f29"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:14:49.132279 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:49.132252    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5890149-4981-4a42-85c8-a3cf3d293f29-kube-api-access-6bst9" (OuterVolumeSpecName: "kube-api-access-6bst9") pod "e5890149-4981-4a42-85c8-a3cf3d293f29" (UID: "e5890149-4981-4a42-85c8-a3cf3d293f29"). InnerVolumeSpecName "kube-api-access-6bst9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:14:49.132366 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:49.132252    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5890149-4981-4a42-85c8-a3cf3d293f29-dshm" (OuterVolumeSpecName: "dshm") pod "e5890149-4981-4a42-85c8-a3cf3d293f29" (UID: "e5890149-4981-4a42-85c8-a3cf3d293f29"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:14:49.132534 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:49.132509    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5890149-4981-4a42-85c8-a3cf3d293f29-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e5890149-4981-4a42-85c8-a3cf3d293f29" (UID: "e5890149-4981-4a42-85c8-a3cf3d293f29"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:14:49.194851 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:49.194794    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5890149-4981-4a42-85c8-a3cf3d293f29-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e5890149-4981-4a42-85c8-a3cf3d293f29" (UID: "e5890149-4981-4a42-85c8-a3cf3d293f29"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:14:49.231597 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:49.231559    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5890149-4981-4a42-85c8-a3cf3d293f29-dshm\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:14:49.231597 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:49.231593    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5890149-4981-4a42-85c8-a3cf3d293f29-home\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:14:49.231597 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:49.231606    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5890149-4981-4a42-85c8-a3cf3d293f29-tmp-dir\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:14:49.231825 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:49.231618    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5890149-4981-4a42-85c8-a3cf3d293f29-tls-certs\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:14:49.231825 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:49.231632    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5890149-4981-4a42-85c8-a3cf3d293f29-kserve-provision-location\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:14:49.231825 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:49.231644    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bst9\" (UniqueName: \"kubernetes.io/projected/e5890149-4981-4a42-85c8-a3cf3d293f29-kube-api-access-6bst9\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:14:49.398841 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:49.398800    2552 generic.go:358] "Generic (PLEG): container finished" podID="e5890149-4981-4a42-85c8-a3cf3d293f29" containerID="cafd4bcf55255e0e00c768f9b10fd7530071a0c4b7c0c3111bf94d9c3d598588" exitCode=0
May 26 16:14:49.399041 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:49.398849    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg" event={"ID":"e5890149-4981-4a42-85c8-a3cf3d293f29","Type":"ContainerDied","Data":"cafd4bcf55255e0e00c768f9b10fd7530071a0c4b7c0c3111bf94d9c3d598588"}
May 26 16:14:49.399041 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:49.398879    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg" event={"ID":"e5890149-4981-4a42-85c8-a3cf3d293f29","Type":"ContainerDied","Data":"7831ad3fe8003df4a01a78512accc269778d7be1844eff2d90f2bfd0f0c50419"}
May 26 16:14:49.399041 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:49.398883    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg"
May 26 16:14:49.399041 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:49.398899    2552 scope.go:117] "RemoveContainer" containerID="cafd4bcf55255e0e00c768f9b10fd7530071a0c4b7c0c3111bf94d9c3d598588"
May 26 16:14:49.407626 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:49.407609    2552 scope.go:117] "RemoveContainer" containerID="97ab06796c1cbca1ac6f92554033f6c4f9ec18b1b92f53557f7b9149b1d95033"
May 26 16:14:49.420864 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:49.420832    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg"]
May 26 16:14:49.428597 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:49.428553    2552 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-c95845974-dqlvg"]
May 26 16:14:49.474107 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:49.474086    2552 scope.go:117] "RemoveContainer" containerID="cafd4bcf55255e0e00c768f9b10fd7530071a0c4b7c0c3111bf94d9c3d598588"
May 26 16:14:49.474437 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:14:49.474416    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cafd4bcf55255e0e00c768f9b10fd7530071a0c4b7c0c3111bf94d9c3d598588\": container with ID starting with cafd4bcf55255e0e00c768f9b10fd7530071a0c4b7c0c3111bf94d9c3d598588 not found: ID does not exist" containerID="cafd4bcf55255e0e00c768f9b10fd7530071a0c4b7c0c3111bf94d9c3d598588"
May 26 16:14:49.474485 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:49.474449    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cafd4bcf55255e0e00c768f9b10fd7530071a0c4b7c0c3111bf94d9c3d598588"} err="failed to get container status \"cafd4bcf55255e0e00c768f9b10fd7530071a0c4b7c0c3111bf94d9c3d598588\": rpc error: code = NotFound desc = could not find container \"cafd4bcf55255e0e00c768f9b10fd7530071a0c4b7c0c3111bf94d9c3d598588\": container with ID starting with cafd4bcf55255e0e00c768f9b10fd7530071a0c4b7c0c3111bf94d9c3d598588 not found: ID does not exist"
May 26 16:14:49.474485 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:49.474474    2552 scope.go:117] "RemoveContainer" containerID="97ab06796c1cbca1ac6f92554033f6c4f9ec18b1b92f53557f7b9149b1d95033"
May 26 16:14:49.474751 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:14:49.474727    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97ab06796c1cbca1ac6f92554033f6c4f9ec18b1b92f53557f7b9149b1d95033\": container with ID starting with 97ab06796c1cbca1ac6f92554033f6c4f9ec18b1b92f53557f7b9149b1d95033 not found: ID does not exist" containerID="97ab06796c1cbca1ac6f92554033f6c4f9ec18b1b92f53557f7b9149b1d95033"
May 26 16:14:49.474812 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:49.474758    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97ab06796c1cbca1ac6f92554033f6c4f9ec18b1b92f53557f7b9149b1d95033"} err="failed to get container status \"97ab06796c1cbca1ac6f92554033f6c4f9ec18b1b92f53557f7b9149b1d95033\": rpc error: code = NotFound desc = could not find container \"97ab06796c1cbca1ac6f92554033f6c4f9ec18b1b92f53557f7b9149b1d95033\": container with ID starting with 97ab06796c1cbca1ac6f92554033f6c4f9ec18b1b92f53557f7b9149b1d95033 not found: ID does not exist"
May 26 16:14:50.889771 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:50.889742    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5890149-4981-4a42-85c8-a3cf3d293f29" path="/var/lib/kubelet/pods/e5890149-4981-4a42-85c8-a3cf3d293f29/volumes"
May 26 16:14:59.501139 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:59.501104    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q"]
May 26 16:14:59.501571 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:59.501407    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5890149-4981-4a42-85c8-a3cf3d293f29" containerName="main"
May 26 16:14:59.501571 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:59.501418    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5890149-4981-4a42-85c8-a3cf3d293f29" containerName="main"
May 26 16:14:59.501571 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:59.501439    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5890149-4981-4a42-85c8-a3cf3d293f29" containerName="storage-initializer"
May 26 16:14:59.501571 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:59.501444    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5890149-4981-4a42-85c8-a3cf3d293f29" containerName="storage-initializer"
May 26 16:14:59.501571 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:59.501498    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="e5890149-4981-4a42-85c8-a3cf3d293f29" containerName="main"
May 26 16:14:59.504990 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:59.504966    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q"
May 26 16:14:59.507761 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:59.507733    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
May 26 16:14:59.507907 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:59.507830    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8m4rr\""
May 26 16:14:59.516391 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:59.516367    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q"]
May 26 16:14:59.621951 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:59.621892    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a346f731-06cd-44ee-846b-233fd8c183a6-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q\" (UID: \"a346f731-06cd-44ee-846b-233fd8c183a6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q"
May 26 16:14:59.622110 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:59.622003    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a346f731-06cd-44ee-846b-233fd8c183a6-kserve-provision-location\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q\" (UID: \"a346f731-06cd-44ee-846b-233fd8c183a6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q"
May 26 16:14:59.622110 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:59.622091    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a346f731-06cd-44ee-846b-233fd8c183a6-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q\" (UID: \"a346f731-06cd-44ee-846b-233fd8c183a6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q"
May 26 16:14:59.622184 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:59.622144    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a346f731-06cd-44ee-846b-233fd8c183a6-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q\" (UID: \"a346f731-06cd-44ee-846b-233fd8c183a6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q"
May 26 16:14:59.622184 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:59.622168    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4q7p\" (UniqueName: \"kubernetes.io/projected/a346f731-06cd-44ee-846b-233fd8c183a6-kube-api-access-x4q7p\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q\" (UID: \"a346f731-06cd-44ee-846b-233fd8c183a6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q"
May 26 16:14:59.622289 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:59.622255    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a346f731-06cd-44ee-846b-233fd8c183a6-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q\" (UID: \"a346f731-06cd-44ee-846b-233fd8c183a6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q"
May 26 16:14:59.622342 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:59.622301    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a346f731-06cd-44ee-846b-233fd8c183a6-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q\" (UID: \"a346f731-06cd-44ee-846b-233fd8c183a6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q"
May 26 16:14:59.723507 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:59.723469    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a346f731-06cd-44ee-846b-233fd8c183a6-kserve-provision-location\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q\" (UID: \"a346f731-06cd-44ee-846b-233fd8c183a6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q"
May 26 16:14:59.723708 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:59.723538    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a346f731-06cd-44ee-846b-233fd8c183a6-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q\" (UID: \"a346f731-06cd-44ee-846b-233fd8c183a6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q"
May 26 16:14:59.723708 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:59.723579    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a346f731-06cd-44ee-846b-233fd8c183a6-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q\" (UID: \"a346f731-06cd-44ee-846b-233fd8c183a6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q"
May 26 16:14:59.723708 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:59.723608    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x4q7p\" (UniqueName: \"kubernetes.io/projected/a346f731-06cd-44ee-846b-233fd8c183a6-kube-api-access-x4q7p\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q\" (UID: \"a346f731-06cd-44ee-846b-233fd8c183a6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q"
May 26 16:14:59.723708 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:59.723635    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a346f731-06cd-44ee-846b-233fd8c183a6-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q\" (UID: \"a346f731-06cd-44ee-846b-233fd8c183a6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q"
May 26 16:14:59.723708 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:59.723669    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a346f731-06cd-44ee-846b-233fd8c183a6-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q\" (UID: \"a346f731-06cd-44ee-846b-233fd8c183a6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q"
May 26 16:14:59.724018 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:59.723708    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a346f731-06cd-44ee-846b-233fd8c183a6-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q\" (UID: \"a346f731-06cd-44ee-846b-233fd8c183a6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q"
May 26 16:14:59.724018 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:59.723969    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a346f731-06cd-44ee-846b-233fd8c183a6-kserve-provision-location\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q\" (UID: \"a346f731-06cd-44ee-846b-233fd8c183a6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q"
May 26 16:14:59.724129 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:59.724040    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a346f731-06cd-44ee-846b-233fd8c183a6-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q\" (UID: \"a346f731-06cd-44ee-846b-233fd8c183a6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q"
May 26 16:14:59.724181 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:59.724143    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a346f731-06cd-44ee-846b-233fd8c183a6-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q\" (UID: \"a346f731-06cd-44ee-846b-233fd8c183a6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q"
May 26 16:14:59.724181 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:59.724163    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a346f731-06cd-44ee-846b-233fd8c183a6-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q\" (UID: \"a346f731-06cd-44ee-846b-233fd8c183a6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q"
May 26 16:14:59.725909 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:59.725887    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a346f731-06cd-44ee-846b-233fd8c183a6-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q\" (UID: \"a346f731-06cd-44ee-846b-233fd8c183a6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q"
May 26 16:14:59.726180 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:59.726159    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a346f731-06cd-44ee-846b-233fd8c183a6-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q\" (UID: \"a346f731-06cd-44ee-846b-233fd8c183a6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q"
May 26 16:14:59.732281 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:59.732257    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4q7p\" (UniqueName: \"kubernetes.io/projected/a346f731-06cd-44ee-846b-233fd8c183a6-kube-api-access-x4q7p\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q\" (UID: \"a346f731-06cd-44ee-846b-233fd8c183a6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q"
May 26 16:14:59.816194 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:59.816160    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q"
May 26 16:14:59.951770 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:14:59.951597    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q"]
May 26 16:14:59.954828 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:14:59.954788    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda346f731_06cd_44ee_846b_233fd8c183a6.slice/crio-3af7eac395f123b26ce02f402f3207f0fbb91811861750c5e9207b31ba4916d7 WatchSource:0}: Error finding container 3af7eac395f123b26ce02f402f3207f0fbb91811861750c5e9207b31ba4916d7: Status 404 returned error can't find the container with id 3af7eac395f123b26ce02f402f3207f0fbb91811861750c5e9207b31ba4916d7
May 26 16:15:00.435283 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:00.435249    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q" event={"ID":"a346f731-06cd-44ee-846b-233fd8c183a6","Type":"ContainerStarted","Data":"c765288a367479eba1f51c7f554721835b6fd99e89770e61a7422826a191a2af"}
May 26 16:15:00.435283 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:00.435287    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q" event={"ID":"a346f731-06cd-44ee-846b-233fd8c183a6","Type":"ContainerStarted","Data":"3af7eac395f123b26ce02f402f3207f0fbb91811861750c5e9207b31ba4916d7"}
May 26 16:15:02.390815 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:02.390781    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q"]
May 26 16:15:02.444016 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:02.443913    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q" podUID="a346f731-06cd-44ee-846b-233fd8c183a6" containerName="storage-initializer" containerID="cri-o://c765288a367479eba1f51c7f554721835b6fd99e89770e61a7422826a191a2af" gracePeriod=30
May 26 16:15:32.548593 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:32.548562    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q_a346f731-06cd-44ee-846b-233fd8c183a6/storage-initializer/0.log"
May 26 16:15:32.548976 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:32.548615    2552 generic.go:358] "Generic (PLEG): container finished" podID="a346f731-06cd-44ee-846b-233fd8c183a6" containerID="c765288a367479eba1f51c7f554721835b6fd99e89770e61a7422826a191a2af" exitCode=137
May 26 16:15:32.548976 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:32.548696    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q" event={"ID":"a346f731-06cd-44ee-846b-233fd8c183a6","Type":"ContainerDied","Data":"c765288a367479eba1f51c7f554721835b6fd99e89770e61a7422826a191a2af"}
May 26 16:15:32.636818 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:32.636791    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q_a346f731-06cd-44ee-846b-233fd8c183a6/storage-initializer/0.log"
May 26 16:15:32.636976 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:32.636869    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q"
May 26 16:15:32.698417 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:32.698320    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a346f731-06cd-44ee-846b-233fd8c183a6-dshm\") pod \"a346f731-06cd-44ee-846b-233fd8c183a6\" (UID: \"a346f731-06cd-44ee-846b-233fd8c183a6\") "
May 26 16:15:32.698417 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:32.698365    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4q7p\" (UniqueName: \"kubernetes.io/projected/a346f731-06cd-44ee-846b-233fd8c183a6-kube-api-access-x4q7p\") pod \"a346f731-06cd-44ee-846b-233fd8c183a6\" (UID: \"a346f731-06cd-44ee-846b-233fd8c183a6\") "
May 26 16:15:32.698417 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:32.698403    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a346f731-06cd-44ee-846b-233fd8c183a6-tmp-dir\") pod \"a346f731-06cd-44ee-846b-233fd8c183a6\" (UID: \"a346f731-06cd-44ee-846b-233fd8c183a6\") "
May 26 16:15:32.698765 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:32.698425    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a346f731-06cd-44ee-846b-233fd8c183a6-tls-certs\") pod \"a346f731-06cd-44ee-846b-233fd8c183a6\" (UID: \"a346f731-06cd-44ee-846b-233fd8c183a6\") "
May 26 16:15:32.698765 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:32.698443    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a346f731-06cd-44ee-846b-233fd8c183a6-home\") pod \"a346f731-06cd-44ee-846b-233fd8c183a6\" (UID: \"a346f731-06cd-44ee-846b-233fd8c183a6\") "
May 26 16:15:32.698765 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:32.698459    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a346f731-06cd-44ee-846b-233fd8c183a6-model-cache\") pod \"a346f731-06cd-44ee-846b-233fd8c183a6\" (UID: \"a346f731-06cd-44ee-846b-233fd8c183a6\") "
May 26 16:15:32.698765 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:32.698525    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a346f731-06cd-44ee-846b-233fd8c183a6-kserve-provision-location\") pod \"a346f731-06cd-44ee-846b-233fd8c183a6\" (UID: \"a346f731-06cd-44ee-846b-233fd8c183a6\") "
May 26 16:15:32.698765 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:32.698716    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a346f731-06cd-44ee-846b-233fd8c183a6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a346f731-06cd-44ee-846b-233fd8c183a6" (UID: "a346f731-06cd-44ee-846b-233fd8c183a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:15:32.698765 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:32.698735    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a346f731-06cd-44ee-846b-233fd8c183a6-model-cache" (OuterVolumeSpecName: "model-cache") pod "a346f731-06cd-44ee-846b-233fd8c183a6" (UID: "a346f731-06cd-44ee-846b-233fd8c183a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:15:32.699274 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:32.699243    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a346f731-06cd-44ee-846b-233fd8c183a6-home" (OuterVolumeSpecName: "home") pod "a346f731-06cd-44ee-846b-233fd8c183a6" (UID: "a346f731-06cd-44ee-846b-233fd8c183a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:15:32.700539 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:32.700518    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a346f731-06cd-44ee-846b-233fd8c183a6-kube-api-access-x4q7p" (OuterVolumeSpecName: "kube-api-access-x4q7p") pod "a346f731-06cd-44ee-846b-233fd8c183a6" (UID: "a346f731-06cd-44ee-846b-233fd8c183a6"). InnerVolumeSpecName "kube-api-access-x4q7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:15:32.700953 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:32.700914    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a346f731-06cd-44ee-846b-233fd8c183a6-dshm" (OuterVolumeSpecName: "dshm") pod "a346f731-06cd-44ee-846b-233fd8c183a6" (UID: "a346f731-06cd-44ee-846b-233fd8c183a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:15:32.701024 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:32.700957    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a346f731-06cd-44ee-846b-233fd8c183a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a346f731-06cd-44ee-846b-233fd8c183a6" (UID: "a346f731-06cd-44ee-846b-233fd8c183a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:15:32.735124 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:32.735074    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a346f731-06cd-44ee-846b-233fd8c183a6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a346f731-06cd-44ee-846b-233fd8c183a6" (UID: "a346f731-06cd-44ee-846b-233fd8c183a6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:15:32.799838 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:32.799800    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4q7p\" (UniqueName: \"kubernetes.io/projected/a346f731-06cd-44ee-846b-233fd8c183a6-kube-api-access-x4q7p\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:15:32.799838 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:32.799832    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a346f731-06cd-44ee-846b-233fd8c183a6-tmp-dir\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:15:32.799838 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:32.799842    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a346f731-06cd-44ee-846b-233fd8c183a6-tls-certs\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:15:32.799838 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:32.799851    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a346f731-06cd-44ee-846b-233fd8c183a6-home\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:15:32.800143 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:32.799860    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a346f731-06cd-44ee-846b-233fd8c183a6-model-cache\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:15:32.800143 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:32.799869    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a346f731-06cd-44ee-846b-233fd8c183a6-kserve-provision-location\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:15:32.800143 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:32.799877    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a346f731-06cd-44ee-846b-233fd8c183a6-dshm\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:15:33.553234 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:33.553205    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q_a346f731-06cd-44ee-846b-233fd8c183a6/storage-initializer/0.log"
May 26 16:15:33.553709 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:33.553258    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q" event={"ID":"a346f731-06cd-44ee-846b-233fd8c183a6","Type":"ContainerDied","Data":"3af7eac395f123b26ce02f402f3207f0fbb91811861750c5e9207b31ba4916d7"}
May 26 16:15:33.553709 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:33.553293    2552 scope.go:117] "RemoveContainer" containerID="c765288a367479eba1f51c7f554721835b6fd99e89770e61a7422826a191a2af"
May 26 16:15:33.553709 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:33.553339    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q"
May 26 16:15:33.586958 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:33.586908    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q"]
May 26 16:15:33.590993 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:33.590963    2552 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-69d69bf4df6vq5q"]
May 26 16:15:34.890296 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:34.890261    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a346f731-06cd-44ee-846b-233fd8c183a6" path="/var/lib/kubelet/pods/a346f731-06cd-44ee-846b-233fd8c183a6/volumes"
May 26 16:15:35.725505 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:35.725465    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks"]
May 26 16:15:35.725904 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:35.725884    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a346f731-06cd-44ee-846b-233fd8c183a6" containerName="storage-initializer"
May 26 16:15:35.725904 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:35.725903    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a346f731-06cd-44ee-846b-233fd8c183a6" containerName="storage-initializer"
May 26 16:15:35.726071 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:35.725980    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="a346f731-06cd-44ee-846b-233fd8c183a6" containerName="storage-initializer"
May 26 16:15:35.729441 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:35.729418    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks"
May 26 16:15:35.740100 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:35.740076    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs\""
May 26 16:15:35.740212 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:35.740107    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8m4rr\""
May 26 16:15:35.752961 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:35.752896    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks"]
May 26 16:15:35.824278 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:35.824233    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/880d495e-c7eb-48d0-8a28-544a6274be52-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks\" (UID: \"880d495e-c7eb-48d0-8a28-544a6274be52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks"
May 26 16:15:35.824471 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:35.824295    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzqnp\" (UniqueName: \"kubernetes.io/projected/880d495e-c7eb-48d0-8a28-544a6274be52-kube-api-access-pzqnp\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks\" (UID: \"880d495e-c7eb-48d0-8a28-544a6274be52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks"
May 26 16:15:35.824471 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:35.824342    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/880d495e-c7eb-48d0-8a28-544a6274be52-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks\" (UID: \"880d495e-c7eb-48d0-8a28-544a6274be52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks"
May 26 16:15:35.824471 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:35.824407    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/880d495e-c7eb-48d0-8a28-544a6274be52-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks\" (UID: \"880d495e-c7eb-48d0-8a28-544a6274be52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks"
May 26 16:15:35.824471 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:35.824440    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/880d495e-c7eb-48d0-8a28-544a6274be52-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks\" (UID: \"880d495e-c7eb-48d0-8a28-544a6274be52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks"
May 26 16:15:35.824643 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:35.824476    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/880d495e-c7eb-48d0-8a28-544a6274be52-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks\" (UID: \"880d495e-c7eb-48d0-8a28-544a6274be52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks"
May 26 16:15:35.824643 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:35.824506    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/880d495e-c7eb-48d0-8a28-544a6274be52-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks\" (UID: \"880d495e-c7eb-48d0-8a28-544a6274be52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks"
May 26 16:15:35.925676 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:35.925628    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/880d495e-c7eb-48d0-8a28-544a6274be52-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks\" (UID: \"880d495e-c7eb-48d0-8a28-544a6274be52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks"
May 26 16:15:35.925676 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:35.925684    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pzqnp\" (UniqueName: \"kubernetes.io/projected/880d495e-c7eb-48d0-8a28-544a6274be52-kube-api-access-pzqnp\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks\" (UID: \"880d495e-c7eb-48d0-8a28-544a6274be52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks"
May 26 16:15:35.926238 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:35.925713    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/880d495e-c7eb-48d0-8a28-544a6274be52-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks\" (UID: \"880d495e-c7eb-48d0-8a28-544a6274be52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks"
May 26 16:15:35.926238 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:35.925733    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/880d495e-c7eb-48d0-8a28-544a6274be52-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks\" (UID: \"880d495e-c7eb-48d0-8a28-544a6274be52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks"
May 26 16:15:35.926238 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:35.925749    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/880d495e-c7eb-48d0-8a28-544a6274be52-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks\" (UID: \"880d495e-c7eb-48d0-8a28-544a6274be52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks"
May 26 16:15:35.926238 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:35.925771    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/880d495e-c7eb-48d0-8a28-544a6274be52-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks\" (UID: \"880d495e-c7eb-48d0-8a28-544a6274be52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks"
May 26 16:15:35.926238 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:35.925791    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/880d495e-c7eb-48d0-8a28-544a6274be52-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks\" (UID: \"880d495e-c7eb-48d0-8a28-544a6274be52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks"
May 26 16:15:35.926238 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:35.926140    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/880d495e-c7eb-48d0-8a28-544a6274be52-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks\" (UID: \"880d495e-c7eb-48d0-8a28-544a6274be52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks"
May 26 16:15:35.926238 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:35.926179    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/880d495e-c7eb-48d0-8a28-544a6274be52-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks\" (UID: \"880d495e-c7eb-48d0-8a28-544a6274be52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks"
May 26 16:15:35.926238 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:35.926204    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/880d495e-c7eb-48d0-8a28-544a6274be52-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks\" (UID: \"880d495e-c7eb-48d0-8a28-544a6274be52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks"
May 26 16:15:35.926550 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:35.926285    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/880d495e-c7eb-48d0-8a28-544a6274be52-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks\" (UID: \"880d495e-c7eb-48d0-8a28-544a6274be52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks"
May 26 16:15:35.928138 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:35.928110    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/880d495e-c7eb-48d0-8a28-544a6274be52-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks\" (UID: \"880d495e-c7eb-48d0-8a28-544a6274be52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks"
May 26 16:15:35.928380 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:35.928363    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/880d495e-c7eb-48d0-8a28-544a6274be52-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks\" (UID: \"880d495e-c7eb-48d0-8a28-544a6274be52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks"
May 26 16:15:35.938160 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:35.938135    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzqnp\" (UniqueName: \"kubernetes.io/projected/880d495e-c7eb-48d0-8a28-544a6274be52-kube-api-access-pzqnp\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks\" (UID: \"880d495e-c7eb-48d0-8a28-544a6274be52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks"
May 26 16:15:36.040394 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:36.040360    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks"
May 26 16:15:36.190268 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:36.190238    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks"]
May 26 16:15:36.193335 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:15:36.193301    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod880d495e_c7eb_48d0_8a28_544a6274be52.slice/crio-11966ab400fdda5be1c82f92b43fa6790fc773f88d31ee1752e95b516dfb54f8 WatchSource:0}: Error finding container 11966ab400fdda5be1c82f92b43fa6790fc773f88d31ee1752e95b516dfb54f8: Status 404 returned error can't find the container with id 11966ab400fdda5be1c82f92b43fa6790fc773f88d31ee1752e95b516dfb54f8
May 26 16:15:36.566866 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:36.566830    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks" event={"ID":"880d495e-c7eb-48d0-8a28-544a6274be52","Type":"ContainerStarted","Data":"a87b477addb4d743c296edf5e3dd2373da72e3879a88e9b5e4b0b510382a5eaf"}
May 26 16:15:36.566866 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:36.566868    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks" event={"ID":"880d495e-c7eb-48d0-8a28-544a6274be52","Type":"ContainerStarted","Data":"11966ab400fdda5be1c82f92b43fa6790fc773f88d31ee1752e95b516dfb54f8"}
May 26 16:15:42.589979 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:42.589919    2552 generic.go:358] "Generic (PLEG): container finished" podID="880d495e-c7eb-48d0-8a28-544a6274be52" containerID="a87b477addb4d743c296edf5e3dd2373da72e3879a88e9b5e4b0b510382a5eaf" exitCode=0
May 26 16:15:42.590346 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:42.589998    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks" event={"ID":"880d495e-c7eb-48d0-8a28-544a6274be52","Type":"ContainerDied","Data":"a87b477addb4d743c296edf5e3dd2373da72e3879a88e9b5e4b0b510382a5eaf"}
May 26 16:15:43.595522 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:43.595489    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks" event={"ID":"880d495e-c7eb-48d0-8a28-544a6274be52","Type":"ContainerStarted","Data":"0ff297d22b8f30f9b21a54fcb22bc17f548f07b908c56bcd912a532ea80ad91c"}
May 26 16:15:43.617818 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:43.617760    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks" podStartSLOduration=8.617745653 podStartE2EDuration="8.617745653s" podCreationTimestamp="2026-05-26 16:15:35 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:15:43.616258575 +0000 UTC m=+893.324713617" watchObservedRunningTime="2026-05-26 16:15:43.617745653 +0000 UTC m=+893.326200750"
May 26 16:15:46.041546 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:46.041504    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks"
May 26 16:15:46.042046 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:46.041651    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks"
May 26 16:15:46.054209 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:46.054181    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks"
May 26 16:15:46.616402 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:46.616373    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks"
May 26 16:15:50.816400 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:50.816363    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sj6db_5c0f9fc6-2bd6-4370-96d0-5ae315363c7f/ovn-acl-logging/0.log"
May 26 16:15:50.817018 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:15:50.816997    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sj6db_5c0f9fc6-2bd6-4370-96d0-5ae315363c7f/ovn-acl-logging/0.log"
May 26 16:16:17.410419 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:17.410382    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks"]
May 26 16:16:17.411067 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:17.410762    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks" podUID="880d495e-c7eb-48d0-8a28-544a6274be52" containerName="main" containerID="cri-o://0ff297d22b8f30f9b21a54fcb22bc17f548f07b908c56bcd912a532ea80ad91c" gracePeriod=30
May 26 16:16:17.661979 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:17.661884    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks"
May 26 16:16:17.671437 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:17.671410    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/880d495e-c7eb-48d0-8a28-544a6274be52-home\") pod \"880d495e-c7eb-48d0-8a28-544a6274be52\" (UID: \"880d495e-c7eb-48d0-8a28-544a6274be52\") "
May 26 16:16:17.671582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:17.671445    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzqnp\" (UniqueName: \"kubernetes.io/projected/880d495e-c7eb-48d0-8a28-544a6274be52-kube-api-access-pzqnp\") pod \"880d495e-c7eb-48d0-8a28-544a6274be52\" (UID: \"880d495e-c7eb-48d0-8a28-544a6274be52\") "
May 26 16:16:17.671582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:17.671465    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/880d495e-c7eb-48d0-8a28-544a6274be52-tls-certs\") pod \"880d495e-c7eb-48d0-8a28-544a6274be52\" (UID: \"880d495e-c7eb-48d0-8a28-544a6274be52\") "
May 26 16:16:17.671582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:17.671510    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/880d495e-c7eb-48d0-8a28-544a6274be52-model-cache\") pod \"880d495e-c7eb-48d0-8a28-544a6274be52\" (UID: \"880d495e-c7eb-48d0-8a28-544a6274be52\") "
May 26 16:16:17.671582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:17.671564    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/880d495e-c7eb-48d0-8a28-544a6274be52-kserve-provision-location\") pod \"880d495e-c7eb-48d0-8a28-544a6274be52\" (UID: \"880d495e-c7eb-48d0-8a28-544a6274be52\") "
May 26 16:16:17.671769 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:17.671598    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/880d495e-c7eb-48d0-8a28-544a6274be52-tmp-dir\") pod \"880d495e-c7eb-48d0-8a28-544a6274be52\" (UID: \"880d495e-c7eb-48d0-8a28-544a6274be52\") "
May 26 16:16:17.671769 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:17.671643    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/880d495e-c7eb-48d0-8a28-544a6274be52-dshm\") pod \"880d495e-c7eb-48d0-8a28-544a6274be52\" (UID: \"880d495e-c7eb-48d0-8a28-544a6274be52\") "
May 26 16:16:17.671769 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:17.671717    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880d495e-c7eb-48d0-8a28-544a6274be52-home" (OuterVolumeSpecName: "home") pod "880d495e-c7eb-48d0-8a28-544a6274be52" (UID: "880d495e-c7eb-48d0-8a28-544a6274be52"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:16:17.671947 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:17.671796    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880d495e-c7eb-48d0-8a28-544a6274be52-model-cache" (OuterVolumeSpecName: "model-cache") pod "880d495e-c7eb-48d0-8a28-544a6274be52" (UID: "880d495e-c7eb-48d0-8a28-544a6274be52"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:16:17.671947 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:17.671906    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880d495e-c7eb-48d0-8a28-544a6274be52-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "880d495e-c7eb-48d0-8a28-544a6274be52" (UID: "880d495e-c7eb-48d0-8a28-544a6274be52"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:16:17.672062 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:17.671923    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/880d495e-c7eb-48d0-8a28-544a6274be52-model-cache\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:16:17.672062 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:17.671970    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/880d495e-c7eb-48d0-8a28-544a6274be52-home\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:16:17.673797 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:17.673766    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/880d495e-c7eb-48d0-8a28-544a6274be52-kube-api-access-pzqnp" (OuterVolumeSpecName: "kube-api-access-pzqnp") pod "880d495e-c7eb-48d0-8a28-544a6274be52" (UID: "880d495e-c7eb-48d0-8a28-544a6274be52"). InnerVolumeSpecName "kube-api-access-pzqnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:16:17.674042 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:17.674024    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880d495e-c7eb-48d0-8a28-544a6274be52-dshm" (OuterVolumeSpecName: "dshm") pod "880d495e-c7eb-48d0-8a28-544a6274be52" (UID: "880d495e-c7eb-48d0-8a28-544a6274be52"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:16:17.674131 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:17.674060    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/880d495e-c7eb-48d0-8a28-544a6274be52-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "880d495e-c7eb-48d0-8a28-544a6274be52" (UID: "880d495e-c7eb-48d0-8a28-544a6274be52"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:16:17.713542 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:17.713507    2552 generic.go:358] "Generic (PLEG): container finished" podID="880d495e-c7eb-48d0-8a28-544a6274be52" containerID="0ff297d22b8f30f9b21a54fcb22bc17f548f07b908c56bcd912a532ea80ad91c" exitCode=0
May 26 16:16:17.713706 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:17.713583    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks" event={"ID":"880d495e-c7eb-48d0-8a28-544a6274be52","Type":"ContainerDied","Data":"0ff297d22b8f30f9b21a54fcb22bc17f548f07b908c56bcd912a532ea80ad91c"}
May 26 16:16:17.713706 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:17.713636    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks" event={"ID":"880d495e-c7eb-48d0-8a28-544a6274be52","Type":"ContainerDied","Data":"11966ab400fdda5be1c82f92b43fa6790fc773f88d31ee1752e95b516dfb54f8"}
May 26 16:16:17.713706 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:17.713640    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks"
May 26 16:16:17.713886 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:17.713658    2552 scope.go:117] "RemoveContainer" containerID="0ff297d22b8f30f9b21a54fcb22bc17f548f07b908c56bcd912a532ea80ad91c"
May 26 16:16:17.723005 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:17.722977    2552 scope.go:117] "RemoveContainer" containerID="a87b477addb4d743c296edf5e3dd2373da72e3879a88e9b5e4b0b510382a5eaf"
May 26 16:16:17.743883 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:17.743838    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880d495e-c7eb-48d0-8a28-544a6274be52-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "880d495e-c7eb-48d0-8a28-544a6274be52" (UID: "880d495e-c7eb-48d0-8a28-544a6274be52"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:16:17.773212 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:17.773177    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/880d495e-c7eb-48d0-8a28-544a6274be52-kserve-provision-location\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:16:17.773212 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:17.773211    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/880d495e-c7eb-48d0-8a28-544a6274be52-tmp-dir\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:16:17.773212 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:17.773222    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/880d495e-c7eb-48d0-8a28-544a6274be52-dshm\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:16:17.773435 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:17.773233    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzqnp\" (UniqueName: \"kubernetes.io/projected/880d495e-c7eb-48d0-8a28-544a6274be52-kube-api-access-pzqnp\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:16:17.773435 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:17.773242    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/880d495e-c7eb-48d0-8a28-544a6274be52-tls-certs\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:16:17.789209 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:17.789188    2552 scope.go:117] "RemoveContainer" containerID="0ff297d22b8f30f9b21a54fcb22bc17f548f07b908c56bcd912a532ea80ad91c"
May 26 16:16:17.789501 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:16:17.789482    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ff297d22b8f30f9b21a54fcb22bc17f548f07b908c56bcd912a532ea80ad91c\": container with ID starting with 0ff297d22b8f30f9b21a54fcb22bc17f548f07b908c56bcd912a532ea80ad91c not found: ID does not exist" containerID="0ff297d22b8f30f9b21a54fcb22bc17f548f07b908c56bcd912a532ea80ad91c"
May 26 16:16:17.789547 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:17.789515    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff297d22b8f30f9b21a54fcb22bc17f548f07b908c56bcd912a532ea80ad91c"} err="failed to get container status \"0ff297d22b8f30f9b21a54fcb22bc17f548f07b908c56bcd912a532ea80ad91c\": rpc error: code = NotFound desc = could not find container \"0ff297d22b8f30f9b21a54fcb22bc17f548f07b908c56bcd912a532ea80ad91c\": container with ID starting with 0ff297d22b8f30f9b21a54fcb22bc17f548f07b908c56bcd912a532ea80ad91c not found: ID does not exist"
May 26 16:16:17.789547 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:17.789536    2552 scope.go:117] "RemoveContainer" containerID="a87b477addb4d743c296edf5e3dd2373da72e3879a88e9b5e4b0b510382a5eaf"
May 26 16:16:17.789832 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:16:17.789809    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a87b477addb4d743c296edf5e3dd2373da72e3879a88e9b5e4b0b510382a5eaf\": container with ID starting with a87b477addb4d743c296edf5e3dd2373da72e3879a88e9b5e4b0b510382a5eaf not found: ID does not exist" containerID="a87b477addb4d743c296edf5e3dd2373da72e3879a88e9b5e4b0b510382a5eaf"
May 26 16:16:17.789893 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:17.789840    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a87b477addb4d743c296edf5e3dd2373da72e3879a88e9b5e4b0b510382a5eaf"} err="failed to get container status \"a87b477addb4d743c296edf5e3dd2373da72e3879a88e9b5e4b0b510382a5eaf\": rpc error: code = NotFound desc = could not find container \"a87b477addb4d743c296edf5e3dd2373da72e3879a88e9b5e4b0b510382a5eaf\": container with ID starting with a87b477addb4d743c296edf5e3dd2373da72e3879a88e9b5e4b0b510382a5eaf not found: ID does not exist"
May 26 16:16:18.034276 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:18.034247    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks"]
May 26 16:16:18.037214 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:18.037188    2552 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6df6c7c777jj5ks"]
May 26 16:16:18.889851 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:18.889812    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="880d495e-c7eb-48d0-8a28-544a6274be52" path="/var/lib/kubelet/pods/880d495e-c7eb-48d0-8a28-544a6274be52/volumes"
May 26 16:16:32.612456 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:32.612424    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g"]
May 26 16:16:32.613841 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:32.612749    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="880d495e-c7eb-48d0-8a28-544a6274be52" containerName="storage-initializer"
May 26 16:16:32.613841 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:32.612760    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="880d495e-c7eb-48d0-8a28-544a6274be52" containerName="storage-initializer"
May 26 16:16:32.613841 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:32.612778    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="880d495e-c7eb-48d0-8a28-544a6274be52" containerName="main"
May 26 16:16:32.613841 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:32.612783    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="880d495e-c7eb-48d0-8a28-544a6274be52" containerName="main"
May 26 16:16:32.613841 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:32.612837    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="880d495e-c7eb-48d0-8a28-544a6274be52" containerName="main"
May 26 16:16:32.620669 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:32.620633    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g"
May 26 16:16:32.623496 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:32.623467    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvdde380eaa9fe1facad32d45131f9e34d-kserve-self-signed-certs\""
May 26 16:16:32.623638 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:32.623517    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8m4rr\""
May 26 16:16:32.626433 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:32.626410    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g"]
May 26 16:16:32.707096 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:32.707051    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g\" (UID: \"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g"
May 26 16:16:32.707096 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:32.707097    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47b95\" (UniqueName: \"kubernetes.io/projected/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-kube-api-access-47b95\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g\" (UID: \"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g"
May 26 16:16:32.707362 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:32.707118    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g\" (UID: \"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g"
May 26 16:16:32.707362 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:32.707136    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g\" (UID: \"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g"
May 26 16:16:32.707362 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:32.707179    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g\" (UID: \"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g"
May 26 16:16:32.707362 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:32.707314    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g\" (UID: \"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g"
May 26 16:16:32.707537 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:32.707397    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g\" (UID: \"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g"
May 26 16:16:32.808558 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:32.808512    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g\" (UID: \"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g"
May 26 16:16:32.808752 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:32.808611    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g\" (UID: \"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g"
May 26 16:16:32.808752 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:32.808650    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g\" (UID: \"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g"
May 26 16:16:32.808752 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:32.808679    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-47b95\" (UniqueName: \"kubernetes.io/projected/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-kube-api-access-47b95\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g\" (UID: \"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g"
May 26 16:16:32.808752 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:32.808708    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g\" (UID: \"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g"
May 26 16:16:32.808752 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:32.808739    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g\" (UID: \"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g"
May 26 16:16:32.809024 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:32.808780    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g\" (UID: \"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g"
May 26 16:16:32.809084 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:32.809018    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g\" (UID: \"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g"
May 26 16:16:32.809129 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:32.809089    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g\" (UID: \"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g"
May 26 16:16:32.809324 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:32.809301    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g\" (UID: \"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g"
May 26 16:16:32.809464 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:32.809332    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g\" (UID: \"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g"
May 26 16:16:32.810842 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:32.810817    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g\" (UID: \"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g"
May 26 16:16:32.811117 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:32.811098    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g\" (UID: \"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g"
May 26 16:16:32.818562 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:32.818536    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47b95\" (UniqueName: \"kubernetes.io/projected/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-kube-api-access-47b95\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g\" (UID: \"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g"
May 26 16:16:32.934550 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:32.934449    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g"
May 26 16:16:33.077434 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:33.077407    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g"]
May 26 16:16:33.080093 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:16:33.080065    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode01c7cba_a6f3_4c45_ac04_a58cd7bb25bb.slice/crio-5bf99150bbb069c0c9d7f206aaae8a5c6195161abcf03234323e016ccaabfa10 WatchSource:0}: Error finding container 5bf99150bbb069c0c9d7f206aaae8a5c6195161abcf03234323e016ccaabfa10: Status 404 returned error can't find the container with id 5bf99150bbb069c0c9d7f206aaae8a5c6195161abcf03234323e016ccaabfa10
May 26 16:16:33.768587 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:33.768544    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g" event={"ID":"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb","Type":"ContainerStarted","Data":"624ded6782f33773645b1c5fbba01fd26f85ab8b08ccfd9c7fc124ff3e0dfa35"}
May 26 16:16:33.768587 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:16:33.768585    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g" event={"ID":"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb","Type":"ContainerStarted","Data":"5bf99150bbb069c0c9d7f206aaae8a5c6195161abcf03234323e016ccaabfa10"}
May 26 16:18:42.197357 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:18:42.197270    2552 generic.go:358] "Generic (PLEG): container finished" podID="e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb" containerID="624ded6782f33773645b1c5fbba01fd26f85ab8b08ccfd9c7fc124ff3e0dfa35" exitCode=0
May 26 16:18:42.197357 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:18:42.197340    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g" event={"ID":"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb","Type":"ContainerDied","Data":"624ded6782f33773645b1c5fbba01fd26f85ab8b08ccfd9c7fc124ff3e0dfa35"}
May 26 16:18:42.198437 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:18:42.198388    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:19:10.308770 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:19:10.308736    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g" event={"ID":"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb","Type":"ContainerStarted","Data":"58309f1a073a89560224eb38456317bfa8e24d68cd8fa93fac41a425df80a1de"}
May 26 16:19:10.332704 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:19:10.332630    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g" podStartSLOduration=130.587844809 podStartE2EDuration="2m38.332611403s" podCreationTimestamp="2026-05-26 16:16:32 +0000 UTC" firstStartedPulling="2026-05-26 16:18:42.198504219 +0000 UTC m=+1071.906959241" lastFinishedPulling="2026-05-26 16:19:09.943270812 +0000 UTC m=+1099.651725835" observedRunningTime="2026-05-26 16:19:10.329895923 +0000 UTC m=+1100.038350963" watchObservedRunningTime="2026-05-26 16:19:10.332611403 +0000 UTC m=+1100.041066445"
May 26 16:19:12.935570 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:19:12.935529    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g"
May 26 16:19:12.935991 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:19:12.935582    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g"
May 26 16:19:12.937050 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:19:12.937021    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g" podUID="e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
May 26 16:19:22.935882 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:19:22.935827    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g" podUID="e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
May 26 16:19:32.935418 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:19:32.935366    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g" podUID="e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
May 26 16:19:42.935679 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:19:42.935631    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g" podUID="e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
May 26 16:19:52.934863 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:19:52.934812    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g" podUID="e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
May 26 16:20:02.935706 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:20:02.935659    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g" podUID="e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
May 26 16:20:12.935182 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:20:12.935082    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g" podUID="e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
May 26 16:20:22.935607 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:20:22.935564    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g" podUID="e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
May 26 16:20:32.935500 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:20:32.935452    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g" podUID="e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
May 26 16:20:42.934886 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:20:42.934838    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g" podUID="e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
May 26 16:20:50.847475 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:20:50.847444    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sj6db_5c0f9fc6-2bd6-4370-96d0-5ae315363c7f/ovn-acl-logging/0.log"
May 26 16:20:50.848028 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:20:50.847452    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sj6db_5c0f9fc6-2bd6-4370-96d0-5ae315363c7f/ovn-acl-logging/0.log"
May 26 16:20:52.945446 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:20:52.945416    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g"
May 26 16:20:52.955221 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:20:52.955190    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g"
May 26 16:20:59.590774 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:20:59.590736    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g"]
May 26 16:20:59.591305 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:20:59.591142    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g" podUID="e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb" containerName="main" containerID="cri-o://58309f1a073a89560224eb38456317bfa8e24d68cd8fa93fac41a425df80a1de" gracePeriod=30
May 26 16:21:25.608697 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:25.608659    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v"]
May 26 16:21:25.612148 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:25.612127    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v"
May 26 16:21:25.614548 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:25.614521    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
May 26 16:21:25.625828 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:25.625802    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v"]
May 26 16:21:25.738849 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:25.738801    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5faafa82-2b7c-4748-82f8-c79f6a4c0719-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-77bdb44765-w6x4v\" (UID: \"5faafa82-2b7c-4748-82f8-c79f6a4c0719\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v"
May 26 16:21:25.739056 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:25.738863    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5faafa82-2b7c-4748-82f8-c79f6a4c0719-dshm\") pod \"custom-route-timeout-test-kserve-77bdb44765-w6x4v\" (UID: \"5faafa82-2b7c-4748-82f8-c79f6a4c0719\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v"
May 26 16:21:25.739056 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:25.738942    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5faafa82-2b7c-4748-82f8-c79f6a4c0719-model-cache\") pod \"custom-route-timeout-test-kserve-77bdb44765-w6x4v\" (UID: \"5faafa82-2b7c-4748-82f8-c79f6a4c0719\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v"
May 26 16:21:25.739056 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:25.739023    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5faafa82-2b7c-4748-82f8-c79f6a4c0719-tls-certs\") pod \"custom-route-timeout-test-kserve-77bdb44765-w6x4v\" (UID: \"5faafa82-2b7c-4748-82f8-c79f6a4c0719\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v"
May 26 16:21:25.739231 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:25.739060    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2rhs\" (UniqueName: \"kubernetes.io/projected/5faafa82-2b7c-4748-82f8-c79f6a4c0719-kube-api-access-b2rhs\") pod \"custom-route-timeout-test-kserve-77bdb44765-w6x4v\" (UID: \"5faafa82-2b7c-4748-82f8-c79f6a4c0719\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v"
May 26 16:21:25.739231 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:25.739095    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5faafa82-2b7c-4748-82f8-c79f6a4c0719-tmp-dir\") pod \"custom-route-timeout-test-kserve-77bdb44765-w6x4v\" (UID: \"5faafa82-2b7c-4748-82f8-c79f6a4c0719\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v"
May 26 16:21:25.739231 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:25.739169    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5faafa82-2b7c-4748-82f8-c79f6a4c0719-home\") pod \"custom-route-timeout-test-kserve-77bdb44765-w6x4v\" (UID: \"5faafa82-2b7c-4748-82f8-c79f6a4c0719\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v"
May 26 16:21:25.840622 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:25.840588    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5faafa82-2b7c-4748-82f8-c79f6a4c0719-dshm\") pod \"custom-route-timeout-test-kserve-77bdb44765-w6x4v\" (UID: \"5faafa82-2b7c-4748-82f8-c79f6a4c0719\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v"
May 26 16:21:25.840622 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:25.840626    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5faafa82-2b7c-4748-82f8-c79f6a4c0719-model-cache\") pod \"custom-route-timeout-test-kserve-77bdb44765-w6x4v\" (UID: \"5faafa82-2b7c-4748-82f8-c79f6a4c0719\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v"
May 26 16:21:25.840876 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:25.840667    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5faafa82-2b7c-4748-82f8-c79f6a4c0719-tls-certs\") pod \"custom-route-timeout-test-kserve-77bdb44765-w6x4v\" (UID: \"5faafa82-2b7c-4748-82f8-c79f6a4c0719\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v"
May 26 16:21:25.840876 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:25.840699    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-b2rhs\" (UniqueName: \"kubernetes.io/projected/5faafa82-2b7c-4748-82f8-c79f6a4c0719-kube-api-access-b2rhs\") pod \"custom-route-timeout-test-kserve-77bdb44765-w6x4v\" (UID: \"5faafa82-2b7c-4748-82f8-c79f6a4c0719\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v"
May 26 16:21:25.840876 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:25.840730    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5faafa82-2b7c-4748-82f8-c79f6a4c0719-tmp-dir\") pod \"custom-route-timeout-test-kserve-77bdb44765-w6x4v\" (UID: \"5faafa82-2b7c-4748-82f8-c79f6a4c0719\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v"
May 26 16:21:25.840876 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:25.840783    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5faafa82-2b7c-4748-82f8-c79f6a4c0719-home\") pod \"custom-route-timeout-test-kserve-77bdb44765-w6x4v\" (UID: \"5faafa82-2b7c-4748-82f8-c79f6a4c0719\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v"
May 26 16:21:25.840876 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:25.840834    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5faafa82-2b7c-4748-82f8-c79f6a4c0719-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-77bdb44765-w6x4v\" (UID: \"5faafa82-2b7c-4748-82f8-c79f6a4c0719\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v"
May 26 16:21:25.841474 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:25.841182    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5faafa82-2b7c-4748-82f8-c79f6a4c0719-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-77bdb44765-w6x4v\" (UID: \"5faafa82-2b7c-4748-82f8-c79f6a4c0719\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v"
May 26 16:21:25.841474 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:25.841278    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5faafa82-2b7c-4748-82f8-c79f6a4c0719-home\") pod \"custom-route-timeout-test-kserve-77bdb44765-w6x4v\" (UID: \"5faafa82-2b7c-4748-82f8-c79f6a4c0719\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v"
May 26 16:21:25.841474 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:25.841426    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5faafa82-2b7c-4748-82f8-c79f6a4c0719-model-cache\") pod \"custom-route-timeout-test-kserve-77bdb44765-w6x4v\" (UID: \"5faafa82-2b7c-4748-82f8-c79f6a4c0719\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v"
May 26 16:21:25.841694 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:25.841457    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5faafa82-2b7c-4748-82f8-c79f6a4c0719-tmp-dir\") pod \"custom-route-timeout-test-kserve-77bdb44765-w6x4v\" (UID: \"5faafa82-2b7c-4748-82f8-c79f6a4c0719\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v"
May 26 16:21:25.843526 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:25.843502    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5faafa82-2b7c-4748-82f8-c79f6a4c0719-dshm\") pod \"custom-route-timeout-test-kserve-77bdb44765-w6x4v\" (UID: \"5faafa82-2b7c-4748-82f8-c79f6a4c0719\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v"
May 26 16:21:25.843925 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:25.843897    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5faafa82-2b7c-4748-82f8-c79f6a4c0719-tls-certs\") pod \"custom-route-timeout-test-kserve-77bdb44765-w6x4v\" (UID: \"5faafa82-2b7c-4748-82f8-c79f6a4c0719\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v"
May 26 16:21:25.852479 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:25.852444    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2rhs\" (UniqueName: \"kubernetes.io/projected/5faafa82-2b7c-4748-82f8-c79f6a4c0719-kube-api-access-b2rhs\") pod \"custom-route-timeout-test-kserve-77bdb44765-w6x4v\" (UID: \"5faafa82-2b7c-4748-82f8-c79f6a4c0719\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v"
May 26 16:21:25.922370 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:25.922286    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v"
May 26 16:21:26.055761 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:26.055628    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v"]
May 26 16:21:26.058017 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:21:26.057984    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5faafa82_2b7c_4748_82f8_c79f6a4c0719.slice/crio-bc4628238331f568217e9ccf72c92deb80b0ab80e6c9c0cda4092239a7004531 WatchSource:0}: Error finding container bc4628238331f568217e9ccf72c92deb80b0ab80e6c9c0cda4092239a7004531: Status 404 returned error can't find the container with id bc4628238331f568217e9ccf72c92deb80b0ab80e6c9c0cda4092239a7004531
May 26 16:21:26.769213 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:26.769177    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v" event={"ID":"5faafa82-2b7c-4748-82f8-c79f6a4c0719","Type":"ContainerStarted","Data":"25952f1433bee9eca0ab14eaf6a4de12a5e9a1e74725196309f04965039ae535"}
May 26 16:21:26.769213 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:26.769212    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v" event={"ID":"5faafa82-2b7c-4748-82f8-c79f6a4c0719","Type":"ContainerStarted","Data":"bc4628238331f568217e9ccf72c92deb80b0ab80e6c9c0cda4092239a7004531"}
May 26 16:21:29.789778 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:29.789742    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g_e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb/main/0.log"
May 26 16:21:29.790216 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:29.790136    2552 generic.go:358] "Generic (PLEG): container finished" podID="e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb" containerID="58309f1a073a89560224eb38456317bfa8e24d68cd8fa93fac41a425df80a1de" exitCode=137
May 26 16:21:29.790216 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:29.790180    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g" event={"ID":"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb","Type":"ContainerDied","Data":"58309f1a073a89560224eb38456317bfa8e24d68cd8fa93fac41a425df80a1de"}
May 26 16:21:29.833872 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:29.833847    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g_e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb/main/0.log"
May 26 16:21:29.834246 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:29.834226    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g"
May 26 16:21:29.976789 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:29.976702    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-kserve-provision-location\") pod \"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb\" (UID: \"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb\") "
May 26 16:21:29.976789 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:29.976778    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-home\") pod \"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb\" (UID: \"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb\") "
May 26 16:21:29.977067 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:29.976815    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-tls-certs\") pod \"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb\" (UID: \"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb\") "
May 26 16:21:29.977067 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:29.976849    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47b95\" (UniqueName: \"kubernetes.io/projected/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-kube-api-access-47b95\") pod \"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb\" (UID: \"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb\") "
May 26 16:21:29.977067 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:29.976867    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-tmp-dir\") pod \"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb\" (UID: \"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb\") "
May 26 16:21:29.977067 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:29.976882    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-dshm\") pod \"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb\" (UID: \"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb\") "
May 26 16:21:29.977288 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:29.976916    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-model-cache\") pod \"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb\" (UID: \"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb\") "
May 26 16:21:29.977288 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:29.977176    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-model-cache" (OuterVolumeSpecName: "model-cache") pod "e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb" (UID: "e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:21:29.977565 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:29.977525    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-home" (OuterVolumeSpecName: "home") pod "e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb" (UID: "e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:21:29.977565 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:29.977541    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-model-cache\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:21:29.979142 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:29.979112    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-dshm" (OuterVolumeSpecName: "dshm") pod "e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb" (UID: "e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:21:29.979239 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:29.979207    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb" (UID: "e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:21:29.979351 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:29.979332    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-kube-api-access-47b95" (OuterVolumeSpecName: "kube-api-access-47b95") pod "e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb" (UID: "e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb"). InnerVolumeSpecName "kube-api-access-47b95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:21:29.989350 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:29.989319    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb" (UID: "e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:21:30.034225 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:30.034181    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb" (UID: "e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:21:30.078879 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:30.078847    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-kserve-provision-location\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:21:30.078879 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:30.078876    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-home\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:21:30.078879 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:30.078886    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-tls-certs\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:21:30.078879 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:30.078897    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47b95\" (UniqueName: \"kubernetes.io/projected/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-kube-api-access-47b95\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:21:30.079154 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:30.078905    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-tmp-dir\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:21:30.079154 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:30.078913    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb-dshm\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:21:30.796038 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:30.796004    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g_e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb/main/0.log"
May 26 16:21:30.796451 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:30.796363    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g" event={"ID":"e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb","Type":"ContainerDied","Data":"5bf99150bbb069c0c9d7f206aaae8a5c6195161abcf03234323e016ccaabfa10"}
May 26 16:21:30.796451 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:30.796398    2552 scope.go:117] "RemoveContainer" containerID="58309f1a073a89560224eb38456317bfa8e24d68cd8fa93fac41a425df80a1de"
May 26 16:21:30.796451 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:30.796406    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g"
May 26 16:21:30.805036 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:30.805019    2552 scope.go:117] "RemoveContainer" containerID="624ded6782f33773645b1c5fbba01fd26f85ab8b08ccfd9c7fc124ff3e0dfa35"
May 26 16:21:30.819043 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:30.819017    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g"]
May 26 16:21:30.822423 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:30.822400    2552 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-7659cbd84fz2m4g"]
May 26 16:21:30.890503 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:30.890466    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb" path="/var/lib/kubelet/pods/e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb/volumes"
May 26 16:21:32.805184 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:32.805149    2552 generic.go:358] "Generic (PLEG): container finished" podID="5faafa82-2b7c-4748-82f8-c79f6a4c0719" containerID="25952f1433bee9eca0ab14eaf6a4de12a5e9a1e74725196309f04965039ae535" exitCode=0
May 26 16:21:32.805184 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:32.805183    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v" event={"ID":"5faafa82-2b7c-4748-82f8-c79f6a4c0719","Type":"ContainerDied","Data":"25952f1433bee9eca0ab14eaf6a4de12a5e9a1e74725196309f04965039ae535"}
May 26 16:21:33.811174 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:33.811135    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v" event={"ID":"5faafa82-2b7c-4748-82f8-c79f6a4c0719","Type":"ContainerStarted","Data":"888fe5286e6e6c16a007f38272582d47047087a85e869054286bfbff6c1cb9b1"}
May 26 16:21:33.834421 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:33.834352    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v" podStartSLOduration=8.834334467 podStartE2EDuration="8.834334467s" podCreationTimestamp="2026-05-26 16:21:25 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:21:33.832143291 +0000 UTC m=+1243.540598332" watchObservedRunningTime="2026-05-26 16:21:33.834334467 +0000 UTC m=+1243.542789513"
May 26 16:21:35.922434 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:35.922397    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v"
May 26 16:21:35.922434 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:35.922440    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v"
May 26 16:21:35.923906 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:35.923874    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v" podUID="5faafa82-2b7c-4748-82f8-c79f6a4c0719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
May 26 16:21:45.923181 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:45.923087    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v" podUID="5faafa82-2b7c-4748-82f8-c79f6a4c0719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
May 26 16:21:55.923721 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:21:55.923679    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v" podUID="5faafa82-2b7c-4748-82f8-c79f6a4c0719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
May 26 16:22:05.923398 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:22:05.923353    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v" podUID="5faafa82-2b7c-4748-82f8-c79f6a4c0719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
May 26 16:22:15.923628 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:22:15.923581    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v" podUID="5faafa82-2b7c-4748-82f8-c79f6a4c0719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
May 26 16:22:25.923067 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:22:25.923019    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v" podUID="5faafa82-2b7c-4748-82f8-c79f6a4c0719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
May 26 16:22:35.923112 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:22:35.923063    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v" podUID="5faafa82-2b7c-4748-82f8-c79f6a4c0719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
May 26 16:22:45.923684 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:22:45.923637    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v" podUID="5faafa82-2b7c-4748-82f8-c79f6a4c0719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
May 26 16:22:55.923389 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:22:55.923345    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v" podUID="5faafa82-2b7c-4748-82f8-c79f6a4c0719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
May 26 16:23:05.923360 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:05.923315    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v" podUID="5faafa82-2b7c-4748-82f8-c79f6a4c0719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
May 26 16:23:15.933167 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:15.933074    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v"
May 26 16:23:15.941107 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:15.941081    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v"
May 26 16:23:22.232865 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:22.232820    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v"]
May 26 16:23:22.233470 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:22.233219    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v" podUID="5faafa82-2b7c-4748-82f8-c79f6a4c0719" containerName="main" containerID="cri-o://888fe5286e6e6c16a007f38272582d47047087a85e869054286bfbff6c1cb9b1" gracePeriod=30
May 26 16:23:40.295285 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:40.295251    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf"]
May 26 16:23:40.295884 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:40.295830    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb" containerName="storage-initializer"
May 26 16:23:40.296073 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:40.295896    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb" containerName="storage-initializer"
May 26 16:23:40.296073 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:40.295968    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb" containerName="main"
May 26 16:23:40.296073 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:40.295979    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb" containerName="main"
May 26 16:23:40.296073 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:40.296074    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="e01c7cba-a6f3-4c45-ac04-a58cd7bb25bb" containerName="main"
May 26 16:23:40.298918 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:40.298897    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf"
May 26 16:23:40.301365 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:40.301348    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
May 26 16:23:40.310965 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:40.310926    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf"]
May 26 16:23:40.402030 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:40.401990    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8qxh\" (UniqueName: \"kubernetes.io/projected/b02ebecd-bab6-4cad-abb2-84e37997ec6f-kube-api-access-s8qxh\") pod \"router-with-refs-test-kserve-67b986fbb5-tpkvf\" (UID: \"b02ebecd-bab6-4cad-abb2-84e37997ec6f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf"
May 26 16:23:40.402030 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:40.402036    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b02ebecd-bab6-4cad-abb2-84e37997ec6f-tmp-dir\") pod \"router-with-refs-test-kserve-67b986fbb5-tpkvf\" (UID: \"b02ebecd-bab6-4cad-abb2-84e37997ec6f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf"
May 26 16:23:40.402234 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:40.402104    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b02ebecd-bab6-4cad-abb2-84e37997ec6f-dshm\") pod \"router-with-refs-test-kserve-67b986fbb5-tpkvf\" (UID: \"b02ebecd-bab6-4cad-abb2-84e37997ec6f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf"
May 26 16:23:40.402234 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:40.402132    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b02ebecd-bab6-4cad-abb2-84e37997ec6f-tls-certs\") pod \"router-with-refs-test-kserve-67b986fbb5-tpkvf\" (UID: \"b02ebecd-bab6-4cad-abb2-84e37997ec6f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf"
May 26 16:23:40.402234 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:40.402158    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b02ebecd-bab6-4cad-abb2-84e37997ec6f-kserve-provision-location\") pod \"router-with-refs-test-kserve-67b986fbb5-tpkvf\" (UID: \"b02ebecd-bab6-4cad-abb2-84e37997ec6f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf"
May 26 16:23:40.402234 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:40.402204    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b02ebecd-bab6-4cad-abb2-84e37997ec6f-home\") pod \"router-with-refs-test-kserve-67b986fbb5-tpkvf\" (UID: \"b02ebecd-bab6-4cad-abb2-84e37997ec6f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf"
May 26 16:23:40.402358 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:40.402249    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b02ebecd-bab6-4cad-abb2-84e37997ec6f-model-cache\") pod \"router-with-refs-test-kserve-67b986fbb5-tpkvf\" (UID: \"b02ebecd-bab6-4cad-abb2-84e37997ec6f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf"
May 26 16:23:40.503273 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:40.503234    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b02ebecd-bab6-4cad-abb2-84e37997ec6f-home\") pod \"router-with-refs-test-kserve-67b986fbb5-tpkvf\" (UID: \"b02ebecd-bab6-4cad-abb2-84e37997ec6f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf"
May 26 16:23:40.503273 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:40.503276    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b02ebecd-bab6-4cad-abb2-84e37997ec6f-model-cache\") pod \"router-with-refs-test-kserve-67b986fbb5-tpkvf\" (UID: \"b02ebecd-bab6-4cad-abb2-84e37997ec6f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf"
May 26 16:23:40.503532 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:40.503311    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s8qxh\" (UniqueName: \"kubernetes.io/projected/b02ebecd-bab6-4cad-abb2-84e37997ec6f-kube-api-access-s8qxh\") pod \"router-with-refs-test-kserve-67b986fbb5-tpkvf\" (UID: \"b02ebecd-bab6-4cad-abb2-84e37997ec6f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf"
May 26 16:23:40.503532 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:40.503336    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b02ebecd-bab6-4cad-abb2-84e37997ec6f-tmp-dir\") pod \"router-with-refs-test-kserve-67b986fbb5-tpkvf\" (UID: \"b02ebecd-bab6-4cad-abb2-84e37997ec6f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf"
May 26 16:23:40.503532 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:40.503361    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b02ebecd-bab6-4cad-abb2-84e37997ec6f-dshm\") pod \"router-with-refs-test-kserve-67b986fbb5-tpkvf\" (UID: \"b02ebecd-bab6-4cad-abb2-84e37997ec6f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf"
May 26 16:23:40.503532 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:40.503382    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b02ebecd-bab6-4cad-abb2-84e37997ec6f-tls-certs\") pod \"router-with-refs-test-kserve-67b986fbb5-tpkvf\" (UID: \"b02ebecd-bab6-4cad-abb2-84e37997ec6f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf"
May 26 16:23:40.503532 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:40.503411    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b02ebecd-bab6-4cad-abb2-84e37997ec6f-kserve-provision-location\") pod \"router-with-refs-test-kserve-67b986fbb5-tpkvf\" (UID: \"b02ebecd-bab6-4cad-abb2-84e37997ec6f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf"
May 26 16:23:40.503795 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:40.503722    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b02ebecd-bab6-4cad-abb2-84e37997ec6f-home\") pod \"router-with-refs-test-kserve-67b986fbb5-tpkvf\" (UID: \"b02ebecd-bab6-4cad-abb2-84e37997ec6f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf"
May 26 16:23:40.503862 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:40.503800    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b02ebecd-bab6-4cad-abb2-84e37997ec6f-kserve-provision-location\") pod \"router-with-refs-test-kserve-67b986fbb5-tpkvf\" (UID: \"b02ebecd-bab6-4cad-abb2-84e37997ec6f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf"
May 26 16:23:40.503862 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:40.503840    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b02ebecd-bab6-4cad-abb2-84e37997ec6f-tmp-dir\") pod \"router-with-refs-test-kserve-67b986fbb5-tpkvf\" (UID: \"b02ebecd-bab6-4cad-abb2-84e37997ec6f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf"
May 26 16:23:40.503985 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:40.503863    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b02ebecd-bab6-4cad-abb2-84e37997ec6f-model-cache\") pod \"router-with-refs-test-kserve-67b986fbb5-tpkvf\" (UID: \"b02ebecd-bab6-4cad-abb2-84e37997ec6f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf"
May 26 16:23:40.505749 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:40.505715    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b02ebecd-bab6-4cad-abb2-84e37997ec6f-dshm\") pod \"router-with-refs-test-kserve-67b986fbb5-tpkvf\" (UID: \"b02ebecd-bab6-4cad-abb2-84e37997ec6f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf"
May 26 16:23:40.505998 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:40.505982    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b02ebecd-bab6-4cad-abb2-84e37997ec6f-tls-certs\") pod \"router-with-refs-test-kserve-67b986fbb5-tpkvf\" (UID: \"b02ebecd-bab6-4cad-abb2-84e37997ec6f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf"
May 26 16:23:40.510987 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:40.510966    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8qxh\" (UniqueName: \"kubernetes.io/projected/b02ebecd-bab6-4cad-abb2-84e37997ec6f-kube-api-access-s8qxh\") pod \"router-with-refs-test-kserve-67b986fbb5-tpkvf\" (UID: \"b02ebecd-bab6-4cad-abb2-84e37997ec6f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf"
May 26 16:23:40.609773 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:40.609682    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf"
May 26 16:23:40.740142 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:40.740111    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf"]
May 26 16:23:40.742399 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:23:40.742371    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb02ebecd_bab6_4cad_abb2_84e37997ec6f.slice/crio-722cfd847da7d12318e8ad6f11fd4dfb7063a39355594d789d658e3d01a1d7f9 WatchSource:0}: Error finding container 722cfd847da7d12318e8ad6f11fd4dfb7063a39355594d789d658e3d01a1d7f9: Status 404 returned error can't find the container with id 722cfd847da7d12318e8ad6f11fd4dfb7063a39355594d789d658e3d01a1d7f9
May 26 16:23:41.247063 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:41.247028    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf" event={"ID":"b02ebecd-bab6-4cad-abb2-84e37997ec6f","Type":"ContainerStarted","Data":"967bb23dce7a5dd40c5fbf653aca8e02bce561af5b4127625fdb3bf3241beb44"}
May 26 16:23:41.247063 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:41.247067    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf" event={"ID":"b02ebecd-bab6-4cad-abb2-84e37997ec6f","Type":"ContainerStarted","Data":"722cfd847da7d12318e8ad6f11fd4dfb7063a39355594d789d658e3d01a1d7f9"}
May 26 16:23:52.504545 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:52.504519    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-77bdb44765-w6x4v_5faafa82-2b7c-4748-82f8-c79f6a4c0719/main/0.log"
May 26 16:23:52.504870 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:52.504857    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v"
May 26 16:23:52.610784 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:52.610746    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5faafa82-2b7c-4748-82f8-c79f6a4c0719-home\") pod \"5faafa82-2b7c-4748-82f8-c79f6a4c0719\" (UID: \"5faafa82-2b7c-4748-82f8-c79f6a4c0719\") "
May 26 16:23:52.610993 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:52.610801    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2rhs\" (UniqueName: \"kubernetes.io/projected/5faafa82-2b7c-4748-82f8-c79f6a4c0719-kube-api-access-b2rhs\") pod \"5faafa82-2b7c-4748-82f8-c79f6a4c0719\" (UID: \"5faafa82-2b7c-4748-82f8-c79f6a4c0719\") "
May 26 16:23:52.610993 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:52.610853    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5faafa82-2b7c-4748-82f8-c79f6a4c0719-kserve-provision-location\") pod \"5faafa82-2b7c-4748-82f8-c79f6a4c0719\" (UID: \"5faafa82-2b7c-4748-82f8-c79f6a4c0719\") "
May 26 16:23:52.610993 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:52.610869    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5faafa82-2b7c-4748-82f8-c79f6a4c0719-dshm\") pod \"5faafa82-2b7c-4748-82f8-c79f6a4c0719\" (UID: \"5faafa82-2b7c-4748-82f8-c79f6a4c0719\") "
May 26 16:23:52.610993 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:52.610900    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5faafa82-2b7c-4748-82f8-c79f6a4c0719-tmp-dir\") pod \"5faafa82-2b7c-4748-82f8-c79f6a4c0719\" (UID: \"5faafa82-2b7c-4748-82f8-c79f6a4c0719\") "
May 26 16:23:52.610993 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:52.610922    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5faafa82-2b7c-4748-82f8-c79f6a4c0719-model-cache\") pod \"5faafa82-2b7c-4748-82f8-c79f6a4c0719\" (UID: \"5faafa82-2b7c-4748-82f8-c79f6a4c0719\") "
May 26 16:23:52.610993 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:52.610977    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5faafa82-2b7c-4748-82f8-c79f6a4c0719-tls-certs\") pod \"5faafa82-2b7c-4748-82f8-c79f6a4c0719\" (UID: \"5faafa82-2b7c-4748-82f8-c79f6a4c0719\") "
May 26 16:23:52.611532 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:52.611503    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5faafa82-2b7c-4748-82f8-c79f6a4c0719-home" (OuterVolumeSpecName: "home") pod "5faafa82-2b7c-4748-82f8-c79f6a4c0719" (UID: "5faafa82-2b7c-4748-82f8-c79f6a4c0719"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:23:52.611661 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:52.611629    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5faafa82-2b7c-4748-82f8-c79f6a4c0719-model-cache" (OuterVolumeSpecName: "model-cache") pod "5faafa82-2b7c-4748-82f8-c79f6a4c0719" (UID: "5faafa82-2b7c-4748-82f8-c79f6a4c0719"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:23:52.613455 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:52.613426    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5faafa82-2b7c-4748-82f8-c79f6a4c0719-kube-api-access-b2rhs" (OuterVolumeSpecName: "kube-api-access-b2rhs") pod "5faafa82-2b7c-4748-82f8-c79f6a4c0719" (UID: "5faafa82-2b7c-4748-82f8-c79f6a4c0719"). InnerVolumeSpecName "kube-api-access-b2rhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:23:52.613586 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:52.613563    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5faafa82-2b7c-4748-82f8-c79f6a4c0719-dshm" (OuterVolumeSpecName: "dshm") pod "5faafa82-2b7c-4748-82f8-c79f6a4c0719" (UID: "5faafa82-2b7c-4748-82f8-c79f6a4c0719"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:23:52.613651 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:52.613627    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5faafa82-2b7c-4748-82f8-c79f6a4c0719-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5faafa82-2b7c-4748-82f8-c79f6a4c0719" (UID: "5faafa82-2b7c-4748-82f8-c79f6a4c0719"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:23:52.623522 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:52.623495    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5faafa82-2b7c-4748-82f8-c79f6a4c0719-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5faafa82-2b7c-4748-82f8-c79f6a4c0719" (UID: "5faafa82-2b7c-4748-82f8-c79f6a4c0719"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:23:52.671289 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:52.671248    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5faafa82-2b7c-4748-82f8-c79f6a4c0719-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5faafa82-2b7c-4748-82f8-c79f6a4c0719" (UID: "5faafa82-2b7c-4748-82f8-c79f6a4c0719"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:23:52.711715 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:52.711683    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2rhs\" (UniqueName: \"kubernetes.io/projected/5faafa82-2b7c-4748-82f8-c79f6a4c0719-kube-api-access-b2rhs\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:23:52.711715 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:52.711709    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5faafa82-2b7c-4748-82f8-c79f6a4c0719-kserve-provision-location\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:23:52.711715 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:52.711719    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5faafa82-2b7c-4748-82f8-c79f6a4c0719-dshm\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:23:52.711959 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:52.711740    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5faafa82-2b7c-4748-82f8-c79f6a4c0719-tmp-dir\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:23:52.711959 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:52.711749    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5faafa82-2b7c-4748-82f8-c79f6a4c0719-model-cache\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:23:52.711959 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:52.711758    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5faafa82-2b7c-4748-82f8-c79f6a4c0719-tls-certs\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:23:52.711959 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:52.711765    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5faafa82-2b7c-4748-82f8-c79f6a4c0719-home\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:23:53.286513 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:53.286486    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-77bdb44765-w6x4v_5faafa82-2b7c-4748-82f8-c79f6a4c0719/main/0.log"
May 26 16:23:53.286851 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:53.286822    2552 generic.go:358] "Generic (PLEG): container finished" podID="5faafa82-2b7c-4748-82f8-c79f6a4c0719" containerID="888fe5286e6e6c16a007f38272582d47047087a85e869054286bfbff6c1cb9b1" exitCode=137
May 26 16:23:53.286925 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:53.286905    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v"
May 26 16:23:53.287018 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:53.286906    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v" event={"ID":"5faafa82-2b7c-4748-82f8-c79f6a4c0719","Type":"ContainerDied","Data":"888fe5286e6e6c16a007f38272582d47047087a85e869054286bfbff6c1cb9b1"}
May 26 16:23:53.287128 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:53.287044    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v" event={"ID":"5faafa82-2b7c-4748-82f8-c79f6a4c0719","Type":"ContainerDied","Data":"bc4628238331f568217e9ccf72c92deb80b0ab80e6c9c0cda4092239a7004531"}
May 26 16:23:53.287128 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:53.287064    2552 scope.go:117] "RemoveContainer" containerID="888fe5286e6e6c16a007f38272582d47047087a85e869054286bfbff6c1cb9b1"
May 26 16:23:53.294866 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:53.294851    2552 scope.go:117] "RemoveContainer" containerID="25952f1433bee9eca0ab14eaf6a4de12a5e9a1e74725196309f04965039ae535"
May 26 16:23:53.311761 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:53.311732    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v"]
May 26 16:23:53.318438 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:53.318413    2552 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-77bdb44765-w6x4v"]
May 26 16:23:53.359957 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:53.359923    2552 scope.go:117] "RemoveContainer" containerID="888fe5286e6e6c16a007f38272582d47047087a85e869054286bfbff6c1cb9b1"
May 26 16:23:53.360307 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:23:53.360286    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"888fe5286e6e6c16a007f38272582d47047087a85e869054286bfbff6c1cb9b1\": container with ID starting with 888fe5286e6e6c16a007f38272582d47047087a85e869054286bfbff6c1cb9b1 not found: ID does not exist" containerID="888fe5286e6e6c16a007f38272582d47047087a85e869054286bfbff6c1cb9b1"
May 26 16:23:53.360357 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:53.360317    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"888fe5286e6e6c16a007f38272582d47047087a85e869054286bfbff6c1cb9b1"} err="failed to get container status \"888fe5286e6e6c16a007f38272582d47047087a85e869054286bfbff6c1cb9b1\": rpc error: code = NotFound desc = could not find container \"888fe5286e6e6c16a007f38272582d47047087a85e869054286bfbff6c1cb9b1\": container with ID starting with 888fe5286e6e6c16a007f38272582d47047087a85e869054286bfbff6c1cb9b1 not found: ID does not exist"
May 26 16:23:53.360357 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:53.360337    2552 scope.go:117] "RemoveContainer" containerID="25952f1433bee9eca0ab14eaf6a4de12a5e9a1e74725196309f04965039ae535"
May 26 16:23:53.360600 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:23:53.360566    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25952f1433bee9eca0ab14eaf6a4de12a5e9a1e74725196309f04965039ae535\": container with ID starting with 25952f1433bee9eca0ab14eaf6a4de12a5e9a1e74725196309f04965039ae535 not found: ID does not exist" containerID="25952f1433bee9eca0ab14eaf6a4de12a5e9a1e74725196309f04965039ae535"
May 26 16:23:53.360643 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:53.360608    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25952f1433bee9eca0ab14eaf6a4de12a5e9a1e74725196309f04965039ae535"} err="failed to get container status \"25952f1433bee9eca0ab14eaf6a4de12a5e9a1e74725196309f04965039ae535\": rpc error: code = NotFound desc = could not find container \"25952f1433bee9eca0ab14eaf6a4de12a5e9a1e74725196309f04965039ae535\": container with ID starting with 25952f1433bee9eca0ab14eaf6a4de12a5e9a1e74725196309f04965039ae535 not found: ID does not exist"
May 26 16:23:54.889854 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:23:54.889819    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5faafa82-2b7c-4748-82f8-c79f6a4c0719" path="/var/lib/kubelet/pods/5faafa82-2b7c-4748-82f8-c79f6a4c0719/volumes"
May 26 16:25:50.868741 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:25:50.868716    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sj6db_5c0f9fc6-2bd6-4370-96d0-5ae315363c7f/ovn-acl-logging/0.log"
May 26 16:25:50.870427 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:25:50.870408    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sj6db_5c0f9fc6-2bd6-4370-96d0-5ae315363c7f/ovn-acl-logging/0.log"
May 26 16:26:16.739803 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:26:16.739764    2552 generic.go:358] "Generic (PLEG): container finished" podID="b02ebecd-bab6-4cad-abb2-84e37997ec6f" containerID="967bb23dce7a5dd40c5fbf653aca8e02bce561af5b4127625fdb3bf3241beb44" exitCode=0
May 26 16:26:16.740171 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:26:16.739842    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf" event={"ID":"b02ebecd-bab6-4cad-abb2-84e37997ec6f","Type":"ContainerDied","Data":"967bb23dce7a5dd40c5fbf653aca8e02bce561af5b4127625fdb3bf3241beb44"}
May 26 16:26:16.740972 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:26:16.740956    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:26:17.744292 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:26:17.744245    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf" event={"ID":"b02ebecd-bab6-4cad-abb2-84e37997ec6f","Type":"ContainerStarted","Data":"650886c33a56157c7babba324868968d34ba7b658c0ef22ad78043f9d60a622e"}
May 26 16:26:17.768941 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:26:17.768879    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf" podStartSLOduration=157.76886026 podStartE2EDuration="2m37.76886026s" podCreationTimestamp="2026-05-26 16:23:40 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:26:17.765699424 +0000 UTC m=+1527.474154463" watchObservedRunningTime="2026-05-26 16:26:17.76886026 +0000 UTC m=+1527.477315303"
May 26 16:26:20.610603 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:26:20.610569    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf"
May 26 16:26:20.610603 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:26:20.610605    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf"
May 26 16:26:20.612357 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:26:20.612328    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf" podUID="b02ebecd-bab6-4cad-abb2-84e37997ec6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
May 26 16:26:30.610605 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:26:30.610558    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf" podUID="b02ebecd-bab6-4cad-abb2-84e37997ec6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
May 26 16:26:40.610209 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:26:40.610154    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf" podUID="b02ebecd-bab6-4cad-abb2-84e37997ec6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
May 26 16:26:50.610908 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:26:50.610862    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf" podUID="b02ebecd-bab6-4cad-abb2-84e37997ec6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
May 26 16:27:00.610871 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:27:00.610829    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf" podUID="b02ebecd-bab6-4cad-abb2-84e37997ec6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
May 26 16:27:10.610243 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:27:10.610200    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf" podUID="b02ebecd-bab6-4cad-abb2-84e37997ec6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
May 26 16:27:20.610407 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:27:20.610360    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf" podUID="b02ebecd-bab6-4cad-abb2-84e37997ec6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
May 26 16:27:30.610883 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:27:30.610841    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf" podUID="b02ebecd-bab6-4cad-abb2-84e37997ec6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
May 26 16:27:40.610590 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:27:40.610493    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf" podUID="b02ebecd-bab6-4cad-abb2-84e37997ec6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
May 26 16:27:50.619995 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:27:50.619966    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf"
May 26 16:27:50.627745 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:27:50.627722    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf"
May 26 16:27:56.683264 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:27:56.683227    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf"]
May 26 16:27:56.683721 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:27:56.683599    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf" podUID="b02ebecd-bab6-4cad-abb2-84e37997ec6f" containerName="main" containerID="cri-o://650886c33a56157c7babba324868968d34ba7b658c0ef22ad78043f9d60a622e" gracePeriod=30
May 26 16:28:04.833371 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.833341    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"]
May 26 16:28:04.833846 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.833634    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5faafa82-2b7c-4748-82f8-c79f6a4c0719" containerName="main"
May 26 16:28:04.833846 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.833645    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="5faafa82-2b7c-4748-82f8-c79f6a4c0719" containerName="main"
May 26 16:28:04.833846 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.833663    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5faafa82-2b7c-4748-82f8-c79f6a4c0719" containerName="storage-initializer"
May 26 16:28:04.833846 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.833670    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="5faafa82-2b7c-4748-82f8-c79f6a4c0719" containerName="storage-initializer"
May 26 16:28:04.833846 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.833761    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="5faafa82-2b7c-4748-82f8-c79f6a4c0719" containerName="main"
May 26 16:28:04.837313 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.837294    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"
May 26 16:28:04.839971 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.839955    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8f1a6f044e8c7a4d31a250e0c4861caf-kserve-self-signed-certs\""
May 26 16:28:04.840070 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.840004    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-dockercfg-29tjb\""
May 26 16:28:04.848284 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.848257    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6"]
May 26 16:28:04.851817 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.851792    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"]
May 26 16:28:04.851924 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.851914    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6"
May 26 16:28:04.862271 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.862232    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6"]
May 26 16:28:04.875638 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.875609    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-889nm\" (UniqueName: \"kubernetes.io/projected/e221ecac-2af0-4182-8b3f-eb7cb5ead412-kube-api-access-889nm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6\" (UID: \"e221ecac-2af0-4182-8b3f-eb7cb5ead412\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6"
May 26 16:28:04.875760 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.875647    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e221ecac-2af0-4182-8b3f-eb7cb5ead412-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6\" (UID: \"e221ecac-2af0-4182-8b3f-eb7cb5ead412\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6"
May 26 16:28:04.875760 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.875665    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e221ecac-2af0-4182-8b3f-eb7cb5ead412-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6\" (UID: \"e221ecac-2af0-4182-8b3f-eb7cb5ead412\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6"
May 26 16:28:04.875760 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.875732    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e221ecac-2af0-4182-8b3f-eb7cb5ead412-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6\" (UID: \"e221ecac-2af0-4182-8b3f-eb7cb5ead412\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6"
May 26 16:28:04.875893 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.875782    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52b5d713-f4db-49ca-93e2-7b8c6fb90530-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr\" (UID: \"52b5d713-f4db-49ca-93e2-7b8c6fb90530\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"
May 26 16:28:04.875893 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.875804    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52b5d713-f4db-49ca-93e2-7b8c6fb90530-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr\" (UID: \"52b5d713-f4db-49ca-93e2-7b8c6fb90530\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"
May 26 16:28:04.875893 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.875834    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52b5d713-f4db-49ca-93e2-7b8c6fb90530-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr\" (UID: \"52b5d713-f4db-49ca-93e2-7b8c6fb90530\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"
May 26 16:28:04.875893 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.875856    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52b5d713-f4db-49ca-93e2-7b8c6fb90530-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr\" (UID: \"52b5d713-f4db-49ca-93e2-7b8c6fb90530\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"
May 26 16:28:04.875893 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.875873    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpgbz\" (UniqueName: \"kubernetes.io/projected/52b5d713-f4db-49ca-93e2-7b8c6fb90530-kube-api-access-tpgbz\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr\" (UID: \"52b5d713-f4db-49ca-93e2-7b8c6fb90530\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"
May 26 16:28:04.876098 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.875894    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e221ecac-2af0-4182-8b3f-eb7cb5ead412-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6\" (UID: \"e221ecac-2af0-4182-8b3f-eb7cb5ead412\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6"
May 26 16:28:04.876098 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.875924    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e221ecac-2af0-4182-8b3f-eb7cb5ead412-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6\" (UID: \"e221ecac-2af0-4182-8b3f-eb7cb5ead412\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6"
May 26 16:28:04.876098 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.875965    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52b5d713-f4db-49ca-93e2-7b8c6fb90530-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr\" (UID: \"52b5d713-f4db-49ca-93e2-7b8c6fb90530\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"
May 26 16:28:04.876098 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.875993    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e221ecac-2af0-4182-8b3f-eb7cb5ead412-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6\" (UID: \"e221ecac-2af0-4182-8b3f-eb7cb5ead412\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6"
May 26 16:28:04.876098 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.876017    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52b5d713-f4db-49ca-93e2-7b8c6fb90530-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr\" (UID: \"52b5d713-f4db-49ca-93e2-7b8c6fb90530\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"
May 26 16:28:04.976350 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.976323    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e221ecac-2af0-4182-8b3f-eb7cb5ead412-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6\" (UID: \"e221ecac-2af0-4182-8b3f-eb7cb5ead412\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6"
May 26 16:28:04.976350 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.976354    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52b5d713-f4db-49ca-93e2-7b8c6fb90530-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr\" (UID: \"52b5d713-f4db-49ca-93e2-7b8c6fb90530\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"
May 26 16:28:04.976538 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.976379    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e221ecac-2af0-4182-8b3f-eb7cb5ead412-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6\" (UID: \"e221ecac-2af0-4182-8b3f-eb7cb5ead412\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6"
May 26 16:28:04.976538 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.976398    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52b5d713-f4db-49ca-93e2-7b8c6fb90530-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr\" (UID: \"52b5d713-f4db-49ca-93e2-7b8c6fb90530\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"
May 26 16:28:04.976538 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.976420    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-889nm\" (UniqueName: \"kubernetes.io/projected/e221ecac-2af0-4182-8b3f-eb7cb5ead412-kube-api-access-889nm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6\" (UID: \"e221ecac-2af0-4182-8b3f-eb7cb5ead412\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6"
May 26 16:28:04.976538 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.976441    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e221ecac-2af0-4182-8b3f-eb7cb5ead412-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6\" (UID: \"e221ecac-2af0-4182-8b3f-eb7cb5ead412\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6"
May 26 16:28:04.976538 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.976456    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e221ecac-2af0-4182-8b3f-eb7cb5ead412-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6\" (UID: \"e221ecac-2af0-4182-8b3f-eb7cb5ead412\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6"
May 26 16:28:04.976538 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.976488    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e221ecac-2af0-4182-8b3f-eb7cb5ead412-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6\" (UID: \"e221ecac-2af0-4182-8b3f-eb7cb5ead412\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6"
May 26 16:28:04.976538 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.976527    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52b5d713-f4db-49ca-93e2-7b8c6fb90530-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr\" (UID: \"52b5d713-f4db-49ca-93e2-7b8c6fb90530\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"
May 26 16:28:04.976882 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.976555    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52b5d713-f4db-49ca-93e2-7b8c6fb90530-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr\" (UID: \"52b5d713-f4db-49ca-93e2-7b8c6fb90530\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"
May 26 16:28:04.976882 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.976586    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52b5d713-f4db-49ca-93e2-7b8c6fb90530-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr\" (UID: \"52b5d713-f4db-49ca-93e2-7b8c6fb90530\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"
May 26 16:28:04.976882 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.976617    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52b5d713-f4db-49ca-93e2-7b8c6fb90530-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr\" (UID: \"52b5d713-f4db-49ca-93e2-7b8c6fb90530\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"
May 26 16:28:04.976882 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.976645    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tpgbz\" (UniqueName: \"kubernetes.io/projected/52b5d713-f4db-49ca-93e2-7b8c6fb90530-kube-api-access-tpgbz\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr\" (UID: \"52b5d713-f4db-49ca-93e2-7b8c6fb90530\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"
May 26 16:28:04.976882 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.976670    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e221ecac-2af0-4182-8b3f-eb7cb5ead412-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6\" (UID: \"e221ecac-2af0-4182-8b3f-eb7cb5ead412\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6"
May 26 16:28:04.976882 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.976755    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e221ecac-2af0-4182-8b3f-eb7cb5ead412-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6\" (UID: \"e221ecac-2af0-4182-8b3f-eb7cb5ead412\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6"
May 26 16:28:04.976882 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.976767    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52b5d713-f4db-49ca-93e2-7b8c6fb90530-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr\" (UID: \"52b5d713-f4db-49ca-93e2-7b8c6fb90530\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"
May 26 16:28:04.976882 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.976813    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e221ecac-2af0-4182-8b3f-eb7cb5ead412-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6\" (UID: \"e221ecac-2af0-4182-8b3f-eb7cb5ead412\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6"
May 26 16:28:04.977277 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.976884    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e221ecac-2af0-4182-8b3f-eb7cb5ead412-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6\" (UID: \"e221ecac-2af0-4182-8b3f-eb7cb5ead412\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6"
May 26 16:28:04.977277 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.977080    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52b5d713-f4db-49ca-93e2-7b8c6fb90530-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr\" (UID: \"52b5d713-f4db-49ca-93e2-7b8c6fb90530\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"
May 26 16:28:04.977277 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.977083    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e221ecac-2af0-4182-8b3f-eb7cb5ead412-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6\" (UID: \"e221ecac-2af0-4182-8b3f-eb7cb5ead412\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6"
May 26 16:28:04.977277 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.977105    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52b5d713-f4db-49ca-93e2-7b8c6fb90530-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr\" (UID: \"52b5d713-f4db-49ca-93e2-7b8c6fb90530\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"
May 26 16:28:04.977477 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.977380    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52b5d713-f4db-49ca-93e2-7b8c6fb90530-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr\" (UID: \"52b5d713-f4db-49ca-93e2-7b8c6fb90530\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"
May 26 16:28:04.979086 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.979058    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52b5d713-f4db-49ca-93e2-7b8c6fb90530-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr\" (UID: \"52b5d713-f4db-49ca-93e2-7b8c6fb90530\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"
May 26 16:28:04.979183 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.979083    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e221ecac-2af0-4182-8b3f-eb7cb5ead412-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6\" (UID: \"e221ecac-2af0-4182-8b3f-eb7cb5ead412\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6"
May 26 16:28:04.979271 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.979249    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e221ecac-2af0-4182-8b3f-eb7cb5ead412-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6\" (UID: \"e221ecac-2af0-4182-8b3f-eb7cb5ead412\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6"
May 26 16:28:04.979441 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.979426    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52b5d713-f4db-49ca-93e2-7b8c6fb90530-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr\" (UID: \"52b5d713-f4db-49ca-93e2-7b8c6fb90530\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"
May 26 16:28:04.988357 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.988329    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-889nm\" (UniqueName: \"kubernetes.io/projected/e221ecac-2af0-4182-8b3f-eb7cb5ead412-kube-api-access-889nm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6\" (UID: \"e221ecac-2af0-4182-8b3f-eb7cb5ead412\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6"
May 26 16:28:04.988479 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:04.988414    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpgbz\" (UniqueName: \"kubernetes.io/projected/52b5d713-f4db-49ca-93e2-7b8c6fb90530-kube-api-access-tpgbz\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr\" (UID: \"52b5d713-f4db-49ca-93e2-7b8c6fb90530\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"
May 26 16:28:05.146404 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:05.146320    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"
May 26 16:28:05.163354 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:05.163330    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6"
May 26 16:28:05.290723 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:05.290673    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"]
May 26 16:28:05.296329 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:28:05.296291    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52b5d713_f4db_49ca_93e2_7b8c6fb90530.slice/crio-37011a53dbda9b25f8d563386bc1312318bdadecfd61c29f1d8f148581cd38bc WatchSource:0}: Error finding container 37011a53dbda9b25f8d563386bc1312318bdadecfd61c29f1d8f148581cd38bc: Status 404 returned error can't find the container with id 37011a53dbda9b25f8d563386bc1312318bdadecfd61c29f1d8f148581cd38bc
May 26 16:28:05.311069 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:05.311047    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6"]
May 26 16:28:05.313305 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:28:05.313281    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode221ecac_2af0_4182_8b3f_eb7cb5ead412.slice/crio-0f0178bb392588bbc62da286638650d76c1acd746299e3e6e59e15eaac73ca8c WatchSource:0}: Error finding container 0f0178bb392588bbc62da286638650d76c1acd746299e3e6e59e15eaac73ca8c: Status 404 returned error can't find the container with id 0f0178bb392588bbc62da286638650d76c1acd746299e3e6e59e15eaac73ca8c
May 26 16:28:06.106402 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:06.106365    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr" event={"ID":"52b5d713-f4db-49ca-93e2-7b8c6fb90530","Type":"ContainerStarted","Data":"37011a53dbda9b25f8d563386bc1312318bdadecfd61c29f1d8f148581cd38bc"}
May 26 16:28:06.107777 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:06.107742    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6" event={"ID":"e221ecac-2af0-4182-8b3f-eb7cb5ead412","Type":"ContainerStarted","Data":"9c5330bc410362159c08c53f806ed7707b87786a54d2cdd28ff1a22aee891727"}
May 26 16:28:06.107777 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:06.107768    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6" event={"ID":"e221ecac-2af0-4182-8b3f-eb7cb5ead412","Type":"ContainerStarted","Data":"0f0178bb392588bbc62da286638650d76c1acd746299e3e6e59e15eaac73ca8c"}
May 26 16:28:07.112082 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:07.112043    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr" event={"ID":"52b5d713-f4db-49ca-93e2-7b8c6fb90530","Type":"ContainerStarted","Data":"d58970c73c008434d1d06898f8b3b661b04bbd7c85f7952433d08cb0a81e10b1"}
May 26 16:28:08.117535 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:08.117501    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr" event={"ID":"52b5d713-f4db-49ca-93e2-7b8c6fb90530","Type":"ContainerStarted","Data":"8034e75dbe46007099c5d741c67fc1cc24531c426ffce84adc17570d03f696e7"}
May 26 16:28:08.117921 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:08.117794    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"
May 26 16:28:11.127503 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:11.127470    2552 generic.go:358] "Generic (PLEG): container finished" podID="52b5d713-f4db-49ca-93e2-7b8c6fb90530" containerID="8034e75dbe46007099c5d741c67fc1cc24531c426ffce84adc17570d03f696e7" exitCode=0
May 26 16:28:11.127861 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:11.127512    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr" event={"ID":"52b5d713-f4db-49ca-93e2-7b8c6fb90530","Type":"ContainerDied","Data":"8034e75dbe46007099c5d741c67fc1cc24531c426ffce84adc17570d03f696e7"}
May 26 16:28:12.139664 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:12.139627    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr" event={"ID":"52b5d713-f4db-49ca-93e2-7b8c6fb90530","Type":"ContainerStarted","Data":"bf2efb595d20276932d9f5cc7bc49a4aebefa9743e03b8b0120488bb951b59f0"}
May 26 16:28:12.164435 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:12.164381    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr" podStartSLOduration=7.129106913 podStartE2EDuration="8.164365656s" podCreationTimestamp="2026-05-26 16:28:04 +0000 UTC" firstStartedPulling="2026-05-26 16:28:05.298607604 +0000 UTC m=+1635.007062625" lastFinishedPulling="2026-05-26 16:28:06.333866327 +0000 UTC m=+1636.042321368" observedRunningTime="2026-05-26 16:28:12.163037944 +0000 UTC m=+1641.871492985" watchObservedRunningTime="2026-05-26 16:28:12.164365656 +0000 UTC m=+1641.872820695"
May 26 16:28:15.147146 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:15.147114    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"
May 26 16:28:15.147146 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:15.147154    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"
May 26 16:28:15.148544 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:15.148506    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr" podUID="52b5d713-f4db-49ca-93e2-7b8c6fb90530" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8001/health\": dial tcp 10.132.0.34:8001: connect: connection refused"
May 26 16:28:25.146875 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:25.146824    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr" podUID="52b5d713-f4db-49ca-93e2-7b8c6fb90530" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8001/health\": dial tcp 10.132.0.34:8001: connect: connection refused"
May 26 16:28:25.161767 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:25.161742    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"
May 26 16:28:26.972726 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:26.972699    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-67b986fbb5-tpkvf_b02ebecd-bab6-4cad-abb2-84e37997ec6f/main/0.log"
May 26 16:28:26.973148 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:26.973131    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf"
May 26 16:28:27.076672 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:27.076640    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b02ebecd-bab6-4cad-abb2-84e37997ec6f-home\") pod \"b02ebecd-bab6-4cad-abb2-84e37997ec6f\" (UID: \"b02ebecd-bab6-4cad-abb2-84e37997ec6f\") "
May 26 16:28:27.076797 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:27.076691    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8qxh\" (UniqueName: \"kubernetes.io/projected/b02ebecd-bab6-4cad-abb2-84e37997ec6f-kube-api-access-s8qxh\") pod \"b02ebecd-bab6-4cad-abb2-84e37997ec6f\" (UID: \"b02ebecd-bab6-4cad-abb2-84e37997ec6f\") "
May 26 16:28:27.076797 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:27.076727    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b02ebecd-bab6-4cad-abb2-84e37997ec6f-tmp-dir\") pod \"b02ebecd-bab6-4cad-abb2-84e37997ec6f\" (UID: \"b02ebecd-bab6-4cad-abb2-84e37997ec6f\") "
May 26 16:28:27.076797 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:27.076745    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b02ebecd-bab6-4cad-abb2-84e37997ec6f-kserve-provision-location\") pod \"b02ebecd-bab6-4cad-abb2-84e37997ec6f\" (UID: \"b02ebecd-bab6-4cad-abb2-84e37997ec6f\") "
May 26 16:28:27.076797 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:27.076788    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b02ebecd-bab6-4cad-abb2-84e37997ec6f-tls-certs\") pod \"b02ebecd-bab6-4cad-abb2-84e37997ec6f\" (UID: \"b02ebecd-bab6-4cad-abb2-84e37997ec6f\") "
May 26 16:28:27.077064 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:27.076831    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b02ebecd-bab6-4cad-abb2-84e37997ec6f-dshm\") pod \"b02ebecd-bab6-4cad-abb2-84e37997ec6f\" (UID: \"b02ebecd-bab6-4cad-abb2-84e37997ec6f\") "
May 26 16:28:27.077064 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:27.076860    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b02ebecd-bab6-4cad-abb2-84e37997ec6f-model-cache\") pod \"b02ebecd-bab6-4cad-abb2-84e37997ec6f\" (UID: \"b02ebecd-bab6-4cad-abb2-84e37997ec6f\") "
May 26 16:28:27.077362 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:27.077333    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02ebecd-bab6-4cad-abb2-84e37997ec6f-home" (OuterVolumeSpecName: "home") pod "b02ebecd-bab6-4cad-abb2-84e37997ec6f" (UID: "b02ebecd-bab6-4cad-abb2-84e37997ec6f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:28:27.077619 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:27.077596    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02ebecd-bab6-4cad-abb2-84e37997ec6f-model-cache" (OuterVolumeSpecName: "model-cache") pod "b02ebecd-bab6-4cad-abb2-84e37997ec6f" (UID: "b02ebecd-bab6-4cad-abb2-84e37997ec6f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:28:27.079199 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:27.079167    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b02ebecd-bab6-4cad-abb2-84e37997ec6f-kube-api-access-s8qxh" (OuterVolumeSpecName: "kube-api-access-s8qxh") pod "b02ebecd-bab6-4cad-abb2-84e37997ec6f" (UID: "b02ebecd-bab6-4cad-abb2-84e37997ec6f"). InnerVolumeSpecName "kube-api-access-s8qxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:28:27.079817 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:27.079778    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b02ebecd-bab6-4cad-abb2-84e37997ec6f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b02ebecd-bab6-4cad-abb2-84e37997ec6f" (UID: "b02ebecd-bab6-4cad-abb2-84e37997ec6f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:28:27.079945 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:27.079832    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02ebecd-bab6-4cad-abb2-84e37997ec6f-dshm" (OuterVolumeSpecName: "dshm") pod "b02ebecd-bab6-4cad-abb2-84e37997ec6f" (UID: "b02ebecd-bab6-4cad-abb2-84e37997ec6f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:28:27.089545 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:27.089519    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02ebecd-bab6-4cad-abb2-84e37997ec6f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b02ebecd-bab6-4cad-abb2-84e37997ec6f" (UID: "b02ebecd-bab6-4cad-abb2-84e37997ec6f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:28:27.138005 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:27.137965    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02ebecd-bab6-4cad-abb2-84e37997ec6f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b02ebecd-bab6-4cad-abb2-84e37997ec6f" (UID: "b02ebecd-bab6-4cad-abb2-84e37997ec6f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:28:27.177523 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:27.177497    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b02ebecd-bab6-4cad-abb2-84e37997ec6f-home\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:28:27.177523 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:27.177520    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8qxh\" (UniqueName: \"kubernetes.io/projected/b02ebecd-bab6-4cad-abb2-84e37997ec6f-kube-api-access-s8qxh\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:28:27.177665 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:27.177531    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b02ebecd-bab6-4cad-abb2-84e37997ec6f-tmp-dir\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:28:27.177665 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:27.177541    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b02ebecd-bab6-4cad-abb2-84e37997ec6f-kserve-provision-location\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:28:27.177665 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:27.177551    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b02ebecd-bab6-4cad-abb2-84e37997ec6f-tls-certs\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:28:27.177665 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:27.177559    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b02ebecd-bab6-4cad-abb2-84e37997ec6f-dshm\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:28:27.177665 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:27.177567    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b02ebecd-bab6-4cad-abb2-84e37997ec6f-model-cache\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:28:27.189856 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:27.189832    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-67b986fbb5-tpkvf_b02ebecd-bab6-4cad-abb2-84e37997ec6f/main/0.log"
May 26 16:28:27.190210 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:27.190189    2552 generic.go:358] "Generic (PLEG): container finished" podID="b02ebecd-bab6-4cad-abb2-84e37997ec6f" containerID="650886c33a56157c7babba324868968d34ba7b658c0ef22ad78043f9d60a622e" exitCode=137
May 26 16:28:27.190294 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:27.190265    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf"
May 26 16:28:27.190294 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:27.190270    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf" event={"ID":"b02ebecd-bab6-4cad-abb2-84e37997ec6f","Type":"ContainerDied","Data":"650886c33a56157c7babba324868968d34ba7b658c0ef22ad78043f9d60a622e"}
May 26 16:28:27.190363 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:27.190309    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf" event={"ID":"b02ebecd-bab6-4cad-abb2-84e37997ec6f","Type":"ContainerDied","Data":"722cfd847da7d12318e8ad6f11fd4dfb7063a39355594d789d658e3d01a1d7f9"}
May 26 16:28:27.190363 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:27.190329    2552 scope.go:117] "RemoveContainer" containerID="650886c33a56157c7babba324868968d34ba7b658c0ef22ad78043f9d60a622e"
May 26 16:28:27.198814 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:27.198732    2552 scope.go:117] "RemoveContainer" containerID="967bb23dce7a5dd40c5fbf653aca8e02bce561af5b4127625fdb3bf3241beb44"
May 26 16:28:27.214457 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:27.214435    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf"]
May 26 16:28:27.217179 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:27.217159    2552 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-67b986fbb5-tpkvf"]
May 26 16:28:27.219656 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:27.219639    2552 scope.go:117] "RemoveContainer" containerID="650886c33a56157c7babba324868968d34ba7b658c0ef22ad78043f9d60a622e"
May 26 16:28:27.219892 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:28:27.219875    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"650886c33a56157c7babba324868968d34ba7b658c0ef22ad78043f9d60a622e\": container with ID starting with 650886c33a56157c7babba324868968d34ba7b658c0ef22ad78043f9d60a622e not found: ID does not exist" containerID="650886c33a56157c7babba324868968d34ba7b658c0ef22ad78043f9d60a622e"
May 26 16:28:27.219970 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:27.219899    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"650886c33a56157c7babba324868968d34ba7b658c0ef22ad78043f9d60a622e"} err="failed to get container status \"650886c33a56157c7babba324868968d34ba7b658c0ef22ad78043f9d60a622e\": rpc error: code = NotFound desc = could not find container \"650886c33a56157c7babba324868968d34ba7b658c0ef22ad78043f9d60a622e\": container with ID starting with 650886c33a56157c7babba324868968d34ba7b658c0ef22ad78043f9d60a622e not found: ID does not exist"
May 26 16:28:27.219970 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:27.219921    2552 scope.go:117] "RemoveContainer" containerID="967bb23dce7a5dd40c5fbf653aca8e02bce561af5b4127625fdb3bf3241beb44"
May 26 16:28:27.220189 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:28:27.220168    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"967bb23dce7a5dd40c5fbf653aca8e02bce561af5b4127625fdb3bf3241beb44\": container with ID starting with 967bb23dce7a5dd40c5fbf653aca8e02bce561af5b4127625fdb3bf3241beb44 not found: ID does not exist" containerID="967bb23dce7a5dd40c5fbf653aca8e02bce561af5b4127625fdb3bf3241beb44"
May 26 16:28:27.220280 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:27.220196    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"967bb23dce7a5dd40c5fbf653aca8e02bce561af5b4127625fdb3bf3241beb44"} err="failed to get container status \"967bb23dce7a5dd40c5fbf653aca8e02bce561af5b4127625fdb3bf3241beb44\": rpc error: code = NotFound desc = could not find container \"967bb23dce7a5dd40c5fbf653aca8e02bce561af5b4127625fdb3bf3241beb44\": container with ID starting with 967bb23dce7a5dd40c5fbf653aca8e02bce561af5b4127625fdb3bf3241beb44 not found: ID does not exist"
May 26 16:28:28.890676 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:28.890640    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b02ebecd-bab6-4cad-abb2-84e37997ec6f" path="/var/lib/kubelet/pods/b02ebecd-bab6-4cad-abb2-84e37997ec6f/volumes"
May 26 16:28:35.147186 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:35.147137    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr" podUID="52b5d713-f4db-49ca-93e2-7b8c6fb90530" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8001/health\": dial tcp 10.132.0.34:8001: connect: connection refused"
May 26 16:28:45.147323 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:45.147271    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr" podUID="52b5d713-f4db-49ca-93e2-7b8c6fb90530" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8001/health\": dial tcp 10.132.0.34:8001: connect: connection refused"
May 26 16:28:55.147202 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:28:55.147141    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr" podUID="52b5d713-f4db-49ca-93e2-7b8c6fb90530" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8001/health\": dial tcp 10.132.0.34:8001: connect: connection refused"
May 26 16:29:05.147460 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:29:05.147407    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr" podUID="52b5d713-f4db-49ca-93e2-7b8c6fb90530" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8001/health\": dial tcp 10.132.0.34:8001: connect: connection refused"
May 26 16:29:15.147203 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:29:15.147102    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr" podUID="52b5d713-f4db-49ca-93e2-7b8c6fb90530" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8001/health\": dial tcp 10.132.0.34:8001: connect: connection refused"
May 26 16:29:24.379714 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:29:24.379677    2552 generic.go:358] "Generic (PLEG): container finished" podID="e221ecac-2af0-4182-8b3f-eb7cb5ead412" containerID="9c5330bc410362159c08c53f806ed7707b87786a54d2cdd28ff1a22aee891727" exitCode=0
May 26 16:29:24.380241 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:29:24.379755    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6" event={"ID":"e221ecac-2af0-4182-8b3f-eb7cb5ead412","Type":"ContainerDied","Data":"9c5330bc410362159c08c53f806ed7707b87786a54d2cdd28ff1a22aee891727"}
May 26 16:29:25.146970 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:29:25.146909    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr" podUID="52b5d713-f4db-49ca-93e2-7b8c6fb90530" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8001/health\": dial tcp 10.132.0.34:8001: connect: connection refused"
May 26 16:29:25.385186 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:29:25.385152    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6" event={"ID":"e221ecac-2af0-4182-8b3f-eb7cb5ead412","Type":"ContainerStarted","Data":"00ad50603dace98259f4b7fb3273bb4b9bf883f210e29e2b6a145aa3c34b46a4"}
May 26 16:29:25.409454 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:29:25.409356    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6" podStartSLOduration=81.409337458 podStartE2EDuration="1m21.409337458s" podCreationTimestamp="2026-05-26 16:28:04 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:29:25.407999519 +0000 UTC m=+1715.116454560" watchObservedRunningTime="2026-05-26 16:29:25.409337458 +0000 UTC m=+1715.117792500"
May 26 16:29:35.147240 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:29:35.147196    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr" podUID="52b5d713-f4db-49ca-93e2-7b8c6fb90530" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8001/health\": dial tcp 10.132.0.34:8001: connect: connection refused"
May 26 16:29:35.163603 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:29:35.163574    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6"
May 26 16:29:35.163743 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:29:35.163618    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6"
May 26 16:29:35.165126 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:29:35.165099    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6" podUID="e221ecac-2af0-4182-8b3f-eb7cb5ead412" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
May 26 16:29:45.157117 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:29:45.157077    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"
May 26 16:29:45.164239 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:29:45.164193    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6" podUID="e221ecac-2af0-4182-8b3f-eb7cb5ead412" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
May 26 16:29:45.179617 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:29:45.179596    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"
May 26 16:29:55.163991 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:29:55.163924    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6" podUID="e221ecac-2af0-4182-8b3f-eb7cb5ead412" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
May 26 16:30:05.164212 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:30:05.164163    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6" podUID="e221ecac-2af0-4182-8b3f-eb7cb5ead412" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
May 26 16:30:15.163775 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:30:15.163736    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6" podUID="e221ecac-2af0-4182-8b3f-eb7cb5ead412" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
May 26 16:30:25.164653 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:30:25.164615    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6" podUID="e221ecac-2af0-4182-8b3f-eb7cb5ead412" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
May 26 16:30:35.163973 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:30:35.163911    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6" podUID="e221ecac-2af0-4182-8b3f-eb7cb5ead412" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
May 26 16:30:45.164510 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:30:45.164455    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6" podUID="e221ecac-2af0-4182-8b3f-eb7cb5ead412" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
May 26 16:30:50.893237 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:30:50.893208    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sj6db_5c0f9fc6-2bd6-4370-96d0-5ae315363c7f/ovn-acl-logging/0.log"
May 26 16:30:50.894978 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:30:50.894954    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sj6db_5c0f9fc6-2bd6-4370-96d0-5ae315363c7f/ovn-acl-logging/0.log"
May 26 16:30:55.174244 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:30:55.174214    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6"
May 26 16:30:55.181856 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:30:55.181831    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6"
May 26 16:31:02.923612 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:02.923579    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6"]
May 26 16:31:02.924239 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:02.923922    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6" podUID="e221ecac-2af0-4182-8b3f-eb7cb5ead412" containerName="main" containerID="cri-o://00ad50603dace98259f4b7fb3273bb4b9bf883f210e29e2b6a145aa3c34b46a4" gracePeriod=30
May 26 16:31:02.926334 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:02.926299    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"]
May 26 16:31:02.926773 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:02.926721    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr" podUID="52b5d713-f4db-49ca-93e2-7b8c6fb90530" containerName="main" containerID="cri-o://bf2efb595d20276932d9f5cc7bc49a4aebefa9743e03b8b0120488bb951b59f0" gracePeriod=30
May 26 16:31:20.142581 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.142544    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"]
May 26 16:31:20.143223 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.143050    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b02ebecd-bab6-4cad-abb2-84e37997ec6f" containerName="main"
May 26 16:31:20.143223 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.143066    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b02ebecd-bab6-4cad-abb2-84e37997ec6f" containerName="main"
May 26 16:31:20.143223 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.143084    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b02ebecd-bab6-4cad-abb2-84e37997ec6f" containerName="storage-initializer"
May 26 16:31:20.143223 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.143093    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b02ebecd-bab6-4cad-abb2-84e37997ec6f" containerName="storage-initializer"
May 26 16:31:20.143223 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.143182    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="b02ebecd-bab6-4cad-abb2-84e37997ec6f" containerName="main"
May 26 16:31:20.146563 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.146541    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"
May 26 16:31:20.147031 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.147005    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"]
May 26 16:31:20.149425 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.149404    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-dockercfg-6zw27\""
May 26 16:31:20.149949 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.149919    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
May 26 16:31:20.275193 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.275160    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac71befe-ce5b-427d-b4a8-045a85af25c0-home\") pod \"custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx\" (UID: \"ac71befe-ce5b-427d-b4a8-045a85af25c0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"
May 26 16:31:20.275378 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.275196    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65rdp\" (UniqueName: \"kubernetes.io/projected/ac71befe-ce5b-427d-b4a8-045a85af25c0-kube-api-access-65rdp\") pod \"custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx\" (UID: \"ac71befe-ce5b-427d-b4a8-045a85af25c0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"
May 26 16:31:20.275378 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.275274    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac71befe-ce5b-427d-b4a8-045a85af25c0-model-cache\") pod \"custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx\" (UID: \"ac71befe-ce5b-427d-b4a8-045a85af25c0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"
May 26 16:31:20.275378 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.275309    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac71befe-ce5b-427d-b4a8-045a85af25c0-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx\" (UID: \"ac71befe-ce5b-427d-b4a8-045a85af25c0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"
May 26 16:31:20.275378 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.275326    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac71befe-ce5b-427d-b4a8-045a85af25c0-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx\" (UID: \"ac71befe-ce5b-427d-b4a8-045a85af25c0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"
May 26 16:31:20.275378 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.275354    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac71befe-ce5b-427d-b4a8-045a85af25c0-dshm\") pod \"custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx\" (UID: \"ac71befe-ce5b-427d-b4a8-045a85af25c0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"
May 26 16:31:20.275668 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.275410    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac71befe-ce5b-427d-b4a8-045a85af25c0-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx\" (UID: \"ac71befe-ce5b-427d-b4a8-045a85af25c0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"
May 26 16:31:20.280795 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.280769    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj"]
May 26 16:31:20.284163 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.284146    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj"
May 26 16:31:20.295130 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.295110    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj"]
May 26 16:31:20.376824 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.376792    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac71befe-ce5b-427d-b4a8-045a85af25c0-model-cache\") pod \"custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx\" (UID: \"ac71befe-ce5b-427d-b4a8-045a85af25c0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"
May 26 16:31:20.377024 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.376834    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcaea896-01c4-4060-9986-f9e5b5aa411c-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj\" (UID: \"bcaea896-01c4-4060-9986-f9e5b5aa411c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj"
May 26 16:31:20.377024 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.376879    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac71befe-ce5b-427d-b4a8-045a85af25c0-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx\" (UID: \"ac71befe-ce5b-427d-b4a8-045a85af25c0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"
May 26 16:31:20.377024 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.376905    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac71befe-ce5b-427d-b4a8-045a85af25c0-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx\" (UID: \"ac71befe-ce5b-427d-b4a8-045a85af25c0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"
May 26 16:31:20.377024 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.376955    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcaea896-01c4-4060-9986-f9e5b5aa411c-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj\" (UID: \"bcaea896-01c4-4060-9986-f9e5b5aa411c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj"
May 26 16:31:20.377024 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.376987    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac71befe-ce5b-427d-b4a8-045a85af25c0-dshm\") pod \"custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx\" (UID: \"ac71befe-ce5b-427d-b4a8-045a85af25c0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"
May 26 16:31:20.377024 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.377013    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac71befe-ce5b-427d-b4a8-045a85af25c0-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx\" (UID: \"ac71befe-ce5b-427d-b4a8-045a85af25c0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"
May 26 16:31:20.377334 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.377038    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcaea896-01c4-4060-9986-f9e5b5aa411c-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj\" (UID: \"bcaea896-01c4-4060-9986-f9e5b5aa411c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj"
May 26 16:31:20.377334 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.377084    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsd87\" (UniqueName: \"kubernetes.io/projected/bcaea896-01c4-4060-9986-f9e5b5aa411c-kube-api-access-bsd87\") pod \"custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj\" (UID: \"bcaea896-01c4-4060-9986-f9e5b5aa411c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj"
May 26 16:31:20.377334 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.377143    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcaea896-01c4-4060-9986-f9e5b5aa411c-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj\" (UID: \"bcaea896-01c4-4060-9986-f9e5b5aa411c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj"
May 26 16:31:20.377334 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.377188    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac71befe-ce5b-427d-b4a8-045a85af25c0-model-cache\") pod \"custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx\" (UID: \"ac71befe-ce5b-427d-b4a8-045a85af25c0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"
May 26 16:31:20.377334 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.377203    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac71befe-ce5b-427d-b4a8-045a85af25c0-home\") pod \"custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx\" (UID: \"ac71befe-ce5b-427d-b4a8-045a85af25c0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"
May 26 16:31:20.377334 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.377245    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-65rdp\" (UniqueName: \"kubernetes.io/projected/ac71befe-ce5b-427d-b4a8-045a85af25c0-kube-api-access-65rdp\") pod \"custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx\" (UID: \"ac71befe-ce5b-427d-b4a8-045a85af25c0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"
May 26 16:31:20.377334 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.377280    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bcaea896-01c4-4060-9986-f9e5b5aa411c-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj\" (UID: \"bcaea896-01c4-4060-9986-f9e5b5aa411c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj"
May 26 16:31:20.377334 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.377332    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac71befe-ce5b-427d-b4a8-045a85af25c0-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx\" (UID: \"ac71befe-ce5b-427d-b4a8-045a85af25c0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"
May 26 16:31:20.377673 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.377341    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcaea896-01c4-4060-9986-f9e5b5aa411c-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj\" (UID: \"bcaea896-01c4-4060-9986-f9e5b5aa411c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj"
May 26 16:31:20.377673 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.377458    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac71befe-ce5b-427d-b4a8-045a85af25c0-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx\" (UID: \"ac71befe-ce5b-427d-b4a8-045a85af25c0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"
May 26 16:31:20.377673 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.377494    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac71befe-ce5b-427d-b4a8-045a85af25c0-home\") pod \"custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx\" (UID: \"ac71befe-ce5b-427d-b4a8-045a85af25c0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"
May 26 16:31:20.379180 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.379159    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac71befe-ce5b-427d-b4a8-045a85af25c0-dshm\") pod \"custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx\" (UID: \"ac71befe-ce5b-427d-b4a8-045a85af25c0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"
May 26 16:31:20.379285 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.379269    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac71befe-ce5b-427d-b4a8-045a85af25c0-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx\" (UID: \"ac71befe-ce5b-427d-b4a8-045a85af25c0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"
May 26 16:31:20.385534 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.385509    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65rdp\" (UniqueName: \"kubernetes.io/projected/ac71befe-ce5b-427d-b4a8-045a85af25c0-kube-api-access-65rdp\") pod \"custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx\" (UID: \"ac71befe-ce5b-427d-b4a8-045a85af25c0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"
May 26 16:31:20.456180 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.456102    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"
May 26 16:31:20.478146 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.478119    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcaea896-01c4-4060-9986-f9e5b5aa411c-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj\" (UID: \"bcaea896-01c4-4060-9986-f9e5b5aa411c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj"
May 26 16:31:20.478251 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.478171    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcaea896-01c4-4060-9986-f9e5b5aa411c-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj\" (UID: \"bcaea896-01c4-4060-9986-f9e5b5aa411c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj"
May 26 16:31:20.478251 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.478212    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcaea896-01c4-4060-9986-f9e5b5aa411c-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj\" (UID: \"bcaea896-01c4-4060-9986-f9e5b5aa411c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj"
May 26 16:31:20.478251 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.478238    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcaea896-01c4-4060-9986-f9e5b5aa411c-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj\" (UID: \"bcaea896-01c4-4060-9986-f9e5b5aa411c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj"
May 26 16:31:20.478404 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.478275    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bsd87\" (UniqueName: \"kubernetes.io/projected/bcaea896-01c4-4060-9986-f9e5b5aa411c-kube-api-access-bsd87\") pod \"custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj\" (UID: \"bcaea896-01c4-4060-9986-f9e5b5aa411c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj"
May 26 16:31:20.478404 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.478302    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcaea896-01c4-4060-9986-f9e5b5aa411c-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj\" (UID: \"bcaea896-01c4-4060-9986-f9e5b5aa411c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj"
May 26 16:31:20.478404 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.478335    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bcaea896-01c4-4060-9986-f9e5b5aa411c-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj\" (UID: \"bcaea896-01c4-4060-9986-f9e5b5aa411c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj"
May 26 16:31:20.478827 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.478796    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcaea896-01c4-4060-9986-f9e5b5aa411c-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj\" (UID: \"bcaea896-01c4-4060-9986-f9e5b5aa411c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj"
May 26 16:31:20.478970 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.478821    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcaea896-01c4-4060-9986-f9e5b5aa411c-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj\" (UID: \"bcaea896-01c4-4060-9986-f9e5b5aa411c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj"
May 26 16:31:20.478970 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.478834    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcaea896-01c4-4060-9986-f9e5b5aa411c-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj\" (UID: \"bcaea896-01c4-4060-9986-f9e5b5aa411c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj"
May 26 16:31:20.478970 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.478892    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bcaea896-01c4-4060-9986-f9e5b5aa411c-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj\" (UID: \"bcaea896-01c4-4060-9986-f9e5b5aa411c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj"
May 26 16:31:20.480876 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.480853    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcaea896-01c4-4060-9986-f9e5b5aa411c-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj\" (UID: \"bcaea896-01c4-4060-9986-f9e5b5aa411c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj"
May 26 16:31:20.481091 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.481074    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcaea896-01c4-4060-9986-f9e5b5aa411c-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj\" (UID: \"bcaea896-01c4-4060-9986-f9e5b5aa411c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj"
May 26 16:31:20.487403 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.487380    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsd87\" (UniqueName: \"kubernetes.io/projected/bcaea896-01c4-4060-9986-f9e5b5aa411c-kube-api-access-bsd87\") pod \"custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj\" (UID: \"bcaea896-01c4-4060-9986-f9e5b5aa411c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj"
May 26 16:31:20.594521 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.594483    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj"
May 26 16:31:20.603745 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.603711    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"]
May 26 16:31:20.607851 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:31:20.607805    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac71befe_ce5b_427d_b4a8_045a85af25c0.slice/crio-8531261c156fbe9a42d23cebcafbe6a29a89d69fee3ab87a4311f57c4f89babe WatchSource:0}: Error finding container 8531261c156fbe9a42d23cebcafbe6a29a89d69fee3ab87a4311f57c4f89babe: Status 404 returned error can't find the container with id 8531261c156fbe9a42d23cebcafbe6a29a89d69fee3ab87a4311f57c4f89babe
May 26 16:31:20.609748 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.609728    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:31:20.734886 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.734861    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj"]
May 26 16:31:20.736469 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:31:20.736444    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcaea896_01c4_4060_9986_f9e5b5aa411c.slice/crio-81ca383809195b74bdeb2e1eb9ebd7e766f87ff326509fab7814c1121f49d016 WatchSource:0}: Error finding container 81ca383809195b74bdeb2e1eb9ebd7e766f87ff326509fab7814c1121f49d016: Status 404 returned error can't find the container with id 81ca383809195b74bdeb2e1eb9ebd7e766f87ff326509fab7814c1121f49d016
May 26 16:31:20.771137 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.771107    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj" event={"ID":"bcaea896-01c4-4060-9986-f9e5b5aa411c","Type":"ContainerStarted","Data":"81ca383809195b74bdeb2e1eb9ebd7e766f87ff326509fab7814c1121f49d016"}
May 26 16:31:20.772290 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.772266    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx" event={"ID":"ac71befe-ce5b-427d-b4a8-045a85af25c0","Type":"ContainerStarted","Data":"1e7e451d9a5634d03e03b422465af3d4055277409161d82ff87e28f1fe3cd6a2"}
May 26 16:31:20.772389 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.772298    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx" event={"ID":"ac71befe-ce5b-427d-b4a8-045a85af25c0","Type":"ContainerStarted","Data":"8531261c156fbe9a42d23cebcafbe6a29a89d69fee3ab87a4311f57c4f89babe"}
May 26 16:31:20.772452 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:20.772413    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"
May 26 16:31:21.778731 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:21.778694    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx" event={"ID":"ac71befe-ce5b-427d-b4a8-045a85af25c0","Type":"ContainerStarted","Data":"8fb0b6833f96146dd3df31d90bfe5b2ecd959e2c7dc18f559a234bac4db37c8e"}
May 26 16:31:21.780316 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:21.780287    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj" event={"ID":"bcaea896-01c4-4060-9986-f9e5b5aa411c","Type":"ContainerStarted","Data":"3131837e33c1714f92d9eac49f7ecc275bfccae31c803ae16aea9e651890ffe1"}
May 26 16:31:32.796957 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:32.796900    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"
May 26 16:31:32.927519 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:32.927481    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr" podUID="52b5d713-f4db-49ca-93e2-7b8c6fb90530" containerName="llm-d-routing-sidecar" containerID="cri-o://d58970c73c008434d1d06898f8b3b661b04bbd7c85f7952433d08cb0a81e10b1" gracePeriod=2
May 26 16:31:33.249692 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.249671    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr_52b5d713-f4db-49ca-93e2-7b8c6fb90530/main/0.log"
May 26 16:31:33.250444 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.250425    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"
May 26 16:31:33.253158 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.253139    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6"
May 26 16:31:33.293070 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.293040    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e221ecac-2af0-4182-8b3f-eb7cb5ead412-home\") pod \"e221ecac-2af0-4182-8b3f-eb7cb5ead412\" (UID: \"e221ecac-2af0-4182-8b3f-eb7cb5ead412\") "
May 26 16:31:33.293225 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.293186    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52b5d713-f4db-49ca-93e2-7b8c6fb90530-dshm\") pod \"52b5d713-f4db-49ca-93e2-7b8c6fb90530\" (UID: \"52b5d713-f4db-49ca-93e2-7b8c6fb90530\") "
May 26 16:31:33.293297 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.293233    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52b5d713-f4db-49ca-93e2-7b8c6fb90530-kserve-provision-location\") pod \"52b5d713-f4db-49ca-93e2-7b8c6fb90530\" (UID: \"52b5d713-f4db-49ca-93e2-7b8c6fb90530\") "
May 26 16:31:33.293297 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.293260    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e221ecac-2af0-4182-8b3f-eb7cb5ead412-model-cache\") pod \"e221ecac-2af0-4182-8b3f-eb7cb5ead412\" (UID: \"e221ecac-2af0-4182-8b3f-eb7cb5ead412\") "
May 26 16:31:33.293412 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.293312    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpgbz\" (UniqueName: \"kubernetes.io/projected/52b5d713-f4db-49ca-93e2-7b8c6fb90530-kube-api-access-tpgbz\") pod \"52b5d713-f4db-49ca-93e2-7b8c6fb90530\" (UID: \"52b5d713-f4db-49ca-93e2-7b8c6fb90530\") "
May 26 16:31:33.296587 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.293522    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e221ecac-2af0-4182-8b3f-eb7cb5ead412-model-cache" (OuterVolumeSpecName: "model-cache") pod "e221ecac-2af0-4182-8b3f-eb7cb5ead412" (UID: "e221ecac-2af0-4182-8b3f-eb7cb5ead412"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:31:33.296587 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.293801    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e221ecac-2af0-4182-8b3f-eb7cb5ead412-home" (OuterVolumeSpecName: "home") pod "e221ecac-2af0-4182-8b3f-eb7cb5ead412" (UID: "e221ecac-2af0-4182-8b3f-eb7cb5ead412"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:31:33.296587 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.293423    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e221ecac-2af0-4182-8b3f-eb7cb5ead412-tls-certs\") pod \"e221ecac-2af0-4182-8b3f-eb7cb5ead412\" (UID: \"e221ecac-2af0-4182-8b3f-eb7cb5ead412\") "
May 26 16:31:33.296587 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.294266    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e221ecac-2af0-4182-8b3f-eb7cb5ead412-kserve-provision-location\") pod \"e221ecac-2af0-4182-8b3f-eb7cb5ead412\" (UID: \"e221ecac-2af0-4182-8b3f-eb7cb5ead412\") "
May 26 16:31:33.296587 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.294300    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52b5d713-f4db-49ca-93e2-7b8c6fb90530-tmp-dir\") pod \"52b5d713-f4db-49ca-93e2-7b8c6fb90530\" (UID: \"52b5d713-f4db-49ca-93e2-7b8c6fb90530\") "
May 26 16:31:33.296587 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.294330    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52b5d713-f4db-49ca-93e2-7b8c6fb90530-home\") pod \"52b5d713-f4db-49ca-93e2-7b8c6fb90530\" (UID: \"52b5d713-f4db-49ca-93e2-7b8c6fb90530\") "
May 26 16:31:33.296587 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.294373    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e221ecac-2af0-4182-8b3f-eb7cb5ead412-tmp-dir\") pod \"e221ecac-2af0-4182-8b3f-eb7cb5ead412\" (UID: \"e221ecac-2af0-4182-8b3f-eb7cb5ead412\") "
May 26 16:31:33.296587 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.294440    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-889nm\" (UniqueName: \"kubernetes.io/projected/e221ecac-2af0-4182-8b3f-eb7cb5ead412-kube-api-access-889nm\") pod \"e221ecac-2af0-4182-8b3f-eb7cb5ead412\" (UID: \"e221ecac-2af0-4182-8b3f-eb7cb5ead412\") "
May 26 16:31:33.296587 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.294497    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52b5d713-f4db-49ca-93e2-7b8c6fb90530-model-cache\") pod \"52b5d713-f4db-49ca-93e2-7b8c6fb90530\" (UID: \"52b5d713-f4db-49ca-93e2-7b8c6fb90530\") "
May 26 16:31:33.296587 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.294519    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e221ecac-2af0-4182-8b3f-eb7cb5ead412-dshm\") pod \"e221ecac-2af0-4182-8b3f-eb7cb5ead412\" (UID: \"e221ecac-2af0-4182-8b3f-eb7cb5ead412\") "
May 26 16:31:33.296587 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.294557    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52b5d713-f4db-49ca-93e2-7b8c6fb90530-tls-certs\") pod \"52b5d713-f4db-49ca-93e2-7b8c6fb90530\" (UID: \"52b5d713-f4db-49ca-93e2-7b8c6fb90530\") "
May 26 16:31:33.296587 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.294863    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e221ecac-2af0-4182-8b3f-eb7cb5ead412-home\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:31:33.296587 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.294881    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e221ecac-2af0-4182-8b3f-eb7cb5ead412-model-cache\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:31:33.297341 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.296748    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b5d713-f4db-49ca-93e2-7b8c6fb90530-model-cache" (OuterVolumeSpecName: "model-cache") pod "52b5d713-f4db-49ca-93e2-7b8c6fb90530" (UID: "52b5d713-f4db-49ca-93e2-7b8c6fb90530"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:31:33.299723 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.299403    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52b5d713-f4db-49ca-93e2-7b8c6fb90530-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "52b5d713-f4db-49ca-93e2-7b8c6fb90530" (UID: "52b5d713-f4db-49ca-93e2-7b8c6fb90530"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:31:33.299723 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.299468    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b5d713-f4db-49ca-93e2-7b8c6fb90530-dshm" (OuterVolumeSpecName: "dshm") pod "52b5d713-f4db-49ca-93e2-7b8c6fb90530" (UID: "52b5d713-f4db-49ca-93e2-7b8c6fb90530"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:31:33.300322 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.300159    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e221ecac-2af0-4182-8b3f-eb7cb5ead412-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e221ecac-2af0-4182-8b3f-eb7cb5ead412" (UID: "e221ecac-2af0-4182-8b3f-eb7cb5ead412"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:31:33.301070 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.300532    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e221ecac-2af0-4182-8b3f-eb7cb5ead412-kube-api-access-889nm" (OuterVolumeSpecName: "kube-api-access-889nm") pod "e221ecac-2af0-4182-8b3f-eb7cb5ead412" (UID: "e221ecac-2af0-4182-8b3f-eb7cb5ead412"). InnerVolumeSpecName "kube-api-access-889nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:31:33.301070 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.300628    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e221ecac-2af0-4182-8b3f-eb7cb5ead412-dshm" (OuterVolumeSpecName: "dshm") pod "e221ecac-2af0-4182-8b3f-eb7cb5ead412" (UID: "e221ecac-2af0-4182-8b3f-eb7cb5ead412"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:31:33.301500 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.301464    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b5d713-f4db-49ca-93e2-7b8c6fb90530-home" (OuterVolumeSpecName: "home") pod "52b5d713-f4db-49ca-93e2-7b8c6fb90530" (UID: "52b5d713-f4db-49ca-93e2-7b8c6fb90530"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:31:33.301741 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.301717    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52b5d713-f4db-49ca-93e2-7b8c6fb90530-kube-api-access-tpgbz" (OuterVolumeSpecName: "kube-api-access-tpgbz") pod "52b5d713-f4db-49ca-93e2-7b8c6fb90530" (UID: "52b5d713-f4db-49ca-93e2-7b8c6fb90530"). InnerVolumeSpecName "kube-api-access-tpgbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:31:33.312423 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.312370    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b5d713-f4db-49ca-93e2-7b8c6fb90530-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "52b5d713-f4db-49ca-93e2-7b8c6fb90530" (UID: "52b5d713-f4db-49ca-93e2-7b8c6fb90530"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:31:33.318869 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.318843    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e221ecac-2af0-4182-8b3f-eb7cb5ead412-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e221ecac-2af0-4182-8b3f-eb7cb5ead412" (UID: "e221ecac-2af0-4182-8b3f-eb7cb5ead412"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:31:33.363198 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.363163    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b5d713-f4db-49ca-93e2-7b8c6fb90530-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "52b5d713-f4db-49ca-93e2-7b8c6fb90530" (UID: "52b5d713-f4db-49ca-93e2-7b8c6fb90530"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:31:33.365786 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.365759    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e221ecac-2af0-4182-8b3f-eb7cb5ead412-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e221ecac-2af0-4182-8b3f-eb7cb5ead412" (UID: "e221ecac-2af0-4182-8b3f-eb7cb5ead412"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:31:33.396137 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.396113    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-889nm\" (UniqueName: \"kubernetes.io/projected/e221ecac-2af0-4182-8b3f-eb7cb5ead412-kube-api-access-889nm\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:31:33.396137 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.396137    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52b5d713-f4db-49ca-93e2-7b8c6fb90530-model-cache\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:31:33.396268 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.396147    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e221ecac-2af0-4182-8b3f-eb7cb5ead412-dshm\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:31:33.396268 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.396157    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52b5d713-f4db-49ca-93e2-7b8c6fb90530-tls-certs\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:31:33.396268 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.396165    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52b5d713-f4db-49ca-93e2-7b8c6fb90530-dshm\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:31:33.396268 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.396174    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52b5d713-f4db-49ca-93e2-7b8c6fb90530-kserve-provision-location\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:31:33.396268 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.396187    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpgbz\" (UniqueName: \"kubernetes.io/projected/52b5d713-f4db-49ca-93e2-7b8c6fb90530-kube-api-access-tpgbz\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:31:33.396268 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.396203    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e221ecac-2af0-4182-8b3f-eb7cb5ead412-tls-certs\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:31:33.396268 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.396213    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e221ecac-2af0-4182-8b3f-eb7cb5ead412-kserve-provision-location\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:31:33.396268 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.396221    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52b5d713-f4db-49ca-93e2-7b8c6fb90530-tmp-dir\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:31:33.396268 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.396229    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52b5d713-f4db-49ca-93e2-7b8c6fb90530-home\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:31:33.396268 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.396238    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e221ecac-2af0-4182-8b3f-eb7cb5ead412-tmp-dir\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:31:33.824650 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.824610    2552 generic.go:358] "Generic (PLEG): container finished" podID="e221ecac-2af0-4182-8b3f-eb7cb5ead412" containerID="00ad50603dace98259f4b7fb3273bb4b9bf883f210e29e2b6a145aa3c34b46a4" exitCode=137
May 26 16:31:33.825132 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.824688    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6"
May 26 16:31:33.825132 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.824691    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6" event={"ID":"e221ecac-2af0-4182-8b3f-eb7cb5ead412","Type":"ContainerDied","Data":"00ad50603dace98259f4b7fb3273bb4b9bf883f210e29e2b6a145aa3c34b46a4"}
May 26 16:31:33.825132 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.824728    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6" event={"ID":"e221ecac-2af0-4182-8b3f-eb7cb5ead412","Type":"ContainerDied","Data":"0f0178bb392588bbc62da286638650d76c1acd746299e3e6e59e15eaac73ca8c"}
May 26 16:31:33.825132 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.824748    2552 scope.go:117] "RemoveContainer" containerID="00ad50603dace98259f4b7fb3273bb4b9bf883f210e29e2b6a145aa3c34b46a4"
May 26 16:31:33.825989 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.825974    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr_52b5d713-f4db-49ca-93e2-7b8c6fb90530/main/0.log"
May 26 16:31:33.826693 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.826668    2552 generic.go:358] "Generic (PLEG): container finished" podID="52b5d713-f4db-49ca-93e2-7b8c6fb90530" containerID="bf2efb595d20276932d9f5cc7bc49a4aebefa9743e03b8b0120488bb951b59f0" exitCode=137
May 26 16:31:33.826693 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.826692    2552 generic.go:358] "Generic (PLEG): container finished" podID="52b5d713-f4db-49ca-93e2-7b8c6fb90530" containerID="d58970c73c008434d1d06898f8b3b661b04bbd7c85f7952433d08cb0a81e10b1" exitCode=0
May 26 16:31:33.826827 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.826712    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr" event={"ID":"52b5d713-f4db-49ca-93e2-7b8c6fb90530","Type":"ContainerDied","Data":"bf2efb595d20276932d9f5cc7bc49a4aebefa9743e03b8b0120488bb951b59f0"}
May 26 16:31:33.826827 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.826724    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"
May 26 16:31:33.826827 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.826756    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr" event={"ID":"52b5d713-f4db-49ca-93e2-7b8c6fb90530","Type":"ContainerDied","Data":"d58970c73c008434d1d06898f8b3b661b04bbd7c85f7952433d08cb0a81e10b1"}
May 26 16:31:33.826827 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.826775    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr" event={"ID":"52b5d713-f4db-49ca-93e2-7b8c6fb90530","Type":"ContainerDied","Data":"37011a53dbda9b25f8d563386bc1312318bdadecfd61c29f1d8f148581cd38bc"}
May 26 16:31:33.837512 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.837365    2552 scope.go:117] "RemoveContainer" containerID="9c5330bc410362159c08c53f806ed7707b87786a54d2cdd28ff1a22aee891727"
May 26 16:31:33.853605 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.853573    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6"]
May 26 16:31:33.854329 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.854313    2552 scope.go:117] "RemoveContainer" containerID="00ad50603dace98259f4b7fb3273bb4b9bf883f210e29e2b6a145aa3c34b46a4"
May 26 16:31:33.854589 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:31:33.854570    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00ad50603dace98259f4b7fb3273bb4b9bf883f210e29e2b6a145aa3c34b46a4\": container with ID starting with 00ad50603dace98259f4b7fb3273bb4b9bf883f210e29e2b6a145aa3c34b46a4 not found: ID does not exist" containerID="00ad50603dace98259f4b7fb3273bb4b9bf883f210e29e2b6a145aa3c34b46a4"
May 26 16:31:33.854660 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.854601    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00ad50603dace98259f4b7fb3273bb4b9bf883f210e29e2b6a145aa3c34b46a4"} err="failed to get container status \"00ad50603dace98259f4b7fb3273bb4b9bf883f210e29e2b6a145aa3c34b46a4\": rpc error: code = NotFound desc = could not find container \"00ad50603dace98259f4b7fb3273bb4b9bf883f210e29e2b6a145aa3c34b46a4\": container with ID starting with 00ad50603dace98259f4b7fb3273bb4b9bf883f210e29e2b6a145aa3c34b46a4 not found: ID does not exist"
May 26 16:31:33.854660 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.854625    2552 scope.go:117] "RemoveContainer" containerID="9c5330bc410362159c08c53f806ed7707b87786a54d2cdd28ff1a22aee891727"
May 26 16:31:33.854974 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:31:33.854918    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c5330bc410362159c08c53f806ed7707b87786a54d2cdd28ff1a22aee891727\": container with ID starting with 9c5330bc410362159c08c53f806ed7707b87786a54d2cdd28ff1a22aee891727 not found: ID does not exist" containerID="9c5330bc410362159c08c53f806ed7707b87786a54d2cdd28ff1a22aee891727"
May 26 16:31:33.854974 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.854971    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c5330bc410362159c08c53f806ed7707b87786a54d2cdd28ff1a22aee891727"} err="failed to get container status \"9c5330bc410362159c08c53f806ed7707b87786a54d2cdd28ff1a22aee891727\": rpc error: code = NotFound desc = could not find container \"9c5330bc410362159c08c53f806ed7707b87786a54d2cdd28ff1a22aee891727\": container with ID starting with 9c5330bc410362159c08c53f806ed7707b87786a54d2cdd28ff1a22aee891727 not found: ID does not exist"
May 26 16:31:33.854974 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.854992    2552 scope.go:117] "RemoveContainer" containerID="bf2efb595d20276932d9f5cc7bc49a4aebefa9743e03b8b0120488bb951b59f0"
May 26 16:31:33.857097 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.857074    2552 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bm9hz6"]
May 26 16:31:33.862169 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.862151    2552 scope.go:117] "RemoveContainer" containerID="8034e75dbe46007099c5d741c67fc1cc24531c426ffce84adc17570d03f696e7"
May 26 16:31:33.868736 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.868715    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"]
May 26 16:31:33.873777 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.873757    2552 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-b867f9f64-pt6rr"]
May 26 16:31:33.933858 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.933616    2552 scope.go:117] "RemoveContainer" containerID="d58970c73c008434d1d06898f8b3b661b04bbd7c85f7952433d08cb0a81e10b1"
May 26 16:31:33.940893 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.940871    2552 scope.go:117] "RemoveContainer" containerID="bf2efb595d20276932d9f5cc7bc49a4aebefa9743e03b8b0120488bb951b59f0"
May 26 16:31:33.941197 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:31:33.941177    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf2efb595d20276932d9f5cc7bc49a4aebefa9743e03b8b0120488bb951b59f0\": container with ID starting with bf2efb595d20276932d9f5cc7bc49a4aebefa9743e03b8b0120488bb951b59f0 not found: ID does not exist" containerID="bf2efb595d20276932d9f5cc7bc49a4aebefa9743e03b8b0120488bb951b59f0"
May 26 16:31:33.941249 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.941206    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf2efb595d20276932d9f5cc7bc49a4aebefa9743e03b8b0120488bb951b59f0"} err="failed to get container status \"bf2efb595d20276932d9f5cc7bc49a4aebefa9743e03b8b0120488bb951b59f0\": rpc error: code = NotFound desc = could not find container \"bf2efb595d20276932d9f5cc7bc49a4aebefa9743e03b8b0120488bb951b59f0\": container with ID starting with bf2efb595d20276932d9f5cc7bc49a4aebefa9743e03b8b0120488bb951b59f0 not found: ID does not exist"
May 26 16:31:33.941249 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.941226    2552 scope.go:117] "RemoveContainer" containerID="8034e75dbe46007099c5d741c67fc1cc24531c426ffce84adc17570d03f696e7"
May 26 16:31:33.941473 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:31:33.941453    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8034e75dbe46007099c5d741c67fc1cc24531c426ffce84adc17570d03f696e7\": container with ID starting with 8034e75dbe46007099c5d741c67fc1cc24531c426ffce84adc17570d03f696e7 not found: ID does not exist" containerID="8034e75dbe46007099c5d741c67fc1cc24531c426ffce84adc17570d03f696e7"
May 26 16:31:33.941536 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.941483    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8034e75dbe46007099c5d741c67fc1cc24531c426ffce84adc17570d03f696e7"} err="failed to get container status \"8034e75dbe46007099c5d741c67fc1cc24531c426ffce84adc17570d03f696e7\": rpc error: code = NotFound desc = could not find container \"8034e75dbe46007099c5d741c67fc1cc24531c426ffce84adc17570d03f696e7\": container with ID starting with 8034e75dbe46007099c5d741c67fc1cc24531c426ffce84adc17570d03f696e7 not found: ID does not exist"
May 26 16:31:33.941536 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.941506    2552 scope.go:117] "RemoveContainer" containerID="d58970c73c008434d1d06898f8b3b661b04bbd7c85f7952433d08cb0a81e10b1"
May 26 16:31:33.941761 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:31:33.941744    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d58970c73c008434d1d06898f8b3b661b04bbd7c85f7952433d08cb0a81e10b1\": container with ID starting with d58970c73c008434d1d06898f8b3b661b04bbd7c85f7952433d08cb0a81e10b1 not found: ID does not exist" containerID="d58970c73c008434d1d06898f8b3b661b04bbd7c85f7952433d08cb0a81e10b1"
May 26 16:31:33.941797 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.941766    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d58970c73c008434d1d06898f8b3b661b04bbd7c85f7952433d08cb0a81e10b1"} err="failed to get container status \"d58970c73c008434d1d06898f8b3b661b04bbd7c85f7952433d08cb0a81e10b1\": rpc error: code = NotFound desc = could not find container \"d58970c73c008434d1d06898f8b3b661b04bbd7c85f7952433d08cb0a81e10b1\": container with ID starting with d58970c73c008434d1d06898f8b3b661b04bbd7c85f7952433d08cb0a81e10b1 not found: ID does not exist"
May 26 16:31:33.941797 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.941781    2552 scope.go:117] "RemoveContainer" containerID="bf2efb595d20276932d9f5cc7bc49a4aebefa9743e03b8b0120488bb951b59f0"
May 26 16:31:33.942057 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.942024    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf2efb595d20276932d9f5cc7bc49a4aebefa9743e03b8b0120488bb951b59f0"} err="failed to get container status \"bf2efb595d20276932d9f5cc7bc49a4aebefa9743e03b8b0120488bb951b59f0\": rpc error: code = NotFound desc = could not find container \"bf2efb595d20276932d9f5cc7bc49a4aebefa9743e03b8b0120488bb951b59f0\": container with ID starting with bf2efb595d20276932d9f5cc7bc49a4aebefa9743e03b8b0120488bb951b59f0 not found: ID does not exist"
May 26 16:31:33.942057 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.942049    2552 scope.go:117] "RemoveContainer" containerID="8034e75dbe46007099c5d741c67fc1cc24531c426ffce84adc17570d03f696e7"
May 26 16:31:33.942263 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.942248    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8034e75dbe46007099c5d741c67fc1cc24531c426ffce84adc17570d03f696e7"} err="failed to get container status \"8034e75dbe46007099c5d741c67fc1cc24531c426ffce84adc17570d03f696e7\": rpc error: code = NotFound desc = could not find container \"8034e75dbe46007099c5d741c67fc1cc24531c426ffce84adc17570d03f696e7\": container with ID starting with 8034e75dbe46007099c5d741c67fc1cc24531c426ffce84adc17570d03f696e7 not found: ID does not exist"
May 26 16:31:33.942310 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.942263    2552 scope.go:117] "RemoveContainer" containerID="d58970c73c008434d1d06898f8b3b661b04bbd7c85f7952433d08cb0a81e10b1"
May 26 16:31:33.942498 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:33.942478    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d58970c73c008434d1d06898f8b3b661b04bbd7c85f7952433d08cb0a81e10b1"} err="failed to get container status \"d58970c73c008434d1d06898f8b3b661b04bbd7c85f7952433d08cb0a81e10b1\": rpc error: code = NotFound desc = could not find container \"d58970c73c008434d1d06898f8b3b661b04bbd7c85f7952433d08cb0a81e10b1\": container with ID starting with d58970c73c008434d1d06898f8b3b661b04bbd7c85f7952433d08cb0a81e10b1 not found: ID does not exist"
May 26 16:31:34.890242 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:34.890210    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52b5d713-f4db-49ca-93e2-7b8c6fb90530" path="/var/lib/kubelet/pods/52b5d713-f4db-49ca-93e2-7b8c6fb90530/volumes"
May 26 16:31:34.890670 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:31:34.890656    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e221ecac-2af0-4182-8b3f-eb7cb5ead412" path="/var/lib/kubelet/pods/e221ecac-2af0-4182-8b3f-eb7cb5ead412/volumes"
May 26 16:32:09.951295 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:32:09.951262    2552 generic.go:358] "Generic (PLEG): container finished" podID="bcaea896-01c4-4060-9986-f9e5b5aa411c" containerID="3131837e33c1714f92d9eac49f7ecc275bfccae31c803ae16aea9e651890ffe1" exitCode=0
May 26 16:32:09.951901 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:32:09.951337    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj" event={"ID":"bcaea896-01c4-4060-9986-f9e5b5aa411c","Type":"ContainerDied","Data":"3131837e33c1714f92d9eac49f7ecc275bfccae31c803ae16aea9e651890ffe1"}
May 26 16:32:10.956079 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:32:10.956045    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj" event={"ID":"bcaea896-01c4-4060-9986-f9e5b5aa411c","Type":"ContainerStarted","Data":"1b9794c726b62439cc925ab8540b8996c210d24d9564d88cb6399a1a70a189c5"}
May 26 16:32:10.980020 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:32:10.979966    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj" podStartSLOduration=50.979947714 podStartE2EDuration="50.979947714s" podCreationTimestamp="2026-05-26 16:31:20 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:32:10.979518022 +0000 UTC m=+1880.687973063" watchObservedRunningTime="2026-05-26 16:32:10.979947714 +0000 UTC m=+1880.688402758"
May 26 16:32:20.595480 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:32:20.595438    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj"
May 26 16:32:20.595480 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:32:20.595479    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj"
May 26 16:32:20.597146 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:32:20.597120    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj" podUID="bcaea896-01c4-4060-9986-f9e5b5aa411c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
May 26 16:32:30.017468 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:32:30.017434    2552 generic.go:358] "Generic (PLEG): container finished" podID="ac71befe-ce5b-427d-b4a8-045a85af25c0" containerID="8fb0b6833f96146dd3df31d90bfe5b2ecd959e2c7dc18f559a234bac4db37c8e" exitCode=0
May 26 16:32:30.017853 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:32:30.017519    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx" event={"ID":"ac71befe-ce5b-427d-b4a8-045a85af25c0","Type":"ContainerDied","Data":"8fb0b6833f96146dd3df31d90bfe5b2ecd959e2c7dc18f559a234bac4db37c8e"}
May 26 16:32:30.595149 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:32:30.595100    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj" podUID="bcaea896-01c4-4060-9986-f9e5b5aa411c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
May 26 16:32:31.022413 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:32:31.022375    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx" event={"ID":"ac71befe-ce5b-427d-b4a8-045a85af25c0","Type":"ContainerStarted","Data":"570d222e9df4b48938a8ecc208d8b2a93384b28445015e350f675a2aeeac33e9"}
May 26 16:32:31.050667 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:32:31.050611    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx" podStartSLOduration=71.050589254 podStartE2EDuration="1m11.050589254s" podCreationTimestamp="2026-05-26 16:31:20 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:32:31.047772008 +0000 UTC m=+1900.756227073" watchObservedRunningTime="2026-05-26 16:32:31.050589254 +0000 UTC m=+1900.759044297"
May 26 16:32:40.456517 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:32:40.456472    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"
May 26 16:32:40.457041 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:32:40.456637    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"
May 26 16:32:40.457041 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:32:40.456846    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx" podUID="ac71befe-ce5b-427d-b4a8-045a85af25c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8001/health\": dial tcp 10.132.0.36:8001: connect: connection refused"
May 26 16:32:40.595019 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:32:40.594975    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj" podUID="bcaea896-01c4-4060-9986-f9e5b5aa411c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
May 26 16:32:50.456580 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:32:50.456530    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx" podUID="ac71befe-ce5b-427d-b4a8-045a85af25c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8001/health\": dial tcp 10.132.0.36:8001: connect: connection refused"
May 26 16:32:50.595218 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:32:50.595179    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj" podUID="bcaea896-01c4-4060-9986-f9e5b5aa411c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
May 26 16:33:00.456465 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:00.456414    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx" podUID="ac71befe-ce5b-427d-b4a8-045a85af25c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8001/health\": dial tcp 10.132.0.36:8001: connect: connection refused"
May 26 16:33:00.595185 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:00.595143    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj" podUID="bcaea896-01c4-4060-9986-f9e5b5aa411c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
May 26 16:33:10.457435 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:10.457383    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx" podUID="ac71befe-ce5b-427d-b4a8-045a85af25c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8001/health\": dial tcp 10.132.0.36:8001: connect: connection refused"
May 26 16:33:10.595698 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:10.595656    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj" podUID="bcaea896-01c4-4060-9986-f9e5b5aa411c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
May 26 16:33:20.457297 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:20.457244    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx" podUID="ac71befe-ce5b-427d-b4a8-045a85af25c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8001/health\": dial tcp 10.132.0.36:8001: connect: connection refused"
May 26 16:33:20.595134 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:20.595092    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj" podUID="bcaea896-01c4-4060-9986-f9e5b5aa411c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
May 26 16:33:30.456708 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:30.456659    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx" podUID="ac71befe-ce5b-427d-b4a8-045a85af25c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8001/health\": dial tcp 10.132.0.36:8001: connect: connection refused"
May 26 16:33:30.595049 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:30.595003    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj" podUID="bcaea896-01c4-4060-9986-f9e5b5aa411c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
May 26 16:33:40.457492 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:40.457394    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx" podUID="ac71befe-ce5b-427d-b4a8-045a85af25c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8001/health\": dial tcp 10.132.0.36:8001: connect: connection refused"
May 26 16:33:40.595317 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:40.595271    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj" podUID="bcaea896-01c4-4060-9986-f9e5b5aa411c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
May 26 16:33:50.456904 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:50.456846    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx" podUID="ac71befe-ce5b-427d-b4a8-045a85af25c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8001/health\": dial tcp 10.132.0.36:8001: connect: connection refused"
May 26 16:33:50.595352 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:50.595299    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj" podUID="bcaea896-01c4-4060-9986-f9e5b5aa411c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
May 26 16:33:58.515655 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.515615    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk"]
May 26 16:33:58.516516 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.516058    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e221ecac-2af0-4182-8b3f-eb7cb5ead412" containerName="main"
May 26 16:33:58.516516 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.516078    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e221ecac-2af0-4182-8b3f-eb7cb5ead412" containerName="main"
May 26 16:33:58.516516 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.516089    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52b5d713-f4db-49ca-93e2-7b8c6fb90530" containerName="llm-d-routing-sidecar"
May 26 16:33:58.516516 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.516095    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="52b5d713-f4db-49ca-93e2-7b8c6fb90530" containerName="llm-d-routing-sidecar"
May 26 16:33:58.516516 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.516120    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e221ecac-2af0-4182-8b3f-eb7cb5ead412" containerName="storage-initializer"
May 26 16:33:58.516516 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.516127    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e221ecac-2af0-4182-8b3f-eb7cb5ead412" containerName="storage-initializer"
May 26 16:33:58.516516 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.516143    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52b5d713-f4db-49ca-93e2-7b8c6fb90530" containerName="storage-initializer"
May 26 16:33:58.516516 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.516155    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="52b5d713-f4db-49ca-93e2-7b8c6fb90530" containerName="storage-initializer"
May 26 16:33:58.516516 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.516163    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52b5d713-f4db-49ca-93e2-7b8c6fb90530" containerName="main"
May 26 16:33:58.516516 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.516170    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="52b5d713-f4db-49ca-93e2-7b8c6fb90530" containerName="main"
May 26 16:33:58.516516 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.516228    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="e221ecac-2af0-4182-8b3f-eb7cb5ead412" containerName="main"
May 26 16:33:58.516516 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.516239    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="52b5d713-f4db-49ca-93e2-7b8c6fb90530" containerName="main"
May 26 16:33:58.516516 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.516246    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="52b5d713-f4db-49ca-93e2-7b8c6fb90530" containerName="llm-d-routing-sidecar"
May 26 16:33:58.518805 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.518787    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk"
May 26 16:33:58.521846 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.521826    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
May 26 16:33:58.532090 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.532059    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk"]
May 26 16:33:58.547015 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.546979    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-home\") pod \"stop-feature-test-kserve-c5d69ddfc-k9mbk\" (UID: \"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk"
May 26 16:33:58.547015 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.547018    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-model-cache\") pod \"stop-feature-test-kserve-c5d69ddfc-k9mbk\" (UID: \"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk"
May 26 16:33:58.547285 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.547045    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhtgm\" (UniqueName: \"kubernetes.io/projected/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-kube-api-access-lhtgm\") pod \"stop-feature-test-kserve-c5d69ddfc-k9mbk\" (UID: \"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk"
May 26 16:33:58.547285 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.547204    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-kserve-provision-location\") pod \"stop-feature-test-kserve-c5d69ddfc-k9mbk\" (UID: \"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk"
May 26 16:33:58.547285 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.547253    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-tls-certs\") pod \"stop-feature-test-kserve-c5d69ddfc-k9mbk\" (UID: \"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk"
May 26 16:33:58.547451 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.547329    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-tmp-dir\") pod \"stop-feature-test-kserve-c5d69ddfc-k9mbk\" (UID: \"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk"
May 26 16:33:58.547451 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.547365    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-dshm\") pod \"stop-feature-test-kserve-c5d69ddfc-k9mbk\" (UID: \"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk"
May 26 16:33:58.648478 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.648443    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-tmp-dir\") pod \"stop-feature-test-kserve-c5d69ddfc-k9mbk\" (UID: \"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk"
May 26 16:33:58.648689 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.648496    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-dshm\") pod \"stop-feature-test-kserve-c5d69ddfc-k9mbk\" (UID: \"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk"
May 26 16:33:58.648689 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.648531    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-home\") pod \"stop-feature-test-kserve-c5d69ddfc-k9mbk\" (UID: \"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk"
May 26 16:33:58.648689 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.648551    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-model-cache\") pod \"stop-feature-test-kserve-c5d69ddfc-k9mbk\" (UID: \"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk"
May 26 16:33:58.648689 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.648586    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lhtgm\" (UniqueName: \"kubernetes.io/projected/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-kube-api-access-lhtgm\") pod \"stop-feature-test-kserve-c5d69ddfc-k9mbk\" (UID: \"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk"
May 26 16:33:58.648689 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.648656    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-kserve-provision-location\") pod \"stop-feature-test-kserve-c5d69ddfc-k9mbk\" (UID: \"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk"
May 26 16:33:58.648689 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.648681    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-tls-certs\") pod \"stop-feature-test-kserve-c5d69ddfc-k9mbk\" (UID: \"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk"
May 26 16:33:58.649030 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.648953    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-tmp-dir\") pod \"stop-feature-test-kserve-c5d69ddfc-k9mbk\" (UID: \"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk"
May 26 16:33:58.649030 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.648983    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-model-cache\") pod \"stop-feature-test-kserve-c5d69ddfc-k9mbk\" (UID: \"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk"
May 26 16:33:58.649282 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.649255    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-kserve-provision-location\") pod \"stop-feature-test-kserve-c5d69ddfc-k9mbk\" (UID: \"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk"
May 26 16:33:58.649467 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.649449    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-home\") pod \"stop-feature-test-kserve-c5d69ddfc-k9mbk\" (UID: \"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk"
May 26 16:33:58.650963 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.650909    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-dshm\") pod \"stop-feature-test-kserve-c5d69ddfc-k9mbk\" (UID: \"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk"
May 26 16:33:58.651254 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.651238    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-tls-certs\") pod \"stop-feature-test-kserve-c5d69ddfc-k9mbk\" (UID: \"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk"
May 26 16:33:58.656880 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.656859    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhtgm\" (UniqueName: \"kubernetes.io/projected/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-kube-api-access-lhtgm\") pod \"stop-feature-test-kserve-c5d69ddfc-k9mbk\" (UID: \"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk"
May 26 16:33:58.831969 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.831889    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk"
May 26 16:33:58.989605 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:58.989543    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk"]
May 26 16:33:58.993073 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:33:58.993036    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcac4fe_2fc5_495b_91ef_3e04fa0c0ce2.slice/crio-9a4617f09e1a01618a1391d357b7cf161116f88699c9d7c7d7862d70aebd2983 WatchSource:0}: Error finding container 9a4617f09e1a01618a1391d357b7cf161116f88699c9d7c7d7862d70aebd2983: Status 404 returned error can't find the container with id 9a4617f09e1a01618a1391d357b7cf161116f88699c9d7c7d7862d70aebd2983
May 26 16:33:59.334184 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:59.334141    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk" event={"ID":"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2","Type":"ContainerStarted","Data":"2b2d35cb7326fee6186e35a639569fa936ba09d6b479f612c7a094f4b5745afe"}
May 26 16:33:59.334355 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:33:59.334191    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk" event={"ID":"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2","Type":"ContainerStarted","Data":"9a4617f09e1a01618a1391d357b7cf161116f88699c9d7c7d7862d70aebd2983"}
May 26 16:34:00.457077 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:34:00.457019    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx" podUID="ac71befe-ce5b-427d-b4a8-045a85af25c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8001/health\": dial tcp 10.132.0.36:8001: connect: connection refused"
May 26 16:34:00.605291 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:34:00.605254    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj"
May 26 16:34:00.614525 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:34:00.614496    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj"
May 26 16:34:10.456818 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:34:10.456765    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx" podUID="ac71befe-ce5b-427d-b4a8-045a85af25c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8001/health\": dial tcp 10.132.0.36:8001: connect: connection refused"
May 26 16:34:20.456763 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:34:20.456713    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx" podUID="ac71befe-ce5b-427d-b4a8-045a85af25c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8001/health\": dial tcp 10.132.0.36:8001: connect: connection refused"
May 26 16:34:30.466018 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:34:30.465984    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"
May 26 16:34:30.478652 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:34:30.478627    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"
May 26 16:34:42.881853 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:34:42.881813    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj"]
May 26 16:34:42.882319 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:34:42.882238    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj" podUID="bcaea896-01c4-4060-9986-f9e5b5aa411c" containerName="main" containerID="cri-o://1b9794c726b62439cc925ab8540b8996c210d24d9564d88cb6399a1a70a189c5" gracePeriod=30
May 26 16:34:42.883985 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:34:42.883960    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"]
May 26 16:34:42.884357 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:34:42.884330    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx" podUID="ac71befe-ce5b-427d-b4a8-045a85af25c0" containerName="main" containerID="cri-o://570d222e9df4b48938a8ecc208d8b2a93384b28445015e350f675a2aeeac33e9" gracePeriod=30
May 26 16:35:00.479791 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.479746    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"]
May 26 16:35:00.482644 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.482621    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"
May 26 16:35:00.485400 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.485383    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
May 26 16:35:00.485479 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.485459    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-dockercfg-qjl2m\""
May 26 16:35:00.511146 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.511120    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs"]
May 26 16:35:00.514366 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.514185    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"]
May 26 16:35:00.514366 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.514323    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs"
May 26 16:35:00.530375 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.530349    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs"]
May 26 16:35:00.575654 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.575627    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/828721da-1970-4217-aebd-6b28d6ccf465-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs\" (UID: \"828721da-1970-4217-aebd-6b28d6ccf465\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs"
May 26 16:35:00.575789 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.575662    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/915357f8-702f-4728-adc0-8adb35de7e57-tmp-dir\") pod \"router-with-refs-pd-test-kserve-655b95864c-px2qs\" (UID: \"915357f8-702f-4728-adc0-8adb35de7e57\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"
May 26 16:35:00.575789 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.575728    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/915357f8-702f-4728-adc0-8adb35de7e57-model-cache\") pod \"router-with-refs-pd-test-kserve-655b95864c-px2qs\" (UID: \"915357f8-702f-4728-adc0-8adb35de7e57\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"
May 26 16:35:00.575789 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.575756    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/828721da-1970-4217-aebd-6b28d6ccf465-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs\" (UID: \"828721da-1970-4217-aebd-6b28d6ccf465\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs"
May 26 16:35:00.575898 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.575773    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/828721da-1970-4217-aebd-6b28d6ccf465-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs\" (UID: \"828721da-1970-4217-aebd-6b28d6ccf465\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs"
May 26 16:35:00.575898 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.575821    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzlcm\" (UniqueName: \"kubernetes.io/projected/915357f8-702f-4728-adc0-8adb35de7e57-kube-api-access-jzlcm\") pod \"router-with-refs-pd-test-kserve-655b95864c-px2qs\" (UID: \"915357f8-702f-4728-adc0-8adb35de7e57\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"
May 26 16:35:00.575898 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.575843    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/828721da-1970-4217-aebd-6b28d6ccf465-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs\" (UID: \"828721da-1970-4217-aebd-6b28d6ccf465\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs"
May 26 16:35:00.575898 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.575890    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms9k5\" (UniqueName: \"kubernetes.io/projected/828721da-1970-4217-aebd-6b28d6ccf465-kube-api-access-ms9k5\") pod \"router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs\" (UID: \"828721da-1970-4217-aebd-6b28d6ccf465\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs"
May 26 16:35:00.576371 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.575908    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/915357f8-702f-4728-adc0-8adb35de7e57-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-655b95864c-px2qs\" (UID: \"915357f8-702f-4728-adc0-8adb35de7e57\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"
May 26 16:35:00.576371 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.575922    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/828721da-1970-4217-aebd-6b28d6ccf465-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs\" (UID: \"828721da-1970-4217-aebd-6b28d6ccf465\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs"
May 26 16:35:00.576371 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.576002    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/915357f8-702f-4728-adc0-8adb35de7e57-dshm\") pod \"router-with-refs-pd-test-kserve-655b95864c-px2qs\" (UID: \"915357f8-702f-4728-adc0-8adb35de7e57\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"
May 26 16:35:00.576371 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.576035    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/915357f8-702f-4728-adc0-8adb35de7e57-home\") pod \"router-with-refs-pd-test-kserve-655b95864c-px2qs\" (UID: \"915357f8-702f-4728-adc0-8adb35de7e57\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"
May 26 16:35:00.576371 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.576053    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/828721da-1970-4217-aebd-6b28d6ccf465-home\") pod \"router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs\" (UID: \"828721da-1970-4217-aebd-6b28d6ccf465\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs"
May 26 16:35:00.576371 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.576091    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/915357f8-702f-4728-adc0-8adb35de7e57-tls-certs\") pod \"router-with-refs-pd-test-kserve-655b95864c-px2qs\" (UID: \"915357f8-702f-4728-adc0-8adb35de7e57\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"
May 26 16:35:00.677267 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.677235    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/915357f8-702f-4728-adc0-8adb35de7e57-dshm\") pod \"router-with-refs-pd-test-kserve-655b95864c-px2qs\" (UID: \"915357f8-702f-4728-adc0-8adb35de7e57\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"
May 26 16:35:00.677436 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.677284    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/915357f8-702f-4728-adc0-8adb35de7e57-home\") pod \"router-with-refs-pd-test-kserve-655b95864c-px2qs\" (UID: \"915357f8-702f-4728-adc0-8adb35de7e57\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"
May 26 16:35:00.677436 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.677312    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/828721da-1970-4217-aebd-6b28d6ccf465-home\") pod \"router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs\" (UID: \"828721da-1970-4217-aebd-6b28d6ccf465\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs"
May 26 16:35:00.677436 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.677352    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/915357f8-702f-4728-adc0-8adb35de7e57-tls-certs\") pod \"router-with-refs-pd-test-kserve-655b95864c-px2qs\" (UID: \"915357f8-702f-4728-adc0-8adb35de7e57\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"
May 26 16:35:00.677436 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.677393    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/828721da-1970-4217-aebd-6b28d6ccf465-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs\" (UID: \"828721da-1970-4217-aebd-6b28d6ccf465\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs"
May 26 16:35:00.677436 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.677427    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/915357f8-702f-4728-adc0-8adb35de7e57-tmp-dir\") pod \"router-with-refs-pd-test-kserve-655b95864c-px2qs\" (UID: \"915357f8-702f-4728-adc0-8adb35de7e57\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"
May 26 16:35:00.677729 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.677487    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/915357f8-702f-4728-adc0-8adb35de7e57-model-cache\") pod \"router-with-refs-pd-test-kserve-655b95864c-px2qs\" (UID: \"915357f8-702f-4728-adc0-8adb35de7e57\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"
May 26 16:35:00.677729 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.677509    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/828721da-1970-4217-aebd-6b28d6ccf465-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs\" (UID: \"828721da-1970-4217-aebd-6b28d6ccf465\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs"
May 26 16:35:00.677729 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.677533    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/828721da-1970-4217-aebd-6b28d6ccf465-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs\" (UID: \"828721da-1970-4217-aebd-6b28d6ccf465\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs"
May 26 16:35:00.677729 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.677564    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jzlcm\" (UniqueName: \"kubernetes.io/projected/915357f8-702f-4728-adc0-8adb35de7e57-kube-api-access-jzlcm\") pod \"router-with-refs-pd-test-kserve-655b95864c-px2qs\" (UID: \"915357f8-702f-4728-adc0-8adb35de7e57\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"
May 26 16:35:00.677729 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.677596    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/828721da-1970-4217-aebd-6b28d6ccf465-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs\" (UID: \"828721da-1970-4217-aebd-6b28d6ccf465\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs"
May 26 16:35:00.677729 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.677651    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ms9k5\" (UniqueName: \"kubernetes.io/projected/828721da-1970-4217-aebd-6b28d6ccf465-kube-api-access-ms9k5\") pod \"router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs\" (UID: \"828721da-1970-4217-aebd-6b28d6ccf465\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs"
May 26 16:35:00.677729 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.677687    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/915357f8-702f-4728-adc0-8adb35de7e57-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-655b95864c-px2qs\" (UID: \"915357f8-702f-4728-adc0-8adb35de7e57\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"
May 26 16:35:00.677729 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.677713    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/828721da-1970-4217-aebd-6b28d6ccf465-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs\" (UID: \"828721da-1970-4217-aebd-6b28d6ccf465\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs"
May 26 16:35:00.678209 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.677769    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/915357f8-702f-4728-adc0-8adb35de7e57-home\") pod \"router-with-refs-pd-test-kserve-655b95864c-px2qs\" (UID: \"915357f8-702f-4728-adc0-8adb35de7e57\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"
May 26 16:35:00.678613 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.678563    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/828721da-1970-4217-aebd-6b28d6ccf465-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs\" (UID: \"828721da-1970-4217-aebd-6b28d6ccf465\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs"
May 26 16:35:00.679309 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.679214    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/915357f8-702f-4728-adc0-8adb35de7e57-tmp-dir\") pod \"router-with-refs-pd-test-kserve-655b95864c-px2qs\" (UID: \"915357f8-702f-4728-adc0-8adb35de7e57\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"
May 26 16:35:00.679424 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.679370    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/915357f8-702f-4728-adc0-8adb35de7e57-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-655b95864c-px2qs\" (UID: \"915357f8-702f-4728-adc0-8adb35de7e57\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"
May 26 16:35:00.679748 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.679594    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/915357f8-702f-4728-adc0-8adb35de7e57-model-cache\") pod \"router-with-refs-pd-test-kserve-655b95864c-px2qs\" (UID: \"915357f8-702f-4728-adc0-8adb35de7e57\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"
May 26 16:35:00.679748 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.679706    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/828721da-1970-4217-aebd-6b28d6ccf465-home\") pod \"router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs\" (UID: \"828721da-1970-4217-aebd-6b28d6ccf465\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs"
May 26 16:35:00.680200 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.680161    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/828721da-1970-4217-aebd-6b28d6ccf465-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs\" (UID: \"828721da-1970-4217-aebd-6b28d6ccf465\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs"
May 26 16:35:00.680523 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.680503    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/828721da-1970-4217-aebd-6b28d6ccf465-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs\" (UID: \"828721da-1970-4217-aebd-6b28d6ccf465\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs"
May 26 16:35:00.680796 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.680779    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/915357f8-702f-4728-adc0-8adb35de7e57-dshm\") pod \"router-with-refs-pd-test-kserve-655b95864c-px2qs\" (UID: \"915357f8-702f-4728-adc0-8adb35de7e57\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"
May 26 16:35:00.681543 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.681275    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/828721da-1970-4217-aebd-6b28d6ccf465-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs\" (UID: \"828721da-1970-4217-aebd-6b28d6ccf465\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs"
May 26 16:35:00.681643 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.681543    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/828721da-1970-4217-aebd-6b28d6ccf465-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs\" (UID: \"828721da-1970-4217-aebd-6b28d6ccf465\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs"
May 26 16:35:00.682557 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.682540    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/915357f8-702f-4728-adc0-8adb35de7e57-tls-certs\") pod \"router-with-refs-pd-test-kserve-655b95864c-px2qs\" (UID: \"915357f8-702f-4728-adc0-8adb35de7e57\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"
May 26 16:35:00.687473 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.687450    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzlcm\" (UniqueName: \"kubernetes.io/projected/915357f8-702f-4728-adc0-8adb35de7e57-kube-api-access-jzlcm\") pod \"router-with-refs-pd-test-kserve-655b95864c-px2qs\" (UID: \"915357f8-702f-4728-adc0-8adb35de7e57\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"
May 26 16:35:00.687804 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.687770    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms9k5\" (UniqueName: \"kubernetes.io/projected/828721da-1970-4217-aebd-6b28d6ccf465-kube-api-access-ms9k5\") pod \"router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs\" (UID: \"828721da-1970-4217-aebd-6b28d6ccf465\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs"
May 26 16:35:00.791986 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.791943    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"
May 26 16:35:00.825741 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.825709    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs"
May 26 16:35:00.949447 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.949414    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"]
May 26 16:35:00.952636 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:35:00.952608    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod915357f8_702f_4728_adc0_8adb35de7e57.slice/crio-cb22cb03473f9a95bd6f452ca47d21c4e31f39473c5391e19953f0f04e95c98f WatchSource:0}: Error finding container cb22cb03473f9a95bd6f452ca47d21c4e31f39473c5391e19953f0f04e95c98f: Status 404 returned error can't find the container with id cb22cb03473f9a95bd6f452ca47d21c4e31f39473c5391e19953f0f04e95c98f
May 26 16:35:00.963297 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:00.963277    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs"]
May 26 16:35:00.964841 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:35:00.964810    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod828721da_1970_4217_aebd_6b28d6ccf465.slice/crio-35ae9086d55c6cf45837f0f742232265fd3a3e3d866a6de03e476bdd65fcfbd8 WatchSource:0}: Error finding container 35ae9086d55c6cf45837f0f742232265fd3a3e3d866a6de03e476bdd65fcfbd8: Status 404 returned error can't find the container with id 35ae9086d55c6cf45837f0f742232265fd3a3e3d866a6de03e476bdd65fcfbd8
May 26 16:35:01.544516 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:01.544468    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs" event={"ID":"828721da-1970-4217-aebd-6b28d6ccf465","Type":"ContainerStarted","Data":"70aaae96419535ad695109a8fe9663cc05a8b999d0508534bec15fe4d2b4f3ff"}
May 26 16:35:01.544973 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:01.544522    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs" event={"ID":"828721da-1970-4217-aebd-6b28d6ccf465","Type":"ContainerStarted","Data":"35ae9086d55c6cf45837f0f742232265fd3a3e3d866a6de03e476bdd65fcfbd8"}
May 26 16:35:01.545777 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:01.545751    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs" event={"ID":"915357f8-702f-4728-adc0-8adb35de7e57","Type":"ContainerStarted","Data":"e000fe0e60f3542d510049c7773e88a0d668fe986f125df36bad0cda40bdb692"}
May 26 16:35:01.545869 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:01.545785    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs" event={"ID":"915357f8-702f-4728-adc0-8adb35de7e57","Type":"ContainerStarted","Data":"cb22cb03473f9a95bd6f452ca47d21c4e31f39473c5391e19953f0f04e95c98f"}
May 26 16:35:01.545912 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:01.545880    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"
May 26 16:35:02.555246 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:02.555195    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs" event={"ID":"915357f8-702f-4728-adc0-8adb35de7e57","Type":"ContainerStarted","Data":"07ff3e3c6fbebdc85d2e043bd97731e0f3bb8e0747f826c4d29b119b04f67223"}
May 26 16:35:05.568480 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:05.568445    2552 generic.go:358] "Generic (PLEG): container finished" podID="828721da-1970-4217-aebd-6b28d6ccf465" containerID="70aaae96419535ad695109a8fe9663cc05a8b999d0508534bec15fe4d2b4f3ff" exitCode=0
May 26 16:35:05.568839 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:05.568523    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs" event={"ID":"828721da-1970-4217-aebd-6b28d6ccf465","Type":"ContainerDied","Data":"70aaae96419535ad695109a8fe9663cc05a8b999d0508534bec15fe4d2b4f3ff"}
May 26 16:35:06.573889 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:06.573850    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs" event={"ID":"828721da-1970-4217-aebd-6b28d6ccf465","Type":"ContainerStarted","Data":"6f5177c45a150852cff06fc46fff90f5399e1ae7f25e19f636bf7f39809ca3fe"}
May 26 16:35:06.575380 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:06.575354    2552 generic.go:358] "Generic (PLEG): container finished" podID="915357f8-702f-4728-adc0-8adb35de7e57" containerID="07ff3e3c6fbebdc85d2e043bd97731e0f3bb8e0747f826c4d29b119b04f67223" exitCode=0
May 26 16:35:06.575510 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:06.575400    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs" event={"ID":"915357f8-702f-4728-adc0-8adb35de7e57","Type":"ContainerDied","Data":"07ff3e3c6fbebdc85d2e043bd97731e0f3bb8e0747f826c4d29b119b04f67223"}
May 26 16:35:06.599973 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:06.599894    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs" podStartSLOduration=6.599873836 podStartE2EDuration="6.599873836s" podCreationTimestamp="2026-05-26 16:35:00 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:35:06.59645376 +0000 UTC m=+2056.304908797" watchObservedRunningTime="2026-05-26 16:35:06.599873836 +0000 UTC m=+2056.308328879"
May 26 16:35:07.581330 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:07.581246    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs" event={"ID":"915357f8-702f-4728-adc0-8adb35de7e57","Type":"ContainerStarted","Data":"5af9f0763f14679785ae34fc8a28415e8e7c207d1f9d1e94dd54b93c48f081eb"}
May 26 16:35:07.609569 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:07.609504    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs" podStartSLOduration=7.609482754 podStartE2EDuration="7.609482754s" podCreationTimestamp="2026-05-26 16:35:00 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:35:07.606237227 +0000 UTC m=+2057.314692268" watchObservedRunningTime="2026-05-26 16:35:07.609482754 +0000 UTC m=+2057.317937794"
May 26 16:35:10.792288 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:10.792254    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"
May 26 16:35:10.792717 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:10.792299    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"
May 26 16:35:10.793849 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:10.793818    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs" podUID="915357f8-702f-4728-adc0-8adb35de7e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8001/health\": dial tcp 10.132.0.39:8001: connect: connection refused"
May 26 16:35:10.826706 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:10.826677    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs"
May 26 16:35:10.826865 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:10.826717    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs"
May 26 16:35:10.828204 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:10.828172    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs" podUID="828721da-1970-4217-aebd-6b28d6ccf465" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
May 26 16:35:12.884518 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:12.884422    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx" podUID="ac71befe-ce5b-427d-b4a8-045a85af25c0" containerName="llm-d-routing-sidecar" containerID="cri-o://1e7e451d9a5634d03e03b422465af3d4055277409161d82ff87e28f1fe3cd6a2" gracePeriod=2
May 26 16:35:13.256418 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.256392    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx_ac71befe-ce5b-427d-b4a8-045a85af25c0/main/0.log"
May 26 16:35:13.257244 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.257218    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"
May 26 16:35:13.267598 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.267577    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj"
May 26 16:35:13.303345 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.300171    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac71befe-ce5b-427d-b4a8-045a85af25c0-kserve-provision-location\") pod \"ac71befe-ce5b-427d-b4a8-045a85af25c0\" (UID: \"ac71befe-ce5b-427d-b4a8-045a85af25c0\") "
May 26 16:35:13.303345 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.300222    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac71befe-ce5b-427d-b4a8-045a85af25c0-dshm\") pod \"ac71befe-ce5b-427d-b4a8-045a85af25c0\" (UID: \"ac71befe-ce5b-427d-b4a8-045a85af25c0\") "
May 26 16:35:13.303345 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.300269    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac71befe-ce5b-427d-b4a8-045a85af25c0-home\") pod \"ac71befe-ce5b-427d-b4a8-045a85af25c0\" (UID: \"ac71befe-ce5b-427d-b4a8-045a85af25c0\") "
May 26 16:35:13.303345 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.300298    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac71befe-ce5b-427d-b4a8-045a85af25c0-tmp-dir\") pod \"ac71befe-ce5b-427d-b4a8-045a85af25c0\" (UID: \"ac71befe-ce5b-427d-b4a8-045a85af25c0\") "
May 26 16:35:13.303345 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.300331    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac71befe-ce5b-427d-b4a8-045a85af25c0-tls-certs\") pod \"ac71befe-ce5b-427d-b4a8-045a85af25c0\" (UID: \"ac71befe-ce5b-427d-b4a8-045a85af25c0\") "
May 26 16:35:13.303345 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.300378    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65rdp\" (UniqueName: \"kubernetes.io/projected/ac71befe-ce5b-427d-b4a8-045a85af25c0-kube-api-access-65rdp\") pod \"ac71befe-ce5b-427d-b4a8-045a85af25c0\" (UID: \"ac71befe-ce5b-427d-b4a8-045a85af25c0\") "
May 26 16:35:13.303345 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.300439    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac71befe-ce5b-427d-b4a8-045a85af25c0-model-cache\") pod \"ac71befe-ce5b-427d-b4a8-045a85af25c0\" (UID: \"ac71befe-ce5b-427d-b4a8-045a85af25c0\") "
May 26 16:35:13.303345 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.301686    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac71befe-ce5b-427d-b4a8-045a85af25c0-model-cache" (OuterVolumeSpecName: "model-cache") pod "ac71befe-ce5b-427d-b4a8-045a85af25c0" (UID: "ac71befe-ce5b-427d-b4a8-045a85af25c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:35:13.303345 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.302212    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac71befe-ce5b-427d-b4a8-045a85af25c0-home" (OuterVolumeSpecName: "home") pod "ac71befe-ce5b-427d-b4a8-045a85af25c0" (UID: "ac71befe-ce5b-427d-b4a8-045a85af25c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:35:13.304089 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.303515    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac71befe-ce5b-427d-b4a8-045a85af25c0-dshm" (OuterVolumeSpecName: "dshm") pod "ac71befe-ce5b-427d-b4a8-045a85af25c0" (UID: "ac71befe-ce5b-427d-b4a8-045a85af25c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:35:13.307628 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.307588    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac71befe-ce5b-427d-b4a8-045a85af25c0-kube-api-access-65rdp" (OuterVolumeSpecName: "kube-api-access-65rdp") pod "ac71befe-ce5b-427d-b4a8-045a85af25c0" (UID: "ac71befe-ce5b-427d-b4a8-045a85af25c0"). InnerVolumeSpecName "kube-api-access-65rdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:35:13.308153 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.308117    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac71befe-ce5b-427d-b4a8-045a85af25c0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ac71befe-ce5b-427d-b4a8-045a85af25c0" (UID: "ac71befe-ce5b-427d-b4a8-045a85af25c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:35:13.321586 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.321553    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac71befe-ce5b-427d-b4a8-045a85af25c0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ac71befe-ce5b-427d-b4a8-045a85af25c0" (UID: "ac71befe-ce5b-427d-b4a8-045a85af25c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:35:13.368503 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.368465    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac71befe-ce5b-427d-b4a8-045a85af25c0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ac71befe-ce5b-427d-b4a8-045a85af25c0" (UID: "ac71befe-ce5b-427d-b4a8-045a85af25c0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:35:13.402042 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.402006    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcaea896-01c4-4060-9986-f9e5b5aa411c-tls-certs\") pod \"bcaea896-01c4-4060-9986-f9e5b5aa411c\" (UID: \"bcaea896-01c4-4060-9986-f9e5b5aa411c\") "
May 26 16:35:13.402220 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.402075    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bsd87\" (UniqueName: \"kubernetes.io/projected/bcaea896-01c4-4060-9986-f9e5b5aa411c-kube-api-access-bsd87\") pod \"bcaea896-01c4-4060-9986-f9e5b5aa411c\" (UID: \"bcaea896-01c4-4060-9986-f9e5b5aa411c\") "
May 26 16:35:13.402220 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.402174    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcaea896-01c4-4060-9986-f9e5b5aa411c-model-cache\") pod \"bcaea896-01c4-4060-9986-f9e5b5aa411c\" (UID: \"bcaea896-01c4-4060-9986-f9e5b5aa411c\") "
May 26 16:35:13.402220 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.402213    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bcaea896-01c4-4060-9986-f9e5b5aa411c-kserve-provision-location\") pod \"bcaea896-01c4-4060-9986-f9e5b5aa411c\" (UID: \"bcaea896-01c4-4060-9986-f9e5b5aa411c\") "
May 26 16:35:13.402386 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.402241    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcaea896-01c4-4060-9986-f9e5b5aa411c-dshm\") pod \"bcaea896-01c4-4060-9986-f9e5b5aa411c\" (UID: \"bcaea896-01c4-4060-9986-f9e5b5aa411c\") "
May 26 16:35:13.402386 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.402271    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcaea896-01c4-4060-9986-f9e5b5aa411c-tmp-dir\") pod \"bcaea896-01c4-4060-9986-f9e5b5aa411c\" (UID: \"bcaea896-01c4-4060-9986-f9e5b5aa411c\") "
May 26 16:35:13.402386 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.402307    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcaea896-01c4-4060-9986-f9e5b5aa411c-home\") pod \"bcaea896-01c4-4060-9986-f9e5b5aa411c\" (UID: \"bcaea896-01c4-4060-9986-f9e5b5aa411c\") "
May 26 16:35:13.402538 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.402448    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcaea896-01c4-4060-9986-f9e5b5aa411c-model-cache" (OuterVolumeSpecName: "model-cache") pod "bcaea896-01c4-4060-9986-f9e5b5aa411c" (UID: "bcaea896-01c4-4060-9986-f9e5b5aa411c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:35:13.402642 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.402620    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65rdp\" (UniqueName: \"kubernetes.io/projected/ac71befe-ce5b-427d-b4a8-045a85af25c0-kube-api-access-65rdp\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:35:13.402729 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.402646    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac71befe-ce5b-427d-b4a8-045a85af25c0-model-cache\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:35:13.402729 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.402662    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcaea896-01c4-4060-9986-f9e5b5aa411c-model-cache\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:35:13.402729 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.402677    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac71befe-ce5b-427d-b4a8-045a85af25c0-kserve-provision-location\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:35:13.402729 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.402693    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac71befe-ce5b-427d-b4a8-045a85af25c0-dshm\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:35:13.402729 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.402708    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac71befe-ce5b-427d-b4a8-045a85af25c0-home\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:35:13.402729 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.402722    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac71befe-ce5b-427d-b4a8-045a85af25c0-tmp-dir\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:35:13.403072 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.402735    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac71befe-ce5b-427d-b4a8-045a85af25c0-tls-certs\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:35:13.403137 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.403114    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcaea896-01c4-4060-9986-f9e5b5aa411c-home" (OuterVolumeSpecName: "home") pod "bcaea896-01c4-4060-9986-f9e5b5aa411c" (UID: "bcaea896-01c4-4060-9986-f9e5b5aa411c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:35:13.404692 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.404669    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcaea896-01c4-4060-9986-f9e5b5aa411c-kube-api-access-bsd87" (OuterVolumeSpecName: "kube-api-access-bsd87") pod "bcaea896-01c4-4060-9986-f9e5b5aa411c" (UID: "bcaea896-01c4-4060-9986-f9e5b5aa411c"). InnerVolumeSpecName "kube-api-access-bsd87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:35:13.405770 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.405744    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcaea896-01c4-4060-9986-f9e5b5aa411c-dshm" (OuterVolumeSpecName: "dshm") pod "bcaea896-01c4-4060-9986-f9e5b5aa411c" (UID: "bcaea896-01c4-4060-9986-f9e5b5aa411c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:35:13.405905 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.405824    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcaea896-01c4-4060-9986-f9e5b5aa411c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bcaea896-01c4-4060-9986-f9e5b5aa411c" (UID: "bcaea896-01c4-4060-9986-f9e5b5aa411c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:35:13.415559 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.415530    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcaea896-01c4-4060-9986-f9e5b5aa411c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bcaea896-01c4-4060-9986-f9e5b5aa411c" (UID: "bcaea896-01c4-4060-9986-f9e5b5aa411c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:35:13.467812 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.467720    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcaea896-01c4-4060-9986-f9e5b5aa411c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bcaea896-01c4-4060-9986-f9e5b5aa411c" (UID: "bcaea896-01c4-4060-9986-f9e5b5aa411c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:35:13.503427 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.503390    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bcaea896-01c4-4060-9986-f9e5b5aa411c-kserve-provision-location\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:35:13.503427 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.503428    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcaea896-01c4-4060-9986-f9e5b5aa411c-dshm\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:35:13.503615 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.503445    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcaea896-01c4-4060-9986-f9e5b5aa411c-tmp-dir\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:35:13.503615 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.503477    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcaea896-01c4-4060-9986-f9e5b5aa411c-home\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:35:13.503615 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.503491    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcaea896-01c4-4060-9986-f9e5b5aa411c-tls-certs\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:35:13.503615 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.503506    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsd87\" (UniqueName: \"kubernetes.io/projected/bcaea896-01c4-4060-9986-f9e5b5aa411c-kube-api-access-bsd87\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:35:13.604313 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.604282    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx_ac71befe-ce5b-427d-b4a8-045a85af25c0/main/0.log"
May 26 16:35:13.604986 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.604957    2552 generic.go:358] "Generic (PLEG): container finished" podID="ac71befe-ce5b-427d-b4a8-045a85af25c0" containerID="570d222e9df4b48938a8ecc208d8b2a93384b28445015e350f675a2aeeac33e9" exitCode=137
May 26 16:35:13.604986 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.604983    2552 generic.go:358] "Generic (PLEG): container finished" podID="ac71befe-ce5b-427d-b4a8-045a85af25c0" containerID="1e7e451d9a5634d03e03b422465af3d4055277409161d82ff87e28f1fe3cd6a2" exitCode=0
May 26 16:35:13.605200 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.605050    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"
May 26 16:35:13.605200 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.605047    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx" event={"ID":"ac71befe-ce5b-427d-b4a8-045a85af25c0","Type":"ContainerDied","Data":"570d222e9df4b48938a8ecc208d8b2a93384b28445015e350f675a2aeeac33e9"}
May 26 16:35:13.605200 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.605166    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx" event={"ID":"ac71befe-ce5b-427d-b4a8-045a85af25c0","Type":"ContainerDied","Data":"1e7e451d9a5634d03e03b422465af3d4055277409161d82ff87e28f1fe3cd6a2"}
May 26 16:35:13.605200 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.605184    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx" event={"ID":"ac71befe-ce5b-427d-b4a8-045a85af25c0","Type":"ContainerDied","Data":"8531261c156fbe9a42d23cebcafbe6a29a89d69fee3ab87a4311f57c4f89babe"}
May 26 16:35:13.605459 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.605210    2552 scope.go:117] "RemoveContainer" containerID="570d222e9df4b48938a8ecc208d8b2a93384b28445015e350f675a2aeeac33e9"
May 26 16:35:13.606757 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.606731    2552 generic.go:358] "Generic (PLEG): container finished" podID="bcaea896-01c4-4060-9986-f9e5b5aa411c" containerID="1b9794c726b62439cc925ab8540b8996c210d24d9564d88cb6399a1a70a189c5" exitCode=137
May 26 16:35:13.606892 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.606811    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj"
May 26 16:35:13.606892 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.606848    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj" event={"ID":"bcaea896-01c4-4060-9986-f9e5b5aa411c","Type":"ContainerDied","Data":"1b9794c726b62439cc925ab8540b8996c210d24d9564d88cb6399a1a70a189c5"}
May 26 16:35:13.606892 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.606874    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj" event={"ID":"bcaea896-01c4-4060-9986-f9e5b5aa411c","Type":"ContainerDied","Data":"81ca383809195b74bdeb2e1eb9ebd7e766f87ff326509fab7814c1121f49d016"}
May 26 16:35:13.616735 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.616714    2552 scope.go:117] "RemoveContainer" containerID="8fb0b6833f96146dd3df31d90bfe5b2ecd959e2c7dc18f559a234bac4db37c8e"
May 26 16:35:13.628100 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.628062    2552 scope.go:117] "RemoveContainer" containerID="1e7e451d9a5634d03e03b422465af3d4055277409161d82ff87e28f1fe3cd6a2"
May 26 16:35:13.631374 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.631344    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"]
May 26 16:35:13.637795 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.637773    2552 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7fb85f4fdc-m6pcx"]
May 26 16:35:13.638131 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.638087    2552 scope.go:117] "RemoveContainer" containerID="570d222e9df4b48938a8ecc208d8b2a93384b28445015e350f675a2aeeac33e9"
May 26 16:35:13.638454 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:35:13.638428    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"570d222e9df4b48938a8ecc208d8b2a93384b28445015e350f675a2aeeac33e9\": container with ID starting with 570d222e9df4b48938a8ecc208d8b2a93384b28445015e350f675a2aeeac33e9 not found: ID does not exist" containerID="570d222e9df4b48938a8ecc208d8b2a93384b28445015e350f675a2aeeac33e9"
May 26 16:35:13.638551 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.638463    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"570d222e9df4b48938a8ecc208d8b2a93384b28445015e350f675a2aeeac33e9"} err="failed to get container status \"570d222e9df4b48938a8ecc208d8b2a93384b28445015e350f675a2aeeac33e9\": rpc error: code = NotFound desc = could not find container \"570d222e9df4b48938a8ecc208d8b2a93384b28445015e350f675a2aeeac33e9\": container with ID starting with 570d222e9df4b48938a8ecc208d8b2a93384b28445015e350f675a2aeeac33e9 not found: ID does not exist"
May 26 16:35:13.638551 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.638489    2552 scope.go:117] "RemoveContainer" containerID="8fb0b6833f96146dd3df31d90bfe5b2ecd959e2c7dc18f559a234bac4db37c8e"
May 26 16:35:13.638798 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:35:13.638769    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fb0b6833f96146dd3df31d90bfe5b2ecd959e2c7dc18f559a234bac4db37c8e\": container with ID starting with 8fb0b6833f96146dd3df31d90bfe5b2ecd959e2c7dc18f559a234bac4db37c8e not found: ID does not exist" containerID="8fb0b6833f96146dd3df31d90bfe5b2ecd959e2c7dc18f559a234bac4db37c8e"
May 26 16:35:13.638888 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.638808    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fb0b6833f96146dd3df31d90bfe5b2ecd959e2c7dc18f559a234bac4db37c8e"} err="failed to get container status \"8fb0b6833f96146dd3df31d90bfe5b2ecd959e2c7dc18f559a234bac4db37c8e\": rpc error: code = NotFound desc = could not find container \"8fb0b6833f96146dd3df31d90bfe5b2ecd959e2c7dc18f559a234bac4db37c8e\": container with ID starting with 8fb0b6833f96146dd3df31d90bfe5b2ecd959e2c7dc18f559a234bac4db37c8e not found: ID does not exist"
May 26 16:35:13.638888 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.638830    2552 scope.go:117] "RemoveContainer" containerID="1e7e451d9a5634d03e03b422465af3d4055277409161d82ff87e28f1fe3cd6a2"
May 26 16:35:13.639144 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:35:13.639124    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e7e451d9a5634d03e03b422465af3d4055277409161d82ff87e28f1fe3cd6a2\": container with ID starting with 1e7e451d9a5634d03e03b422465af3d4055277409161d82ff87e28f1fe3cd6a2 not found: ID does not exist" containerID="1e7e451d9a5634d03e03b422465af3d4055277409161d82ff87e28f1fe3cd6a2"
May 26 16:35:13.639208 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.639152    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e7e451d9a5634d03e03b422465af3d4055277409161d82ff87e28f1fe3cd6a2"} err="failed to get container status \"1e7e451d9a5634d03e03b422465af3d4055277409161d82ff87e28f1fe3cd6a2\": rpc error: code = NotFound desc = could not find container \"1e7e451d9a5634d03e03b422465af3d4055277409161d82ff87e28f1fe3cd6a2\": container with ID starting with 1e7e451d9a5634d03e03b422465af3d4055277409161d82ff87e28f1fe3cd6a2 not found: ID does not exist"
May 26 16:35:13.639208 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.639174    2552 scope.go:117] "RemoveContainer" containerID="570d222e9df4b48938a8ecc208d8b2a93384b28445015e350f675a2aeeac33e9"
May 26 16:35:13.639460 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.639439    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"570d222e9df4b48938a8ecc208d8b2a93384b28445015e350f675a2aeeac33e9"} err="failed to get container status \"570d222e9df4b48938a8ecc208d8b2a93384b28445015e350f675a2aeeac33e9\": rpc error: code = NotFound desc = could not find container \"570d222e9df4b48938a8ecc208d8b2a93384b28445015e350f675a2aeeac33e9\": container with ID starting with 570d222e9df4b48938a8ecc208d8b2a93384b28445015e350f675a2aeeac33e9 not found: ID does not exist"
May 26 16:35:13.639549 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.639462    2552 scope.go:117] "RemoveContainer" containerID="8fb0b6833f96146dd3df31d90bfe5b2ecd959e2c7dc18f559a234bac4db37c8e"
May 26 16:35:13.639701 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.639681    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fb0b6833f96146dd3df31d90bfe5b2ecd959e2c7dc18f559a234bac4db37c8e"} err="failed to get container status \"8fb0b6833f96146dd3df31d90bfe5b2ecd959e2c7dc18f559a234bac4db37c8e\": rpc error: code = NotFound desc = could not find container \"8fb0b6833f96146dd3df31d90bfe5b2ecd959e2c7dc18f559a234bac4db37c8e\": container with ID starting with 8fb0b6833f96146dd3df31d90bfe5b2ecd959e2c7dc18f559a234bac4db37c8e not found: ID does not exist"
May 26 16:35:13.639755 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.639703    2552 scope.go:117] "RemoveContainer" containerID="1e7e451d9a5634d03e03b422465af3d4055277409161d82ff87e28f1fe3cd6a2"
May 26 16:35:13.639982 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.639962    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e7e451d9a5634d03e03b422465af3d4055277409161d82ff87e28f1fe3cd6a2"} err="failed to get container status \"1e7e451d9a5634d03e03b422465af3d4055277409161d82ff87e28f1fe3cd6a2\": rpc error: code = NotFound desc = could not find container \"1e7e451d9a5634d03e03b422465af3d4055277409161d82ff87e28f1fe3cd6a2\": container with ID starting with 1e7e451d9a5634d03e03b422465af3d4055277409161d82ff87e28f1fe3cd6a2 not found: ID does not exist"
May 26 16:35:13.640054 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.639983    2552 scope.go:117] "RemoveContainer" containerID="1b9794c726b62439cc925ab8540b8996c210d24d9564d88cb6399a1a70a189c5"
May 26 16:35:13.649095 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.649048    2552 scope.go:117] "RemoveContainer" containerID="3131837e33c1714f92d9eac49f7ecc275bfccae31c803ae16aea9e651890ffe1"
May 26 16:35:13.651087 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.651062    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj"]
May 26 16:35:13.655093 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.655070    2552 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7f554b9658-gq2kj"]
May 26 16:35:13.721411 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.721386    2552 scope.go:117] "RemoveContainer" containerID="1b9794c726b62439cc925ab8540b8996c210d24d9564d88cb6399a1a70a189c5"
May 26 16:35:13.721742 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:35:13.721724    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b9794c726b62439cc925ab8540b8996c210d24d9564d88cb6399a1a70a189c5\": container with ID starting with 1b9794c726b62439cc925ab8540b8996c210d24d9564d88cb6399a1a70a189c5 not found: ID does not exist" containerID="1b9794c726b62439cc925ab8540b8996c210d24d9564d88cb6399a1a70a189c5"
May 26 16:35:13.721837 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.721768    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b9794c726b62439cc925ab8540b8996c210d24d9564d88cb6399a1a70a189c5"} err="failed to get container status \"1b9794c726b62439cc925ab8540b8996c210d24d9564d88cb6399a1a70a189c5\": rpc error: code = NotFound desc = could not find container \"1b9794c726b62439cc925ab8540b8996c210d24d9564d88cb6399a1a70a189c5\": container with ID starting with 1b9794c726b62439cc925ab8540b8996c210d24d9564d88cb6399a1a70a189c5 not found: ID does not exist"
May 26 16:35:13.721837 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.721787    2552 scope.go:117] "RemoveContainer" containerID="3131837e33c1714f92d9eac49f7ecc275bfccae31c803ae16aea9e651890ffe1"
May 26 16:35:13.722144 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:35:13.722123    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3131837e33c1714f92d9eac49f7ecc275bfccae31c803ae16aea9e651890ffe1\": container with ID starting with 3131837e33c1714f92d9eac49f7ecc275bfccae31c803ae16aea9e651890ffe1 not found: ID does not exist" containerID="3131837e33c1714f92d9eac49f7ecc275bfccae31c803ae16aea9e651890ffe1"
May 26 16:35:13.722221 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:13.722153    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3131837e33c1714f92d9eac49f7ecc275bfccae31c803ae16aea9e651890ffe1"} err="failed to get container status \"3131837e33c1714f92d9eac49f7ecc275bfccae31c803ae16aea9e651890ffe1\": rpc error: code = NotFound desc = could not find container \"3131837e33c1714f92d9eac49f7ecc275bfccae31c803ae16aea9e651890ffe1\": container with ID starting with 3131837e33c1714f92d9eac49f7ecc275bfccae31c803ae16aea9e651890ffe1 not found: ID does not exist"
May 26 16:35:14.889773 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:14.889729    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac71befe-ce5b-427d-b4a8-045a85af25c0" path="/var/lib/kubelet/pods/ac71befe-ce5b-427d-b4a8-045a85af25c0/volumes"
May 26 16:35:14.890414 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:14.890393    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcaea896-01c4-4060-9986-f9e5b5aa411c" path="/var/lib/kubelet/pods/bcaea896-01c4-4060-9986-f9e5b5aa411c/volumes"
May 26 16:35:20.792626 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:20.792567    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs" podUID="915357f8-702f-4728-adc0-8adb35de7e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8001/health\": dial tcp 10.132.0.39:8001: connect: connection refused"
May 26 16:35:20.804876 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:20.804851    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"
May 26 16:35:20.826607 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:20.826576    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs" podUID="828721da-1970-4217-aebd-6b28d6ccf465" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
May 26 16:35:30.792569 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:30.792522    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs" podUID="915357f8-702f-4728-adc0-8adb35de7e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8001/health\": dial tcp 10.132.0.39:8001: connect: connection refused"
May 26 16:35:30.826967 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:30.826915    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs" podUID="828721da-1970-4217-aebd-6b28d6ccf465" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
May 26 16:35:40.793858 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:40.793797    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs" podUID="915357f8-702f-4728-adc0-8adb35de7e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8001/health\": dial tcp 10.132.0.39:8001: connect: connection refused"
May 26 16:35:40.826985 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:40.826915    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs" podUID="828721da-1970-4217-aebd-6b28d6ccf465" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
May 26 16:35:50.793306 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:50.793254    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs" podUID="915357f8-702f-4728-adc0-8adb35de7e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8001/health\": dial tcp 10.132.0.39:8001: connect: connection refused"
May 26 16:35:50.826264 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:50.826224    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs" podUID="828721da-1970-4217-aebd-6b28d6ccf465" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
May 26 16:35:50.917354 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:50.917323    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sj6db_5c0f9fc6-2bd6-4370-96d0-5ae315363c7f/ovn-acl-logging/0.log"
May 26 16:35:50.919145 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:35:50.919124    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sj6db_5c0f9fc6-2bd6-4370-96d0-5ae315363c7f/ovn-acl-logging/0.log"
May 26 16:36:00.793044 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:36:00.792992    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs" podUID="915357f8-702f-4728-adc0-8adb35de7e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8001/health\": dial tcp 10.132.0.39:8001: connect: connection refused"
May 26 16:36:00.825982 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:36:00.825953    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs" podUID="828721da-1970-4217-aebd-6b28d6ccf465" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
May 26 16:36:10.793176 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:36:10.793132    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs" podUID="915357f8-702f-4728-adc0-8adb35de7e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8001/health\": dial tcp 10.132.0.39:8001: connect: connection refused"
May 26 16:36:10.826966 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:36:10.826919    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs" podUID="828721da-1970-4217-aebd-6b28d6ccf465" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
May 26 16:36:20.793346 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:36:20.793286    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs" podUID="915357f8-702f-4728-adc0-8adb35de7e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8001/health\": dial tcp 10.132.0.39:8001: connect: connection refused"
May 26 16:36:20.826156 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:36:20.826118    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs" podUID="828721da-1970-4217-aebd-6b28d6ccf465" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
May 26 16:36:30.793341 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:36:30.793234    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs" podUID="915357f8-702f-4728-adc0-8adb35de7e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8001/health\": dial tcp 10.132.0.39:8001: connect: connection refused"
May 26 16:36:30.826305 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:36:30.826268    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs" podUID="828721da-1970-4217-aebd-6b28d6ccf465" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
May 26 16:36:37.902716 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:36:37.902678    2552 generic.go:358] "Generic (PLEG): container finished" podID="ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2" containerID="2b2d35cb7326fee6186e35a639569fa936ba09d6b479f612c7a094f4b5745afe" exitCode=0
May 26 16:36:37.903121 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:36:37.902743    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk" event={"ID":"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2","Type":"ContainerDied","Data":"2b2d35cb7326fee6186e35a639569fa936ba09d6b479f612c7a094f4b5745afe"}
May 26 16:36:37.903872 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:36:37.903856    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:36:38.907640 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:36:38.907604    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk" event={"ID":"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2","Type":"ContainerStarted","Data":"d9df1d3db3411977aeaf2b4abbc4dad71ad9d746bbae03b54aed674c9b18cc19"}
May 26 16:36:38.934226 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:36:38.934181    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk" podStartSLOduration=160.934164973 podStartE2EDuration="2m40.934164973s" podCreationTimestamp="2026-05-26 16:33:58 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:36:38.93281051 +0000 UTC m=+2148.641265549" watchObservedRunningTime="2026-05-26 16:36:38.934164973 +0000 UTC m=+2148.642620014"
May 26 16:36:40.793273 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:36:40.792893    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs" podUID="915357f8-702f-4728-adc0-8adb35de7e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8001/health\": dial tcp 10.132.0.39:8001: connect: connection refused"
May 26 16:36:40.827796 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:36:40.827745    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs" podUID="828721da-1970-4217-aebd-6b28d6ccf465" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
May 26 16:36:48.832428 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:36:48.832386    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk"
May 26 16:36:48.832428 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:36:48.832438    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk"
May 26 16:36:48.834145 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:36:48.834106    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk" podUID="ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
May 26 16:36:50.792313 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:36:50.792264    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs" podUID="915357f8-702f-4728-adc0-8adb35de7e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8001/health\": dial tcp 10.132.0.39:8001: connect: connection refused"
May 26 16:36:50.826812 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:36:50.826764    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs" podUID="828721da-1970-4217-aebd-6b28d6ccf465" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
May 26 16:36:58.832921 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:36:58.832872    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk" podUID="ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
May 26 16:37:00.792416 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:00.792363    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs" podUID="915357f8-702f-4728-adc0-8adb35de7e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8001/health\": dial tcp 10.132.0.39:8001: connect: connection refused"
May 26 16:37:00.826760 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:00.826707    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs" podUID="828721da-1970-4217-aebd-6b28d6ccf465" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
May 26 16:37:08.833215 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:08.833163    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk" podUID="ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
May 26 16:37:10.808333 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:10.808300    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"
May 26 16:37:10.820413 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:10.820380    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"
May 26 16:37:10.826631 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:10.826598    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs" podUID="828721da-1970-4217-aebd-6b28d6ccf465" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
May 26 16:37:18.832347 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:18.832308    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk" podUID="ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
May 26 16:37:20.836627 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:20.836593    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs"
May 26 16:37:20.845399 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:20.845372    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs"
May 26 16:37:28.832698 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:28.832648    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk" podUID="ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
May 26 16:37:32.994000 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:32.993951    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"]
May 26 16:37:32.994516 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:32.994272    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs" podUID="915357f8-702f-4728-adc0-8adb35de7e57" containerName="main" containerID="cri-o://5af9f0763f14679785ae34fc8a28415e8e7c207d1f9d1e94dd54b93c48f081eb" gracePeriod=30
May 26 16:37:33.020360 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:33.020318    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs"]
May 26 16:37:33.020599 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:33.020574    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs" podUID="828721da-1970-4217-aebd-6b28d6ccf465" containerName="main" containerID="cri-o://6f5177c45a150852cff06fc46fff90f5399e1ae7f25e19f636bf7f39809ca3fe" gracePeriod=30
May 26 16:37:38.833106 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:38.833048    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk" podUID="ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
May 26 16:37:44.190051 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.190008    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl"]
May 26 16:37:44.190506 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.190488    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcaea896-01c4-4060-9986-f9e5b5aa411c" containerName="main"
May 26 16:37:44.190550 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.190512    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcaea896-01c4-4060-9986-f9e5b5aa411c" containerName="main"
May 26 16:37:44.190550 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.190533    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac71befe-ce5b-427d-b4a8-045a85af25c0" containerName="main"
May 26 16:37:44.190550 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.190542    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac71befe-ce5b-427d-b4a8-045a85af25c0" containerName="main"
May 26 16:37:44.190646 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.190560    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac71befe-ce5b-427d-b4a8-045a85af25c0" containerName="storage-initializer"
May 26 16:37:44.190646 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.190570    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac71befe-ce5b-427d-b4a8-045a85af25c0" containerName="storage-initializer"
May 26 16:37:44.190646 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.190582    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac71befe-ce5b-427d-b4a8-045a85af25c0" containerName="llm-d-routing-sidecar"
May 26 16:37:44.190646 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.190591    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac71befe-ce5b-427d-b4a8-045a85af25c0" containerName="llm-d-routing-sidecar"
May 26 16:37:44.190646 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.190608    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcaea896-01c4-4060-9986-f9e5b5aa411c" containerName="storage-initializer"
May 26 16:37:44.190646 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.190617    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcaea896-01c4-4060-9986-f9e5b5aa411c" containerName="storage-initializer"
May 26 16:37:44.190825 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.190705    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="ac71befe-ce5b-427d-b4a8-045a85af25c0" containerName="llm-d-routing-sidecar"
May 26 16:37:44.190825 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.190721    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="ac71befe-ce5b-427d-b4a8-045a85af25c0" containerName="main"
May 26 16:37:44.190825 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.190733    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="bcaea896-01c4-4060-9986-f9e5b5aa411c" containerName="main"
May 26 16:37:44.193875 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.193854    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl"
May 26 16:37:44.196489 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.196465    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8de1d74aab16d9cabd8b5aafeb5248e8-kserve-self-signed-certs\""
May 26 16:37:44.204573 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.204550    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl"]
May 26 16:37:44.273985 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.273925    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b2a69a8-b745-4203-b81b-2c4b45a60333-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl\" (UID: \"4b2a69a8-b745-4203-b81b-2c4b45a60333\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl"
May 26 16:37:44.273985 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.273978    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b2a69a8-b745-4203-b81b-2c4b45a60333-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl\" (UID: \"4b2a69a8-b745-4203-b81b-2c4b45a60333\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl"
May 26 16:37:44.274250 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.274053    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b2a69a8-b745-4203-b81b-2c4b45a60333-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl\" (UID: \"4b2a69a8-b745-4203-b81b-2c4b45a60333\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl"
May 26 16:37:44.274250 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.274164    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b2a69a8-b745-4203-b81b-2c4b45a60333-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl\" (UID: \"4b2a69a8-b745-4203-b81b-2c4b45a60333\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl"
May 26 16:37:44.274250 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.274200    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b2a69a8-b745-4203-b81b-2c4b45a60333-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl\" (UID: \"4b2a69a8-b745-4203-b81b-2c4b45a60333\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl"
May 26 16:37:44.274250 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.274226    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkbsd\" (UniqueName: \"kubernetes.io/projected/4b2a69a8-b745-4203-b81b-2c4b45a60333-kube-api-access-pkbsd\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl\" (UID: \"4b2a69a8-b745-4203-b81b-2c4b45a60333\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl"
May 26 16:37:44.274250 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.274248    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b2a69a8-b745-4203-b81b-2c4b45a60333-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl\" (UID: \"4b2a69a8-b745-4203-b81b-2c4b45a60333\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl"
May 26 16:37:44.375479 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.375435    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b2a69a8-b745-4203-b81b-2c4b45a60333-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl\" (UID: \"4b2a69a8-b745-4203-b81b-2c4b45a60333\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl"
May 26 16:37:44.375479 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.375480    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b2a69a8-b745-4203-b81b-2c4b45a60333-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl\" (UID: \"4b2a69a8-b745-4203-b81b-2c4b45a60333\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl"
May 26 16:37:44.375735 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.375611    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b2a69a8-b745-4203-b81b-2c4b45a60333-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl\" (UID: \"4b2a69a8-b745-4203-b81b-2c4b45a60333\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl"
May 26 16:37:44.375735 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.375728    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b2a69a8-b745-4203-b81b-2c4b45a60333-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl\" (UID: \"4b2a69a8-b745-4203-b81b-2c4b45a60333\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl"
May 26 16:37:44.375871 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.375777    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b2a69a8-b745-4203-b81b-2c4b45a60333-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl\" (UID: \"4b2a69a8-b745-4203-b81b-2c4b45a60333\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl"
May 26 16:37:44.375871 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.375811    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pkbsd\" (UniqueName: \"kubernetes.io/projected/4b2a69a8-b745-4203-b81b-2c4b45a60333-kube-api-access-pkbsd\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl\" (UID: \"4b2a69a8-b745-4203-b81b-2c4b45a60333\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl"
May 26 16:37:44.375871 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.375843    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b2a69a8-b745-4203-b81b-2c4b45a60333-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl\" (UID: \"4b2a69a8-b745-4203-b81b-2c4b45a60333\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl"
May 26 16:37:44.376067 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.375922    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b2a69a8-b745-4203-b81b-2c4b45a60333-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl\" (UID: \"4b2a69a8-b745-4203-b81b-2c4b45a60333\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl"
May 26 16:37:44.376135 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.376089    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b2a69a8-b745-4203-b81b-2c4b45a60333-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl\" (UID: \"4b2a69a8-b745-4203-b81b-2c4b45a60333\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl"
May 26 16:37:44.376135 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.376104    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b2a69a8-b745-4203-b81b-2c4b45a60333-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl\" (UID: \"4b2a69a8-b745-4203-b81b-2c4b45a60333\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl"
May 26 16:37:44.376244 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.376197    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b2a69a8-b745-4203-b81b-2c4b45a60333-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl\" (UID: \"4b2a69a8-b745-4203-b81b-2c4b45a60333\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl"
May 26 16:37:44.377739 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.377716    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b2a69a8-b745-4203-b81b-2c4b45a60333-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl\" (UID: \"4b2a69a8-b745-4203-b81b-2c4b45a60333\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl"
May 26 16:37:44.378348 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.378324    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b2a69a8-b745-4203-b81b-2c4b45a60333-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl\" (UID: \"4b2a69a8-b745-4203-b81b-2c4b45a60333\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl"
May 26 16:37:44.384459 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.384434    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkbsd\" (UniqueName: \"kubernetes.io/projected/4b2a69a8-b745-4203-b81b-2c4b45a60333-kube-api-access-pkbsd\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl\" (UID: \"4b2a69a8-b745-4203-b81b-2c4b45a60333\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl"
May 26 16:37:44.506254 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.506157    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl"
May 26 16:37:44.655069 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:44.655042    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl"]
May 26 16:37:44.657261 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:37:44.657232    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2a69a8_b745_4203_b81b_2c4b45a60333.slice/crio-7b375fbcf9b567854601765eb0d429ce18a54076a94acfaf39125b5f4d4aaac2 WatchSource:0}: Error finding container 7b375fbcf9b567854601765eb0d429ce18a54076a94acfaf39125b5f4d4aaac2: Status 404 returned error can't find the container with id 7b375fbcf9b567854601765eb0d429ce18a54076a94acfaf39125b5f4d4aaac2
May 26 16:37:45.146592 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:45.146555    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl" event={"ID":"4b2a69a8-b745-4203-b81b-2c4b45a60333","Type":"ContainerStarted","Data":"18e3edebab5a38e2d198636c3effdab97a8b861953c6c215a59211a198367f57"}
May 26 16:37:45.146592 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:45.146595    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl" event={"ID":"4b2a69a8-b745-4203-b81b-2c4b45a60333","Type":"ContainerStarted","Data":"7b375fbcf9b567854601765eb0d429ce18a54076a94acfaf39125b5f4d4aaac2"}
May 26 16:37:48.833303 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:48.833253    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk" podUID="ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
May 26 16:37:49.164743 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:49.164653    2552 generic.go:358] "Generic (PLEG): container finished" podID="4b2a69a8-b745-4203-b81b-2c4b45a60333" containerID="18e3edebab5a38e2d198636c3effdab97a8b861953c6c215a59211a198367f57" exitCode=0
May 26 16:37:49.164743 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:49.164692    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl" event={"ID":"4b2a69a8-b745-4203-b81b-2c4b45a60333","Type":"ContainerDied","Data":"18e3edebab5a38e2d198636c3effdab97a8b861953c6c215a59211a198367f57"}
May 26 16:37:50.169772 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:50.169738    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl" event={"ID":"4b2a69a8-b745-4203-b81b-2c4b45a60333","Type":"ContainerStarted","Data":"75d72fc40bd2e1ce6f2fa6fe71b87c272b217b3ab8f9e22b088ebc71eb8fa6df"}
May 26 16:37:50.196228 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:50.196172    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl" podStartSLOduration=6.196134137 podStartE2EDuration="6.196134137s" podCreationTimestamp="2026-05-26 16:37:44 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:37:50.194092695 +0000 UTC m=+2219.902547736" watchObservedRunningTime="2026-05-26 16:37:50.196134137 +0000 UTC m=+2219.904589179"
May 26 16:37:54.506622 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:54.506585    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl"
May 26 16:37:54.506622 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:54.506635    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl"
May 26 16:37:54.507867 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:54.507831    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl" podUID="4b2a69a8-b745-4203-b81b-2c4b45a60333" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
May 26 16:37:58.833022 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:37:58.832966    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk" podUID="ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
May 26 16:38:02.995303 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:02.995226    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs" podUID="915357f8-702f-4728-adc0-8adb35de7e57" containerName="llm-d-routing-sidecar" containerID="cri-o://e000fe0e60f3542d510049c7773e88a0d668fe986f125df36bad0cda40bdb692" gracePeriod=2
May 26 16:38:03.224849 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.224806    2552 generic.go:358] "Generic (PLEG): container finished" podID="828721da-1970-4217-aebd-6b28d6ccf465" containerID="6f5177c45a150852cff06fc46fff90f5399e1ae7f25e19f636bf7f39809ca3fe" exitCode=137
May 26 16:38:03.225059 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.224950    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs" event={"ID":"828721da-1970-4217-aebd-6b28d6ccf465","Type":"ContainerDied","Data":"6f5177c45a150852cff06fc46fff90f5399e1ae7f25e19f636bf7f39809ca3fe"}
May 26 16:38:03.231241 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.231210    2552 generic.go:358] "Generic (PLEG): container finished" podID="915357f8-702f-4728-adc0-8adb35de7e57" containerID="e000fe0e60f3542d510049c7773e88a0d668fe986f125df36bad0cda40bdb692" exitCode=0
May 26 16:38:03.231364 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.231251    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs" event={"ID":"915357f8-702f-4728-adc0-8adb35de7e57","Type":"ContainerDied","Data":"e000fe0e60f3542d510049c7773e88a0d668fe986f125df36bad0cda40bdb692"}
May 26 16:38:03.434137 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.434112    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs"
May 26 16:38:03.437330 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.437310    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-655b95864c-px2qs_915357f8-702f-4728-adc0-8adb35de7e57/main/0.log"
May 26 16:38:03.438028 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.438011    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"
May 26 16:38:03.445322 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.445297    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/915357f8-702f-4728-adc0-8adb35de7e57-dshm\") pod \"915357f8-702f-4728-adc0-8adb35de7e57\" (UID: \"915357f8-702f-4728-adc0-8adb35de7e57\") "
May 26 16:38:03.445445 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.445353    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ms9k5\" (UniqueName: \"kubernetes.io/projected/828721da-1970-4217-aebd-6b28d6ccf465-kube-api-access-ms9k5\") pod \"828721da-1970-4217-aebd-6b28d6ccf465\" (UID: \"828721da-1970-4217-aebd-6b28d6ccf465\") "
May 26 16:38:03.445445 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.445382    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/828721da-1970-4217-aebd-6b28d6ccf465-home\") pod \"828721da-1970-4217-aebd-6b28d6ccf465\" (UID: \"828721da-1970-4217-aebd-6b28d6ccf465\") "
May 26 16:38:03.445445 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.445421    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/915357f8-702f-4728-adc0-8adb35de7e57-tls-certs\") pod \"915357f8-702f-4728-adc0-8adb35de7e57\" (UID: \"915357f8-702f-4728-adc0-8adb35de7e57\") "
May 26 16:38:03.445618 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.445460    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/828721da-1970-4217-aebd-6b28d6ccf465-kserve-provision-location\") pod \"828721da-1970-4217-aebd-6b28d6ccf465\" (UID: \"828721da-1970-4217-aebd-6b28d6ccf465\") "
May 26 16:38:03.445618 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.445484    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/828721da-1970-4217-aebd-6b28d6ccf465-model-cache\") pod \"828721da-1970-4217-aebd-6b28d6ccf465\" (UID: \"828721da-1970-4217-aebd-6b28d6ccf465\") "
May 26 16:38:03.445801 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.445753    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828721da-1970-4217-aebd-6b28d6ccf465-model-cache" (OuterVolumeSpecName: "model-cache") pod "828721da-1970-4217-aebd-6b28d6ccf465" (UID: "828721da-1970-4217-aebd-6b28d6ccf465"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:38:03.445801 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.445789    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/828721da-1970-4217-aebd-6b28d6ccf465-dshm\") pod \"828721da-1970-4217-aebd-6b28d6ccf465\" (UID: \"828721da-1970-4217-aebd-6b28d6ccf465\") "
May 26 16:38:03.446049 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.445868    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/915357f8-702f-4728-adc0-8adb35de7e57-tmp-dir\") pod \"915357f8-702f-4728-adc0-8adb35de7e57\" (UID: \"915357f8-702f-4728-adc0-8adb35de7e57\") "
May 26 16:38:03.446049 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.445897    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/915357f8-702f-4728-adc0-8adb35de7e57-model-cache\") pod \"915357f8-702f-4728-adc0-8adb35de7e57\" (UID: \"915357f8-702f-4728-adc0-8adb35de7e57\") "
May 26 16:38:03.446179 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.446122    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828721da-1970-4217-aebd-6b28d6ccf465-home" (OuterVolumeSpecName: "home") pod "828721da-1970-4217-aebd-6b28d6ccf465" (UID: "828721da-1970-4217-aebd-6b28d6ccf465"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:38:03.446427 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.446233    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/828721da-1970-4217-aebd-6b28d6ccf465-tmp-dir\") pod \"828721da-1970-4217-aebd-6b28d6ccf465\" (UID: \"828721da-1970-4217-aebd-6b28d6ccf465\") "
May 26 16:38:03.446427 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.446327    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/915357f8-702f-4728-adc0-8adb35de7e57-home\") pod \"915357f8-702f-4728-adc0-8adb35de7e57\" (UID: \"915357f8-702f-4728-adc0-8adb35de7e57\") "
May 26 16:38:03.446427 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.446362    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzlcm\" (UniqueName: \"kubernetes.io/projected/915357f8-702f-4728-adc0-8adb35de7e57-kube-api-access-jzlcm\") pod \"915357f8-702f-4728-adc0-8adb35de7e57\" (UID: \"915357f8-702f-4728-adc0-8adb35de7e57\") "
May 26 16:38:03.446427 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.446390    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/828721da-1970-4217-aebd-6b28d6ccf465-tls-certs\") pod \"828721da-1970-4217-aebd-6b28d6ccf465\" (UID: \"828721da-1970-4217-aebd-6b28d6ccf465\") "
May 26 16:38:03.446427 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.446415    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/915357f8-702f-4728-adc0-8adb35de7e57-kserve-provision-location\") pod \"915357f8-702f-4728-adc0-8adb35de7e57\" (UID: \"915357f8-702f-4728-adc0-8adb35de7e57\") "
May 26 16:38:03.446738 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.446689    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/828721da-1970-4217-aebd-6b28d6ccf465-model-cache\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:38:03.446738 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.446706    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/828721da-1970-4217-aebd-6b28d6ccf465-home\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:38:03.447144 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.446901    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915357f8-702f-4728-adc0-8adb35de7e57-model-cache" (OuterVolumeSpecName: "model-cache") pod "915357f8-702f-4728-adc0-8adb35de7e57" (UID: "915357f8-702f-4728-adc0-8adb35de7e57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:38:03.448002 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.447688    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915357f8-702f-4728-adc0-8adb35de7e57-home" (OuterVolumeSpecName: "home") pod "915357f8-702f-4728-adc0-8adb35de7e57" (UID: "915357f8-702f-4728-adc0-8adb35de7e57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:38:03.448002 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.447789    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915357f8-702f-4728-adc0-8adb35de7e57-dshm" (OuterVolumeSpecName: "dshm") pod "915357f8-702f-4728-adc0-8adb35de7e57" (UID: "915357f8-702f-4728-adc0-8adb35de7e57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:38:03.461379 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.461334    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/915357f8-702f-4728-adc0-8adb35de7e57-kube-api-access-jzlcm" (OuterVolumeSpecName: "kube-api-access-jzlcm") pod "915357f8-702f-4728-adc0-8adb35de7e57" (UID: "915357f8-702f-4728-adc0-8adb35de7e57"). InnerVolumeSpecName "kube-api-access-jzlcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:38:03.461621 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.461587    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/915357f8-702f-4728-adc0-8adb35de7e57-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "915357f8-702f-4728-adc0-8adb35de7e57" (UID: "915357f8-702f-4728-adc0-8adb35de7e57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:38:03.464570 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.463213    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915357f8-702f-4728-adc0-8adb35de7e57-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "915357f8-702f-4728-adc0-8adb35de7e57" (UID: "915357f8-702f-4728-adc0-8adb35de7e57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:38:03.464570 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.463280    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/828721da-1970-4217-aebd-6b28d6ccf465-kube-api-access-ms9k5" (OuterVolumeSpecName: "kube-api-access-ms9k5") pod "828721da-1970-4217-aebd-6b28d6ccf465" (UID: "828721da-1970-4217-aebd-6b28d6ccf465"). InnerVolumeSpecName "kube-api-access-ms9k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:38:03.464570 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.464440    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828721da-1970-4217-aebd-6b28d6ccf465-dshm" (OuterVolumeSpecName: "dshm") pod "828721da-1970-4217-aebd-6b28d6ccf465" (UID: "828721da-1970-4217-aebd-6b28d6ccf465"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:38:03.464570 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.464529    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/828721da-1970-4217-aebd-6b28d6ccf465-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "828721da-1970-4217-aebd-6b28d6ccf465" (UID: "828721da-1970-4217-aebd-6b28d6ccf465"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:38:03.478252 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.478225    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828721da-1970-4217-aebd-6b28d6ccf465-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "828721da-1970-4217-aebd-6b28d6ccf465" (UID: "828721da-1970-4217-aebd-6b28d6ccf465"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:38:03.523808 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.523740    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828721da-1970-4217-aebd-6b28d6ccf465-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "828721da-1970-4217-aebd-6b28d6ccf465" (UID: "828721da-1970-4217-aebd-6b28d6ccf465"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:38:03.547403 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.547371    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/828721da-1970-4217-aebd-6b28d6ccf465-dshm\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:38:03.547403 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.547406    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/915357f8-702f-4728-adc0-8adb35de7e57-tmp-dir\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:38:03.547582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.547421    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/915357f8-702f-4728-adc0-8adb35de7e57-model-cache\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:38:03.547582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.547435    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/828721da-1970-4217-aebd-6b28d6ccf465-tmp-dir\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:38:03.547582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.547448    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/915357f8-702f-4728-adc0-8adb35de7e57-home\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:38:03.547582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.547461    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzlcm\" (UniqueName: \"kubernetes.io/projected/915357f8-702f-4728-adc0-8adb35de7e57-kube-api-access-jzlcm\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:38:03.547582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.547474    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/828721da-1970-4217-aebd-6b28d6ccf465-tls-certs\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:38:03.547582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.547487    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/915357f8-702f-4728-adc0-8adb35de7e57-dshm\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:38:03.547582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.547500    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms9k5\" (UniqueName: \"kubernetes.io/projected/828721da-1970-4217-aebd-6b28d6ccf465-kube-api-access-ms9k5\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:38:03.547582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.547511    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/915357f8-702f-4728-adc0-8adb35de7e57-tls-certs\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:38:03.547582 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.547524    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/828721da-1970-4217-aebd-6b28d6ccf465-kserve-provision-location\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:38:03.548924 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.548901    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915357f8-702f-4728-adc0-8adb35de7e57-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "915357f8-702f-4728-adc0-8adb35de7e57" (UID: "915357f8-702f-4728-adc0-8adb35de7e57"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:38:03.648353 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:03.648316    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/915357f8-702f-4728-adc0-8adb35de7e57-kserve-provision-location\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:38:04.235909 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:04.235876    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs"
May 26 16:38:04.235909 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:04.235885    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs" event={"ID":"828721da-1970-4217-aebd-6b28d6ccf465","Type":"ContainerDied","Data":"35ae9086d55c6cf45837f0f742232265fd3a3e3d866a6de03e476bdd65fcfbd8"}
May 26 16:38:04.236454 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:04.235950    2552 scope.go:117] "RemoveContainer" containerID="6f5177c45a150852cff06fc46fff90f5399e1ae7f25e19f636bf7f39809ca3fe"
May 26 16:38:04.237418 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:04.237388    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-655b95864c-px2qs_915357f8-702f-4728-adc0-8adb35de7e57/main/0.log"
May 26 16:38:04.238111 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:04.238082    2552 generic.go:358] "Generic (PLEG): container finished" podID="915357f8-702f-4728-adc0-8adb35de7e57" containerID="5af9f0763f14679785ae34fc8a28415e8e7c207d1f9d1e94dd54b93c48f081eb" exitCode=137
May 26 16:38:04.238223 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:04.238160    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"
May 26 16:38:04.238223 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:04.238169    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs" event={"ID":"915357f8-702f-4728-adc0-8adb35de7e57","Type":"ContainerDied","Data":"5af9f0763f14679785ae34fc8a28415e8e7c207d1f9d1e94dd54b93c48f081eb"}
May 26 16:38:04.238223 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:04.238213    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs" event={"ID":"915357f8-702f-4728-adc0-8adb35de7e57","Type":"ContainerDied","Data":"cb22cb03473f9a95bd6f452ca47d21c4e31f39473c5391e19953f0f04e95c98f"}
May 26 16:38:04.247385 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:04.247366    2552 scope.go:117] "RemoveContainer" containerID="70aaae96419535ad695109a8fe9663cc05a8b999d0508534bec15fe4d2b4f3ff"
May 26 16:38:04.257167 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:04.257144    2552 scope.go:117] "RemoveContainer" containerID="5af9f0763f14679785ae34fc8a28415e8e7c207d1f9d1e94dd54b93c48f081eb"
May 26 16:38:04.260248 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:04.260222    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs"]
May 26 16:38:04.264399 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:04.264377    2552 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5d6b8b7978-fltrs"]
May 26 16:38:04.267652 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:04.267628    2552 scope.go:117] "RemoveContainer" containerID="07ff3e3c6fbebdc85d2e043bd97731e0f3bb8e0747f826c4d29b119b04f67223"
May 26 16:38:04.277885 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:04.277805    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"]
May 26 16:38:04.277885 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:04.277879    2552 scope.go:117] "RemoveContainer" containerID="e000fe0e60f3542d510049c7773e88a0d668fe986f125df36bad0cda40bdb692"
May 26 16:38:04.284350 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:04.284298    2552 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-655b95864c-px2qs"]
May 26 16:38:04.288208 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:04.288190    2552 scope.go:117] "RemoveContainer" containerID="5af9f0763f14679785ae34fc8a28415e8e7c207d1f9d1e94dd54b93c48f081eb"
May 26 16:38:04.288528 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:38:04.288507    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5af9f0763f14679785ae34fc8a28415e8e7c207d1f9d1e94dd54b93c48f081eb\": container with ID starting with 5af9f0763f14679785ae34fc8a28415e8e7c207d1f9d1e94dd54b93c48f081eb not found: ID does not exist" containerID="5af9f0763f14679785ae34fc8a28415e8e7c207d1f9d1e94dd54b93c48f081eb"
May 26 16:38:04.288612 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:04.288541    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5af9f0763f14679785ae34fc8a28415e8e7c207d1f9d1e94dd54b93c48f081eb"} err="failed to get container status \"5af9f0763f14679785ae34fc8a28415e8e7c207d1f9d1e94dd54b93c48f081eb\": rpc error: code = NotFound desc = could not find container \"5af9f0763f14679785ae34fc8a28415e8e7c207d1f9d1e94dd54b93c48f081eb\": container with ID starting with 5af9f0763f14679785ae34fc8a28415e8e7c207d1f9d1e94dd54b93c48f081eb not found: ID does not exist"
May 26 16:38:04.288612 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:04.288565    2552 scope.go:117] "RemoveContainer" containerID="07ff3e3c6fbebdc85d2e043bd97731e0f3bb8e0747f826c4d29b119b04f67223"
May 26 16:38:04.288840 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:38:04.288819    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07ff3e3c6fbebdc85d2e043bd97731e0f3bb8e0747f826c4d29b119b04f67223\": container with ID starting with 07ff3e3c6fbebdc85d2e043bd97731e0f3bb8e0747f826c4d29b119b04f67223 not found: ID does not exist" containerID="07ff3e3c6fbebdc85d2e043bd97731e0f3bb8e0747f826c4d29b119b04f67223"
May 26 16:38:04.289056 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:04.288842    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07ff3e3c6fbebdc85d2e043bd97731e0f3bb8e0747f826c4d29b119b04f67223"} err="failed to get container status \"07ff3e3c6fbebdc85d2e043bd97731e0f3bb8e0747f826c4d29b119b04f67223\": rpc error: code = NotFound desc = could not find container \"07ff3e3c6fbebdc85d2e043bd97731e0f3bb8e0747f826c4d29b119b04f67223\": container with ID starting with 07ff3e3c6fbebdc85d2e043bd97731e0f3bb8e0747f826c4d29b119b04f67223 not found: ID does not exist"
May 26 16:38:04.289056 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:04.288858    2552 scope.go:117] "RemoveContainer" containerID="e000fe0e60f3542d510049c7773e88a0d668fe986f125df36bad0cda40bdb692"
May 26 16:38:04.289181 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:38:04.289120    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e000fe0e60f3542d510049c7773e88a0d668fe986f125df36bad0cda40bdb692\": container with ID starting with e000fe0e60f3542d510049c7773e88a0d668fe986f125df36bad0cda40bdb692 not found: ID does not exist" containerID="e000fe0e60f3542d510049c7773e88a0d668fe986f125df36bad0cda40bdb692"
May 26 16:38:04.289181 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:04.289139    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e000fe0e60f3542d510049c7773e88a0d668fe986f125df36bad0cda40bdb692"} err="failed to get container status \"e000fe0e60f3542d510049c7773e88a0d668fe986f125df36bad0cda40bdb692\": rpc error: code = NotFound desc = could not find container \"e000fe0e60f3542d510049c7773e88a0d668fe986f125df36bad0cda40bdb692\": container with ID starting with e000fe0e60f3542d510049c7773e88a0d668fe986f125df36bad0cda40bdb692 not found: ID does not exist"
May 26 16:38:04.507518 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:04.507477    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl" podUID="4b2a69a8-b745-4203-b81b-2c4b45a60333" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
May 26 16:38:04.889912 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:04.889879    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="828721da-1970-4217-aebd-6b28d6ccf465" path="/var/lib/kubelet/pods/828721da-1970-4217-aebd-6b28d6ccf465/volumes"
May 26 16:38:04.890334 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:04.890321    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="915357f8-702f-4728-adc0-8adb35de7e57" path="/var/lib/kubelet/pods/915357f8-702f-4728-adc0-8adb35de7e57/volumes"
May 26 16:38:08.832974 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:08.832864    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk" podUID="ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
May 26 16:38:14.506809 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:14.506767    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl" podUID="4b2a69a8-b745-4203-b81b-2c4b45a60333" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
May 26 16:38:18.842105 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:18.842072    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk"
May 26 16:38:18.850155 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:18.850129    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk"
May 26 16:38:19.783561 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:19.783530    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk"]
May 26 16:38:20.297875 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:20.297812    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk" podUID="ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2" containerName="main" containerID="cri-o://d9df1d3db3411977aeaf2b4abbc4dad71ad9d746bbae03b54aed674c9b18cc19" gracePeriod=30
May 26 16:38:24.506965 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:24.506907    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl" podUID="4b2a69a8-b745-4203-b81b-2c4b45a60333" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
May 26 16:38:34.507381 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:34.507338    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl" podUID="4b2a69a8-b745-4203-b81b-2c4b45a60333" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
May 26 16:38:35.612694 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.612662    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4"]
May 26 16:38:35.613210 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.613011    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="915357f8-702f-4728-adc0-8adb35de7e57" containerName="llm-d-routing-sidecar"
May 26 16:38:35.613210 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.613025    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="915357f8-702f-4728-adc0-8adb35de7e57" containerName="llm-d-routing-sidecar"
May 26 16:38:35.613210 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.613037    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="915357f8-702f-4728-adc0-8adb35de7e57" containerName="storage-initializer"
May 26 16:38:35.613210 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.613043    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="915357f8-702f-4728-adc0-8adb35de7e57" containerName="storage-initializer"
May 26 16:38:35.613210 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.613049    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="828721da-1970-4217-aebd-6b28d6ccf465" containerName="storage-initializer"
May 26 16:38:35.613210 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.613056    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="828721da-1970-4217-aebd-6b28d6ccf465" containerName="storage-initializer"
May 26 16:38:35.613210 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.613067    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="828721da-1970-4217-aebd-6b28d6ccf465" containerName="main"
May 26 16:38:35.613210 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.613072    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="828721da-1970-4217-aebd-6b28d6ccf465" containerName="main"
May 26 16:38:35.613210 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.613082    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="915357f8-702f-4728-adc0-8adb35de7e57" containerName="main"
May 26 16:38:35.613210 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.613087    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="915357f8-702f-4728-adc0-8adb35de7e57" containerName="main"
May 26 16:38:35.613210 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.613141    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="915357f8-702f-4728-adc0-8adb35de7e57" containerName="main"
May 26 16:38:35.613210 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.613152    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="915357f8-702f-4728-adc0-8adb35de7e57" containerName="llm-d-routing-sidecar"
May 26 16:38:35.613210 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.613159    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="828721da-1970-4217-aebd-6b28d6ccf465" containerName="main"
May 26 16:38:35.615428 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.615407    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4"
May 26 16:38:35.629922 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.629898    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4"]
May 26 16:38:35.722853 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.722825    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b759932b-9db9-4c73-b28c-d1871bed94ff-home\") pod \"stop-feature-test-kserve-c5d69ddfc-qx5h4\" (UID: \"b759932b-9db9-4c73-b28c-d1871bed94ff\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4"
May 26 16:38:35.723039 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.722865    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsgxv\" (UniqueName: \"kubernetes.io/projected/b759932b-9db9-4c73-b28c-d1871bed94ff-kube-api-access-wsgxv\") pod \"stop-feature-test-kserve-c5d69ddfc-qx5h4\" (UID: \"b759932b-9db9-4c73-b28c-d1871bed94ff\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4"
May 26 16:38:35.723039 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.722886    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b759932b-9db9-4c73-b28c-d1871bed94ff-tls-certs\") pod \"stop-feature-test-kserve-c5d69ddfc-qx5h4\" (UID: \"b759932b-9db9-4c73-b28c-d1871bed94ff\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4"
May 26 16:38:35.723039 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.723002    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b759932b-9db9-4c73-b28c-d1871bed94ff-dshm\") pod \"stop-feature-test-kserve-c5d69ddfc-qx5h4\" (UID: \"b759932b-9db9-4c73-b28c-d1871bed94ff\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4"
May 26 16:38:35.723154 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.723042    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b759932b-9db9-4c73-b28c-d1871bed94ff-kserve-provision-location\") pod \"stop-feature-test-kserve-c5d69ddfc-qx5h4\" (UID: \"b759932b-9db9-4c73-b28c-d1871bed94ff\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4"
May 26 16:38:35.723154 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.723092    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b759932b-9db9-4c73-b28c-d1871bed94ff-model-cache\") pod \"stop-feature-test-kserve-c5d69ddfc-qx5h4\" (UID: \"b759932b-9db9-4c73-b28c-d1871bed94ff\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4"
May 26 16:38:35.723154 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.723143    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b759932b-9db9-4c73-b28c-d1871bed94ff-tmp-dir\") pod \"stop-feature-test-kserve-c5d69ddfc-qx5h4\" (UID: \"b759932b-9db9-4c73-b28c-d1871bed94ff\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4"
May 26 16:38:35.824046 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.824005    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b759932b-9db9-4c73-b28c-d1871bed94ff-home\") pod \"stop-feature-test-kserve-c5d69ddfc-qx5h4\" (UID: \"b759932b-9db9-4c73-b28c-d1871bed94ff\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4"
May 26 16:38:35.824235 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.824084    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wsgxv\" (UniqueName: \"kubernetes.io/projected/b759932b-9db9-4c73-b28c-d1871bed94ff-kube-api-access-wsgxv\") pod \"stop-feature-test-kserve-c5d69ddfc-qx5h4\" (UID: \"b759932b-9db9-4c73-b28c-d1871bed94ff\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4"
May 26 16:38:35.824235 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.824123    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b759932b-9db9-4c73-b28c-d1871bed94ff-tls-certs\") pod \"stop-feature-test-kserve-c5d69ddfc-qx5h4\" (UID: \"b759932b-9db9-4c73-b28c-d1871bed94ff\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4"
May 26 16:38:35.824235 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.824173    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b759932b-9db9-4c73-b28c-d1871bed94ff-dshm\") pod \"stop-feature-test-kserve-c5d69ddfc-qx5h4\" (UID: \"b759932b-9db9-4c73-b28c-d1871bed94ff\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4"
May 26 16:38:35.824235 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.824204    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b759932b-9db9-4c73-b28c-d1871bed94ff-kserve-provision-location\") pod \"stop-feature-test-kserve-c5d69ddfc-qx5h4\" (UID: \"b759932b-9db9-4c73-b28c-d1871bed94ff\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4"
May 26 16:38:35.824471 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.824256    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b759932b-9db9-4c73-b28c-d1871bed94ff-model-cache\") pod \"stop-feature-test-kserve-c5d69ddfc-qx5h4\" (UID: \"b759932b-9db9-4c73-b28c-d1871bed94ff\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4"
May 26 16:38:35.824471 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.824322    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b759932b-9db9-4c73-b28c-d1871bed94ff-tmp-dir\") pod \"stop-feature-test-kserve-c5d69ddfc-qx5h4\" (UID: \"b759932b-9db9-4c73-b28c-d1871bed94ff\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4"
May 26 16:38:35.824471 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.824338    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b759932b-9db9-4c73-b28c-d1871bed94ff-home\") pod \"stop-feature-test-kserve-c5d69ddfc-qx5h4\" (UID: \"b759932b-9db9-4c73-b28c-d1871bed94ff\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4"
May 26 16:38:35.824720 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.824612    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b759932b-9db9-4c73-b28c-d1871bed94ff-model-cache\") pod \"stop-feature-test-kserve-c5d69ddfc-qx5h4\" (UID: \"b759932b-9db9-4c73-b28c-d1871bed94ff\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4"
May 26 16:38:35.824720 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.824665    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b759932b-9db9-4c73-b28c-d1871bed94ff-tmp-dir\") pod \"stop-feature-test-kserve-c5d69ddfc-qx5h4\" (UID: \"b759932b-9db9-4c73-b28c-d1871bed94ff\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4"
May 26 16:38:35.824873 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.824838    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b759932b-9db9-4c73-b28c-d1871bed94ff-kserve-provision-location\") pod \"stop-feature-test-kserve-c5d69ddfc-qx5h4\" (UID: \"b759932b-9db9-4c73-b28c-d1871bed94ff\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4"
May 26 16:38:35.826589 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.826564    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b759932b-9db9-4c73-b28c-d1871bed94ff-dshm\") pod \"stop-feature-test-kserve-c5d69ddfc-qx5h4\" (UID: \"b759932b-9db9-4c73-b28c-d1871bed94ff\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4"
May 26 16:38:35.826884 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.826859    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b759932b-9db9-4c73-b28c-d1871bed94ff-tls-certs\") pod \"stop-feature-test-kserve-c5d69ddfc-qx5h4\" (UID: \"b759932b-9db9-4c73-b28c-d1871bed94ff\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4"
May 26 16:38:35.843163 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.843135    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsgxv\" (UniqueName: \"kubernetes.io/projected/b759932b-9db9-4c73-b28c-d1871bed94ff-kube-api-access-wsgxv\") pod \"stop-feature-test-kserve-c5d69ddfc-qx5h4\" (UID: \"b759932b-9db9-4c73-b28c-d1871bed94ff\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4"
May 26 16:38:35.925461 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:35.925365    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4"
May 26 16:38:36.070788 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:36.070719    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4"]
May 26 16:38:36.073719 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:38:36.073697    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb759932b_9db9_4c73_b28c_d1871bed94ff.slice/crio-041e37d2430d7a4841db04ebca29aa17df89ca3a48e1e00fde348fd349cd3f30 WatchSource:0}: Error finding container 041e37d2430d7a4841db04ebca29aa17df89ca3a48e1e00fde348fd349cd3f30: Status 404 returned error can't find the container with id 041e37d2430d7a4841db04ebca29aa17df89ca3a48e1e00fde348fd349cd3f30
May 26 16:38:36.358594 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:36.358557    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4" event={"ID":"b759932b-9db9-4c73-b28c-d1871bed94ff","Type":"ContainerStarted","Data":"941a1fa001eaae030ce3af34f6146338b2f5e9e17ebe82571c9ffcdd93ff0520"}
May 26 16:38:36.358594 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:36.358599    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4" event={"ID":"b759932b-9db9-4c73-b28c-d1871bed94ff","Type":"ContainerStarted","Data":"041e37d2430d7a4841db04ebca29aa17df89ca3a48e1e00fde348fd349cd3f30"}
May 26 16:38:44.506828 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:44.506783    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl" podUID="4b2a69a8-b745-4203-b81b-2c4b45a60333" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
May 26 16:38:50.564463 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:50.564440    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-c5d69ddfc-k9mbk_ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2/main/0.log"
May 26 16:38:50.564881 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:50.564861    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk"
May 26 16:38:50.760810 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:50.760778    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-dshm\") pod \"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2\" (UID: \"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2\") "
May 26 16:38:50.760997 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:50.760832    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-tls-certs\") pod \"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2\" (UID: \"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2\") "
May 26 16:38:50.760997 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:50.760858    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-tmp-dir\") pod \"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2\" (UID: \"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2\") "
May 26 16:38:50.760997 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:50.760888    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-model-cache\") pod \"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2\" (UID: \"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2\") "
May 26 16:38:50.760997 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:50.760905    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-home\") pod \"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2\" (UID: \"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2\") "
May 26 16:38:50.760997 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:50.760947    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-kserve-provision-location\") pod \"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2\" (UID: \"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2\") "
May 26 16:38:50.760997 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:50.760994    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhtgm\" (UniqueName: \"kubernetes.io/projected/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-kube-api-access-lhtgm\") pod \"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2\" (UID: \"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2\") "
May 26 16:38:50.763073 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:50.763035    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2" (UID: "ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:38:50.763195 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:50.763101    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-kube-api-access-lhtgm" (OuterVolumeSpecName: "kube-api-access-lhtgm") pod "ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2" (UID: "ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2"). InnerVolumeSpecName "kube-api-access-lhtgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:38:50.766431 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:50.766344    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-model-cache" (OuterVolumeSpecName: "model-cache") pod "ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2" (UID: "ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:38:50.766431 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:50.766413    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-dshm" (OuterVolumeSpecName: "dshm") pod "ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2" (UID: "ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:38:50.766685 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:50.766659    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-home" (OuterVolumeSpecName: "home") pod "ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2" (UID: "ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:38:50.781918 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:50.781887    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2" (UID: "ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:38:50.824976 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:50.824866    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2" (UID: "ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:38:50.861909 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:50.861867    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-dshm\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:38:50.861909 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:50.861895    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-tls-certs\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:38:50.861909 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:50.861905    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-tmp-dir\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:38:50.861909 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:50.861914    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-model-cache\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:38:50.861909 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:50.861922    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-home\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:38:50.862343 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:50.861954    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-kserve-provision-location\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:38:50.862343 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:50.861964    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhtgm\" (UniqueName: \"kubernetes.io/projected/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2-kube-api-access-lhtgm\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:38:51.419311 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:51.419272    2552 generic.go:358] "Generic (PLEG): container finished" podID="ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2" containerID="d9df1d3db3411977aeaf2b4abbc4dad71ad9d746bbae03b54aed674c9b18cc19" exitCode=137
May 26 16:38:51.419474 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:51.419366    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk" event={"ID":"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2","Type":"ContainerDied","Data":"d9df1d3db3411977aeaf2b4abbc4dad71ad9d746bbae03b54aed674c9b18cc19"}
May 26 16:38:51.419474 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:51.419370    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk"
May 26 16:38:51.419474 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:51.419413    2552 scope.go:117] "RemoveContainer" containerID="d9df1d3db3411977aeaf2b4abbc4dad71ad9d746bbae03b54aed674c9b18cc19"
May 26 16:38:51.419641 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:51.419401    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk" event={"ID":"ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2","Type":"ContainerDied","Data":"9a4617f09e1a01618a1391d357b7cf161116f88699c9d7c7d7862d70aebd2983"}
May 26 16:38:51.427502 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:51.427482    2552 scope.go:117] "RemoveContainer" containerID="2b2d35cb7326fee6186e35a639569fa936ba09d6b479f612c7a094f4b5745afe"
May 26 16:38:51.462179 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:51.462145    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk"]
May 26 16:38:51.467114 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:51.467088    2552 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-k9mbk"]
May 26 16:38:51.495527 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:51.495496    2552 scope.go:117] "RemoveContainer" containerID="d9df1d3db3411977aeaf2b4abbc4dad71ad9d746bbae03b54aed674c9b18cc19"
May 26 16:38:51.495865 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:38:51.495837    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9df1d3db3411977aeaf2b4abbc4dad71ad9d746bbae03b54aed674c9b18cc19\": container with ID starting with d9df1d3db3411977aeaf2b4abbc4dad71ad9d746bbae03b54aed674c9b18cc19 not found: ID does not exist" containerID="d9df1d3db3411977aeaf2b4abbc4dad71ad9d746bbae03b54aed674c9b18cc19"
May 26 16:38:51.495970 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:51.495868    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9df1d3db3411977aeaf2b4abbc4dad71ad9d746bbae03b54aed674c9b18cc19"} err="failed to get container status \"d9df1d3db3411977aeaf2b4abbc4dad71ad9d746bbae03b54aed674c9b18cc19\": rpc error: code = NotFound desc = could not find container \"d9df1d3db3411977aeaf2b4abbc4dad71ad9d746bbae03b54aed674c9b18cc19\": container with ID starting with d9df1d3db3411977aeaf2b4abbc4dad71ad9d746bbae03b54aed674c9b18cc19 not found: ID does not exist"
May 26 16:38:51.495970 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:51.495889    2552 scope.go:117] "RemoveContainer" containerID="2b2d35cb7326fee6186e35a639569fa936ba09d6b479f612c7a094f4b5745afe"
May 26 16:38:51.496240 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:38:51.496220    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b2d35cb7326fee6186e35a639569fa936ba09d6b479f612c7a094f4b5745afe\": container with ID starting with 2b2d35cb7326fee6186e35a639569fa936ba09d6b479f612c7a094f4b5745afe not found: ID does not exist" containerID="2b2d35cb7326fee6186e35a639569fa936ba09d6b479f612c7a094f4b5745afe"
May 26 16:38:51.496307 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:51.496251    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b2d35cb7326fee6186e35a639569fa936ba09d6b479f612c7a094f4b5745afe"} err="failed to get container status \"2b2d35cb7326fee6186e35a639569fa936ba09d6b479f612c7a094f4b5745afe\": rpc error: code = NotFound desc = could not find container \"2b2d35cb7326fee6186e35a639569fa936ba09d6b479f612c7a094f4b5745afe\": container with ID starting with 2b2d35cb7326fee6186e35a639569fa936ba09d6b479f612c7a094f4b5745afe not found: ID does not exist"
May 26 16:38:52.890097 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:52.890058    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2" path="/var/lib/kubelet/pods/ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2/volumes"
May 26 16:38:54.506851 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:38:54.506798    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl" podUID="4b2a69a8-b745-4203-b81b-2c4b45a60333" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
May 26 16:39:04.507260 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:04.507214    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl" podUID="4b2a69a8-b745-4203-b81b-2c4b45a60333" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
May 26 16:39:14.506731 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:14.506681    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl" podUID="4b2a69a8-b745-4203-b81b-2c4b45a60333" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
May 26 16:39:24.516875 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:24.516843    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl"
May 26 16:39:24.524526 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:24.524505    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl"
May 26 16:39:28.928854 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:28.928818    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl"]
May 26 16:39:28.929345 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:28.929216    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl" podUID="4b2a69a8-b745-4203-b81b-2c4b45a60333" containerName="main" containerID="cri-o://75d72fc40bd2e1ce6f2fa6fe71b87c272b217b3ab8f9e22b088ebc71eb8fa6df" gracePeriod=30
May 26 16:39:34.737455 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:34.737421    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 26 16:39:34.737842 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:34.737731    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2" containerName="storage-initializer"
May 26 16:39:34.737842 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:34.737743    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2" containerName="storage-initializer"
May 26 16:39:34.737842 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:34.737760    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2" containerName="main"
May 26 16:39:34.737842 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:34.737766    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2" containerName="main"
May 26 16:39:34.737842 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:34.737819    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="ffcac4fe-2fc5-495b-91ef-3e04fa0c0ce2" containerName="main"
May 26 16:39:34.741094 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:34.741077    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 16:39:34.753588 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:34.753566    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 26 16:39:34.755556 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:34.755538    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs\""
May 26 16:39:34.755757 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:34.755743    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-f312f5-cb7fb8cf-dockercfg-2m5pn\""
May 26 16:39:34.823308 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:34.823275    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ea5f6b2c-4776-4505-87a2-3e78e8154aaa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 16:39:34.823478 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:34.823315    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ea5f6b2c-4776-4505-87a2-3e78e8154aaa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 16:39:34.823478 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:34.823331    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ea5f6b2c-4776-4505-87a2-3e78e8154aaa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 16:39:34.823478 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:34.823433    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ea5f6b2c-4776-4505-87a2-3e78e8154aaa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 16:39:34.823612 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:34.823482    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ea5f6b2c-4776-4505-87a2-3e78e8154aaa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 16:39:34.823612 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:34.823540    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ea5f6b2c-4776-4505-87a2-3e78e8154aaa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 16:39:34.823612 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:34.823584    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjfkq\" (UniqueName: \"kubernetes.io/projected/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-kube-api-access-jjfkq\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ea5f6b2c-4776-4505-87a2-3e78e8154aaa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 16:39:34.924995 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:34.924965    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ea5f6b2c-4776-4505-87a2-3e78e8154aaa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 16:39:34.925171 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:34.925004    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ea5f6b2c-4776-4505-87a2-3e78e8154aaa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 16:39:34.925171 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:34.925051    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ea5f6b2c-4776-4505-87a2-3e78e8154aaa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 16:39:34.925171 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:34.925088    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ea5f6b2c-4776-4505-87a2-3e78e8154aaa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 16:39:34.925171 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:34.925135    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ea5f6b2c-4776-4505-87a2-3e78e8154aaa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 16:39:34.925393 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:34.925177    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jjfkq\" (UniqueName: \"kubernetes.io/projected/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-kube-api-access-jjfkq\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ea5f6b2c-4776-4505-87a2-3e78e8154aaa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 16:39:34.925393 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:34.925239    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ea5f6b2c-4776-4505-87a2-3e78e8154aaa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 16:39:34.925393 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:34.925331    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ea5f6b2c-4776-4505-87a2-3e78e8154aaa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 16:39:34.925561 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:34.925388    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ea5f6b2c-4776-4505-87a2-3e78e8154aaa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 16:39:34.925561 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:34.925477    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ea5f6b2c-4776-4505-87a2-3e78e8154aaa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 16:39:34.925561 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:34.925492    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ea5f6b2c-4776-4505-87a2-3e78e8154aaa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 16:39:34.927273 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:34.927252    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ea5f6b2c-4776-4505-87a2-3e78e8154aaa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 16:39:34.927616 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:34.927598    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ea5f6b2c-4776-4505-87a2-3e78e8154aaa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 16:39:34.933911 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:34.933883    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjfkq\" (UniqueName: \"kubernetes.io/projected/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-kube-api-access-jjfkq\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"ea5f6b2c-4776-4505-87a2-3e78e8154aaa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 16:39:35.050657 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:35.050625    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 16:39:35.184633 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:35.184479    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 26 16:39:35.186817 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:39:35.186787    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea5f6b2c_4776_4505_87a2_3e78e8154aaa.slice/crio-ff587565371ba77903e5edc83c2acd365a7a19dae570cc4d5792cb5a0a6d55d9 WatchSource:0}: Error finding container ff587565371ba77903e5edc83c2acd365a7a19dae570cc4d5792cb5a0a6d55d9: Status 404 returned error can't find the container with id ff587565371ba77903e5edc83c2acd365a7a19dae570cc4d5792cb5a0a6d55d9
May 26 16:39:35.575991 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:35.575953    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"ea5f6b2c-4776-4505-87a2-3e78e8154aaa","Type":"ContainerStarted","Data":"e4178133c0fb9bc7f75348689e3d469c11e200850c8b3629915cf174bd2b6b5a"}
May 26 16:39:35.575991 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:35.575995    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"ea5f6b2c-4776-4505-87a2-3e78e8154aaa","Type":"ContainerStarted","Data":"ff587565371ba77903e5edc83c2acd365a7a19dae570cc4d5792cb5a0a6d55d9"}
May 26 16:39:39.590917 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:39.590830    2552 generic.go:358] "Generic (PLEG): container finished" podID="ea5f6b2c-4776-4505-87a2-3e78e8154aaa" containerID="e4178133c0fb9bc7f75348689e3d469c11e200850c8b3629915cf174bd2b6b5a" exitCode=0
May 26 16:39:39.591268 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:39.590907    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"ea5f6b2c-4776-4505-87a2-3e78e8154aaa","Type":"ContainerDied","Data":"e4178133c0fb9bc7f75348689e3d469c11e200850c8b3629915cf174bd2b6b5a"}
May 26 16:39:40.595908 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:40.595871    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"ea5f6b2c-4776-4505-87a2-3e78e8154aaa","Type":"ContainerStarted","Data":"4a795e888df6f283d3c2519197233ad1823a97fbd843f7ae66fc290b2de26b44"}
May 26 16:39:40.618779 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:40.618736    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podStartSLOduration=6.618722267 podStartE2EDuration="6.618722267s" podCreationTimestamp="2026-05-26 16:39:34 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:39:40.61711175 +0000 UTC m=+2330.325566790" watchObservedRunningTime="2026-05-26 16:39:40.618722267 +0000 UTC m=+2330.327177305"
May 26 16:39:45.051457 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:45.051421    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 16:39:45.052853 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:45.052826    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="ea5f6b2c-4776-4505-87a2-3e78e8154aaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 26 16:39:55.051654 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:55.051612    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="ea5f6b2c-4776-4505-87a2-3e78e8154aaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 26 16:39:59.175773 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:59.175749    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl_4b2a69a8-b745-4203-b81b-2c4b45a60333/main/0.log"
May 26 16:39:59.176171 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:59.176156    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl"
May 26 16:39:59.225151 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:59.225074    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkbsd\" (UniqueName: \"kubernetes.io/projected/4b2a69a8-b745-4203-b81b-2c4b45a60333-kube-api-access-pkbsd\") pod \"4b2a69a8-b745-4203-b81b-2c4b45a60333\" (UID: \"4b2a69a8-b745-4203-b81b-2c4b45a60333\") "
May 26 16:39:59.225151 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:59.225113    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b2a69a8-b745-4203-b81b-2c4b45a60333-kserve-provision-location\") pod \"4b2a69a8-b745-4203-b81b-2c4b45a60333\" (UID: \"4b2a69a8-b745-4203-b81b-2c4b45a60333\") "
May 26 16:39:59.225151 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:59.225141    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b2a69a8-b745-4203-b81b-2c4b45a60333-model-cache\") pod \"4b2a69a8-b745-4203-b81b-2c4b45a60333\" (UID: \"4b2a69a8-b745-4203-b81b-2c4b45a60333\") "
May 26 16:39:59.225410 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:59.225166    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b2a69a8-b745-4203-b81b-2c4b45a60333-dshm\") pod \"4b2a69a8-b745-4203-b81b-2c4b45a60333\" (UID: \"4b2a69a8-b745-4203-b81b-2c4b45a60333\") "
May 26 16:39:59.225410 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:59.225200    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b2a69a8-b745-4203-b81b-2c4b45a60333-tmp-dir\") pod \"4b2a69a8-b745-4203-b81b-2c4b45a60333\" (UID: \"4b2a69a8-b745-4203-b81b-2c4b45a60333\") "
May 26 16:39:59.225410 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:59.225250    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b2a69a8-b745-4203-b81b-2c4b45a60333-tls-certs\") pod \"4b2a69a8-b745-4203-b81b-2c4b45a60333\" (UID: \"4b2a69a8-b745-4203-b81b-2c4b45a60333\") "
May 26 16:39:59.225410 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:59.225312    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b2a69a8-b745-4203-b81b-2c4b45a60333-home\") pod \"4b2a69a8-b745-4203-b81b-2c4b45a60333\" (UID: \"4b2a69a8-b745-4203-b81b-2c4b45a60333\") "
May 26 16:39:59.225410 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:59.225355    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b2a69a8-b745-4203-b81b-2c4b45a60333-model-cache" (OuterVolumeSpecName: "model-cache") pod "4b2a69a8-b745-4203-b81b-2c4b45a60333" (UID: "4b2a69a8-b745-4203-b81b-2c4b45a60333"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:39:59.225666 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:59.225564    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b2a69a8-b745-4203-b81b-2c4b45a60333-model-cache\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:39:59.226152 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:59.226122    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b2a69a8-b745-4203-b81b-2c4b45a60333-home" (OuterVolumeSpecName: "home") pod "4b2a69a8-b745-4203-b81b-2c4b45a60333" (UID: "4b2a69a8-b745-4203-b81b-2c4b45a60333"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:39:59.227510 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:59.227483    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b2a69a8-b745-4203-b81b-2c4b45a60333-kube-api-access-pkbsd" (OuterVolumeSpecName: "kube-api-access-pkbsd") pod "4b2a69a8-b745-4203-b81b-2c4b45a60333" (UID: "4b2a69a8-b745-4203-b81b-2c4b45a60333"). InnerVolumeSpecName "kube-api-access-pkbsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:39:59.227701 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:59.227674    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b2a69a8-b745-4203-b81b-2c4b45a60333-dshm" (OuterVolumeSpecName: "dshm") pod "4b2a69a8-b745-4203-b81b-2c4b45a60333" (UID: "4b2a69a8-b745-4203-b81b-2c4b45a60333"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:39:59.228058 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:59.228028    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b2a69a8-b745-4203-b81b-2c4b45a60333-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4b2a69a8-b745-4203-b81b-2c4b45a60333" (UID: "4b2a69a8-b745-4203-b81b-2c4b45a60333"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:39:59.240298 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:59.240256    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b2a69a8-b745-4203-b81b-2c4b45a60333-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4b2a69a8-b745-4203-b81b-2c4b45a60333" (UID: "4b2a69a8-b745-4203-b81b-2c4b45a60333"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:39:59.287693 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:59.283647    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b2a69a8-b745-4203-b81b-2c4b45a60333-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4b2a69a8-b745-4203-b81b-2c4b45a60333" (UID: "4b2a69a8-b745-4203-b81b-2c4b45a60333"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:39:59.326031 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:59.325995    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkbsd\" (UniqueName: \"kubernetes.io/projected/4b2a69a8-b745-4203-b81b-2c4b45a60333-kube-api-access-pkbsd\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:39:59.326031 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:59.326027    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b2a69a8-b745-4203-b81b-2c4b45a60333-kserve-provision-location\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:39:59.326031 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:59.326037    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b2a69a8-b745-4203-b81b-2c4b45a60333-dshm\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:39:59.326252 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:59.326048    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b2a69a8-b745-4203-b81b-2c4b45a60333-tmp-dir\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:39:59.326252 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:59.326056    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b2a69a8-b745-4203-b81b-2c4b45a60333-tls-certs\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:39:59.326252 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:59.326065    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b2a69a8-b745-4203-b81b-2c4b45a60333-home\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:39:59.658269 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:59.658241    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl_4b2a69a8-b745-4203-b81b-2c4b45a60333/main/0.log"
May 26 16:39:59.658584 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:59.658560    2552 generic.go:358] "Generic (PLEG): container finished" podID="4b2a69a8-b745-4203-b81b-2c4b45a60333" containerID="75d72fc40bd2e1ce6f2fa6fe71b87c272b217b3ab8f9e22b088ebc71eb8fa6df" exitCode=137
May 26 16:39:59.658671 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:59.658638    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl" event={"ID":"4b2a69a8-b745-4203-b81b-2c4b45a60333","Type":"ContainerDied","Data":"75d72fc40bd2e1ce6f2fa6fe71b87c272b217b3ab8f9e22b088ebc71eb8fa6df"}
May 26 16:39:59.658736 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:59.658672    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl" event={"ID":"4b2a69a8-b745-4203-b81b-2c4b45a60333","Type":"ContainerDied","Data":"7b375fbcf9b567854601765eb0d429ce18a54076a94acfaf39125b5f4d4aaac2"}
May 26 16:39:59.658736 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:59.658675    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl"
May 26 16:39:59.658844 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:59.658689    2552 scope.go:117] "RemoveContainer" containerID="75d72fc40bd2e1ce6f2fa6fe71b87c272b217b3ab8f9e22b088ebc71eb8fa6df"
May 26 16:39:59.667700 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:59.667681    2552 scope.go:117] "RemoveContainer" containerID="18e3edebab5a38e2d198636c3effdab97a8b861953c6c215a59211a198367f57"
May 26 16:39:59.677621 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:59.677599    2552 scope.go:117] "RemoveContainer" containerID="75d72fc40bd2e1ce6f2fa6fe71b87c272b217b3ab8f9e22b088ebc71eb8fa6df"
May 26 16:39:59.677924 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:39:59.677899    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75d72fc40bd2e1ce6f2fa6fe71b87c272b217b3ab8f9e22b088ebc71eb8fa6df\": container with ID starting with 75d72fc40bd2e1ce6f2fa6fe71b87c272b217b3ab8f9e22b088ebc71eb8fa6df not found: ID does not exist" containerID="75d72fc40bd2e1ce6f2fa6fe71b87c272b217b3ab8f9e22b088ebc71eb8fa6df"
May 26 16:39:59.678013 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:59.677951    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75d72fc40bd2e1ce6f2fa6fe71b87c272b217b3ab8f9e22b088ebc71eb8fa6df"} err="failed to get container status \"75d72fc40bd2e1ce6f2fa6fe71b87c272b217b3ab8f9e22b088ebc71eb8fa6df\": rpc error: code = NotFound desc = could not find container \"75d72fc40bd2e1ce6f2fa6fe71b87c272b217b3ab8f9e22b088ebc71eb8fa6df\": container with ID starting with 75d72fc40bd2e1ce6f2fa6fe71b87c272b217b3ab8f9e22b088ebc71eb8fa6df not found: ID does not exist"
May 26 16:39:59.678013 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:59.677978    2552 scope.go:117] "RemoveContainer" containerID="18e3edebab5a38e2d198636c3effdab97a8b861953c6c215a59211a198367f57"
May 26 16:39:59.678240 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:39:59.678220    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18e3edebab5a38e2d198636c3effdab97a8b861953c6c215a59211a198367f57\": container with ID starting with 18e3edebab5a38e2d198636c3effdab97a8b861953c6c215a59211a198367f57 not found: ID does not exist" containerID="18e3edebab5a38e2d198636c3effdab97a8b861953c6c215a59211a198367f57"
May 26 16:39:59.678300 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:59.678248    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18e3edebab5a38e2d198636c3effdab97a8b861953c6c215a59211a198367f57"} err="failed to get container status \"18e3edebab5a38e2d198636c3effdab97a8b861953c6c215a59211a198367f57\": rpc error: code = NotFound desc = could not find container \"18e3edebab5a38e2d198636c3effdab97a8b861953c6c215a59211a198367f57\": container with ID starting with 18e3edebab5a38e2d198636c3effdab97a8b861953c6c215a59211a198367f57 not found: ID does not exist"
May 26 16:39:59.684374 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:59.684348    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl"]
May 26 16:39:59.687567 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:39:59.687528    2552 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-75cdb9b469cn8zl"]
May 26 16:40:00.890763 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:40:00.890725    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b2a69a8-b745-4203-b81b-2c4b45a60333" path="/var/lib/kubelet/pods/4b2a69a8-b745-4203-b81b-2c4b45a60333/volumes"
May 26 16:40:03.675075 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:40:03.675043    2552 generic.go:358] "Generic (PLEG): container finished" podID="b759932b-9db9-4c73-b28c-d1871bed94ff" containerID="941a1fa001eaae030ce3af34f6146338b2f5e9e17ebe82571c9ffcdd93ff0520" exitCode=0
May 26 16:40:03.675446 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:40:03.675119    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4" event={"ID":"b759932b-9db9-4c73-b28c-d1871bed94ff","Type":"ContainerDied","Data":"941a1fa001eaae030ce3af34f6146338b2f5e9e17ebe82571c9ffcdd93ff0520"}
May 26 16:40:04.683611 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:40:04.683575    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4" event={"ID":"b759932b-9db9-4c73-b28c-d1871bed94ff","Type":"ContainerStarted","Data":"6fa86dca470c1d9ce9fdabb2d0253ce1f91b3e35ef0be8722f7475d8f54fd67c"}
May 26 16:40:04.708868 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:40:04.708813    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4" podStartSLOduration=89.70879589 podStartE2EDuration="1m29.70879589s" podCreationTimestamp="2026-05-26 16:38:35 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:40:04.705866126 +0000 UTC m=+2354.414321166" watchObservedRunningTime="2026-05-26 16:40:04.70879589 +0000 UTC m=+2354.417250931"
May 26 16:40:05.051455 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:40:05.051417    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 16:40:05.051823 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:40:05.051792    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="ea5f6b2c-4776-4505-87a2-3e78e8154aaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 26 16:40:05.925638 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:40:05.925595    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4"
May 26 16:40:05.925638 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:40:05.925649    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4"
May 26 16:40:05.927310 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:40:05.927280    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4" podUID="b759932b-9db9-4c73-b28c-d1871bed94ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
May 26 16:40:15.052081 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:40:15.052038    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="ea5f6b2c-4776-4505-87a2-3e78e8154aaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 26 16:40:15.926184 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:40:15.926143    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4" podUID="b759932b-9db9-4c73-b28c-d1871bed94ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
May 26 16:40:25.051698 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:40:25.051649    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="ea5f6b2c-4776-4505-87a2-3e78e8154aaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 26 16:40:25.926420 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:40:25.926370    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4" podUID="b759932b-9db9-4c73-b28c-d1871bed94ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
May 26 16:40:35.051772 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:40:35.051720    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="ea5f6b2c-4776-4505-87a2-3e78e8154aaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 26 16:40:35.926892 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:40:35.926845    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4" podUID="b759932b-9db9-4c73-b28c-d1871bed94ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
May 26 16:40:45.051811 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:40:45.051759    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="ea5f6b2c-4776-4505-87a2-3e78e8154aaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 26 16:40:45.926230 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:40:45.926178    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4" podUID="b759932b-9db9-4c73-b28c-d1871bed94ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
May 26 16:40:50.945058 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:40:50.945028    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sj6db_5c0f9fc6-2bd6-4370-96d0-5ae315363c7f/ovn-acl-logging/0.log"
May 26 16:40:50.947444 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:40:50.947422    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sj6db_5c0f9fc6-2bd6-4370-96d0-5ae315363c7f/ovn-acl-logging/0.log"
May 26 16:40:55.051558 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:40:55.051508    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="ea5f6b2c-4776-4505-87a2-3e78e8154aaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 26 16:40:55.926322 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:40:55.926271    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4" podUID="b759932b-9db9-4c73-b28c-d1871bed94ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
May 26 16:41:05.051377 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:05.051335    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="ea5f6b2c-4776-4505-87a2-3e78e8154aaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 26 16:41:05.926335 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:05.926291    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4" podUID="b759932b-9db9-4c73-b28c-d1871bed94ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
May 26 16:41:15.051773 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:15.051668    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="ea5f6b2c-4776-4505-87a2-3e78e8154aaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 26 16:41:15.925869 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:15.925824    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4" podUID="b759932b-9db9-4c73-b28c-d1871bed94ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
May 26 16:41:25.060976 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:25.060918    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 16:41:25.068366 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:25.068340    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 16:41:25.926788 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:25.926749    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4" podUID="b759932b-9db9-4c73-b28c-d1871bed94ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
May 26 16:41:35.331958 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:35.331899    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 26 16:41:35.332548 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:35.332313    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="ea5f6b2c-4776-4505-87a2-3e78e8154aaa" containerName="main" containerID="cri-o://4a795e888df6f283d3c2519197233ad1823a97fbd843f7ae66fc290b2de26b44" gracePeriod=30
May 26 16:41:35.926587 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:35.926548    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4" podUID="b759932b-9db9-4c73-b28c-d1871bed94ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
May 26 16:41:36.314869 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:36.314844    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 16:41:36.400955 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:36.400894    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-kserve-provision-location\") pod \"ea5f6b2c-4776-4505-87a2-3e78e8154aaa\" (UID: \"ea5f6b2c-4776-4505-87a2-3e78e8154aaa\") "
May 26 16:41:36.401422 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:36.400971    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-model-cache\") pod \"ea5f6b2c-4776-4505-87a2-3e78e8154aaa\" (UID: \"ea5f6b2c-4776-4505-87a2-3e78e8154aaa\") "
May 26 16:41:36.401422 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:36.401026    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-home\") pod \"ea5f6b2c-4776-4505-87a2-3e78e8154aaa\" (UID: \"ea5f6b2c-4776-4505-87a2-3e78e8154aaa\") "
May 26 16:41:36.401422 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:36.401080    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-tmp-dir\") pod \"ea5f6b2c-4776-4505-87a2-3e78e8154aaa\" (UID: \"ea5f6b2c-4776-4505-87a2-3e78e8154aaa\") "
May 26 16:41:36.401422 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:36.401111    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-dshm\") pod \"ea5f6b2c-4776-4505-87a2-3e78e8154aaa\" (UID: \"ea5f6b2c-4776-4505-87a2-3e78e8154aaa\") "
May 26 16:41:36.401422 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:36.401153    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjfkq\" (UniqueName: \"kubernetes.io/projected/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-kube-api-access-jjfkq\") pod \"ea5f6b2c-4776-4505-87a2-3e78e8154aaa\" (UID: \"ea5f6b2c-4776-4505-87a2-3e78e8154aaa\") "
May 26 16:41:36.401422 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:36.401204    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-tls-certs\") pod \"ea5f6b2c-4776-4505-87a2-3e78e8154aaa\" (UID: \"ea5f6b2c-4776-4505-87a2-3e78e8154aaa\") "
May 26 16:41:36.401422 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:36.401272    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-model-cache" (OuterVolumeSpecName: "model-cache") pod "ea5f6b2c-4776-4505-87a2-3e78e8154aaa" (UID: "ea5f6b2c-4776-4505-87a2-3e78e8154aaa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:41:36.401734 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:36.401499    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-model-cache\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:41:36.401734 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:36.401696    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-home" (OuterVolumeSpecName: "home") pod "ea5f6b2c-4776-4505-87a2-3e78e8154aaa" (UID: "ea5f6b2c-4776-4505-87a2-3e78e8154aaa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:41:36.403614 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:36.403579    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-kube-api-access-jjfkq" (OuterVolumeSpecName: "kube-api-access-jjfkq") pod "ea5f6b2c-4776-4505-87a2-3e78e8154aaa" (UID: "ea5f6b2c-4776-4505-87a2-3e78e8154aaa"). InnerVolumeSpecName "kube-api-access-jjfkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:41:36.403872 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:36.403853    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-dshm" (OuterVolumeSpecName: "dshm") pod "ea5f6b2c-4776-4505-87a2-3e78e8154aaa" (UID: "ea5f6b2c-4776-4505-87a2-3e78e8154aaa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:41:36.403971 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:36.403915    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ea5f6b2c-4776-4505-87a2-3e78e8154aaa" (UID: "ea5f6b2c-4776-4505-87a2-3e78e8154aaa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:41:36.420797 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:36.420745    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ea5f6b2c-4776-4505-87a2-3e78e8154aaa" (UID: "ea5f6b2c-4776-4505-87a2-3e78e8154aaa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:41:36.460643 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:36.460612    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ea5f6b2c-4776-4505-87a2-3e78e8154aaa" (UID: "ea5f6b2c-4776-4505-87a2-3e78e8154aaa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:41:36.501874 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:36.501819    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-home\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:41:36.501874 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:36.501842    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-tmp-dir\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:41:36.501874 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:36.501853    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-dshm\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:41:36.501874 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:36.501861    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjfkq\" (UniqueName: \"kubernetes.io/projected/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-kube-api-access-jjfkq\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:41:36.501874 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:36.501871    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-tls-certs\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:41:36.502110 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:36.501880    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea5f6b2c-4776-4505-87a2-3e78e8154aaa-kserve-provision-location\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:41:37.018197 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:37.018154    2552 generic.go:358] "Generic (PLEG): container finished" podID="ea5f6b2c-4776-4505-87a2-3e78e8154aaa" containerID="4a795e888df6f283d3c2519197233ad1823a97fbd843f7ae66fc290b2de26b44" exitCode=0
May 26 16:41:37.018373 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:37.018274    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 16:41:37.018373 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:37.018296    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"ea5f6b2c-4776-4505-87a2-3e78e8154aaa","Type":"ContainerDied","Data":"4a795e888df6f283d3c2519197233ad1823a97fbd843f7ae66fc290b2de26b44"}
May 26 16:41:37.018373 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:37.018341    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"ea5f6b2c-4776-4505-87a2-3e78e8154aaa","Type":"ContainerDied","Data":"ff587565371ba77903e5edc83c2acd365a7a19dae570cc4d5792cb5a0a6d55d9"}
May 26 16:41:37.018373 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:37.018363    2552 scope.go:117] "RemoveContainer" containerID="4a795e888df6f283d3c2519197233ad1823a97fbd843f7ae66fc290b2de26b44"
May 26 16:41:37.026126 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:37.026104    2552 scope.go:117] "RemoveContainer" containerID="e4178133c0fb9bc7f75348689e3d469c11e200850c8b3629915cf174bd2b6b5a"
May 26 16:41:37.041829 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:37.041743    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 26 16:41:37.044100 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:37.044073    2552 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 26 16:41:37.091399 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:37.091372    2552 scope.go:117] "RemoveContainer" containerID="4a795e888df6f283d3c2519197233ad1823a97fbd843f7ae66fc290b2de26b44"
May 26 16:41:37.091723 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:41:37.091701    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a795e888df6f283d3c2519197233ad1823a97fbd843f7ae66fc290b2de26b44\": container with ID starting with 4a795e888df6f283d3c2519197233ad1823a97fbd843f7ae66fc290b2de26b44 not found: ID does not exist" containerID="4a795e888df6f283d3c2519197233ad1823a97fbd843f7ae66fc290b2de26b44"
May 26 16:41:37.091824 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:37.091731    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a795e888df6f283d3c2519197233ad1823a97fbd843f7ae66fc290b2de26b44"} err="failed to get container status \"4a795e888df6f283d3c2519197233ad1823a97fbd843f7ae66fc290b2de26b44\": rpc error: code = NotFound desc = could not find container \"4a795e888df6f283d3c2519197233ad1823a97fbd843f7ae66fc290b2de26b44\": container with ID starting with 4a795e888df6f283d3c2519197233ad1823a97fbd843f7ae66fc290b2de26b44 not found: ID does not exist"
May 26 16:41:37.091824 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:37.091750    2552 scope.go:117] "RemoveContainer" containerID="e4178133c0fb9bc7f75348689e3d469c11e200850c8b3629915cf174bd2b6b5a"
May 26 16:41:37.092083 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:41:37.092063    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4178133c0fb9bc7f75348689e3d469c11e200850c8b3629915cf174bd2b6b5a\": container with ID starting with e4178133c0fb9bc7f75348689e3d469c11e200850c8b3629915cf174bd2b6b5a not found: ID does not exist" containerID="e4178133c0fb9bc7f75348689e3d469c11e200850c8b3629915cf174bd2b6b5a"
May 26 16:41:37.092142 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:37.092090    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4178133c0fb9bc7f75348689e3d469c11e200850c8b3629915cf174bd2b6b5a"} err="failed to get container status \"e4178133c0fb9bc7f75348689e3d469c11e200850c8b3629915cf174bd2b6b5a\": rpc error: code = NotFound desc = could not find container \"e4178133c0fb9bc7f75348689e3d469c11e200850c8b3629915cf174bd2b6b5a\": container with ID starting with e4178133c0fb9bc7f75348689e3d469c11e200850c8b3629915cf174bd2b6b5a not found: ID does not exist"
May 26 16:41:38.890641 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:38.890606    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea5f6b2c-4776-4505-87a2-3e78e8154aaa" path="/var/lib/kubelet/pods/ea5f6b2c-4776-4505-87a2-3e78e8154aaa/volumes"
May 26 16:41:45.925814 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:45.925769    2552 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4" podUID="b759932b-9db9-4c73-b28c-d1871bed94ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
May 26 16:41:55.936138 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:55.936104    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4"
May 26 16:41:55.943548 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:55.943524    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4"
May 26 16:41:57.030614 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:57.030579    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4"]
May 26 16:41:57.089162 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:41:57.089109    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4" podUID="b759932b-9db9-4c73-b28c-d1871bed94ff" containerName="main" containerID="cri-o://6fa86dca470c1d9ce9fdabb2d0253ce1f91b3e35ef0be8722f7475d8f54fd67c" gracePeriod=30
May 26 16:42:27.339651 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:27.339628    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-c5d69ddfc-qx5h4_b759932b-9db9-4c73-b28c-d1871bed94ff/main/0.log"
May 26 16:42:27.340038 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:27.340023    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4"
May 26 16:42:27.407646 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:27.407576    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b759932b-9db9-4c73-b28c-d1871bed94ff-tmp-dir\") pod \"b759932b-9db9-4c73-b28c-d1871bed94ff\" (UID: \"b759932b-9db9-4c73-b28c-d1871bed94ff\") "
May 26 16:42:27.407777 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:27.407653    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b759932b-9db9-4c73-b28c-d1871bed94ff-model-cache\") pod \"b759932b-9db9-4c73-b28c-d1871bed94ff\" (UID: \"b759932b-9db9-4c73-b28c-d1871bed94ff\") "
May 26 16:42:27.407777 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:27.407690    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b759932b-9db9-4c73-b28c-d1871bed94ff-home\") pod \"b759932b-9db9-4c73-b28c-d1871bed94ff\" (UID: \"b759932b-9db9-4c73-b28c-d1871bed94ff\") "
May 26 16:42:27.407777 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:27.407712    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b759932b-9db9-4c73-b28c-d1871bed94ff-tls-certs\") pod \"b759932b-9db9-4c73-b28c-d1871bed94ff\" (UID: \"b759932b-9db9-4c73-b28c-d1871bed94ff\") "
May 26 16:42:27.407777 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:27.407737    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsgxv\" (UniqueName: \"kubernetes.io/projected/b759932b-9db9-4c73-b28c-d1871bed94ff-kube-api-access-wsgxv\") pod \"b759932b-9db9-4c73-b28c-d1871bed94ff\" (UID: \"b759932b-9db9-4c73-b28c-d1871bed94ff\") "
May 26 16:42:27.407777 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:27.407771    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b759932b-9db9-4c73-b28c-d1871bed94ff-dshm\") pod \"b759932b-9db9-4c73-b28c-d1871bed94ff\" (UID: \"b759932b-9db9-4c73-b28c-d1871bed94ff\") "
May 26 16:42:27.408061 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:27.407804    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b759932b-9db9-4c73-b28c-d1871bed94ff-kserve-provision-location\") pod \"b759932b-9db9-4c73-b28c-d1871bed94ff\" (UID: \"b759932b-9db9-4c73-b28c-d1871bed94ff\") "
May 26 16:42:27.408061 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:27.407997    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b759932b-9db9-4c73-b28c-d1871bed94ff-model-cache" (OuterVolumeSpecName: "model-cache") pod "b759932b-9db9-4c73-b28c-d1871bed94ff" (UID: "b759932b-9db9-4c73-b28c-d1871bed94ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:42:27.408178 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:27.408093    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b759932b-9db9-4c73-b28c-d1871bed94ff-model-cache\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:42:27.408465 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:27.408438    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b759932b-9db9-4c73-b28c-d1871bed94ff-home" (OuterVolumeSpecName: "home") pod "b759932b-9db9-4c73-b28c-d1871bed94ff" (UID: "b759932b-9db9-4c73-b28c-d1871bed94ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:42:27.409889 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:27.409857    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b759932b-9db9-4c73-b28c-d1871bed94ff-dshm" (OuterVolumeSpecName: "dshm") pod "b759932b-9db9-4c73-b28c-d1871bed94ff" (UID: "b759932b-9db9-4c73-b28c-d1871bed94ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:42:27.410019 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:27.409968    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b759932b-9db9-4c73-b28c-d1871bed94ff-kube-api-access-wsgxv" (OuterVolumeSpecName: "kube-api-access-wsgxv") pod "b759932b-9db9-4c73-b28c-d1871bed94ff" (UID: "b759932b-9db9-4c73-b28c-d1871bed94ff"). InnerVolumeSpecName "kube-api-access-wsgxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:42:27.410107 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:27.410088    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b759932b-9db9-4c73-b28c-d1871bed94ff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b759932b-9db9-4c73-b28c-d1871bed94ff" (UID: "b759932b-9db9-4c73-b28c-d1871bed94ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:42:27.430227 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:27.430202    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b759932b-9db9-4c73-b28c-d1871bed94ff-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b759932b-9db9-4c73-b28c-d1871bed94ff" (UID: "b759932b-9db9-4c73-b28c-d1871bed94ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:42:27.482256 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:27.482214    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b759932b-9db9-4c73-b28c-d1871bed94ff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b759932b-9db9-4c73-b28c-d1871bed94ff" (UID: "b759932b-9db9-4c73-b28c-d1871bed94ff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:42:27.508870 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:27.508851    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b759932b-9db9-4c73-b28c-d1871bed94ff-home\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:42:27.508949 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:27.508875    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b759932b-9db9-4c73-b28c-d1871bed94ff-tls-certs\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:42:27.508949 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:27.508888    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsgxv\" (UniqueName: \"kubernetes.io/projected/b759932b-9db9-4c73-b28c-d1871bed94ff-kube-api-access-wsgxv\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:42:27.508949 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:27.508897    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b759932b-9db9-4c73-b28c-d1871bed94ff-dshm\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:42:27.508949 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:27.508906    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b759932b-9db9-4c73-b28c-d1871bed94ff-kserve-provision-location\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:42:27.508949 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:27.508914    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b759932b-9db9-4c73-b28c-d1871bed94ff-tmp-dir\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:42:28.187890 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:28.187861    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-c5d69ddfc-qx5h4_b759932b-9db9-4c73-b28c-d1871bed94ff/main/0.log"
May 26 16:42:28.188202 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:28.188178    2552 generic.go:358] "Generic (PLEG): container finished" podID="b759932b-9db9-4c73-b28c-d1871bed94ff" containerID="6fa86dca470c1d9ce9fdabb2d0253ce1f91b3e35ef0be8722f7475d8f54fd67c" exitCode=137
May 26 16:42:28.188278 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:28.188215    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4" event={"ID":"b759932b-9db9-4c73-b28c-d1871bed94ff","Type":"ContainerDied","Data":"6fa86dca470c1d9ce9fdabb2d0253ce1f91b3e35ef0be8722f7475d8f54fd67c"}
May 26 16:42:28.188278 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:28.188239    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4" event={"ID":"b759932b-9db9-4c73-b28c-d1871bed94ff","Type":"ContainerDied","Data":"041e37d2430d7a4841db04ebca29aa17df89ca3a48e1e00fde348fd349cd3f30"}
May 26 16:42:28.188278 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:28.188245    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4"
May 26 16:42:28.188278 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:28.188259    2552 scope.go:117] "RemoveContainer" containerID="6fa86dca470c1d9ce9fdabb2d0253ce1f91b3e35ef0be8722f7475d8f54fd67c"
May 26 16:42:28.197552 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:28.197535    2552 scope.go:117] "RemoveContainer" containerID="941a1fa001eaae030ce3af34f6146338b2f5e9e17ebe82571c9ffcdd93ff0520"
May 26 16:42:28.208281 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:28.208261    2552 scope.go:117] "RemoveContainer" containerID="6fa86dca470c1d9ce9fdabb2d0253ce1f91b3e35ef0be8722f7475d8f54fd67c"
May 26 16:42:28.208510 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:42:28.208493    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fa86dca470c1d9ce9fdabb2d0253ce1f91b3e35ef0be8722f7475d8f54fd67c\": container with ID starting with 6fa86dca470c1d9ce9fdabb2d0253ce1f91b3e35ef0be8722f7475d8f54fd67c not found: ID does not exist" containerID="6fa86dca470c1d9ce9fdabb2d0253ce1f91b3e35ef0be8722f7475d8f54fd67c"
May 26 16:42:28.208579 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:28.208517    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fa86dca470c1d9ce9fdabb2d0253ce1f91b3e35ef0be8722f7475d8f54fd67c"} err="failed to get container status \"6fa86dca470c1d9ce9fdabb2d0253ce1f91b3e35ef0be8722f7475d8f54fd67c\": rpc error: code = NotFound desc = could not find container \"6fa86dca470c1d9ce9fdabb2d0253ce1f91b3e35ef0be8722f7475d8f54fd67c\": container with ID starting with 6fa86dca470c1d9ce9fdabb2d0253ce1f91b3e35ef0be8722f7475d8f54fd67c not found: ID does not exist"
May 26 16:42:28.208579 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:28.208534    2552 scope.go:117] "RemoveContainer" containerID="941a1fa001eaae030ce3af34f6146338b2f5e9e17ebe82571c9ffcdd93ff0520"
May 26 16:42:28.208787 ip-10-0-131-96 kubenswrapper[2552]: E0526 16:42:28.208754    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"941a1fa001eaae030ce3af34f6146338b2f5e9e17ebe82571c9ffcdd93ff0520\": container with ID starting with 941a1fa001eaae030ce3af34f6146338b2f5e9e17ebe82571c9ffcdd93ff0520 not found: ID does not exist" containerID="941a1fa001eaae030ce3af34f6146338b2f5e9e17ebe82571c9ffcdd93ff0520"
May 26 16:42:28.208891 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:28.208795    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"941a1fa001eaae030ce3af34f6146338b2f5e9e17ebe82571c9ffcdd93ff0520"} err="failed to get container status \"941a1fa001eaae030ce3af34f6146338b2f5e9e17ebe82571c9ffcdd93ff0520\": rpc error: code = NotFound desc = could not find container \"941a1fa001eaae030ce3af34f6146338b2f5e9e17ebe82571c9ffcdd93ff0520\": container with ID starting with 941a1fa001eaae030ce3af34f6146338b2f5e9e17ebe82571c9ffcdd93ff0520 not found: ID does not exist"
May 26 16:42:28.211319 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:28.211300    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4"]
May 26 16:42:28.215693 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:28.215671    2552 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-c5d69ddfc-qx5h4"]
May 26 16:42:28.890166 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:28.890133    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b759932b-9db9-4c73-b28c-d1871bed94ff" path="/var/lib/kubelet/pods/b759932b-9db9-4c73-b28c-d1871bed94ff/volumes"
May 26 16:42:34.973227 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:34.973199    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-7f6dfc774f-znr6c"]
May 26 16:42:34.973600 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:34.973439    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve/llmisvc-controller-manager-7f6dfc774f-znr6c" podUID="880bb567-ef75-4853-ad67-794058f47650" containerName="manager" containerID="cri-o://c2089bf3dcdd62483b5f929e34d9a378085ef4e1a336d58121490fc7aa1496b5" gracePeriod=30
May 26 16:42:35.212460 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:35.212430    2552 generic.go:358] "Generic (PLEG): container finished" podID="880bb567-ef75-4853-ad67-794058f47650" containerID="c2089bf3dcdd62483b5f929e34d9a378085ef4e1a336d58121490fc7aa1496b5" exitCode=0
May 26 16:42:35.212559 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:35.212500    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-7f6dfc774f-znr6c" event={"ID":"880bb567-ef75-4853-ad67-794058f47650","Type":"ContainerDied","Data":"c2089bf3dcdd62483b5f929e34d9a378085ef4e1a336d58121490fc7aa1496b5"}
May 26 16:42:35.212559 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:35.212535    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-7f6dfc774f-znr6c" event={"ID":"880bb567-ef75-4853-ad67-794058f47650","Type":"ContainerDied","Data":"d04c1a6805a2d4baf2acb971c8ea8a393285b5ad0cd03fb0e199c526d00802ff"}
May 26 16:42:35.212559 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:35.212544    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d04c1a6805a2d4baf2acb971c8ea8a393285b5ad0cd03fb0e199c526d00802ff"
May 26 16:42:35.217471 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:35.217451    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-7f6dfc774f-znr6c"
May 26 16:42:35.267940 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:35.267872    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/880bb567-ef75-4853-ad67-794058f47650-cert\") pod \"880bb567-ef75-4853-ad67-794058f47650\" (UID: \"880bb567-ef75-4853-ad67-794058f47650\") "
May 26 16:42:35.268036 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:35.267924    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4prp\" (UniqueName: \"kubernetes.io/projected/880bb567-ef75-4853-ad67-794058f47650-kube-api-access-t4prp\") pod \"880bb567-ef75-4853-ad67-794058f47650\" (UID: \"880bb567-ef75-4853-ad67-794058f47650\") "
May 26 16:42:35.269850 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:35.269825    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/880bb567-ef75-4853-ad67-794058f47650-cert" (OuterVolumeSpecName: "cert") pod "880bb567-ef75-4853-ad67-794058f47650" (UID: "880bb567-ef75-4853-ad67-794058f47650"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:42:35.269965 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:35.269946    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/880bb567-ef75-4853-ad67-794058f47650-kube-api-access-t4prp" (OuterVolumeSpecName: "kube-api-access-t4prp") pod "880bb567-ef75-4853-ad67-794058f47650" (UID: "880bb567-ef75-4853-ad67-794058f47650"). InnerVolumeSpecName "kube-api-access-t4prp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:42:35.369219 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:35.369195    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4prp\" (UniqueName: \"kubernetes.io/projected/880bb567-ef75-4853-ad67-794058f47650-kube-api-access-t4prp\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:42:35.369219 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:35.369219    2552 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/880bb567-ef75-4853-ad67-794058f47650-cert\") on node \"ip-10-0-131-96.ec2.internal\" DevicePath \"\""
May 26 16:42:36.215440 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:36.215404    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-7f6dfc774f-znr6c"
May 26 16:42:36.237044 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:36.237012    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-7f6dfc774f-znr6c"]
May 26 16:42:36.241340 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:36.241316    2552 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve/llmisvc-controller-manager-7f6dfc774f-znr6c"]
May 26 16:42:36.889940 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:36.889907    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="880bb567-ef75-4853-ad67-794058f47650" path="/var/lib/kubelet/pods/880bb567-ef75-4853-ad67-794058f47650/volumes"
May 26 16:42:51.019521 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:42:51.019493    2552 scope.go:117] "RemoveContainer" containerID="c2089bf3dcdd62483b5f929e34d9a378085ef4e1a336d58121490fc7aa1496b5"
May 26 16:43:22.744134 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:22.744104    2552 ???:1] "http2: server: error reading preface from client 10.0.131.96:51826: read tcp 10.0.131.96:10250->10.0.131.96:51826: read: connection reset by peer"
May 26 16:43:22.748665 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:22.748631    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-6gstw_d1f769c9-1a3d-4212-8317-9b4b357feb0e/istio-proxy/0.log"
May 26 16:43:23.727949 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:23.727895    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-6gstw_d1f769c9-1a3d-4212-8317-9b4b357feb0e/istio-proxy/0.log"
May 26 16:43:24.695920 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:24.695893    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-6gstw_d1f769c9-1a3d-4212-8317-9b4b357feb0e/istio-proxy/0.log"
May 26 16:43:25.646718 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:25.646692    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-6gstw_d1f769c9-1a3d-4212-8317-9b4b357feb0e/istio-proxy/0.log"
May 26 16:43:26.591390 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:26.591351    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-6gstw_d1f769c9-1a3d-4212-8317-9b4b357feb0e/istio-proxy/0.log"
May 26 16:43:27.539998 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:27.539971    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-6gstw_d1f769c9-1a3d-4212-8317-9b4b357feb0e/istio-proxy/0.log"
May 26 16:43:28.490539 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:28.490507    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-6gstw_d1f769c9-1a3d-4212-8317-9b4b357feb0e/istio-proxy/0.log"
May 26 16:43:29.424169 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:29.424138    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-6gstw_d1f769c9-1a3d-4212-8317-9b4b357feb0e/istio-proxy/0.log"
May 26 16:43:30.393941 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:30.393879    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-6gstw_d1f769c9-1a3d-4212-8317-9b4b357feb0e/istio-proxy/0.log"
May 26 16:43:31.331719 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:31.331686    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-6gstw_d1f769c9-1a3d-4212-8317-9b4b357feb0e/istio-proxy/0.log"
May 26 16:43:32.298528 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:32.298482    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-6gstw_d1f769c9-1a3d-4212-8317-9b4b357feb0e/istio-proxy/0.log"
May 26 16:43:33.251411 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:33.251383    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-6gstw_d1f769c9-1a3d-4212-8317-9b4b357feb0e/istio-proxy/0.log"
May 26 16:43:34.250696 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:34.250664    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-6gstw_d1f769c9-1a3d-4212-8317-9b4b357feb0e/istio-proxy/0.log"
May 26 16:43:35.260164 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:35.260106    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-6gstw_d1f769c9-1a3d-4212-8317-9b4b357feb0e/istio-proxy/0.log"
May 26 16:43:36.349336 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:36.349303    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-798958bb55-mm958_ce1e9de0-8ee7-43fb-9091-553fe6cfe394/discovery/0.log"
May 26 16:43:36.362762 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:36.362739    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-849cdfbf5f-m95mt_8310f0a3-b09d-4746-8561-11c489116cf8/istio-proxy/0.log"
May 26 16:43:37.270039 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:37.270008    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-798958bb55-mm958_ce1e9de0-8ee7-43fb-9091-553fe6cfe394/discovery/0.log"
May 26 16:43:37.285275 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:37.285248    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-849cdfbf5f-m95mt_8310f0a3-b09d-4746-8561-11c489116cf8/istio-proxy/0.log"
May 26 16:43:38.128803 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:38.128772    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7587b89b76-ntp27_f8572d1e-9ffd-4442-bd73-d63b4a458cce/manager/0.log"
May 26 16:43:38.278200 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:38.278172    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-c7fb4c8d5-rbb7k_a07ead44-5922-4db0-aea0-2f827a0c0f4c/manager/0.log"
May 26 16:43:43.517354 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:43.517322    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-p5zcf_3369da97-ee5f-42cb-a95f-e6066680f349/global-pull-secret-syncer/0.log"
May 26 16:43:43.632413 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:43.632378    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-v7qxk_50bb2f40-6580-4a07-abf3-e6cd8ca11465/konnectivity-agent/0.log"
May 26 16:43:43.658005 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:43.657980    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-96.ec2.internal_b789aeffe1887cd67b4335801350d604/haproxy/0.log"
May 26 16:43:48.019330 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:48.019295    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7587b89b76-ntp27_f8572d1e-9ffd-4442-bd73-d63b4a458cce/manager/0.log"
May 26 16:43:48.206188 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:48.206157    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-c7fb4c8d5-rbb7k_a07ead44-5922-4db0-aea0-2f827a0c0f4c/manager/0.log"
May 26 16:43:49.614505 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:49.614471    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d7p8f_3f008cf1-69a4-45fc-aca1-35a597b97653/node-exporter/0.log"
May 26 16:43:49.643628 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:49.643600    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d7p8f_3f008cf1-69a4-45fc-aca1-35a597b97653/kube-rbac-proxy/0.log"
May 26 16:43:49.676911 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:49.676873    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d7p8f_3f008cf1-69a4-45fc-aca1-35a597b97653/init-textfile/0.log"
May 26 16:43:52.538321 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.538285    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9f8jk/perf-node-gather-daemonset-lf7tk"]
May 26 16:43:52.538737 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.538578    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b2a69a8-b745-4203-b81b-2c4b45a60333" containerName="storage-initializer"
May 26 16:43:52.538737 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.538589    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b2a69a8-b745-4203-b81b-2c4b45a60333" containerName="storage-initializer"
May 26 16:43:52.538737 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.538597    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b2a69a8-b745-4203-b81b-2c4b45a60333" containerName="main"
May 26 16:43:52.538737 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.538603    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b2a69a8-b745-4203-b81b-2c4b45a60333" containerName="main"
May 26 16:43:52.538737 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.538612    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b759932b-9db9-4c73-b28c-d1871bed94ff" containerName="storage-initializer"
May 26 16:43:52.538737 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.538617    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b759932b-9db9-4c73-b28c-d1871bed94ff" containerName="storage-initializer"
May 26 16:43:52.538737 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.538626    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="880bb567-ef75-4853-ad67-794058f47650" containerName="manager"
May 26 16:43:52.538737 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.538635    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="880bb567-ef75-4853-ad67-794058f47650" containerName="manager"
May 26 16:43:52.538737 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.538643    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b759932b-9db9-4c73-b28c-d1871bed94ff" containerName="main"
May 26 16:43:52.538737 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.538647    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b759932b-9db9-4c73-b28c-d1871bed94ff" containerName="main"
May 26 16:43:52.538737 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.538661    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea5f6b2c-4776-4505-87a2-3e78e8154aaa" containerName="main"
May 26 16:43:52.538737 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.538666    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea5f6b2c-4776-4505-87a2-3e78e8154aaa" containerName="main"
May 26 16:43:52.538737 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.538671    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea5f6b2c-4776-4505-87a2-3e78e8154aaa" containerName="storage-initializer"
May 26 16:43:52.538737 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.538676    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea5f6b2c-4776-4505-87a2-3e78e8154aaa" containerName="storage-initializer"
May 26 16:43:52.538737 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.538728    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="ea5f6b2c-4776-4505-87a2-3e78e8154aaa" containerName="main"
May 26 16:43:52.538737 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.538739    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="4b2a69a8-b745-4203-b81b-2c4b45a60333" containerName="main"
May 26 16:43:52.539252 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.538746    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="880bb567-ef75-4853-ad67-794058f47650" containerName="manager"
May 26 16:43:52.539252 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.538753    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="b759932b-9db9-4c73-b28c-d1871bed94ff" containerName="main"
May 26 16:43:52.540603 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.540587    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-lf7tk"
May 26 16:43:52.543651 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.543631    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9f8jk\"/\"default-dockercfg-bg6lc\""
May 26 16:43:52.544561 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.544543    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9f8jk\"/\"kube-root-ca.crt\""
May 26 16:43:52.544668 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.544578    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9f8jk\"/\"openshift-service-ca.crt\""
May 26 16:43:52.554609 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.554586    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9f8jk/perf-node-gather-daemonset-lf7tk"]
May 26 16:43:52.640733 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.640698    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8f83039a-7225-4dd8-a348-365fc3f927b8-lib-modules\") pod \"perf-node-gather-daemonset-lf7tk\" (UID: \"8f83039a-7225-4dd8-a348-365fc3f927b8\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-lf7tk"
May 26 16:43:52.640920 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.640754    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9595w\" (UniqueName: \"kubernetes.io/projected/8f83039a-7225-4dd8-a348-365fc3f927b8-kube-api-access-9595w\") pod \"perf-node-gather-daemonset-lf7tk\" (UID: \"8f83039a-7225-4dd8-a348-365fc3f927b8\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-lf7tk"
May 26 16:43:52.640920 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.640860    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f83039a-7225-4dd8-a348-365fc3f927b8-sys\") pod \"perf-node-gather-daemonset-lf7tk\" (UID: \"8f83039a-7225-4dd8-a348-365fc3f927b8\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-lf7tk"
May 26 16:43:52.640920 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.640904    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8f83039a-7225-4dd8-a348-365fc3f927b8-proc\") pod \"perf-node-gather-daemonset-lf7tk\" (UID: \"8f83039a-7225-4dd8-a348-365fc3f927b8\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-lf7tk"
May 26 16:43:52.641089 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.640997    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8f83039a-7225-4dd8-a348-365fc3f927b8-podres\") pod \"perf-node-gather-daemonset-lf7tk\" (UID: \"8f83039a-7225-4dd8-a348-365fc3f927b8\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-lf7tk"
May 26 16:43:52.741409 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.741378    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f83039a-7225-4dd8-a348-365fc3f927b8-sys\") pod \"perf-node-gather-daemonset-lf7tk\" (UID: \"8f83039a-7225-4dd8-a348-365fc3f927b8\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-lf7tk"
May 26 16:43:52.741409 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.741416    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8f83039a-7225-4dd8-a348-365fc3f927b8-proc\") pod \"perf-node-gather-daemonset-lf7tk\" (UID: \"8f83039a-7225-4dd8-a348-365fc3f927b8\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-lf7tk"
May 26 16:43:52.741659 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.741486    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f83039a-7225-4dd8-a348-365fc3f927b8-sys\") pod \"perf-node-gather-daemonset-lf7tk\" (UID: \"8f83039a-7225-4dd8-a348-365fc3f927b8\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-lf7tk"
May 26 16:43:52.741659 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.741547    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8f83039a-7225-4dd8-a348-365fc3f927b8-proc\") pod \"perf-node-gather-daemonset-lf7tk\" (UID: \"8f83039a-7225-4dd8-a348-365fc3f927b8\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-lf7tk"
May 26 16:43:52.741659 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.741576    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8f83039a-7225-4dd8-a348-365fc3f927b8-podres\") pod \"perf-node-gather-daemonset-lf7tk\" (UID: \"8f83039a-7225-4dd8-a348-365fc3f927b8\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-lf7tk"
May 26 16:43:52.741659 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.741599    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8f83039a-7225-4dd8-a348-365fc3f927b8-lib-modules\") pod \"perf-node-gather-daemonset-lf7tk\" (UID: \"8f83039a-7225-4dd8-a348-365fc3f927b8\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-lf7tk"
May 26 16:43:52.741659 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.741632    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9595w\" (UniqueName: \"kubernetes.io/projected/8f83039a-7225-4dd8-a348-365fc3f927b8-kube-api-access-9595w\") pod \"perf-node-gather-daemonset-lf7tk\" (UID: \"8f83039a-7225-4dd8-a348-365fc3f927b8\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-lf7tk"
May 26 16:43:52.741818 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.741700    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8f83039a-7225-4dd8-a348-365fc3f927b8-podres\") pod \"perf-node-gather-daemonset-lf7tk\" (UID: \"8f83039a-7225-4dd8-a348-365fc3f927b8\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-lf7tk"
May 26 16:43:52.741818 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.741736    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8f83039a-7225-4dd8-a348-365fc3f927b8-lib-modules\") pod \"perf-node-gather-daemonset-lf7tk\" (UID: \"8f83039a-7225-4dd8-a348-365fc3f927b8\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-lf7tk"
May 26 16:43:52.752161 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.752111    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9595w\" (UniqueName: \"kubernetes.io/projected/8f83039a-7225-4dd8-a348-365fc3f927b8-kube-api-access-9595w\") pod \"perf-node-gather-daemonset-lf7tk\" (UID: \"8f83039a-7225-4dd8-a348-365fc3f927b8\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-lf7tk"
May 26 16:43:52.850741 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.850653    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-lf7tk"
May 26 16:43:52.973223 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.973143    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9f8jk/perf-node-gather-daemonset-lf7tk"]
May 26 16:43:52.975990 ip-10-0-131-96 kubenswrapper[2552]: W0526 16:43:52.975961    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8f83039a_7225_4dd8_a348_365fc3f927b8.slice/crio-d90a0e9b83b4c4cadf47e5de94be411d60f7c12108130def8aa6ea4c39c15f97 WatchSource:0}: Error finding container d90a0e9b83b4c4cadf47e5de94be411d60f7c12108130def8aa6ea4c39c15f97: Status 404 returned error can't find the container with id d90a0e9b83b4c4cadf47e5de94be411d60f7c12108130def8aa6ea4c39c15f97
May 26 16:43:52.977715 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:52.977700    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:43:53.449201 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:53.449168    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-lf7tk" event={"ID":"8f83039a-7225-4dd8-a348-365fc3f927b8","Type":"ContainerStarted","Data":"2a9dcd1f2aa5fc6ed585a877a091690147677f159a4aa667f3be5e8450362b27"}
May 26 16:43:53.449201 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:53.449205    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-lf7tk" event={"ID":"8f83039a-7225-4dd8-a348-365fc3f927b8","Type":"ContainerStarted","Data":"d90a0e9b83b4c4cadf47e5de94be411d60f7c12108130def8aa6ea4c39c15f97"}
May 26 16:43:53.449432 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:53.449395    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-lf7tk"
May 26 16:43:53.469240 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:53.469194    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-lf7tk" podStartSLOduration=1.469179409 podStartE2EDuration="1.469179409s" podCreationTimestamp="2026-05-26 16:43:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:43:53.467606634 +0000 UTC m=+2583.176061686" watchObservedRunningTime="2026-05-26 16:43:53.469179409 +0000 UTC m=+2583.177634466"
May 26 16:43:54.129223 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:54.129197    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6nz6h_076454e2-eee8-47cb-a7e6-dc5e0a0270ae/dns/0.log"
May 26 16:43:54.155694 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:54.155658    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6nz6h_076454e2-eee8-47cb-a7e6-dc5e0a0270ae/kube-rbac-proxy/0.log"
May 26 16:43:54.307592 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:54.307560    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j7hlf_4475d020-5281-4257-bf03-1a264049169d/dns-node-resolver/0.log"
May 26 16:43:54.849524 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:54.849496    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5xp7f_092e4858-1372-4cb9-b51b-dde9e50e0f9d/node-ca/0.log"
May 26 16:43:55.805591 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:55.805557    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-798958bb55-mm958_ce1e9de0-8ee7-43fb-9091-553fe6cfe394/discovery/0.log"
May 26 16:43:55.831292 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:55.831252    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-849cdfbf5f-m95mt_8310f0a3-b09d-4746-8561-11c489116cf8/istio-proxy/0.log"
May 26 16:43:56.433732 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:56.433702    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-thjfg_dfdaec24-5a44-429c-bf06-3422d868294d/serve-healthcheck-canary/0.log"
May 26 16:43:56.904984 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:56.904947    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2759w_c844ea1b-4c3b-4273-a8b7-6667de9b3fd9/kube-rbac-proxy/0.log"
May 26 16:43:56.928336 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:56.928309    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2759w_c844ea1b-4c3b-4273-a8b7-6667de9b3fd9/exporter/0.log"
May 26 16:43:56.954720 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:56.954690    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2759w_c844ea1b-4c3b-4273-a8b7-6667de9b3fd9/extractor/0.log"
May 26 16:43:59.461346 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:43:59.461322    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-lf7tk"
May 26 16:44:00.312840 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:44:00.312811    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-6f7545b5b9-grtw8_a9ef56ea-aca9-4eea-9cd9-7426c3a1c87c/manager/0.log"
May 26 16:44:00.371162 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:44:00.371135    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-86f7b4b499-7pzn4_e5076f02-1823-47b1-9c24-af72ea72ec6a/server/0.log"
May 26 16:44:00.714528 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:44:00.714446    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-79f457656b-xwhcb_7232bab9-5261-4dc4-8fe6-c0bcbc19eec0/seaweedfs/0.log"
May 26 16:44:06.808898 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:44:06.808866    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bq8pz_b90e9cb1-f2de-4919-ab89-8bc29c2476ea/kube-multus-additional-cni-plugins/0.log"
May 26 16:44:06.829876 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:44:06.829843    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bq8pz_b90e9cb1-f2de-4919-ab89-8bc29c2476ea/egress-router-binary-copy/0.log"
May 26 16:44:06.853951 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:44:06.853896    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bq8pz_b90e9cb1-f2de-4919-ab89-8bc29c2476ea/cni-plugins/0.log"
May 26 16:44:06.875815 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:44:06.875790    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bq8pz_b90e9cb1-f2de-4919-ab89-8bc29c2476ea/bond-cni-plugin/0.log"
May 26 16:44:06.897770 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:44:06.897744    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bq8pz_b90e9cb1-f2de-4919-ab89-8bc29c2476ea/routeoverride-cni/0.log"
May 26 16:44:06.918723 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:44:06.918697    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bq8pz_b90e9cb1-f2de-4919-ab89-8bc29c2476ea/whereabouts-cni-bincopy/0.log"
May 26 16:44:06.943027 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:44:06.942998    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bq8pz_b90e9cb1-f2de-4919-ab89-8bc29c2476ea/whereabouts-cni/0.log"
May 26 16:44:07.423373 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:44:07.423345    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-wpv87_1f8c521d-e87e-4504-af4b-b30fa7ece51b/kube-multus/0.log"
May 26 16:44:07.591109 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:44:07.591072    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-gj94f_655a0641-f708-444d-91bc-fe7518694ed3/network-metrics-daemon/0.log"
May 26 16:44:07.610797 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:44:07.610774    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-gj94f_655a0641-f708-444d-91bc-fe7518694ed3/kube-rbac-proxy/0.log"
May 26 16:44:09.115296 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:44:09.115267    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sj6db_5c0f9fc6-2bd6-4370-96d0-5ae315363c7f/ovn-controller/0.log"
May 26 16:44:09.132535 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:44:09.132499    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sj6db_5c0f9fc6-2bd6-4370-96d0-5ae315363c7f/ovn-acl-logging/0.log"
May 26 16:44:09.154741 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:44:09.154713    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sj6db_5c0f9fc6-2bd6-4370-96d0-5ae315363c7f/ovn-acl-logging/1.log"
May 26 16:44:09.176228 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:44:09.176205    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sj6db_5c0f9fc6-2bd6-4370-96d0-5ae315363c7f/kube-rbac-proxy-node/0.log"
May 26 16:44:09.198717 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:44:09.198692    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sj6db_5c0f9fc6-2bd6-4370-96d0-5ae315363c7f/kube-rbac-proxy-ovn-metrics/0.log"
May 26 16:44:09.224196 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:44:09.224170    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sj6db_5c0f9fc6-2bd6-4370-96d0-5ae315363c7f/northd/0.log"
May 26 16:44:09.246485 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:44:09.246461    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sj6db_5c0f9fc6-2bd6-4370-96d0-5ae315363c7f/nbdb/0.log"
May 26 16:44:09.267886 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:44:09.267864    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sj6db_5c0f9fc6-2bd6-4370-96d0-5ae315363c7f/sbdb/0.log"
May 26 16:44:09.432273 ip-10-0-131-96 kubenswrapper[2552]: I0526 16:44:09.432184    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sj6db_5c0f9fc6-2bd6-4370-96d0-5ae315363c7f/ovnkube-controller/0.log"